How is this to be used?  Do we want to use this when we can't get robots.txt and we just want to treat a non-GETtyable robots.txt like an "Allow all" robots.txt?