granted, requesting robots.txt is probably not requested very frequently.