take out potentially bad robots.txt
filter compression logic.