A fix to honor robots.txt. Without it e are leaking info. Credits go to tmk.