allow file caching of robots.txt
remove trailing spaces
import zlib
remove unneeded origin calculation
enable robots txt caching in webui