Extended with the option of specifying hostname for known crawlers like findthatfile.com