Crawler disabled http moved functionality