Revert "Crawler disabled http moved functionality"

This reverts commit e7df302379b6e924a754a6d717773cd719562893.