added respective rspec tests and working redirect on crawler fail