improved crawler tests - added support for 404s and time-outs