working robust crawler
more verbose on the errors