handle crawler errors