Updated crawler with retries