crawl method now receives entire Response object instead of just the read()