return responses in crawl result for working urls also