records all urls with crawling