There are plenty of Web crawler tools available which help in easily crawling a website URLs. These tools help in doing website audit to identify any broken links, redirects and other SEO parameters for improving a website. Screaming Frog is one of the best web crawler tool which can crawl 500 URLs in free version with plenty of crawled information. Visual SEO Studio and 80legs are also crawling tools with both free and paid options.