Basic crawler

An basic web crawler that crawls an entire website for urls.