Added visited urls to front end. Added user-agent. Fixed an issue that allowed for the crawler to stop if it hit an anchor with no url.