Merge pull request #1 from agjmills/master

add 502 response
Merge pull request #2 from agjmills/master

altered responses to be a dict, in order to handle any type of http resp...
added script to get all links from a supplied url
changed url scraper into funtion and added incomplete check for existing urls
Prints a list of (full?) site pages via trawling
added arguments to specify the site to trawl
Merge pull request #2 from datkinson/master

start of a site trawler