Crawler should now respect robots.txt.

 - Added 'robots' library to fetch and parse robots.txt, and check URL permissions
 - Server only crawls pages that the parser says are OK to crawl.
 - For sites without a valid robots.txt, all encountered links are considered valid.
Fixed queue length indicator, only crawl pathname, not query.

 - Fixed queue length indicator (number in parentheses after log message)
 - Discard query parameters before crawling (reduces total number of pages)
Does not crawl images (paths matching .png, .jpg, so on).

 - Check each encountered link against a set of image extension regexes, only attempt to
   download links that aren't images.