saltybeagle/Spider — pull request #27 by mfairchild365 (2014-03-18 21:44:18+00:00)
Title: Support custom user agents

This adds support for custom user agents for the default downloader class and robots.txt filter