saltybeagle/Spider — pull request #28 by mfairchild365 (2014-03-20 13:41:21+00:00)
Title: Only check if the domain is not set

This was only working for the first request. Subsequent requests to other domains were causing an undefined index error because `empty(self::$robotstxt)` was `false`