No longer ignores prereqs.
DNS will still be ignored since its not HTTP, but this will allow dedup
of robots.txt.
Use provided method on CrawlURI to detect whether or not it is an HTTP
transaction
Also, no longer filter out curi's with null content types. Never happens
as the method actually returns 'unknown' in those instances. Dealing
with these should be done in the should process rules for the processor.