Make Statistics thread safe
Updated statistics javadoc
Rename methods to match search strategy names.
Now invokes UrlDigestFallback correctly.
Plus minor cleanup.
Account for duplicate type by direct examination
not by what the search strategy was used. An exact url match can be made
even when looking by digest.
Always index digest. 
URLs are indexed by default.
Canonicalized URLs are included by default.
Improved search strategy for DIGEST_ANY
Now will find exact url hits if URLs are indexed without the cost of an
additional query. 
Dropped the two 'fallback' search strategies. They no longer make sense.
Use Processor's shouldProcessRule for mimefilter.
Improved instructions
Fixed comments/instructions.
No longer ignores prereqs.
DNS will still be ignored since its not HTTP, but this will allow dedup
of robots.txt.
Use provided method on CrawlURI to detect whether or not it is an HTTP
transaction
Also, no longer filter out curi's with null content types. Never happens
as the method actually returns 'unknown' in those instances. Dealing
with these should be done in the should process rules for the processor.
Merge pull request #4 from kris-sigur/master

Improved search strategies