NUTCH-1031 Delegate parsing of robots.txt to crawler-commons

git-svn-id: https://svn.apache.org/repos/asf/nutch/trunk@1465159 13f79535-47bb-0310-9956-ffa450edef68
NUTCH-1554 org.apache.nutch.net.protocols.HttpDateFormat should NOT be Locale.US aware

git-svn-id: https://svn.apache.org/repos/asf/nutch/trunk@1465741 13f79535-47bb-0310-9956-ffa450edef68
revert NUTCH-1554 org.apache.nutch.net.protocols.HttpDateFormat should NOT be Locale.US aware

git-svn-id: https://svn.apache.org/repos/asf/nutch/trunk@1465831 13f79535-47bb-0310-9956-ffa450edef68
NUTCH-1501 Harmonize behavior of parsechecker and indexchecker

git-svn-id: https://svn.apache.org/repos/asf/nutch/trunk@1469100 13f79535-47bb-0310-9956-ffa450edef68
Fixed detection of job file for distributed mode

git-svn-id: https://svn.apache.org/repos/asf/nutch/trunk@1469231 13f79535-47bb-0310-9956-ffa450edef68
NUTCH-829 duplicate hadoop temp files

git-svn-id: https://svn.apache.org/repos/asf/nutch/trunk@1476702 13f79535-47bb-0310-9956-ffa450edef68
NUTCH-346 Improve readability of logs/hadoop.log

git-svn-id: https://svn.apache.org/repos/asf/nutch/trunk@1476859 13f79535-47bb-0310-9956-ffa450edef68
NUTCH-1549 Fix deprecated use of Tika MimeType API in o.a.n.util.MimeUtil

git-svn-id: https://svn.apache.org/repos/asf/nutch/trunk@1477821 13f79535-47bb-0310-9956-ffa450edef68
NUTCH-1334 NPE in FetcherOutputFormat

git-svn-id: https://svn.apache.org/repos/asf/nutch/trunk@1477847 13f79535-47bb-0310-9956-ffa450edef68
Merge branch 'trunk' into cc