NUTCH-1031 Delegate parsing of robots.txt to crawler-commons

git-svn-id: https://svn.apache.org/repos/asf/nutch/branches/2.x@1477319 13f79535-47bb-0310-9956-ffa450edef68