NUTCH-1513 Support Robots.txt for Ftp urls

git-svn-id: https://svn.apache.org/repos/asf/nutch/trunk@1484638 13f79535-47bb-0310-9956-ffa450edef68