discard ftp crawl;add gz ,lzo convert;refactor file to hdfs.