commoncrawl/example-warc-java — pull request #2 by pirroh (2014-01-29 00:58:03+00:00)
Title: Support I/O on non-default Hadoop FileSystems

Currently, combining the code in commoncrawl-examples with this lib
leads to the following exception: "java.lang.IllegalArgumentException:
This file system object (hdfs://x.x.x.x:9000) does not support
access to the request path 's3n://test-bucket/file.warc.gz; You
possibly called FileSystem.get(conf) when you should have called
FileSystem.get(uri, conf) to obtain a file system supporting your path."
This patch works very similar to the solution used by Apache Crunch:
https://issues.apache.org/jira/browse/CRUNCH-47
Already tested with multiple Elastic MapReduce jobs, and everything
worked flawlessly.

I've also completed the workaround on line endings, and made the project
Eclipse-friendly.