metamx/druid — pull request #726 by davideanastasia (2014-09-10 16:27:06+00:00)
Title: Use URI scheme instead of FileSystem type to write the correct value in the DB

On EMR, the default FileSystem for S3 is com.amazon.ws.emr.hadoop.fs.EmrFileSystem, which is automatically configured with Key and SecretKey during the cluster startup. However, Druid is expecting to use NativeS3FileSystem to write into S3, which fails on EMR with the following stack trace:

Error: com.metamx.common.ISE: Unknown file system[class com.amazon.ws.emr.hadoop.fs.EmrFileSystem] at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.serializeOutIndex(IndexGeneratorJob.java:453) at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:387) at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:252) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:635) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

While this can be corrected using

-Dhadoop.fs.s3n.impl=org.apache.hadoop.fs.s3native.NativeS3FileSystem

and many other option to setup the access key and secret key, this patch will remove the problem