Aloisius/hadoop-s3a — pull request #4 by tarnfeld (2014-08-07 13:31:52+00:00)
Title: Ensure the default block size is passed into all FileStatus objects

Hadoop doesn't manage passing the default block size to FileStatus objects for us, so we need to ensure we do that. Currently every FileStatus object has a block size of zero.

For reference, the S3Native filesystem does exactly this, passing `getDefaultBlockSize()` into every new `FileStatus` object.

We found this as a result of a strange memory issue with the `FileInputFormat.getSplits` method. It relies heavily in the block size to do the split computations, and with a size of zero, it will continually append splits until there's no more memory left. If launching the same job on a machine with a significant amount of memory, we then get an exception `java.io.IOException: Split metadata size exceeded 10000000`.

```
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:237)
at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1106)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1098)
at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:177)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:995)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:948)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:948)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:922)
at org.apache.hadoop.streaming.StreamJob.submitAndMonitorJob(StreamJob.java:922)
at org.apache.hadoop.streaming.StreamJob.run(StreamJob.java:123)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
```

I'm a little thrown back as to why this hasn't been found elsewhere. Perhaps it only happens in certain circumstances within the `getSplits` method, or nobody is using this low level input format.

Would love any thoughts! Not particularly familiar with the internals of the hadoop filesystem code so this may not be the appropriate fix.