Add support for building against hadoop2-yarn : adding new maven profile for it
Checkpoint commit - compiles and passes a lot of tests - not all though, looking into FileSuite issues
Checkpoint commit - compiles and passes a lot of tests - not all though, looking into FileSuite issues
Fix issue with FileSuite failing
Commit job to persist files
Use hostname instead of hostport for executor, fix creation of workdir
Fix bug with atomic update
Fix npe
Scala does not prevent variable shadowing ! Sick error due to it ...
Fix exception when checkpoint path does not exist (no data in rdd which is being checkpointed for example)
remove spurious debug statements
Move from master to driver
Fudge order of classpath - so that our jars take precedence over what is in CLASSPATH variable. Sounds logical, hope there is no issue cos of it
Ensure all ask/await block for spark.akka.askTimeout - so that it is controllable : instead of arbitrary timeouts spread across codebase. In our tests, we use 30 seconds, though default of 10 is maintained
Move back to 2.0.2-alpha, since 2.0.3-alpha is not available in cloudera yet
Move back to 2.0.2-alpha, since 2.0.3-alpha is not available in cloudera yet. Also, add netty dependency explicitly to prevent resolving to older 2.3x version. Additionally, comment out retrievePattern to ensure correct netty is picked up
Add a small note on spark.tasks.schedule.aggression
Cache pattern, add (commented out) alternatives for check* apis
Add some basic documentation
Fix review comments, add a new api to SparkHadoopUtil to create appropriate Configuration. Modify an example to show how to use SplitInfo