Added constructors and made other tweaks while testing running all the jobs on 3 physical nodes. Changed Split to Fields in the DemoMapper.
Created per node emittedStore with some simple methods. Wrote a partial test for the mittedStore.
Changed my EmittedStore to be called EmittedStorage to resolve a conflict. Introducing Emitter interface as a slightly tweaked approach to Emitting so that Mapper writers never need to know about jobIds, taskIds, and never need to be able to read intermediates (Emitter is a very thin wrapper of EmittedStorage).
SimpleEmitter implementation of Emitter being used to Emit inside of Mapper Map method so that jobID, taskId, and the local MapReduceNode's emittedStorage instance are used. There is no longer any need to send an IntermediateAccessor/Emitter across te network in a Task.
New slice of values was not being stored in Emitter.putEmitted, so now it is.
Fixed problem where MapTask.JobId was not being set in the constructor.
Removed TaskId from Emitter. Added in hashing/partitioning function so that intermediate keys are partitioned correctly for reducers.
Reduce tasks are now created. ReduceTask.execute() now gathers all the intermediate KVPairs that will need to be reduced (from all the nodes). Just need to write the reduce function that will process these KVPairs.
Added a reducer implementation to count up all the words. Seems to work, just needs to write to final output.
Merged with tgarv-dev, added some stylistic changes, and a few comments.
Restructuring how Reduce is called so it is invoked per key in a ReduceTask which now has an associated PartitionNumber rather than a 'Key' as MapTasks do. Reorg has broken some things to work, so wait for further commits before merging. JobConfig now passed across the network as information in it may need to be transparently used at receiving workers (before executing client Reduce method).
Intermediate pairs are now sorted by key and executed in groups so that each Reduce call only handles a single key, as described in the Map Reduce paper. This can be important for some type of Map Reduce operations. The Reduce type signature has changed accordingly.
The signatures of the MapTask and ReduceTasks have diverged as they need to accept different arguments. workerRole type assertion conditionally executed the correct Task type. Now making use of the EmittedReader so ReduceTask is a bit cleaner and now sorts by intermediates and executes Reduce for each key.
Added S3Outputer and its Output method. Reducers now write their output to S3!
In master_role, it now loops until all tasks have been marked as "complete" by the TaskManager (instead of just sleeping for 5 seconds and then continuing).
Using environment variables to set the S3 bucket name and AWS access credentials.
Merge branch 'tgarv-dev' into dghubble-dev
Cleaned up some code.
Fixed an issue with counting words, which also fixed an issue with memory use. Commented out some prints.
Started work on a JobManager which can be queried to determine when Jobs are complete and store other JobState information that would previously have been thrown in the Job struct.