elsevierlabs/logistic-regression-sgd-mapreduce — pull request #3 by bradleypallen (2013-02-05 05:54:41+00:00)
Title: Refactored to better support Hadoop streaming, fixed model averaging

Changed train_reducer.py to produce TSV files with model parameter
values summed across models prodcued by train_mapper.py, and added
model_encoder.py to take the TSV files and output the model JSON object
expected by the prediction, test and query scripts. These changes also
introduced a way of counting the total number of train mapper tasks
(equal to the number of models) to use to generate the average weight
for the parameters in the final model. Previously, the average was
incorrectly performed by using the number of weights accumulated for a
given parameter, which given the split of data across mapper tasks may
be less than the total number of mapper tasks, if one or more splits
yielded a set of instances that did not have an occurrence of the
feature associated with the parameter. When the scripts are used in the
Hadoop streaming context and multiple reducer tasks are run, simple
concatenating the output TSV files and piping the concatentation
through model_encoder provides a simple way to build a model file;
previously, separately produced JSON files would have to be manually
combined or the number of reducers limited to one to get usable output.