apache/mahout — pull request #36 by pferrel (2014-07-30 20:13:08+00:00)
Title: MAHOUT-1541

Parts of this address MAHOUT-1541, MAHOUT-1568, and MAHOUT-1569

The previous merge of MAHOUT-1541 was supporting A'A primarily, this merge support A'B as well with all features. Lots of refactoring and new tests for A and B of different cardinality and using different item ID spaces. Took the forced cardinality matching from Math and put in the data prep part. This means passing in an nrow to drmWrap, which may be larger than the actual number of rows embodied in the drm/rdd. I've added tests for B.t %*% A as well as the actual driver for these cases (missing row cases).

Can't complete the full epinions cross-cooccurrence on a single machine with an out of Java heap exception. So I'm now testing it on a cluster. The cooccurrence of A'A does complete on a single machine.

One known improvement is to limit the use of dictionaries if they are not need and to look at replacing the Guava HashBiMap with a minimal Scala verison. This version uses dictionaries for IDs even if the input is using Mahout sequential int IDs.

MAHOUT-1568: Proposed standards for text versions of DRM-ish output. These preserve the IDs of the application while using Mahout IDs internally. In other words output has application IDs. There are several configurable readers and writers of TD files. Reading Tuples into a DRM is implemented, Writing a DRM-ish TD file is also implemented.

MAHOUT-1569: There is a refactored MahoutOptionParser and MahoutDriver with some default behavior that should make creating drivers a bit easier and DRYer than the last merge. The options are being proposed as standards across all drivers so we have one way to specify the formats of input/output files and other common options.

This is for comment, I won't merge until several larger dataset are working on a cluster.