apache/mahout — pull request #31 by pferrel (2014-07-17 20:34:37+00:00)
Title: MAHOUT-1541

Covers MAHOUT-1464, and MAHOUT-1541

Fixes cross-cooccurrence, which originally was of the wrong order. If A is the primary interaction matrix, them cross should be A'B but was B'A. Also added tests for A and B of different column cardinality, which should is allowed.

Looked a bit deeper regarding @dev thread on changing cardinality of a sparse matrix. This is needed in ItemSimilarity because only once all cross-interaction matrices have been read in can the true cardinality of all be known. The much more typical case is that they can be computed from the input. So there needs to be a method to update the cardinality after the matrices have been read in.

This implementation creates a new abstract method in CheckpointedDrm and implementation in CheckpointedDrmSpark. Here is the reasoning.

1) for sparse DRMs there is no need for any representation of an empty row (or column) not even the keys need to be known only the cardinality. You only have to think about a transpose of sparse vectors to see that this must be so. Further it works and I’ve relied on it since the Hadoop mr version. Baring any revelation from the math gods--it is so.

2) rbind semantics apply to dense matrices. This IMO should be avoided in this case because even if we rejigger rbind to only change the cardinality without inserting real rows it would seem to violate its semantics. Sparse matrices don’t fit the default R semantics in a few areas (in my non-expert opinion) and this is one. Unless someone feels strongly it will be in CheckpointedDrm as abstract and implemented in CheckpoimtedDrmSpark#addToRowCardinality(n: Int): Unit. Creating an op that returns a new CheckpointedDrm is also possible if there is something unsafe about my implementation, but rbind?

3) I have implemented this so that there is no call to drm.nrow neither to read nor modify it. So it will remain lazy evaluated until needed by other math.

4) ItemSimilarity for the A’B case now passes several asymmetric input cases and outputs the correct external IDs.