apache/spark — pull request #493 by tmyklebu (2014-04-23 02:21:33+00:00)
Title: [SPARK-1580] Estimate ALS communication and computation costs.

This pull request adds a function `evaluatePartitioner` to the `ALS` object. This function takes a dataset, a rank, and user and product Partitioners and returns estimates of the communication cost (in bytes) and computation cost (currently roughly in flops) of a single iteration of alternating least squares.

The communication cost estimate considers only the number of bytes needed for the data in the feature vectors. It does not consider the `InLinkBlock`s or `OutLinkBlocks` or array headers for the feature vectors or anything. It also is oblivious to compression. Thus it is inaccurate. However, the changes in this pull request are sufficient to meet the goals described in [SPARK-1580] and, for better or worse, are independent of those particular implementation details.

Likewise, the computation cost estimate considers only the number of floating-point operations needed to form and factorise the matrices that come up. No attention is paid to loop overhead or any other vagaries of the current implementation.

This pull request also appears to contain a bunch of commits that have already made it into trunk. I'm not really sure why that's the case or how to fix it; I'd appreciate guidance.