apache/spark — pull request #3288 by dbtsai (2014-11-16 01:08:30+00:00)
Title: [SPARK-4431][MLlib] Implement efficient activeIterator for dense and sparse vector

Previously, we were using Breeze's activeIterator to access the non-zero elements
in sparse vector, and explicitly skipping the zero in dense/sparse vector using
pattern matching. Due to the overhead, we switched back to native `while loop`
in #SPARK-4129.

However, #SPARK-4129 requires de-reference the dv.values/sv.values in
each access to the value, and the zeros in dense vector and sparse vector
if exist are skipped in the add function call; the overall penalty will be
around 10% compared with de-reference once outside the while block,
and checking if zero before calling the add function. The code is branched out
for dense and sparse vector, and it's not easy to maintain in the long term.

Not only this activeIterator implementation increases the performance,
but the abstraction of accessing the non-zero elements in different
vector type also helps the maintainability of codebase. In this PR,
only MultivariateOnlineSummarizer uses new API as example,
and others can be migrated to activeIterator later.

Benchmarking with mnist8m dataset on single JVM
with first 200 samples loaded in memory, and repeating 5000 times.

Before change:
Sparse Vector - 30.02
Dense Vector - 38.27

After this optimization:
Sparse Vector - 27.54
Dense Vector - 35.13