That's a nice observation.  However, using the ```variance``` calculation in ```StatCounter``` might be slow since the merge method recomputes ```n, mu, m2``` for each value. Also, it won't fit well with the ```binCombOp``` operation in the ```aggregate``` function. One can probably optimize the ```def merge(values: TraversableOnce[Double]): StatCounter``` method in the ```Variance``` class by doing a batch or mini-batch update for both speed and precision but that's a separate discussion.

I see your concern with computing ```sumSquares``` for a large fraction of the instances and I think it will be be best to leverage the ```def merge(other: StatCounter): StatCounter``` method. 

We can calculate StatCounter per partition using ```count, sum, sumSquares```  and then merge during ```binCombOp``` for numerical stability. I won't be hard to implement. Let me know what you think.