twitter/scalding — pull request #661 by johnynek (2013-10-17 08:51:47+00:00)
Title: sumByLocalKeys, Counters and Task Numbers

This introduces three concepts I have resisted adding for a while. They are: mapside-only summing, reading the task number, and incrementing counters. I've resisted them because I had no great API to clone, and I was not sure a good one. That said, they keep coming up, so I took a stab at it.

I'd appreciate any feedback.

I only did this for the TypedAPI, but since they are supported with cascading Operations they are easy to call with each/eachTo (in fact, I do that in the methods). I view these as more expert level APIs and I think the type safety is good to encourage people to get the usage correct.

The only remaining issue I can think of with Hadoop is setting the keepAlive, but that is again an expert level thing. That can be controlled with Hadoop settings or with a custom cascading operation, and is even less commonly needed. So, I'm going to punt on that here.