Yelp/mrjob — pull request #429 by goldenberg (2012-04-03 00:07:21+00:00)
Title: Add error_counter_wrapper and corresponding test

At Yelp, we currently handle Python errors slightly differently in a couple different MRJob subclasses. It would be nice to centralize that logic and provide it for any MRJob.

This pull requests provides a wrapper function to be used in steps() to catch all Python errors and increment hadoop counters in a similar format to the one used at Yelp

e.g.

def steps(self):
return [self.mr(mapper=self.error_counter_wrapper(self.mapper),
reducer=self.error_counter_wrapper(self.reducer,
msg='executing first reducer')]

I originally tried to make it accessible as a function decorator to be used on mapper(), etc. But I couldn't figure out a clean way to do that, since it needs access to self.increment_counter. I'm all ears if someone knows how to do that.

I have no particular attachment to this counter format if anyone has ideas for cleaning it up.