Netflix/Lipstick — pull request #59 by piaozhexiu (2014-06-29 15:43:50+00:00)
Title: JobClient.getMap/ReduceTaskReport() causes OOM for large jobs

For jobs that have 100K+ tasks, JobClient.getMap/ReduceTaskReport() causes OOM in Lipstick. There are 2 places to fix-

1) BasicP2LClient.java:L365
I made it configurable to call JobClient.getMap/ReduceTaskReport() via a new property "pig.stats.notaskreport". In fact, this property controls the same behavior in Apache Pig too. (See below.)

2) BasicP2LClient.java:L408
I rewrote the code for getting the # of tasks using counters. There is a slight difference between the original and new code: the original code only counts completed tasks while the new code counts failed/killed tasks as well as completed tasks. So the # of tasks is slightly greater than before, but I believe this is okay.

Note that this issue also exists in Apache Pig, and I am fixing it in PIG-4043-
https://issues.apache.org/jira/browse/PIG-4043