tried to run against real data and ran out of RAM,  this time with 12GB allocated.

It's very possible I'm doing something wrong,  and also possible this can be made to fit with further shoehorning,  but Hadoop can do this kind of problem easily.  It would be extremely challenging to do weeklies or dailies at this rate.

So let's check the code in and move on to the next thing.