cwensel/cascading — pull request #31 by gerashegalov (2014-08-05 10:31:00+00:00)
Title: Add DistributedCache support for Hfs to optimize HashJoin in large jobs similar to Pig and Hive

Unlike Pig and Hive that add the right-hand side of the replicated Join to DistributedCache, there seems to be no way of achieving this in Cascading even manually with HashJoin. In large jobs on a large cluster HashJoin is currently prohibitive, because mappers first cause an additional RPC storm on the NN to open the rhs source to locate its blocks, and then overwhelm few (3) DN's that hold the replicas of rhs. When done via DistributedCache localization, it would happen at most once per node (order of magnitude less than the number of map containers).

In this prototype patch, the DC functionality is achieved via a FileSystem API in order to leave InputFormats unconstrained.