Project: duracloud
Issue DURACLOUD-285: Bottleneck in hadoop processing
Two runs of the Bulk Bit Integrity service over the same 1TB set of BHL content took approximately the same amount of time (a few minutes over 8 hours.) The problem is that the first run used 10 small instances and the second run used 10 x-large instances. The x-large instance should have completed the task much faster, not only because they have higher bandwidth and compute capacities, but also because they are set to use 8 mappers per instance rather than the small instance setting of 2. This task is to discover and fix the bottleneck which is not allowing the x-large instances to run at full capacity.
Story points: 3