Microsoft-CISL/REEF — pull request #1024 by johnyangk (2014-10-22 13:11:20+00:00)
Title: Improve YARN allocation performance

This PR addresses https://issues.apache.org/jira/browse/REEF-6.
It changes the policy of sending requests to YARN RM from "one-request-at-a-time" to "one-kind-of-requests-at-a-time".

To circumvent YARN-314, instead of using lastContainerMemory, this PR uses a matching algorithm and avoids sending zero-container requests. This is because re-requesting every time we do not get a match can result in requesting a lot of extra containers. But this method needs to be revisited when we implement locality-strictness.

2 test sets of different sequence of (numberOfRequests, memory size) were tested

Test set 1:

(7, 1GB)
(5, 2GB)
(5, 1GB)
(3, 2GB)

Test set 2:

(1, 1GB)
(1, 6GB)
(1, 2GB)
(1, 5GB)
(1, 3GB)
(1, 4GB)
(1, 4GB)
(1, 3GB)
(1, 5GB)
(1, 2GB)
(1, 6GB)
(1, 1GB)