apache/incubator-reef — pull request #4 by johnyangk (2014-10-24 09:11:14+00:00)
Title: Improve YARN allocation performance

This PR addresses https://issues.apache.org/jira/browse/REEF-6. It changes the policy of sending requests to YARN RM from "one-request-at-a-time" to "one-kind-of-requests-at-a-time".

Similar to https://github.com/Microsoft-CISL/REEF/pull/1021, to circumvent YARN-314, we do not remove the last request of a batch of same-kind requests.

But since now multiple requests are allocated at once, different kinds of same-priority requests can be in flight at once(i.e. a request from decResourceRequest + a request from addResourceRequest). This is not a problem when only considering memory/vcores since we can simply match bigger containers.

But this can cause problems when we implement locality-strictness. I'd like to revisit this later when we work on locality-specific requests.

2 test sets of different sequence of (numberOfRequests, memory size) were tested

Test set 1:
(7, 1GB)
(5, 2GB)
(5, 1GB)
(3, 2GB)

Test set 2:
(1, 1GB)
(1, 6GB)
(1, 2GB)
(1, 5GB)
(1, 3GB)
(1, 4GB)
(1, 4GB)
(1, 3GB)
(1, 5GB)
(1, 2GB)
(1, 6GB)
(1, 1GB)

Note that for the existing code(before this PR), when we have the following sequence of same-priority requests:
(1, 1GB)
(1, 2GB)
(1, 1GB)
(1, 3GB)

Because the first request was not removed, the third request results in requesting 2 containers. The existing code does not handle this well. (i.e. we may end up allocating 3X1GB + 1X2GB)