apache/kafka — pull request #34 by mdykman (2014-10-06 14:58:56+00:00)
Title: changed List type in RequestPurgatory.Watchers

class Watchers is declared, having member

private val requests = new util.ArrayList[T]

in purgeSatisfied(), an iterator is created which conditionally removes elements from the list as it goes:

RequestPurgatory.scala, lines 193-201:

val iter = requests.iterator()
var purged = 0
while(iter.hasNext) {
val curr = iter.next
if(curr.satisfied.get()) {
iter.remove()
purged += 1
}
}

Using the .remove operation on an ArrayList iterator is very expensive as the ArrayList promises a contiguous backing array and all higher elements must be shifted on every operation.

A LinkedList is optimized for for the removal of arbitrary elements. Therefore recommending:

- private val requests = new util.ArrayList[T]
+ private val requests = new util.LinkedList[T]

This case was identified due to an observed failure In a high-load production environment, which found one or more cores pinned (ie. @100 usage) insider this loop. Once the pin was established, it tended to remain until the system was restarted.

As the loop is within a synchronized block, checkAndMaybeAdd (0.8.2) or add (0.8.1) which are also synchronized on the same object, may become blocked by this inefficiency.