chuyskywalker/rolling-curl — pull request #3 by polyfractal (2013-05-30 19:31:51+00:00)
Title: Performance fix for large buffers of pending requests

I'll preface this by saying I'm not sure if you want to merge it. It's a performance enhancement, but it trades some memory for a (rather drastic) increase in speed.

**The Problem**
Basically, large numbers of pending requests slow to a crawl. The method getNextPendingRequests() uses array_splice on the pending queue to get the next range of requests to act on.

This is fine for small array sizes, but causes considerable memory thrashing when the buffer is large. PHP copies the portion to be removed to a new array, then copies the remaining values to a new array variable.

Given an array of 10,000 requests and splicing them out one at a time, you can see how most of the time is spent simply moving bytes around in memory. Profiling in XDebug confirmed this as well.

**My Solution**
My solution is pretty simple, and arguably not very elegant. Instead of splicing the array, I'm simply traversing the array with a pointer to the current position. This is *a lot* faster, but leaves the entire queue sitting in memory until it is cleaned up with getCompletedRequests().

For my use, this is a reasonable tradeoff. If the queue residing in memory is big enough to cause memory problems, the data should be streamed from disk anyway (using the callback functionality of RollingCurl).