graphite-project/graphite-web — pull request #1073 by jraby (2014-12-30 23:26:36+00:00)
Title: Do a single http req per backend - and double the perf in some cases

Hi guys,

I know the 0.9.13 merge window is basically closed, but I've cleaned up my patch from issue #1045 and ported it to the new thread stuff from #1026.

There's already a long explanation of how this works in #1045 and in https://github.com/datacratic/graphite-web/pull/2 (this one has some benchmark at the bottom)
In brief, this reduces the number of http calls done between the frontend and backends by fetching all the requested targets in a single http request.
It helps tremendously when having multiple remote backends with some latency, and with graphs with a few targets (>4-5)

NOTE: both the backends and the frontend MUST run with this code.
The backend must return a new field: 'pathExpression' in its replies,
if it doesn't the current code will raise an exception.

The code is also disabled by default, and must be enabled by setting `PREFETCH_REMOTE_DATA` to true.

It work as follows:
- extract all the pathExpressions (metric name/patterns) from the targets
- fetch all pathExpressions from all backends using a single http request
- store the result in a properly keyed hash table:
cache[backend][originalTarget-start-end] = [ series_from_backend]
- Store that hash table in the requestContext
- Continue processing as before (via evaluateTarget -> fetchData)
- In the fetchData method do a hashtable lookup for the requested data.
- If the data is there use the cached data.
- If there is a cache miss, do a regular remote fetch.

This last case is pretty rare, it happens when functions needs data out of the timerange of the original query.
For example: timeshift, movingAverage, etc.

As noted in the code, it could be nice to disable prefetching if the backend doesn't support it (instead of raising an exception).
If there's enough interest, I'll add that.

Any comments? what do you think of the concept?