graphite-project/graphite-web — pull request #613 by baites (2014-02-03 16:35:01+00:00)
Title: Bug fix remote connection multithread issues between sending and receivi...

This is and simplification to fix the problem of sending and receiving queries to discover metrics in a graphite cluster as implemented in remote_storage.py.

I have documented some of my findings in the issues #592 and #582. The problem centered in the sending and parsing of the request to discover those metrics not locally present in graphite server. The gist of the issue was that the GET request and parsing of that request response was not done in the same thread.

The request is handle by the *connection* variable defined by the first thread to reach fetch function. That created the issue that *connection* might not even define by the time another thread was trying to read the response of the request. For all the problematic cases I saw, the assignment of *connection* was successful before leaving fetch function, but after another thread tried to read the response, triggering the exception as documented in #582.

My first two solutions were to try to add a lock and later an event in an attempt to make sure the request is read after it was sent and responded. For reason it not clear to me this did not worked out. I was fooled a few times by this bug as to be fixed because of its sporadic nature.

Therefore I decided to simplify the code by reducing the number of locks and events. For this the thread that sends the request is the the same that waits for the response and parses it. This eliminated the need for synchronizing the send/receiving and parsing of the request by different threads. This also eliminated the need for a thread for sending/receiving and another thread for parsing the request.

I have being heavily testing this new code and at the moment worked in all the cases.