robotframework/RIDE — pull request #16 by dragonfyre13 (2013-09-24 17:24:56+00:00)
Title: Fix memory leak running tests (in testrunner plugin)

There exists a memory leak related to the use of Pickle for sending and receiving data over the socket. Sustained use of the same pickle object for dumping and particularly loading can/will cause all sent and received data to retain a reference in the pickle object, due to memoization of data.

The memory leak is a known issue in python since 2.0, per this defect in python (closed as "Won't Fix", per guido, "don't do this"): http://bugs.python.org/issue229810

The "dump" method of a pickle object has a workaround added, in clearing the memo (http://docs.python.org/2/library/pickle.html#pickle.Pickler.clear_memo) however Unpickler.load does not. Even if Pickler.dump clears it's memo, Unpickler.load does not know it is cleared, and thus continues to retain a reference to every object received over the socket/file descriptor.

The modifications here add a new class, StreamHandler, to the testrunner code. This handler uses JSON to store the data rather than pickle, and properly adds a message header with type of object and length. If Python 2.5 is used, and SimpleJson is not loaded, the class falls back to using pickle.loads and pickle.dumps rather than their file writing alternatives. These do not have the same memory leaks, as Pickler/Unpickler instances are created for each object being pickled. Using pickle.loads and pickle.dumps is not an ideal solution where alternatives are available largely because of CPU usage, and the performance hit of creating a new StringIO and Pickler/Unpickler object with each message, but it's a fallback for versions of python prior to 2.6 which includes json as a default library.

JSON was chosen for a serialization format as the standard library does not have the same memory leak with JSON as with pickle, and it was the most readily available alternative in python standard library, as well as being well understood by a majority of other developers (contrary to using structs or other packaging formats for the information). Should an odd situation come up where the listener is running in a different python version, or in 2.5 where simplejson is loaded, and pybot is not (thus the actual execution occurs in a context where pickle is used rather than json to serialize the data), the StreamHandler class manages this by virtue of a "msgtype" sent in each header, differentiating between pickle serialized and json serialized messages.

The code was finished yesterday afternoon, and tested rigerously through a test set of over 18,000 tests producing an average of over 4k messages each from testrunner back to RIDE. There still exist other memory leaks/held references throughout the test run, but the reduction is in stark contrast. Previously to the modification a run of 500 tests would increase memory from 86MB to nearly 1GB for the RIDE process alone, and from an original 65MB to 450 or 500MB for testrunner. After the modifications in this fork, memory increase is limited to a final 300MB usage (up from ~86MB original) after a test run for the RIDE process, and ~95MB after a test run for the testrunner process (up from ~65MB original).

Memory usage still increases with each subsequent test run, but other issues with persistently held references to objects will be tackled at a later time, in other areas of the RIDE code.

Performance impact of the code is negligible (~5% average) for anything approaching a reasonable use case, however if logging is performed as the sole action in a test, wrapped in a keyword called in a for loop over 10k iterations, our testing showed an increase in resource utilization of nearly 38% shared between the testrunner and RIDE processes. Performance was measured through stdlib profilers, lineprofiler/kernprof as well as my in-house developed python profiling tools, using an external process monitor to verify results holistically.

The code has been tested on windows and linux machines (not OSX, though do not expect vast differences in behavior) running python 2.5, 2.6 and 2.7 releases, with and without simplejson loaded. Also have tested using non-C based libraries of pickle and StringIO by removing appropriate import lines while using same python versions, as import statements in the code default to C based libraries while falling back to pure python versions where import of cPickle and cStringIO fail (e.g. pypy, jython, etc.). Note PyPy, Jython and IronPython were not explicitly tested as target platforms due largely to lack of an existing test environment running these.