reverbrain/elliptics — pull request #561 by agend (2014-11-01 22:13:02+00:00)
Title: improved perfromance for recovery of many keys - removed final_merge step

MR for fix issue#556 When there are lots of keys to recover in final_merge pickle dump consume lots of memory and CPU as it's perform checking already pickled objects (https://docs.python.org/2/library/pickle.html#relationship-to-other-python-modules "The pickle module keeps track of the objects it has already serialized, so that later references to the same object won’t be serialized again. marshal doesn’t do this.")
I have found that it's possible to put all data in final merged file to remove usless disk IO.