ooici/marine-integrations — pull request #85 by n1ywb (2013-05-26 04:38:40+00:00)
Title: Instrument Driver Performance Enhancements

I profiled the instrument driver process by sending packets representative of what I expect to see from Antelope, and found two major bottlenecks, both in particle generation; the deepcopy and json encoding of the particle dict.

Simply turning off key sorting gave a huge improvement.

Switching to simplejson gives another huge improvement, due to it's C speedups. It's API compatible with the standard lib json module (in fact the std lib json module IS simplejson minus the C speedups). I made it fall back to json if simplejson is not installed. Note: the instrument agent may want to change json libs as well.

Finally replacing the deepcopy with a shallow dict copy constructor gives another significant improvement. I'm not sure why the copy is necessary at all, but the dict only holds immutable values, so the deepcopy is clearly unnecessary.

These few small changes improve particle generation performance on antelope-dev01 from 1400 to 4000 particles per second, a factor of 2.86. I estimate we'll need to handle a rate of 2532 particles per second to publish all of the antelope data. I realize that full load will probably not be taken up by a single driver process, but at the same time you want to have some overhead in your CPU cycles. And we probably will want to be able to handle 1200 sps in a single driver process. Prior to these changes that would really be pushing it. With these changes it should be loafing.

It passes bin/nosetests mi/core/instrument/test