numenta/nupic.core — pull request #130 by vsinha (2014-06-25 16:44:30+00:00)
Title: Replaces current serialization system with protocol buffers (don't merge)

Not ready to merge. If anyone wants to look at the code, here it is.

One main problem: with this implementation the save and load functions take filenames rather than filestreams. I think this can be fixed if we were to link the non-"lite" version of protocol buffers. Currently this uses the lite version as specified by `option optimize_for = LITE_RUNTIME;` in SpatialPooler.proto. I had problems getting the full version to link.

Another problem: I'm currently including the compiled protocol buffer files (SpatialPooler.pb.cc SpatialPooler.pb.h). These shouldn't be included and should be part of the build process in the final version.

Performance-wise, this protocol buffers solution is about twice as fast to save or load than the current serialization implementation. This solution doesn't use the protocol buffer in memory, it allocates and populates the protobuf when save() is called. When load() is called, the protobuf is loaded from disk and variables are copied over to the spatial pooler object's fields.

Here are the results from a single test of 100 saves and loads. Repeated runs yield approximately the same numbers. This test creates a spatial pooler, serializes it, and then loads it into a second spatial pooler, 100 times. (see SpatialPoolerTest.cpp)

current implementation time spent saving: 0.038861
current implementation time spent loading: 0.030781

protocol buffers time spent saving: 0.018369
protocol buffers time spent loading: 0.009601

I also ran tests to see where the time was being spent: An order of magnitude more time is spent actually serializing the protocol buffer and writing it to disk than is spent creating/allocating the protocol buffer and copying local variables over into it. (this single run of 100 saves/loads is again indicative of performance over multiple trials) (this test is in SpatialPoolerTest::testPerformance)

total time spent saving: 0.018035
time spent creating: 0.001575
time spent serializing: 0.016257

total time spent loading: 0.009382
time spent deserializing: 0.006598
time spent copying locally: 0.002559

@scottpurdy @rhyolight