pixelogik/NearPy — pull request #29 by wanji (2014-10-26 07:32:47+00:00)
Title: more compact serialization; fix dtype issue

1. Serialization for Redis is requires much more memory than the in-memory way. A `4096-D` single precision float vector requires `83KB` memory, while the useful data only requires `16KB` memory.
I have replaced `tostring()` with `tolist()`, and replaced `json.dumps` with `cPickle.dumps`. The amount of required memory reduced to about `16KB`.

2. While rebuilding `numpy.array` from Redis, `float64` is used as `dtype`, which causes errors for other data type. I recored the `dtype` field of `numpy.array` in `var_dict`, and used it for rebuilding `numpy.array`