floere/picky — Issue #71: Can loading and dumping with the standard backend be faster?
Author: rogerbraun
Created: 2012-01-31 11:43:00+00:00
State: open

When running Picky integrated into another application (in my case, a Rails app), it gets started and stopped a lot. If you do dumping and reloading, this means most of the time of startup and shutdown is spent in Picky.

The obvious way out is not to use Picky like this, instead mounting a separate Picky server. I do like the simplicity of the integrated approach, though. Maybe something can be done to make loading and dumping a bit faster. This is all for the in-memory index. Loading for other backends should already be fast.

Ideas:
1. File count could be reduced. Maybe if the one large file could be read instead of many small ones, this would improve loading. The same may be true for dumping.
2. Parallelize loading and dumping. I don't know if this would change anything, but it might be worth a try.

Any more ideas?