coleifer/huey — pull request #52 by jbaiter (2014-02-25 21:54:38+00:00)
Title: LevelDB backend

After experimenting with the SQLite backend for a while (#50) and finding its performance to be very much lacking during benchmarking (see table below), I had a go at writing another non-distributed backend, this time using [LevelDB](http://code.google.com/p/leveldb/).

It makes use of the [Plyvel](http://plyvel.readthedocs) bindings, which work with Python2/3 and PyPy.

Here are some very unscientific benchmarks I made:

Backend | Submit/Read 1000 tasks |
-------------------|------------------------- |
LevelDB | 950ms |
SQLite (ramdisk) | 1,120ms |
SQLite (disk) [1]| 242,200ms (*yep...*) |
Redis | 620ms |

[1] 7200rpm SATA 3.0Gb/s formatted with ext4

Together with the SQLite backend, this gives the user some great choices, depending on their needs:

- Redis:
* :+1: Fastest, distributed
* :-1: Requires external service
- SQLite:
* :+1: No dependencies, doesn't require an external service
* :-1: Slow, not distributed
- LevelDB:
* :+1: Fast, doesn't require an external service
* :-1: Not distributed