Package: dredis
DRedis: Disk-based Redis implementationRedis is a great key-value database and it's extremely fast because it's in-memory.
Some people want Redis's rich data types without having to worry about the memory limitations. For those
that can afford slower performance and want unlimited storage, DRedis may be an alternative.WARNING: This project is still experimental and it doesn't implement all Redis commands!InstallingMake sure to install theLevelDBC++ library (apt-get install libleveldb-devorbrew install leveldb) and then run:$pipinstalldredisNote: The LMDB backend doesn't require external dependencies.Running$dredis--dir/tmp/dredis-dataTo know about all of the options, use--help:$dredis--help
usage:dredis[-h][-v][--hostHOST][--portPORT][--dirDIR][--backend{lmdb,leveldb,memory}][--backend-optionBACKEND_OPTION][--rdbRDB][--debug][--flushall][--readonly][--requirepassREQUIREPASS][--gc-intervalGC_INTERVAL][--gc-batch-sizeGC_BATCH_SIZE]optionalarguments:-h,--helpshowthishelpmessageandexit-v,--versionshowprogram'sversionnumberandexit--hostHOSTserverhost(defaultsto127.0.0.1)--portPORTserverport(defaultsto6377)--dirDIRdirectorytosavedata(defaultstoatemporarydirectory)--backend{lmdb,leveldb,memory}key/valuedatabasebackend(defaultstoleveldb)--backend-optionBACKEND_OPTIONdatabasebackendoptions(e.g.,--backend-optionmap_size=BYTES)--rdbRDBRDBfiletoseeddredis--debugenabledebuglogs--flushallrunFLUSHALLonstartup--readonlyacceptread-onlycommands--requirepassREQUIREPASSrequireclientstoissueAUTH<password>beforeprocessinganyothercommands--gc-intervalGC_INTERVALkeygcintervalinmilliseconds(defaultsto500)--gc-batch-sizeGC_BATCH_SIZEkeygcbatchsize(defaultsto10000)Running dredis with Docker locally (port 6377 on the host):$docker-composeupAsynchronous Deletions (Key Garbage Collection)This is a new and experimental feature that was necessary at Yipit to offload dredis onRENAMEandDELoperations.
When a sorted set, hash, or set is stored using any of the backends, dredis creates a few keys in the backend for each element of those collections.
With this new feature, onRENAMEandDELETE, only the key holding the key ID/pointer is replaced/deleted.
There's a background thread that periodically deletes those related keys from the storage backend (you can tweak the GC options via--gc-intervaland--gc-batch-size).
Despite the asynchronous deletions, the deletion operation still ensures strong consistency because of the use of key IDs/pointers.If you don't want this experimental feature, you need to go back to DRedis 2.5.3.BackendsThere's support for LevelDB, LMDB, and an experimental memory backend.
All backend options should be passed in the command line as--backend-option NAME1=value1 --backend-option NAME2=value2(the values must be JSON-compatible).LevelDBLevelDB is the easiest persistent backend because it doesn't require any option tweaking to get it to work reliably.OptionsWe useplyvelas the LevelDB backend. All available options are parameters ofplyvel.DB.The current default options for LevelDB are:name: The same value as the--diroptioncreate_if_missing:TrueLMDBThe performance of LMDB can be better than LevelDB and we're considering making it the default backend in the future.OptionsWe usepy-lmdbas the LMDB backend. All available options are parameters oflmdb.Environment.
We recommend that you think ahead and change themap_sizeparameter according to your needs — this is the maximum size of the LMDB database file on disk.The current default options for LMDB are:path: The same value as the--diroptionmap_size:1073741824(1GB)map_async:Truewritemap:Truereadahead:Falsemetasync:FalseMemoryThis is experimental and doesn't persist to disk. It was created to have a baseline to compare persistent backends.OptionsNone.Supported CommandsCommand signatureTypeCOMMAND*ServerCONFIG GET parameterServerCONFIG HELPServerCONFIG SET parameter valueServerDBSIZEServerFLUSHALLServerFLUSHDBServerSAVEServerDEL key [key ...]KeysDUMP keyKeysEXISTS key [key ...]KeysEXPIRE key ttl**KeysKEYS patternKeysRENAME key newkeyKeysRESTORE key ttl serialized-value [REPLACE]***KeysTTL keyKeysTYPE keyKeysAUTHConnectionPING [msg]ConnectionSELECT dbConnectionGET keyStringsGETRANGE key start endStringsINCR keyStringsINCRBY key incrementStringsSET key valueStringsSADD key value [value ..]SetsSCARD keySetsSISMEMBER key valueSetsSMEMBERS keySetsEVAL script numkeys [key ...] [arg ...]ScriptingZADD key [NX|XX] score member [score member ...]Sorted SetsZCARD keySorted SetsZCOUNT key min_score max_scoreSorted SetsZRANGE key start top [WITHSCORES]Sorted SetsZRANGEBYSCORE key min_score max_score [WITHSCORES] [LIMIT offset count]Sorted SetsZRANK key memberSorted SetsZREM key member [member ...]Sorted SetsZSCAN key cursor [MATCH pattern] [COUNT count]Sorted SetsZSCORE key memberSorted SetsZUNIONSTORE destination numkeys key [key ...] [WEIGHTS weight [weight ...]]Sorted SetsHDEL key field [field ...]HashesHGET key valueHashesHGETALL keyHashesHINCRBY key field incrementHashesHKEYS keyHashesHLEN keyHashesHSET key field value [field value ...]HashesHSETNX key field valueHashesHVALS valueHashesFootnotes:*COMMAND's reply is incompatible at the moment, it returns a flat array with command names (their arity, flags, positions, or step count are not returned).**EXPIREdoesn't set key expiration yet, it's a no-op command***RESTOREdoesn't work with Redis strings compressed with LZF or encoded asOBJ_ENCODING_INT; also doesn't work with sets encoded asOBJ_ENCODING_INTSET, nor hashes and sorted sets encoded asOBJ_ENCODING_ZIPLIST.CONFIG GET,CONFIG HELP, andCONFIG SETare specific to dredis. The commands' signature and behavior are equivalent to the ones in RedisHow is DRedis implementedInitially DRedis had its own filesystem structure, but then it was converted to useLevelDB, which is a lot more reliable and faster (nowadays there's also the LMDB backend).Other projects implement similar features to what's available on DRedis, but they aren't what Yipit needed when the project started. Some of them
rely on multiple threads and compromise on consistency, don't implement Lua scripts, or don't implement sorted sets correctly. We ran the DRedis tests against a few solutions and they failed (which means they're not fully compatible).Similar projects:https://github.com/yinqiwen/ardbardbseems to be the most similar in scope and a good candidate for contributions or a fork. Their sorted sets implementation has a bug with negative scores. It's a large C++ project with lots of features.https://github.com/Qihoo360/pikano Lua support. This is a large C++ project and its documentation is in Chinese. The project seems to be stable and is used by large Chinese companies.https://github.com/KernelMaker/blackwidowa C++ library, not a Redis-like serverhttps://github.com/siddontang/ledisdbSimilar to Redis but different commandshttps://github.com/reborndb/qdbNo Lua support and no longer maintainedhttps://github.com/alash3al/redixNo Lua and no sorted set supporthttps://github.com/meitu/titanNo Lua support and not enough sorted set supportLua supportLua is supported through thelupalibrary.ChallengesData ConsistencyWe rely on the backends' consistency properties and we use batches/transactions to stay consistent. Tweaking the backend options may impair consistency (e.g.,sync=falsefor LMDB).Cluster mode & ReplicationReplication, key distribution, and cluster mode are not supported.
If you want higher availability you can create multiple servers that share or replicate a disk (consistency may suffer when replicating).
Use DNS routing or a network load balancer to route requests properly.BackupsThe commandSAVEcreates a snapshot in the same format as Redis's RDB version 7 (compatible with Redis 3.x).
We recommend you to runSAVEon a secondarydredisprocess, otherwise the server will hang during the snapshot (consistency guarantees are higher with LMDB as the backend).
The commandBGSAVEmay be supported in the future.Other backups solutions involve backing up the files created by the backend.
A straightforward approach is to have periodic backups to an object storage such as Amazon S3 orr use a block storage solution and perform periodic backups (e.g., AWS EBS).If you useSAVEfrom a secondary process or backup the data directory, there shouldn't be any significant impact on the main server.Why PythonBecause it's a good language to get things off the ground quickly and everybody at Yipit knows it well.
If this becomes a valuable project, other languages will be evaluated — the language of choice won't affect much of the I/O bottleneck, but it may bring good performance benefits.
We're experimenting withCythonto get better performance without having to rewrite large chunks in C.The project will migrate to Python 3 soon.Didn't you have better names?@andrewgrosssuggestedREDISK. The name will only matter if this project survives, it's still an experiment.
Also,other projects use the name redisk.@nadlerjessiesuggested we pronounce dredis as "Doctor Redis".Releasing dredisMake sure you have all important changes in the top section ofCHANGELOG.mdMake sure your PyPI credentials are correct in~/.pypircRunmake releaseEnter the new version (e.g.,1.0.0)