Package: ckydb
ckydbA simple fast memory-first thread-safe (or goroutine-safe for Go) key-value embedded database that persists data on disk.It is read as 'skydb' This the python implementation of ckydbQuick StartCreate a new project and activate your virtual environmentmkdirckydb_examplecdckydb_example
python3-mvenvenvsourceenv/bin/activateInstall ckydbpipinstallckydbCreate a main.py file in the project folder and add the following codeif__name__=='__main__':importckydbkeys=["hey","hi","salut","bonjour","hola","oi","mulimuta"]values=["English","English","French","French","Spanish","Portuguese","Runyoro"]withckydb.connect("db",max_file_size_kb=(4*1024),vacuum_interval_sec=(5*60))asdb:# setting the keysfork,vinzip(keys,values):db.set(k,v)fori,kinenumerate(keys):assertvalues[i]==db.get(k)# updating keysnew_values=["Jane","John","Jean","Marie","Santos","Ronaldo","Aliguma"]fork,vinzip(keys,new_values):db.set(k,v)fori,kinenumerate(keys):assertnew_values[i]==db.get(k)# deleting the keysforkinkeys[:2]:db.delete(k)fork,vinzip(keys[2:],new_values[2:]):assertv==db.get(k)errors=[]forkinkeys[:2]:try:v=db.get(k)exceptckydb.exc.NotFoundErrorasexc:errors.append(exc)assertlen(errors)==len(keys[:2])# clear the databaseerrors.clear()db.clear()forkinkeys:try:v=db.get(k)exceptckydb.exc.NotFoundErrorasexc:errors.append(exc)assertlen(errors)==len(keys)Run themain.pymodule and observe the terminalpythonmain.pyHow to Run TestsClone the repogitclonegit@github.com:sopherapps/ckydb.gitEnter the python implementation foldercdckydb/implementations/py_ckydbCreate and activate the python 3.7+ virtual environmentpython3-mvenvenvsourceenv/bin/activateRun the test commandpython-munittestUnder the HoodEvery key has a TIMESTAMP prefix, added to it on creation. This TIMESTAMPED key is the one used to store data in a
sorted way for easy retrieval.The actual key known by user, however, is kept in the index. When ckydb is initialized, the index is loaded into
memory from the index file (a ".idx" file). The index is basically a map ofkey: TIMESTAMPED-keyThe TIMESTAMPED-key and its value are stored first in a log file (a ".log" file). This current log file has an
in-memory copy we callmemtableWhen the current log file exceeds a predefined size (4MBs by default), it is converted to a sorted data file (a ".cky"
file) andmemtablerefreshed and a new log file created.The names of each ".cky" or ".log" file are the timestamps when they were created. Do note that conversion of ".log"
to "cky" just changes the file extension.There is always one ".log" file in the database folder. If on initialization, there is no ".log" file, a new one is
created.There is an in-memory sorted list of ".cky" files calleddata_filesthat is kept updated everytime a ".log" file is
converted into ".cky".The name of the current log (current_log_file) file is also kept in memory, and updated when a new log file is
created.There is also a ".del" file that holds all thekey: TIMESTAMPED-keypairs that have been marked for deletion.At a predefined interval (5 minutes by default), a background task deletes the values from ".cky" and ".log" files
corresponding to thekey: TIMESTAMPED-keypairs found in the ".del" file. Each deleted pair is then removed from
the ".del" file.On initial load, any keys in .del should have their values deleted in the corresponding ".log" or ".cky" filesOperationsOnckydb.set(key, value):the corresponding TIMESTAMPED key is searched for in the indexif the key does not exist:a new TIMESTAMPED key is created and added to the index with its user-defined keythe user-defined key and its TIMESTAMPED key are then added to the index file (".idx")this TIMESTAMPED key and its value are then added tomemtable.this TIMESTAMPED key and its value are then added to the current log file (".log")A check is made on the size of the log file. If the log file is bigger than the max size allowed,
it is rolled into a .cky file and a new log file created, and thememtablerefreshed.if the key exists:its timestamp is extracted and compared to the current_log file to see if it is later than the current_log
fileif it is later or equal,memtableand the current log file are updatedelse the timestamp is compared to cache's "start" and "stop" to see if it lies within the cacheif it exists in the cache, then the cache data and its corresponding data file are updatedelse, the data file in which the timestamp exists is located within the data_files. This is done by finding
the two data files between which the timestamp exists when the list is sorted in ascending order. The file to
the left is the one containing the timestamp.the key-values from the data file are then extracted and they new key-value insertedthe new data is then loaded into the cachethe new data is also loaded into the data fileIf any error occurs on any of these steps, the preceding steps are reversed and the error returned/raised/thrown
in the callOnckydb.delete(key):Itskey: TIMESTAMPED-keypair is removed from the in-memory index.Itskey: TIMESTAMPED-keypair is removed from the ".idx" fileItskey: TIMESTAMPED-keyis added to the ".del" fileIf any error occurs on any of these steps, the preceding steps are reversed and the error returned/raised/thrown
in the callOnckydb.get(key):the corresponding TIMESTAMPED key is searched for in the indexif the key does not exist, a NotFoundError is thrown/raised/returned.if the key exists, its TIMESTAMP is extracted and checked if it is greater (later) than the name of the current
log file.if this TIMESTAMP is later, its value is quickly got frommemtablein memory. If for some crazy reason, it does
not exist there, a CorruptedDataError is thrown/raised/returned.If this TIMESTAMP is earlier than the name of the current log file, the TIMESTAMP is compared to the range in the
memorycache, if it falls there in, its value is got fromcache. If the value is not found for some reason, a
CorruptedDataError is thrown/raise/returnedOtherwise the ".cky" file whose name is earlier than the TIMESTAMP but whose neighbour to the right, in the
in-memory sorteddata_fileslist, is later than TIMESTAMP is loaded into an in-memorycachewhose range is set
to two ".cky" filenames between which it falls.the value is then got fromcache's data. If it is not found for some reason, a CorruptedDataError is
thrown/raise/returnedOnckydb.clear():memtableis resetcacheis resetindexin memory is resetdata_filesin memory is resetall files in the database folder are deletedA new ".log" file is createdFile formatsThe file format of the ".idx" index files is just "key<key_value_separator>TIMESTAMPED-key" separated by a
unique token e.g. "{&*/%}" and a key_value_separator e.g. "[><?&(^#]"goat[><?&(^#]1655304770518678-goat{&*/%}hen[><?&(^#]1655304670510698-hen{&*/%}pig[><?&(^#]1655304770534578-pig{&*/%}fish[><?&(^#]1655303775538278-fish$%#@*&^&The file format of the ".del" files is just "TIMESTAMPED-key" separated by a unique token e.g. "{&*/%}"1655304770518678-goat{&*/%}1655304670510698-hen{&*/%}1655304770534578-pig{&*/%}1655303775538278-fish$%#@*&^&The file format of the ".log" and ".cky" files is just  "TIMESTAMPED-key<key_value_separator>value" separated
by a unique token e.g. "{&*/%}" and a key_value_separator like "[><?&(^#]"1655304770518678-goat[><?&(^#]678 months{&*/%}1655304670510698-hen[><?&(^#]567 months{&*/%}1655304770534578-pig[><?&(^#]70 months{&*/%}1655303775538278-fish[><?&(^#]8990 months$%#@*&^&Note: There is configuration that one can enable to escape the "token" in any user-defined key or value just to avoid
weird errors. However, the escaping is expensive and it is thus turned off by default.Ideas For ImprovementExplicitly allow for multiple concurrent reads (e.g. don't lock at all on read)Explicitly allow for conditional multiple concurrent writes (e.g. lock on key, not on store)Distribute the database across different machines or nodes (
e.g. have multiple backend nodes, and let each node's timestamped key range be recorded on the master/main/gateway
node(s). The gateway nodes themselves could be replicated. Clients read/update data through the gateway node)Multiple Concurrent Reads, Single Writes at a timeHave no lock on the main routine ofckydb.get.ckydb.gethas propsindex,memtableandcacheas its source of truth.To avoid using a stalecacheand yet also avoid data races betweenstore.setandstore.get, both, of old keys,
we have acache_locklock. This lock is to be obtained by eitherstore.getorstore.setboth for old keysHave the samemut_locklock on theckydb.deleteandckydb.set. If you had separate locks, there would be chance
for a data race.Forckydb.clear, updateindexfirst.Forckydb.delete, updateindexlast.Forckydb.setof a new key (i.e. not an update), updateindexlast.Forckydb.setof pre-existing key, updatememtableorcachelastas index would already be up-to-date.Forstore.vacuumtask andstore.delete, there will be adel_file_lockwithin store to avoid conflicts.AcknowledgmentsWe can do nothing without God (John 15: 5). Glory be to Him.Some of these ideas were adapted fromleveldb. Thanks.LicenseCopyright (c) 2022Martin Ahindura. All implementations are licensed under
theMIT License