Package: jk-keyvaluestore
jk_keyvaluestoreIntroductionThis python module implements a simple key-value data base based on JSON data stored in a directory.Information about this module can be found here:github.orgpypi.python.orgGoalsThis module provides a simple key-value-based data store. The current implementation does not aim for highes performance but for ease of use, even in interprocess domain. This means that ...You can instantiate a data store class right away and use it to read and write key-value data. The only thing you need to provide is a writable data directory.A data store can be read-write or read-only.A key can be associated with arbitrary amounts of JSON data.Other data store instances (sharing the same directory) can perform writes as well. These will be visible to the other data store instances.LimitationsThe current implementation is pretty straight forward. Every change causes a write of a JSON file that contains that data. This way a) persistency is ensured and b) other instances of a data store will receive the new data. Therefore the concurrency model is very simple: THe most recent write wins.Though this concept has drawbacks:Managing thousands of key-value-pairs will be inefficient as for every key-value-pair a single file is required to hold that data.High frequent changes of data will be inefficient as well.As time stamps are used to synchronize one or more data store instances if you use multiple processes residing on multiple hosts you need to ensure that the system time will not be much different accross all nodes.This key-value store is ment for use cases where you desire to keep things simple in your software as the amount of data you need to manage is quite limited.Use casesUse cases for this implementation is:Write out state information of an application and monitor itImplement simple interprocess communicationModify configuration data of an application while the application is runningBenefits of this approachNevertheless this approach has a variety of benefits:Portability: If you (would) require the cooperation of programs written in different languages this can be achieved very easily. Porting this python implementation to other programming languages would be very simple.How to use this moduleImport this modulePlease include this module into your application using the following code:fromjk_keyvaluestoreimportDirBasedKeyValueStoreImporting this way is recommended as currentlyDirBasedKeyValueStoreis the only class. (This might change in the future.)Using a single instance of the data storeCreate an instanceTo set up a data store instance is easy:ds=DirBasedKeyValueStore(dirPath="my/data/directory",identifier=1)For clarification all arguments have been named in this example (in their order of declaration).dirPathmust refer to an existing directory that will hold the data. If multiple instances are created they must refer to the same directory.If this is not a read only instanceidentifiermust be either a valid string or a valid integer value. If it is a string the identifier is matched againstr[a-zA-Z0-9_+-\.]+. If it is an integer the value must be greater or equal to zero. If you specifyNone(which is the default) the data store will be read only.Write dataWriting data is very simple. Example:ds.put("someKey",["some","value"])or:ds["someKey"]=["some","value"]The keys must be strings of arbitrary size. Values can be anything that is storable in JSON.Read dataReading data is very simple. Example:value=ds.get("someKey")or:value=ds["someKey"]This operation will returnNoneby default if the key does not exist. (No exception will be raised.)Check if a key existsPerforming a check for existance of a key in the data store is easy. Example:bKeyValuePairExists=ds.contains("someKey")Please note that the data store will maintain a list of all deleted entries to be able to still synchronize such information with other instances. The data returned bycontains()willnotreflect deleted entries. Example:ds.put("someKey","someValue")ds.delete("someKey")assertds.contains("someKey")==FalseDelete a single key value pairDeleting data is very simple. Example:ds.remove("someKey")or:delds["someKey"]This operation is indempotent. No exception will be raised if the key has already been deleted.Please note that in ordert to maintain synchronization capabilities with other data stores information about this delete will be kept internally in the data store.Delete all key value pairsRemoving all data from a data store is simple. Example:ds.clear()This operation is indempotent. No exception will be raised if the data store is already empty.Please note that in order to maintain synchronization capabilities with other data stores information about this delete will be kept internally in the data store.Get a list of all keysIt is possible to get a list of all keys currently in use in the data store. Example:allKeys=ds.keys()The methhodkeys()will return a list of keys.Please note that the data store will maintain a list of all deleted entries. The data returned bykeys()willnotcontain deleted entries.Using a multipe instances of the data storeInstantiationUsing multiple instances is easy. Example:# In program AdsA=DirBasedKeyValueStore(dirPath="my/data/directory",identifier=1)# In program BdsB=DirBasedKeyValueStore(dirPath="my/data/directory",identifier=2)Please note that both instances must be distinguishable. Therefore you need to provide unique identifiers for each instance matching the regular expressionr[a-zA-Z0-9_+-\.]+. (If you specify an identifier ofNoneyou will create a read only instance.)After this you can useget(),put(),remove()and other methods.SynchronizationIf intermediate changes occurred in other instances of a data store these changes are not synchronized automatically to other data store. Synchronization is entirely up to you: As synchronization is not cheap you as the developer has to decide when a synchronization should be performed.During synchronization the directory of files storing the key-value pairs is scanned for new entries. If new entries are found they will get loaded. The information contained in these files will then be incorporated into the current data store instance. As this is directly dependend on the number of files written since the last time synchronization has been performed, this might be a bit costly. Therefore it is up to you as a developer to decide when exactly these synchronizations should occur.PerformanceAs files are read and written in order to manage all data the time required for file operations is the limiting factor. Here are some performance values based on experience in Python on Linux with a regular SATA-SSD:OperationPerformancegetLimited by python interpreter performance only.put~15mssynchronize single change~10msAs during instantiationsynchronize()is called to read all existing data, instantiation performance is very similar to synchronization performance.(Possible) Future DevelopmentThe current implementation is based on synchroneous I/O operations. Future implementation options therefore would be:provide an implementation based on asynchroneous I/Oprovide an implementation for C, Java, C#, maybe even JavaScriptmaybe provide an implementation accessing data via SFTPexplore even better ways of implementing such a data store:using named pipes to communicate to a single processOther things to do:test the current implementation on an NFS shareIf you would be interested in improving or porting the current implementation you're welcome. Feel free to contact me.Contact InformationThis is Open Source code. That not only gives you the possibility of freely using this code it also
allows you to contribute. Feel free to contact the author(s) of this software listed below, either
for comments, collaboration requests, suggestions for improvement or reporting bugs:Jürgen Knauth:jknauth@uni-goettingen.de,pubsrc@binary-overflow.deLicenseThis software is provided under the following license:Apache Software License 2.0