Package: lsm
lsmFast Python bindings forSQLite's LSM key/value store.
The LSM storage engine was initially written as part of the experimental
SQLite4 rewrite (now abandoned). More recently, the LSM source code was moved
into the SQLite3source treeand has seen some improvements and fixes. This project uses the LSM code from
the SQLite3 source tree.Features:Embedded zero-conf database.Keys support in-order traversal using cursors.Transactional (including nested transactions).Single writer/multiple reader MVCC based transactional concurrency model.On-disk database stored in a single file.Data is durable in the face of application or power failure.Thread-safe.Releases GIL for read and write operations
(each connection has own mutex)Page compression (lz4 or zstd)Zero dependency static libraryPython 3.x.Limitations:The source for Python lsm ishosted on GitHub.If you encounter any bugs in the library, pleaseopen an issue,
including a description of the bug and any related traceback.Quick-startBelow is a sample interactive console session designed to show some of the
basic features and functionality of thelsmPython library.To begin, instantiate aLSMobject, specifying a path to a database file.fromlsmimportLSMdb=LSM('test.ldb')assertdb.open()More pythonic variant is using context manager:fromlsmimportLSMwithLSM("test.ldb")asdb:assertdb.info()Not opened database will raise a RuntimeError:importpytestfromlsmimportLSMdb=LSM('test.ldb')withpytest.raises(RuntimeError):db.info()Binary/string modeYou should select mode for opening the database withbinary: bool = Trueargument.For example when you want to store strings just passbinary=False:fromlsmimportLSMwithLSM("test_0.ldb",binary=False)asdb:# must be str for keys and valuesdb['foo']='bar'assertdb['foo']=="bar"Otherwise, you must pass keys and values adbytes(default behaviour):fromlsmimportLSMwithLSM("test.ldb")asdb:db[b'foo']=b'bar'assertdb[b'foo']==b'bar'Key/Value Featureslsmis a key/value store, and has a dictionary-like API:fromlsmimportLSMwithLSM("test.ldb",binary=False)asdb:db['foo']='bar'assertdb['foo']=='bar'Database apply changes as soon as possible:importpytestfromlsmimportLSMwithLSM("test.ldb",binary=False)asdb:foriinrange(4):db[f'k{i}']=str(i)assert'k3'indbassert'k4'notindbdeldb['k3']withpytest.raises(KeyError):print(db['k3'])By default, when you attempt to look up a key,lsmwill search for an
exact match. You can also search for the closest key, if the specific key you
are searching for does not exist:importpytestfromlsmimportLSM,SEEK_LE,SEEK_GE,SEEK_LEFASTwithLSM("test.ldb",binary=False)asdb:foriinrange(4):db[f'k{i}']=str(i)# Here we will match "k1".assertdb['k1xx',SEEK_LE]=='1'# Here we will match "k1" but do not fetch a value# In this case the value will always be ``True`` or there will# be an exception if the key is not foundassertdb['k1xx',SEEK_LEFAST]isTruewithpytest.raises(KeyError):print(db['000',SEEK_LEFAST])# Here we will match "k2".assertdb['k1xx',SEEK_GE]=="2"LSMsupports other common dictionary methods such as:keys()values()items()update()Slices and IterationThe database can be iterated through directly, or sliced. When you are slicing
the database the start and end keys need not exist --lsmwill find the
closest key (details can be found in theLSM.fetch_range()documentation).fromlsmimportLSMwithLSM("test_slices.ldb",binary=False)asdb:# clean databaseforkeyindb.keys():deldb[key]db['foo']='bar'foriinrange(3):db[f'k{i}']=str(i)# Can easily iterate over the database itemsassert(sorted(itemforitemindb.items())==[('foo','bar'),('k0','0'),('k1','1'),('k2','2')])# However, you will not read the entire database into memory, as special# iterator objects are used.assertstr(db['k0':'k99']).startswith("<lsm_slice object at")# But you can cast it to the list for exampleassertlist(db['k0':'k99'])==[('k0','0'),('k1','1'),('k2','2')]You can use open-ended slices. If the lower- or upper-bound is outside the
range of keys an empty list is returned.withLSM("test_slices.ldb",binary=False,readonly=True)asdb:assertlist(db['k0':])==[('k0','0'),('k1','1'),('k2','2')]assertlist(db[:'k1'])==[('foo','bar'),('k0','0'),('k1','1')]assertlist(db[:'aaa'])==[]To retrieve keys in reverse order or stepping over more than one item,
simply use a third slice argument as usual.
Negative step value means reverse order, but first and second arguments
must be ordinarily ordered.withLSM("test_slices.ldb",binary=False,readonly=True)asdb:assertlist(db['k0':'k99':2])==[('k0','0'),('k2','2')]assertlist(db['k0'::-1])==[('k2','2'),('k1','1'),('k0','0')]assertlist(db['k0'::-2])==[('k2','2'),('k0','0')]assertlist(db['k0'::3])==[('k0','0')]You can alsodeleteslices of keys, but note that deletewill notinclude the keys themselves:withLSM("test_slices.ldb",binary=False)asdb:deldb['k0':'k99']# Note that 'k0' still exists.assertlist(db.items())==[('foo','bar'),('k0','0')]CursorsWhile slicing may cover most use-cases, for finer-grained control you can use
cursors for traversing records.fromlsmimportLSM,SEEK_GE,SEEK_LEwithLSM("test_cursors.ldb",binary=False)asdb:deldb["a":"z"]db["spam"]="spam"withdb.cursor()ascursor:cursor.seek('spam')key,value=cursor.retrieve()assertkey=='spam'assertvalue=='spam'Seeking over cursors:withLSM("test_cursors.ldb",binary=False)asdb:db.update({'k0':'0','k1':'1','k2':'2','k3':'3','foo':'bar'})withdb.cursor()ascursor:cursor.first()key,value=cursor.retrieve()assertkey=="foo"assertvalue=="bar"cursor.last()key,value=cursor.retrieve()assertkey=="spam"assertvalue=="spam"cursor.previous()key,value=cursor.retrieve()assertkey=="k3"assertvalue=="3"Finding the first match that is greater than or equal to'k0'and move
forward until the key is less than'k99'withLSM("test_cursors.ldb",binary=False)asdb:withdb.cursor()ascursor:cursor.seek("k0",SEEK_GE)results=[]whilecursor.compare("k99")>0:key,value=cursor.retrieve()results.append((key,value))cursor.next()assertresults==[('k0','0'),('k1','1'),('k2','2'),('k3','3')]Finding the last match that is lower than or equal to'k99'and move
backward until the key is less than'k0'withLSM("test_cursors.ldb",binary=False)asdb:withdb.cursor()ascursor:cursor.seek("k99",SEEK_LE)results=[]whilecursor.compare("k0")>=0:key,value=cursor.retrieve()results.append((key,value))cursor.previous()assertresults==[('k3','3'),('k2','2'),('k1','1'),('k0','0')]It is very important to close a cursor when you are through using it. For this
reason, it is recommended you use theLSM.cursor()context-manager, which
ensures the cursor is closed properly.Transactionslsmsupports nested transactions. The simplest way to use transactions
is with theLSM.transaction()method, which returns a context-manager:fromlsmimportLSMwithLSM("test_tx.ldb",binary=False)asdb:deldb["a":"z"]foriinrange(10):db[f"k{i}"]=f"{i}"withLSM("test_tx.ldb",binary=False)asdb:withdb.transaction()astx1:db['k1']='1-mod'withdb.transaction()astx2:db['k2']='2-mod'tx2.rollback()assertdb['k1']=='1-mod'assertdb['k2']=='2'You can commit or roll-back transactions part-way through a wrapped block:fromlsmimportLSMwithLSM("test_tx_2.ldb",binary=False)asdb:deldb["a":"z"]foriinrange(10):db[f"k{i}"]=f"{i}"withLSM("test_tx_2.ldb",binary=False)asdb:withdb.transaction()astxn:db['k1']='outer txn'# The write operation is preserved.txn.commit()db['k1']='outer txn-2'withdb.transaction()astxn2:# This is committed after the block ends.db['k1']='inner-txn'assertdb['k1']=="inner-txn"# Rolls back both the changes from txn2 and the preceding write.txn.rollback()assertdb['k1']=='outer txn',db['k1']If you like, you can also explicitly callLSM.begin(),LSM.commit(), andLSM.rollback().fromlsmimportLSM# fill dbwithLSM("test_db_tx.ldb",binary=False)asdb:deldb["k":"z"]foriinrange(10):db[f"k{i}"]=f"{i}"withLSM("test_db_tx.ldb",binary=False)asdb:# start transactiondb.begin()db['k1']='1-mod'# nested transactiondb.begin()db['k2']='2-mod'# rolling back nested transactiondb.rollback()# comitting top-level transactiondb.commit()assertdb['k1']=='1-mod'assertdb['k2']=='2'Thanks to@coleifer- this project was inspired bycoleifer/python-lsm-db.