Package: udb-py
Udb is an in-memory database based on theZope Foundation BTrees, theRtreeand on the native python’s dict.
Udb provides indexes support and limited MongoDB-like queries.
Udb does not support any type of transactions for now.Table of contentsRequirementsInstallationQuick startData schema for default valuesFunctional fieldsIndexesIndex declarationFloat precisionQueryingQuery validationComparison orderGetting planScan operationsStoragesSelect operationDelete operationInsert operationUpdate operationLimitationsRunning testsBenchmarksRequirementsPython 2.7, Python 3.6Installationpipinstalludb_pyTo enable BTree indexes support installZope Foundation BTreespackage:pipinstallBTreesTo enable RTree indexes support installRtreepackage (requireslibspatialindex):pipinstallRtreeQuick startCreate the Udb instance with the indexes declaration:fromudb_pyimportUdb,UdbBtreeIndexdb=Udb({'a':UdbBtreeIndex(['a']),'b':UdbBtreeIndex(['b']),'cde':UdbBtreeIndex(['c','d','e']),})Insert records:db.insert({'a':1,'b':1,'c':3,'d':4,'e':5})db.insert({'a':2,'b':2,'c':3,'d':4,'e':5})db.insert({'a':3,'b':3,'c':3,'d':4,'e':5})db.insert({'a':4,'b':4,'c':3,'d':4,'e':6})db.insert({'a':5,'b':5,'c':3,'d':4,'e':7})Select records:a=list(db.select({'a':1})[{'a':1,'b':1,'c':3,'d':4,'e':5}]b=list(db.select({'b':0})[]# no records with b=0c=list(db.select({'c':3,'d':4},limit=2)[{'a':3,'b':3,'c':3,'d':4,'e':5},{'a':4,'b':4,'c':3,'d':4,'e':6}]Data schema for default valuesData schema allows to fill the inserted record with default values.
The default value can be defined as a primitive value or callable:fromudb_pyimportUdbdb=Udb(schema={'a':'a','b':'b','c':lambdakey,record:'b'ifrecord['b']=='b'else'c',})Functional fieldsauto_id- generates unique id (uuid v1 by default)fromudb_pyimportUdb,auto_iddb=Udb(schema={'id':auto_id(),})current_timestamp- uses current timestamp (as int value)fromudb_pyimportUdb,current_timestampdb=Udb(schema={'timestamp':current_timestamp(),})fn- calls custom functionfromudb_pyimportUdb,fndb=Udb(schema={'timestamp':fn(lambdarecord:record['a']+record['b']),})optional- returns “None” valuefromudb_pyimportUdb,optionaldb=Udb(schema={'a':optional,})IndexesTo speed up the search for records, the necessary fields can be indexed.
The Udb also includes a simple query optimiser that can select the most appropriate index.BTree indexes:UdbBtreeMultivaluedIndex- btree based multivalued index supporting multiple records with the same index key.UdbBtreeMultivaluedEmbeddedIndex- same as theUdbBtreeMultivaluedIndex, but supports embedded list of values.UdbBtreeUniqIndex- btree based index operating with always single records, but the second record insertion with the same index key will raise IndexConstraintError.UdbBtreeIndex- btree based index operating with always single records, so that the second record insertion with the same index key will overwrite the old one. Can be used when the inserting record definitely generates a unique index key.Hash indexes:UdbHashMultivaluedIndex- hash based multivalued index supporting multiple records with the same index key.UdbHashMultivaluedEmbeddedIndex- same as theUdbHashMultivaluedIndex, but supports embedded list of values.UdbHashUniqIndex- hash based index operating with always single records, but the second record insertion with the same index key will raise IndexConstraintError.UdbHashIndex- hash based index operating with always single records, so that the second record insertion with the same index key will overwrite the old one. Can be used when the inserting record definitely generates a unique index key.Spatial indexes:UdbRtreeIndex- spatial index that supports “intersection with rectangle” and “near to point” searchIndex declarationAs it was shownabove, for the index declaration the Udb instance should be created with theindexesparameter that provides dict with the key as an index name and value as an index instance.
The index instance should be created with the sequence of fields (1 at least) which will be fetched in the declared order from the indexed record.
By this sequence of fields, the index key will be generated and will be associated with the indexed record.fromudb_pyimportUdb,UdbBtreeIndexdb=Udb(indexes={'abc':UdbBtreeIndex(['a','b','c'])# "a", "b" and "c" fields will be fetched from the indexed record})record={'a':'A','b':'B','c':'C'}# index key=ABCIn this case of declaration in order that the record to be indexed it must contain all of the fields declared in the sequence of index fields.fromudb_pyimportUdb,UdbBtreeIndexdb=Udb(indexes={'abc':UdbBtreeIndex(['a','b','c'])# "a", "b" and "c" fields will be fetched from the indexed record})record={'a':'A','b':'B'}# won't be indexed, raises FieldRequiredErrorUsing dictionary in case of Python 3:fromudb_pyimportUdb,UdbBtreeIndex,requireddb=Udb(indexes={'abc':UdbBtreeIndex({'a':required,'b':required,'c':required})# "a", "b" and "c" fields will be fetched from the indexed record})record={'a':'A','b':'B'}# won't be indexed, raises FieldRequiredErrorUsing list of tuples in case of Python 2 (to keep key order):fromudb_pyimportUdb,UdbBtreeIndex,requireddb=Udb(indexes={'abc':UdbBtreeIndex([('a',required),('b',required),('c',required)])# "a", "b" and "c" fields will be fetched from the indexed record})record={'a':'A','b':'B'}# won't be indexed, raises FieldRequiredErrorThe default value for missing field can be defined as a primitive value or callable (functional index):fromudb_pyimportUdb,UdbBtreeIndexdb=Udb(indexes={'abc':UdbBtreeIndex({'a':'a','b':'b','c':'c'})})record={'a':'A','c':'C'}# index key=AbCfromudb_pyimportUdb,UdbBtreeIndexdb=Udb(indexes={'abc':UdbBtreeIndex({'a':'a','b':lambdakey,values:'b','c':'c'})})record={'a':'A','c':'C'}# index key=AbCFloat precisionTo be able to index float values enable the float mode with necessary precision (number of decimals):fromudb_pyimportUdb,UdbBtreeIndexdb=Udb(indexes={'abc':UdbBtreeIndex(['a']).set_float_precision(10)})db.insert({'a':3.1415926525})QueryingUdb supports limited MongoDB-like queries which can be used in the delete, select or update operations.
The query generally is a python’s dict with the key as a field and value as a primitive value or an equality condition over the field.
The query dict ismutable, therefore it needs to be initialized every time anew.Supported query operations:$eq- equal to a valueudb.select({'a':{'$eq':5}})$gt- greater then valueudb.select({'a':{'$gt':5}})$gte- greater or equal to a valueudb.select({'a':{'$gte':5}})$in- equal to an any value in the list of a valuesudb.select({'a':{'$in':5}})$intersection- intersection with rectangleudb.select({'a':{'$intersection':{'minX':5,'minY':5,'maxX':1,'maxY':5}}})$lt- less then valueudb.select({'a':{'$lt':5}})$lte- less or equal to a valueudb.select({'a':{'$lte':5}})$ne- not equal to a valueudb.select({'a':{'$ne':5}})performs “seq” scan.$near- near to point with optional min and max distancesudb.select({'a':{'$near':{'x':5,'y':5,'minDistance':1,'maxDistance':5}}})allocates sort buffer is case of “seq” scanselects all records in case of unsetmaxDistanceand setminDistance.$nin- not equal to an any value in the list of a valuesudb.select({'a':{'$nin':[1,2,3]}})performs “seq” scan.primitive value- equal to a valueudb.select({'a':5})Example:records=list(udb.select({'a':1}))records=list(udb.select({'a':{'$gte':1,'$lte':3}}))records=list(udb.select({'a':{'$in':[1,2,3],'$lte':2}}))Query validationBy default Udb does not check the query dict validity.
To check its validity usevalidate_querymethod.udb.validate_query({'a':{'$gte':[1,2,3]}})# raises InvalidScanOperationValueError('a.$gte')Comparison orderDue to the fact that the Udb database is not strictly typed for stored values, there is the following order of ascending comparisons for values ​​of different types:Noneboolean -falseless thentrueint, floatstringSo, for example, the record containingintvalue always greater than the record containingbooleanvalue for the same field.
Also, it means, that the records having indexed field will be fetched in the provided order.Getting planTo get the query plan useselectmethod withget_plan=True:fromudb_pyimportUdb,UdbBtreeIndexdb=Udb(indexes={'abc':UdbBtreeIndex({'a':'a','b':lambdakey,values:'b','c':'c'})})db.select({'a':3},sort='-a',get_plan=True)# [(<udb.index.udb_btree_index.UdbBtreeIndex object at 0x104994080>, 'const', 1, 2), (None, 'sort', 0, 0, 'a', False)]Scan operationsBTree index:const- an index covers only one record by the index keyin- an index covers multiple records by the list of the index keys, each of which covers exactly one recordrange- an index covers multiple records by the index keys set by the minimum and maximum valuesprefix- an index covers range of records by the partial index keyprefix_in- an index covers multiple records by the list of the partial index keys, each of which covers range of recordsRTree index:intersection- an index covers records intersected by the rectanglenear- an index covers records near to the pointNo index:seq- scanning that is not covered by any index, all records will be scanned (worst case)StoragesThe storage allows keeping data persistent.UdbJsonFileStoragestores data in the JSON file.fromudb_pyimportUdbJsonFileStoragedb=Udb(storage=UdbJsonFileStorage('db'))db.load_db()db.insert({'a':'a'})db.save_db()UdbWalStoragestores data of delete, insert and update operations in the WAL (Write-Ahead-Logging) file chronologically.fromudb_pyimportUdbWalStoragedb=Udb(storage=UdbWalStorage('db'))db.load_db()db.insert({'a':'a'})db.save_db()# does nothing; delete, insert and update data will be stored on the flySelect operationSelected records aremutable, so avoid to update them directly.
Otherwise use copy on select mode:udb.set_copy_on_select()To limit the result subset to particular number of records uselimitparameter:records=list(udb.select({'a':1},limit=5)To fetch the result subset from the particular offset useoffsetparameter:records=list(udb.select({'a':1},offset=5)Delete operationudb.delete(q={'a':1},offset=5)Insert operationudb.insert({'a':1})Update operationudb.update({'a':2},q={'a':1},offset=5)Running tests with pytestpytest.--ignore=virtualenv-vLimitationsNested paths for indexing and querying are not supported, only the root levelTransactions are not supportedBenchmarksIntel Core i7, 3.58 GHz, 4 cores, disabled HT16GB 1600 MHz RAMPyPy3INSERT (BTREE, 1ST INDEX COVERS 1 FIELD)

Total time: 2.9712460041046143 sec., per sample: 2.971246004104614e-06 sec., samples per second: 336559.1400437912, total samples: 1000000

SELECT (BTREE, 1ST INDEX COVERS 1 FIELD)

Total time: 1.7301840782165527 sec., per sample: 1.7301840782165527e-06 sec., samples per second: 577973.1836573046, total samples: 1000000

INSERT (BTREE, 1ST INDEX COVERS 1 FIELD, 2ND INDEX COVERS 1 FIELD, 3RD INDEX COVERS 2 FIELDS)

Total time: 6.8810200691223145 sec., per sample: 6.881020069122315e-06 sec., samples per second: 145327.29013353275, total samples: 1000000

SELECT (BTREE, 1ST INDEX COVERS 1 FIELD, 2ND INDEX COVERS 1 FIELD, 3RD INDEX COVERS 2 FIELDS)

Total time: 1.8345210552215576 sec., per sample: 1.8345210552215576e-06 sec., samples per second: 545101.4024361953, total samples: 1000000

INSERT (HASH, 1ST INDEX COVERS 1 FIELD)

Total time: 1.781458854675293 sec., per sample: 1.781458854675293e-06 sec., samples per second: 561337.6909467103, total samples: 1000000

SELECT (HASH, 1ST INDEX COVERS 1 FIELD)

Total time: 0.8209011554718018 sec., per sample: 8.209011554718018e-07 sec., samples per second: 1218173.458929125, total samples: 1000000

INSERT (HASH, 1ST INDEX COVERS 1 FIELD, 2ND INDEX COVERS 1 FIELD, 3RD INDEX COVERS 2 FIELDS)

Total time: 4.138401985168457 sec., per sample: 4.138401985168457e-06 sec., samples per second: 241639.16496847855, total samples: 1000000

SELECT (HASH, 1ST INDEX COVERS 1 FIELD, 2ND INDEX COVERS 1 FIELD, 3RD INDEX COVERS 2 FIELDS)

Total time: 1.001291036605835 sec., per sample: 1.001291036605835e-06 sec., samples per second: 998710.628020589, total samples: 1000000

INSERT (RTREE, 1ST INDEX COVERS 1 FIELD)

Total time: 9.943094968795776 sec., per sample: 9.943094968795777e-05 sec., samples per second: 10057.230702696503, total samples: 100000

SELECT (RTREE, 1ST INDEX COVERS 1 FIELD, LIMIT = 5)

Total time: 11.716284990310669 sec., per sample: 0.00011716284990310669 sec., samples per second: 8535.128676256994, total samples: 100000