Package: lmdb-dict-full
lmdb-dict-fullThe full-featureddictinterface to the LMDB "Lightning" Database.Internally optimized vialmdblibrary cursors. Optional LRU caching of deserialized values. Thread-safe operations. No added reserved keys,etc.Provides value-serializingSafeLmdbDictand str-onlyStrLmdbDict, as well as abstract base classLmdbDictfor customization of database encoding.Unique-key, labeled and unlabeled databases and read-write sessions supported.Table of ContentsInstallationUseGeneral useCachingStr-onlyLicenseInstallationpip install lmdb-dict-fullUseGeneral useSafeLmdbDictprovides the fulldictinterface to a LMDB database at a given filesystem path. (An empty database is automatically provisioned within a directory without one.)Values are automatically serialized (deserialized) and compressed (decompressed) usingPyYAMLandzlib.fromlmdb_dictimportSafeLmdbDictdbdict=SafeLmdbDict('/path/to/db/directory/0/')dbdict['aaa']={'values':[0,1,'x']}One or more named databases are also supported.LMDB requires that the maximum number of named databases is specified up-front. Below we'll only need two named databases.users=SafeLmdbDict('/path/to/db/directory/1/',name='users',max_dbs=2)hats=SafeLmdbDict('/path/to/db/directory/1/',name='hats',max_dbs=2)Note that it would otherwise be unsafe to hold open multiplelmdbclient objects within a single process at once. This is handled automatically: a weak reference is kept to the client opened for each filesystem path and reused for eachLmdbDictrequiring it.CachingCaching of LMDB itselfshould not be necessary. The database "fully exploits the operating system’s buffer cache" and memory mapping[ref].Moreover,lmdb-dict-fullmakes every effort to uselmdbefficiently, such that the user need not be concerned with undue overhead of interacting with the database-backed dictionary.That said: the value serialization layer ofSafeLmdbDictis another matter. Given sufficiently hefty values to deserialize, itmaybe worthwhile to engage thelmdb-dict-fullcaching layer, along with the trade-offs that it entails.Caveatslmdb-dict-fullcaching is thread-safeThis is achieved with behind-the-scenes locking – narrowly applied to singular keys where feasible – but the small overhead of which applies when caching.lmdb-dict-fullcaching isnot(yet)automaticallyprocess-safeCaching is thread-safe thanks to thread locks and (again) weak references to caches which must be shared across dictionaries backed by the same databases.Achieving the same under a multiprocessing regime would be another matter.Users may nonetheless make use oflmdb-dict-fullwhile multiprocessing, either without caching or with thoughtful application of caches across processes.OptionsCaching is built into all concrete subclasses ofLmdbDict; however, it is disabled by default, in that it is set toDummyCache– a mapping capable of storing zero items.Subclasses ofLmdbDictcheck their cache for its maximum capacity by means of:getattr(cache, 'maxsize', …). A cache reportingmaxsize=0– such as theDummyCache– will be givendummy locks, such that locking is disabled for this dictionary.A cache reporting any othermaxsize– or lacking this property – is treated as a proper cache, and locking will be applied.Caching may be specified – toSafeLmdbDictfor example – via an instance, a class, or any callable returning an instance of a mapping for use as a deserialization cache. Either an instance or a class are strongly recommended, as these enable checking any cache retrieved from the weak reference registry against the user's instantiation argument.fromlmdb_dict.cacheimportLRUCache128SafeLmdbDict('/path/to/db/directory/',cache=LRUCache128)Above, we've specified that ourSafeLmdbDictshould cache deserialized values using an instance ofLRUCache128– that is, a subclass of theLRUCacheprovided bycachetools.LRUCache128distinguishes itself only in that it requires no initialization arguments – a requirement of supplying a callable in lieu of a cache instance – and it setsmaxsize=128.As a shortcut to the above,lmdb-dict-fullprovidesCachedLmdbDict:fromlmdb_dictimportCachedLmdbDictCachedLmdbDict('/path/to/db/directory/')CachedLmdbDictdiffers from other subclasses ofLmdbDictin that it defaults to caching viaLRUCache128. Other caches may be specified via thecacheargument. Supplying an entity with propertymaxsize=0– such as theDummyCache– will raise aTypeError.Str-onlyThe above concrete subclasses ofLmdbDictsupport arbitrary serializable values in order to best mimic the functionality of the Pythondict.For use-cases supporting str-only (and/or bytes-only) values, all of the above concerns over serialization, caching and locking may be sidestepped.StrLmdbDictprovides the same full-featureddictinterface to LMDB, but only for values of typestrandbytes.fromlmdb_dictimportStrLmdbDictStrLmdbDict('/path/to/db/directory/')StrLmdbDictfurther differs from other subclasses ofLmdbDictin that it accepts nocacheargument, and may not perform caching.Licenselmdb-dict-fullis distributed under the terms of theMITlicense.