Package: amoredb
AmoreDBSimple append-only database for Python with rich record formats and compression.For impatientspipinstallamoredbExample 1importasynciofromamoredbimportAmoreDBasyncdefmain():asyncwithAmoreDB('test','w')asdb:awaitdb.append(b'foo')awaitdb.append(b'bar')awaitdb.append(b'baz')asyncforrecordindb:print(record)asyncio.run(main())Result:b'foo'
b'bar'
b'baz'Example 2importasynciofromamoredb.jsonimportJsonAmoreDBasyncdefmain():asyncwithJsonAmoreDB('test.json','w')asdb:awaitdb.append({'foo':'bar'})awaitdb.append({'bar':'foo'})asyncforrecordindb:print(record)asyncio.run(main())Result:{'foo': 'bar'}
{'bar': 'foo'}Record formatsThe basic format for database records is bytes object. Subclasses may support other formats,
as demonstrated above in the Example 2. AmoreDB provides support for the following formats:JSON:JsonMixin,JsonAmoreDBfromamoredb.jsonstrings:StrMixin,StrAmoreDBfromamoredb.strstructures:StructMixin,StructAmoreDBfromamoredb.structBSON:BsonMixin,BsonAmoreDBfromamoredb.bson, requiressimple_bsonpackageRecords are converted to the binary data by mixins and AmoreDB provides
predefined classes, such, for example, asclassJsonAmoreDB(JsonMixin,AmoreDB):passRecord compressionSimilar to record format conversion, compression is implemented by mix-ins.
AmoreDB provides a few for the following formats:gzip:GzipMixinfromamoredb.gziplzma:LzmaMixinfromamoredb.lzmalz4:Lz4Mixinfromamoredb.lzma, requireslz4packagebrotli:BrotliMixinfromamoredb.brotli, requiresbrotlipackagesnappy:SnappyMixinfromamoredb.snappy, requirespython-snappypackageThere are no predefined classes for compression, it's up to end users to define ones for their needs.
For example,fromamoredbimportAmoreDBfromamoredb.jsonimportJsonMixinfromamoredb.gzipimportGzipMixinclassMyDB(JsonMixin,GzipMixin,AmoreDB):passasyncwithMyDB('test.json.gz','w',compresslevel=5)asdb:awaitdb.append({'foo':'bar'})awaitdb.append({'bar':'foo'})asyncforrecordindb:print(record)Record transformation pipelineRecords in AmoreDB are processed by the following methods:defrecord_to_raw_data(self,record_data):# do custom conversion here# ...# call base methodreturnsuper().record_to_raw_data(record_data)defrecord_from_raw_data(self,record_data):# do custom conversion here# ...# call base methodreturnsuper().record_from_raw_data(record_data)Mix-ins override these methods and to make pipeline working, mix-ins should be defined in the right order.
As we have seen above,classMyDB(JsonMixin,GzipMixin,AmoreDB):passGzipMixinis placed in between, because compression takes place after converting record from JSON to binary data
and before writing this data to file. Same for opposite direction.Database structureThe database consists of data file and index file. Optional metadata file in JSON format may contain
the structure of database class.Index file contains positions of records except the first one which is always zero.
The first element in index file is the offset of the next record.
Thus, the number of items in the index file equals to the number of records.Record id is implicit, it is the index of the record.
Thus, to get a record by id, read its offset from the index file and then read the record from data file.