Package: aiodiskdb
Minimal, embeddable on-disk DB, tailored for asyncio.aiodiskdbis a lightweight, fast, simpleappend onlydatabase.To be used in theasyncioevent loop.InstallpipinstallaiodiskdbUsageStart the DB by fire and forget:fromaiodiskdbimportAioDiskDB,ItemLocationdb=AioDiskDB('/tmp/aiodiskdb')loop.create_task(db.start())Use the db API to write and read data from a coroutine.asyncdefread_and_write():new_data_location:ItemLocation=awaitdb.add(b'data')data:bytes=awaitdb.read(location)assertdata==b'data'noted_location=ItemLocation(index=0,position=80,size=1024333)prev_saved_data:bytes=awaitdb.read(noted_location)assertlen(prev_saved_data)==1024333Stop the DB before closing the application.awaitdb.stop()Be alerted when data is actually persisted to disk:asyncdefcallback(timestamp:int,event:WriteEvent):human_time=datetime.fromtimestamp(timestamp).isoformat()log(f'{human_time}-{event}persisted to disk.')awaitdo_something(location)db.events.on_write=callbackOr hook to other events:db.events.on_start=...db.events.on_stop=...db.events.on_failure=...db.events.on_index_drop=...Asynchronous non-blockingHandle file writes with no locks.
Data is appended in RAM and persisted asynchronously, according to customizable settings.Transactional"All or nothing" commit.
Lock all the DB write operations during commits, still allowing the reads.
Ensure an arbitrary sequence of data is persisted to disk.Transaction is scoped. Data added into a transaction is not available outside until committed.transaction=awaitdb.transaction()transaction.add(b'cafe')transaction.add(b'babe')transaction.add(b'deadbeef')locations:typing.Sequence[ItemLocation]=awaittransaction.commit()Not-so-append-onlyAiodiskdbis an append-only database. It means you'll never see methods todeleteorremovesingle entries.However, data pruning is supported, with the following methods:db.enable_overwrite()db.rtrim(0,400)db.ltrim(8,900)db.drop_index(3)db.disable_overwrite()These three methods respectively:prune data from the right, at index0, starting from the location400to the index end (rtrim)prune data from the left, at index8, starting from the beginning to the location900(ltrim)drop the whole index3, resulting in a file deletion:drop_indexAll the items locations not involved into a TRIM operation remains unmodified, even after anltrim.Highly customizableThe default parameters:_FILE_SIZE=128_FILE_PREFIX='data'_FILE_ZEROS_PADDING=5_BUFFER_SIZE=16_BUFFER_ITEMS=1000_FLUSH_INTERVAL=30_TIMEOUT=30_CONCURRENCY=32can be easily customized. In the following example the files max size is 16 MB,
and data is persisted to disk every 1 MBORevery 100 new itemsORevery minute.db=AioDiskDB(max_file_size=16max_buffer_size=1,max_buffer_items=100,flush_interval=60)The max DB size ismax_file_size * max_files.
Withfile_padding=5the max number of files is 10,000.A DB created withfile_padding=5andmax_file_size=16is capable to store up to 160 GB, or 167,772,160,000 items,
at its maximum capacity will allocate 10,000 files.Try to do its bestHook the blockingon_stop_signalmethod to avoid data losses on exit.importsignalfromaiodiskdbimportAioDiskDBdb=AioDiskDB(...)signal.signal(signal.SIGINT,db.on_stop_signal)signal.signal(signal.SIGTERM,db.on_stop_signal)signal.signal(signal.SIGKILL,db.on_stop_signal)Quite enough fast for some use casesConcurrency tests, part of the unit tests, can be replicated as system benchmark.
The followings are performed on a common consumer SSD:Duration: 14.12s,
Reads: 2271 (~162/s),
Writes: 2014 (~143/s),
Bandwidth: 1000MB (71MB/s),
Avg file size: 508.0kBDuration: 18.97s,
Reads: 10244 (~540/s),
Writes: 10245 (~540/s),
Bandwidth: 20MB (1.05MB/s),
Avg file size: 1.0kBLimitationsassertlen(data)<=max_buffer_sizeassertmax_transaction_size<RAMassertmax_file_size<4096Ifrtrimis applied on thecurrentindex, the space is reused, otherwise no.
Withltrim, once the space is freed, it is not allocated again.
Withdrop_indexthe discarded index is not reused.With a lot of data turn-over (pruning by trimming), it may be necessary to set an unusual highfile_padding, and
increase the database potential size.CreditsInspired by the raw block data storage of thebitcoincore blocks database.Logo by mepheesto.NotesAlpha stage. Still under development, use with care and expect data losses.Donate :heart:Bitcointo: 3FVGopUDc6tyAP6t4P8f3GkYTJ5JD5tPwV orpaypal