Package: atomic-store
atomic_storeEasier than a DBMS, but more fault-resistant than just a file.Sometimes you need to manage a bit of state across executions.
Sometimes, a fully-blown database is just too much.This library makes it easy to keep astoreof stuff in a JSON file,
in an atomic and fault-resistant manner.Other formats (like pickle and bson) are also supported,
and arbitrary formats are possible.Table of ContentsInstallUsageTODOsNOTDOsContributeInstallJustpip install atomic_store.  Or, if you must,pip install -r requirements.txtNote that the only dependency isatomicwrites, which has no dependencies.UsageBy default, the store is encoded as json, written to a temporary file,
and then atomically replaces the old file.  When reading, if the file does
not exist, a default value is used.  The default default value isNone.Context ManagerThis program remembers all start times:importatomic_storeimporttimewithatomic_store.open('runs.json',default=[])asstore:print('Previous executions:')print(store.value)new_entry=time.strftime('%Y-%m-%d%H:%M:%S%z')store.value.append(new_entry)Leaving the context manager takes care of all writes.
No intermediate values get written to disk.This is ideal if the task runs short, and in case of any error
you only want to keep the old state anyway.For advanced uses, also see the subsection onreentrancy.Manual controlThis program remembers all start times:importatomic_storemy_store=atomic_store.open('gathered.json',default=dict())my_store.value['state']='running'my_store.value['thought']='I would not eat green eggs and ham.'my_store.commit()# ... some calculations ...my_store.value['state']='done'my_store.value['thought']='I do so like Green eggs and ham!'my_store.commit()Only calls tocommit()cause writes to the disk.
Again, no intermediate values get written to disk.This is ideal if you have a long-running job with clear steps,
and each step's output is valuable.Note thatcommit()is also available in the context manager.Format tweaksIf you're using the json backend, and want to keep the JSON file as small as possible,
you can callopenwithdump_kwargs=dict(separators=(',', ':')).
The keywordload_kwargsalso exists.Non-JSON formatsYou can use arbitrary other formats, using theformatkeyword:atomic_store.open('runs.json',default=[],format=MY_FORMAT)Supported values areNone(for JSON),'json','pickle','bson'(requires bson to be installed), and also any module or object
providingdump/loadordumps/loads.
By default,atomic_storeassumes you operate on binary files, except when JSON is involved.
To override this, you can setis_binary.
Note that this means you can use the modulesjson,pickle, andbsonas they are.For convenience, you can also override the abstract classesatomic_store.AbstractFormatFileoratomic_store.AbstractFormatBstr.In all cases,load_kwargsanddump_kwargsare still supported.ReentrancyIf the sameatomic_storeis used as a context manager more than once,
the default behavior is to write the file only when the lastwithis exited:# Assume `state.json` contains only `"before"`.mngr=atomic_store.open('mystate.json',default=[])withmngrasstore:store.value='outer'# File contains `"before"`: We haven't exited any context manager yet.withmngrasstore:store.value='inner'# File contains `"before"`: We haven't exited any context manager yet.# File now contains `"inner"`, because the inner `with`-statement wrote it.# Read the Reentrancy section if you consider this undesired behavior.# File now contains `"inner"`, because the outer `with`-statement wrote it again.If you consider this behavior undesirable, you can either just use multiple context managers (by callingatomic_store.openmultiple times), or by using the keywordignore_inner_exits=True, like this:# Assume `state.json` contains only `"before"`.mngr=atomic_store.open('mystate.json',default=[],ignore_inner_exits=True)withmngrasstore:store.value='outer'# File contains `"before"`: We haven't exited any context manager yet.withmngrasstore:store.value='inner'# File contains `"before"`: We haven't exited any context manager yet.# File *still* contains `"before"`, as the manager detected that it is still active.# File now contains `"outer"`, because the outer `with`-statement wrote it.Atomic is not magicThis library is not magical.If two threads (or two processes, or whatever) open a store,
modify something, and then write concurrently, one of the results may be lost.
However, the writes are guaranteed to be atomic,
so the data is merely lost, but not corrupted.TODOsFigure out how to makebsonoptionalPublish on PyPINOTDOsHere are some things this project will not support:Any DB backend.Any multi-file backend.More advanced semantics than justcommit.This includes rollback.  It's just not obvious which behavior is desired when the file does not exist (Re-usedefaultvalue?  What if it was modified, as it happens with lists and dicts?), and with stacked context managers (should it rollback to the file's state?  Or to the beginning of thewith?)ContributeFeel free to dive in!Open an issueor submit PRs.