Package: dynafile
Consider the project as a proof of concept! Definitely not production ready!DynafileEmbedded pure Python NoSQL database following DynamoDB concepts.pipinstalldynafile# with string filter support using filtrationpipinstall"dynafile[filter]"# bloody edgepipinstallgit+https://github.com/eruvanos/dynafile.git
pipinstallfiltrationOverviewDynafile stores items within partitions, which are stored as separate files. Each partition contains a SortedDict
fromsortedcontainerswhich are sorted by the sort key attribute.Dynafile does not implement the interface or functionality of DynamoDB, but provides familiar API patterns.Differences:Embedded, file basedNo paginationFeaturespersistenceput itemget itemdelete itemscan - without parametersquery - starts_withquery - index directionquery - filterscan - filterbatch writeratomic file writeevent stream hooks (put, delete)TTLRoadmapGSI - global secondary indexupdate itembatch getthread safenessLSI - local secondary indexsplit partitionsparallel scans - pre defined scan segmentstransactionsoptimise disc load time (cache partitions in memory, invalidate on file change)conditional put itemimprove file consistency (options: acidfile)APIfromdynafileimport*# init DB interfacedb=Dynafile(path=".",pk_attribute="PK",sk_attribute="SK")# put itemsdb.put_item(item={"PK":"user#1","SK":"user#1","name":"Bob"})db.put_item(item={"PK":"user#1","SK":"role#1","TYPE":"sender"})db.put_item(item={"PK":"user#2","SK":"user#2","name":"Alice"})# more performant batch operationwithdb.batch_writer()aswriter:db.put_item(item={"PK":"user#3","SK":"user#3","name":"Steve"})db.delete_item(key={"PK":"user#3","SK":"user#3"})# retrieve itemsitem=db.get_item(key={"PK":"user#1","SK":"user#1"})# query item collection by pkitems=list(db.query(pk="user#1"))# scan full tableitems=list(db.scan())# add event stream listener to retrieve item modificationdefprint_listener(event:Event):print(event.action)print(event.old)print(event.new)db.add_stream_listener(print_listener)Filterqueryandscansupport filter, you can provide callables as filter like lambda expressions.Another option arefiltrationexpressions.Equal ("==")Not equal ("!=")Less than ("<")Less than or equal ("<=")Greater than (">")Greater than or equal (">=")Contains ("in")RHS must be a list or a SubnetRegular expression ("=~")RHS must be a regex tokenExamples:SK =~ /^a/- SK starts with aSK == 1- SK is equal 1SK == 1- SK is equal 1nested.a == 1- accesses nested structureitem.nested.aTTL - Time To LiveTTL provides the option to expire items on read time (get, query, scan).importtimefromdynafileimport*db=Dynafile(path=".",pk_attribute="PK",sk_attribute="SK",ttl_attribute="ttl")item={"PK":"1","SK":"2","ttl":time.time()-1000}# expired ttldb.put_item(item=item)list(db.scan())# -> []ArchitectureFile Structure--- ROOT ---
./db/

--- MAIN DB ---

|- meta.json - meta information
|- _partitions/
    |- <hash>/
        |- data.pickle - Contains partition data by sort key (SortedDict)
        |- lsi-attr1.pickle - Contains partition data by lsi attr (SortedDict)

--- GSI ---
|- _gsi-<gsi-name>/
    |- _partitions/
        |- <hash>/
            |- data.pickle - Contains partition data by sort key (SortedDict)