Package: dinkycache
dinkycache for python projectsA very small and flexible name/value cache for python.Intended for quick set up, in development and small scale projects.Usessqlitefor storage andlzstringfor compression.Stores any data that can be parsed in to a string withjson.dumps()andjson.loads()Returnsint,dictandstrjust fine, but returns alistif supplied atupleInstallFrom pippython-mpipinstalldinkycacheFrom github# Copy the 'dinkycache' directory and# requirements.txt into your cwd# Install dependencies:python-mpipinstall-rrequirements.txtHow to useImportfromdinkycacheimportDinky3 main methods called like so:Dinky().read(str:id)Dinky().write(str:id,str:data,float:ttl)Dinky().delete(str:id)->intExamples with default settingsfromdinkycacheimportDinky#gets data from some slow sourcedeffetch_data(id):return"some data"id="001"Then where you would normaly write:results=fetch_data(id)Write these two lines instead:if(result:=Dinky().read(id)==False):Dinky().write(id,result:=fetch_data(id))If you are running Python < 3.8 or just don't likewalruses:results=Dinky().read(id)ifresults==False:results=fetch_data(id)Dinky().write(id,results)This is also an option, its there, its fully supported, however not further documented:#Write:d=Dinky()d.id="test"d.data={"whatever":"floats"}d.setTTL(24)#hrd.write()print(d.data)#Read:d=Dinky()d.id="testresults=d.read()print(results)In either caseresultswill contain the data from cache if its there and within the specified TTL. Or it will call your get_some_data() to try and fetch the data instead.SettingsAvaialble settings and default valuesdbfile:str="dinkycache.db",# name of sqlite filettl:float=2160,# time to live in hours, default 2160 = 90 days, 0 = no expirypurge_rows:bool=True,# will enforce row_limit if truerow_limit:int=10000,# maximum number of rows in dbrow_overflow:int=1000,# buffer zone above row_limit before anything is deletedclean_expired:bool=True,# will delete outdated entries if trueclean_hrs:int=24,# time between cleanups of expried entriesclean_iterations:int=100,# iterations (reads/writes) between cleanupsSet them in one of the following ways# Positional arguments:Dinky('preferred.db',24).read(id)OR# Keyword arguments:Dinky(dbfile='preferred.db').read(id)OR# Unpack list as positional arguments:settings=['preferred.db',24]Dinky(*settings).read(id)OR# Unpack dict as keyword arguments:settings={'dbfile'='preferred.db','ttl'=24,}Dinky(**settings).read(id)Examples of use with user-defined settingsYou can destruct a dict an pass it as settings each time you invokeDinky(**settings),
or assign the newDinky objectto avariableand re-use it that way:Invoke on every use:settings={'dbfile'='preferred.db','purge_rows'=True,'clean_expired'=False,'row_limit'=100,'ttl'=0,}if(result:=Dinky(**settings).read(id)==False):Dinky(**settings).write(id,result:=fetch_data(id))Retain Dinky object:d=Dinky(dbfile='preferred.db',purge_rows=True,clean_expired=False,row_limit=100,ttl=0,)if(result:=d.read(id)==False):d.write(id,result:=fetch_data(id))clean_expired, clean_hrs and clean_iterationsIfclean_expired = True, script will try to clean out expired entries every time data iswrittenif one of the following conditions are met.It has been minimumclean_hrs: int = 24hours since last cleanupORThere have been more thanclean_iterations: int = 100calls since last cleanupThe cleanup function comes at a 75% performance cost, so if it runs on every 100 write, that amounts to a 7.5% average performance cost.clean_expiredmight therefore be a much better alternative than usingpurge_rowsfor larger amounts of data.purge_rows, row_limit and row_overflowIfpurge_rows = True, script will try to clean out overflowing lines every time data iswritten.row_limit = intsets the maximum lines in the database.row_overflow = inthow many lines overrow_limitbeforerow_limitis enforcedThis comes at a great performance cost for larger databases. 462 ms to sort 100k rows on a 1.8 ghz Intel Core i5. For this reasonrow_overflowis added as a buffer threshold, so that deletion dont happen on every call to.write.It is probably best used for small databases and/or databases with small entries.Public methods.read()Argumentsid(string, required)Returns data corresponding toid, or False if there is no dataCan be called without arguments on existing object if id has alredy been set..write()Argumentsid(string, required),data(string, required),tll(int, optional)Stores the supplieddatato thatid,tllcan be set here if not already passed on invocationReturns the hashedidor FalseWill doclean_expiredandpurge_rowsif they are set TrueCan be called without arguments on existing object if id and data has alredy been set..delete()Argumentsid(string, required)Deletes entry corresponding to thatidReturns number of rows deletet,1or0.Can be called without arguments on existing object if id has alredy been set.Setting TTL to seconds, months etc:Its been a bit of a discussion whats the most sensible choice of default time unit for TTL, we landed on hours as a float.The idea is that hours will be sufficient and sensible enough in most usercases. However it also allows for a workaround if you need to set lower or higher values:10seconds:Dinky().write(id="1",data="some data",ttl=10/3600)10minutes:Dinky().write(id="1",data="some data",ttl=10/60)10months:Dinky().write(id="1",data="some data",ttl=10*720)10years:Dinky().write(id="1",data="some data",ttl=10*8760)Alternatively you can expiry in seconds directly on the Dinky object like sod=Dinky()d.expires=10+d.nowd.write(id="1",data="some data")PerformanceThis wont ever compete with Redis, MongoDB or anything like that. This is ment to be a small, easy solution for small scale use cases where you dont want or need any big dependencies. Hence performance will be less, but might still be orders of magnitude faster than repeatedly parsing the data from some website.Tests:Reads from DB 110k entries of 40 to 1500 characters:

1 read = 0.018 to 0.003 sec
100 reads = 0.6 sec (0.006 avg)Reads from DB 210k entries of 40 to 150000 characters:
1 read = 0.003 to 0.022 sec
100 reads = 1.1 to 2.4 sec (0.015 avg)Test DB 3:38.1mb: 100k writes str_len 40~1500: avg 11.3ms (incl generation)

10k reads: 6.57 ms avgSecurityIds are hashed, so you may put anything in there
Data is compressed to a string of base 64 characters, so you may put anything in there.Lzstring seem to have very high integrity, we have not been able to produce a test result where the input and output has not been equal.That said, what you put in is what you'll get out. There is no checking for html-tags and such. Just something to bevare of if for some reason you'll use it to store and later display user provided data.CompressionLzstring is not great for shorter strings, and does sometimes even increase to string lenght. However in testing we found that short strings (80 to 1500 chars) have an average compression rate of 98%, while strings longer than 60000 characters have an average compression rate of 48%. Testing was done with random as well as real world data.So there is most likely some performance loss, but it is outweighed by smaller database files and the fact that base 64 strings makes life very easy.