Package: pickledir
pickledirFile-based key-value storage.Keys and values are serialized
withpickle. Data is kept in
files in the specified directory.CI-tested with Python 3.8-3.9 on macOS, Ubuntu and Windows.The storage has zero initialization time, fast random access, fast reads and
writes.Unlikeshelve, the data saved
by PickleDir is cross-platform: you can write it on Linux and read the same
files on Windows. Unlike most database-based caching solutions (including the
shelve), the PickleDir does not require the "open" and "close" storage. It's
always open since it's just a directory in the file system.PickleDir is better for casual data storage. Database-based solutions are
preferred when your storage has many elements (3 thousand or more). They will
also be faster when working with a predictably high load in terms of reading and
writing.Install$pip3installpickledirUseCreatefrompickledirimportPickleDircache=PickleDir('path/to/my_cache_dir')WriteKeys do not need to be hashable. They only need to be serializable withpickle.When you assign a value, the data is literally written to a file.cache['key']='hello, user!'cache[5]=23cache[{'a','b','c'}]='abc'Readprint(cache['key'])print(cache[5])print(cache[{'a','b','c'}])Read all valuesforkey,valueincache.items():print(key,value)Delete itemdelcache['key']Type hints# declaring PickleDir with string keys and integer values:cache:PickleDir[str,int]=PickleDir('path/to/my_cache_dir')Set expiration time on writingThe expired items will be removed from the storage.cache.set('a',1000,max_age=datetime.timedelta(seconds=1))print(cache.get('a'))# 1000time.sleep(2)print(cache.get('a'))# None (and removed from storage)Set expiration time on readingThe expired items will not be returned, but kept in the storage.cache['b']=1000time.sleep(2)cache.get('b'max_age=datetime.timedelta(seconds=1))# Nonecache.get('b'max_age=datetime.timedelta(seconds=9))# 1000Set data versionSetting the data version makes it easy to mark old data as obsolete.For example, you cached the result of a function, and then changed the
implementation of that function. In this case, there is no need to delete old
files from the cache. Just change the version number.cache=PickleDir('path/to/dir',version=1)cache['a']='some_data'You can read all stored data while theversionvalue is1.cache=PickleDir('path/to/dir',version=1)print(cache.get('a'))# 'some_data'If you decide that all the data in the cache is out of date, just pass the
constructor a version number that you haven't used before.cache=PickleDir('path/to/dir',version=2)print(cache.get('a'))# NoneNow all that is saved with version2is actual data. Any other version is
considered obsolete and will be gradually removed.Do not create thePickleDirwith an old version number. It will make the data
unpredictable.cacheV1=PickleDir('path/to/dir',version=1)# okcacheV1['a']='old A'cacheV1['b']='old B'cacheV2=PickleDir('path/to/dir',version=2)# okcacheV2['a']='new A'cacheV1=PickleDir('path/to/dir',version=1)# don't do thisprint(cacheV1.get('b'))# Schrödinger's data ('old B' or None)BenchmarksCasually saving 10 items and reading them again:foriinrange(10):cache[str(i)]={"data":i,"other":None}foriinrange(10):_=cache[str(i)]StorageTimePickleDir0.42shelve6.68diskcache.Cache1.09Measured on macOS, Python 3.8, SATA HDD (not SSD), Journaled HFS+.See sources
inbenchmarkdir.The main advantage ofpicklediris the lack of time required to create a database or initialize
tables. If we did not save 10 items, but 1000 in a row,shelveanddiskcachewould be faster thanpickledir.Under the hoodSerialized data is stored inside files in the same directory. Each file contains
one or more items. The maximum number of files is limited to 4096. The values
are uniformly distributed between the files.Reading is slower when a file contains more than one item. Therefore, the
PickleDir is better suited for cases with the number of items within a few
thousand.