Package: tarn
A generic framework for key-value storageInstallpipinstalltarnRecipesA simple datalakeLet's start small and create a simple disk-based datalake. It will store various files, and the keys will be theirsha256digest:fromtarnimportHashKeyStoragestorage=HashKeyStorage('/path/to/some/folder')# here `key` is the sha256 digestkey=storage.write('/path/to/some/file.png')# now we can use the key to read the file at a later timewithstorage.read(key)asvalue:# this will output something like Path('/path/to/some/folder/a0/ff9ae8987..')print(value.resolve())# you can also store values directly from memory# - either byte stringskey=storage.write(b'my-bytes')# - or file-like objects#  in this example we stream data from an url directly to the datalakeimportrequestskey=storage.write(requests.get('https://example.com').raw)Smart cache to diskA really cool feature oftarnismemoizationwith automatic invalidation:fromtarnimportsmart_cache@smart_cache('/path/to/storage')defmy_expensive_function(x):y=x**2returnmy_other_function(x,y)defmy_other_function(x,y):...z=x*yreturnx+y+zNow the calls tomy_expensive_functionwill be automatically cached to disk.But that's not all! Let's assume thatmy_expensive_functionandmy_other_functionare often prone to change,
and we would like to invalidate the cache when they do. Just annotate these function with a decorator:fromtarnimportsmart_cache,mark_unstable@smart_cache('/path/to/storage')@mark_unstabledefmy_expensive_function(x):...@mark_unstabledefmy_other_function(x,y):...Now any change to these functions, will cause the cache to invalidate itself!Other storage locationsWe support multiple storage locations out of the box.Didn't find the location you were looking for? Create anissue.S3fromtarnimportHashKeyStorage,S3storage=HashKeyStorage(S3('my-storage-url','my-bucket'))RedisIf your files are small, and you want a fast in-memory storageRedisis a great optionfromtarnimportHashKeyStorage,RedisLocationstorage=HashKeyStorage(RedisLocation('localhost'))SFTPfromtarnimportHashKeyStorage,SFTPstorage=HashKeyStorage(SFTP('myserver','/path/to/root/folder'))SCPfromtarnimportHashKeyStorage,SCPstorage=HashKeyStorage(SCP('myserver','/path/to/root/folder'))NginxNginx has anautoindexoption, that
allows to serve files and list directory contents. This is useful when you want to access files over http/https:fromtarnimportHashKeyStorage,Nginxstorage=HashKeyStorage(Nginx('https://example.com/storage'))AdvancedHere we'll show more specific (but useful!) use-casesFanoutYou might have several HDDs, and you may want to keep your datalake on both without creating a RAID array:fromtarnimportHashKeyStorage,Fanoutstorage=HashKeyStorage(Fanout('/mount/hdd1/lake','/mount/hdd2/lake',))Now both disks are used, and we'll start writing to/mount/hdd2/lakeafter/mount/hdd1/lakebecomes full.You can even use other types of locations:fromtarnimportHashKeyStorage,Fanout,S3storage=HashKeyStorage(Fanout(S3('server1','bucket1'),S3('server2','bucket2')))Or mix and match them as you please:fromtarnimportHashKeyStorage,Fanout,S3# write to s3, then start writing to HDD1 after s3 becomes fullstorage=HashKeyStorage(Fanout(S3('server2','bucket2'),'/mount/hdd1/lake'))Lazy migrationLet's say you want to seamlessly replicate an old storage to a new location, but copy only the needed files first:fromtarnimportHashKeyStorage,Levelsstorage=HashKeyStorage(Levels('/mount/new-hdd/lake','/mount/old-hdd/lake',))This will create something like acache hierarchywith copy-on-read
behaviour. Each time we read a key, if we don't find it in/mount/new-hdd/lake, we read it from/mount/old-hdd/lakeand save a copy to/mount/new-hdd/lake.Cache levelsThe samecache hierarchylogic can be used if you have a combination of
HDDs and SSD which will seriously speed up the reading:fromtarnimportHashKeyStorage,Levels,Levelstorage=HashKeyStorage(Levels(Level('/mount/fast-ssd/lake',write=False),Level('/mount/slow-hdd/lake',write=False),'/mount/slower-nfs/lake',))The setup above is similar to the one we use in our lab:we have a slow buthugeNFS-mounted storagea faster but smaller HDDand a super fast but even smaller SSDNow, we only write to the NFS storage, but the data gets lazily replicated to the local HDD and SSD to speed up the
reads.Caching small files to RedisWe can take this approach even further and use ultra fast in-memory storages, such as Redis:fromtarnimportHashKeyStorage,Levels,Small,RedisLocationstorage=HashKeyStorage(Levels(# max file size = 100KiBSmall(RedisLocation('my-host'),max_size=100*1024),'/mount/hdd/lake',))Here we useSmall- a wrapper that only allows small (<=100KiB in this case) files to be written to it.
In our experiments we observed a 10x speedup for reading small files.ComposabilityBecause all the locations implement the same interface, you can start creating more complex storage logic specifically
tailored to your needs. You can make setups as crazy as you want!fromtarnimportHashKeyStorage,Levels,Fanout,RedisLocation,Small,S3,SFTPstorage=HashKeyStorage(Levels(Small(RedisLocation('my-host'),max_size=10*1024**2),'/mount/fast-ssd/lake',Fanout('/mount/hdd1/lake','/mount/hdd2/lake','/mount/hdd3/lake',# nested locations are not a problem!Levels(# apparently we want mirrored locations here'/mount/hdd3/lake','/mount/old-hdd/lake',),),'/mount/slower-nfs/lake',S3('my-s3-host','my-bucket'),# pull missing files over sftp when neededSFTP('remove-host','/path/to/remote/folder'),))AcknowledgementsSome parts of our cache invalidation machinery were heavily inspired by
thecloudpickleproject.