Package: deltalake-redis-lock
deltalake-redis-lockA library creating an interface for a write lock fordelta-rs.Library UsageWhen using this client, it can be used from multiple hosts. Below follow a minimal example
to mimic this behaviour.Redis Env VariablesMake sure to set theseenvsbefore executing code.REDIS_HOST=<host>REDIS_PORT=<port>REDIS_DB=<0>Concurrent Write Example# run.pyimportloggingimportosfrommultiprocessingimportPoolfromdeltalakeimportDeltaTablefrompandasimportDataFramefromdeltalake_redis_lockimportwrite_redis_lock_deltalake,optimize_redis_lock_deltalakelogging.basicConfig(level=logging.INFO,format='%(asctime)s[%(levelname)s]%(message)s',datefmt='%Y-%m-%d%H:%M:%S')deffake_worker(args)->None:df,table_name=argstable_path=f"{os.getcwd()}/{table_name}"write_redis_lock_deltalake(table_or_uri=table_path,lock_table_name=table_name,mode="append",data=df,overwrite_schema=True,)defdefine_datasets(_table_name:str)->None:df1=DataFrame({"id":[1]})df2=DataFrame({"id":[2]})df3=DataFrame({"id":[3]})df4=DataFrame({"id":[4]})datasets=[(df1,table_name),(df2,table_name),(df3,table_name),(df4,table_name)]withPool()aspool:pool.map(fake_worker,datasets)if__name__=='__main__':table_name=f"test_run"table_path=f"{os.getcwd()}/{table_name}"define_datasets(_table_name=table_name)df=DeltaTable(table_uri=table_path).to_pandas().to_string()logging.info(df)2023-07-1821:28:47[INFO]TrytoAcquireRedisLock...2023-07-1821:28:47[INFO]TrytoAcquireRedisLock...2023-07-1821:28:47[INFO]Acquiredlock,blocking:True2023-07-1821:28:47[INFO]AcquiredRedisLock...2023-07-1821:28:47[INFO]Lockacquired.Executingfunction...2023-07-1821:28:47[INFO]TrytoAcquireRedisLock...2023-07-1821:28:47[INFO]TrytoAcquireRedisLock...2023-07-1821:28:47[INFO]Releasinglock...2023-07-18T20:28:47.3786302023-07-1821:28:47[INFO]Acquiredlock,blocking:True2023-07-1821:28:47[INFO]AcquiredRedisLock...2023-07-1821:28:47[INFO]Lockacquired.Executingfunction...2023-07-1821:28:47[INFO]Releasinglock...2023-07-18T20:28:47.4193732023-07-1821:28:47[INFO]Acquiredlock,blocking:True2023-07-1821:28:47[INFO]AcquiredRedisLock...2023-07-1821:28:47[INFO]Lockacquired.Executingfunction...2023-07-1821:28:47[INFO]Releasinglock...2023-07-18T20:28:47.4764112023-07-1821:28:47[INFO]Acquiredlock,blocking:True2023-07-1821:28:47[INFO]AcquiredRedisLock...2023-07-1821:28:47[INFO]Lockacquired.Executingfunction...2023-07-1821:28:47[INFO]Releasinglock...2023-07-18T20:28:47.517992id01132234Structuretest_run
├──0-a2811af1-e9fa-4984-9824-3956acdbaba8-0.parquet
├──1-87889b2d-1971-4e9b-8244-5e0d4a222458-0.parquet
├──2-a2f0ac25-df02-43b7-945d-014db522b19f-0.parquet
├──3-e57eae65-3cc7-4539-9eb6-b41ba52642bc-0.parquet
└──_delta_log├──00000000000000000000.json├──00000000000000000001.json├──00000000000000000002.json└──00000000000000000003.json1directory,8filesConcurrent Write With Optimize Example# run.pyimportloggingimportosfrommultiprocessingimportPoolfromdeltalakeimportDeltaTablefrompandasimportDataFramefromdeltalake_redis_lockimportwrite_redis_lock_deltalake,optimize_redis_lock_deltalakelogging.basicConfig(level=logging.INFO,format='%(asctime)s[%(levelname)s]%(message)s',datefmt='%Y-%m-%d%H:%M:%S')deffake_worker(args)->None:df,table_name=argstable_path=f"{os.getcwd()}/{table_name}"write_redis_lock_deltalake(table_or_uri=table_path,lock_table_name=table_name,mode="append",data=df,overwrite_schema=True,)optimize_redis_lock_deltalake(table_or_uri=table_path,lock_table_name=table_name,retention_hours=0,dry_run=False,enforce_retention_duration=False,)defdefine_datasets(_table_name:str)->None:df1=DataFrame({"id":[1]})df2=DataFrame({"id":[2]})df3=DataFrame({"id":[3]})df4=DataFrame({"id":[4]})datasets=[(df1,table_name),(df2,table_name),(df3,table_name),(df4,table_name)]withPool()aspool:pool.map(fake_worker,datasets)if__name__=='__main__':table_name=f"test_run"table_path=f"{os.getcwd()}/{table_name}"define_datasets(_table_name=table_name)df=DeltaTable(table_uri=table_path).to_pandas().to_string()logging.info(df)Output2023-07-1821:26:42[INFO]TrytoAcquireRedisLock...2023-07-1821:26:42[INFO]TrytoAcquireRedisLock...2023-07-1821:26:42[INFO]Acquiredlock,blocking:True2023-07-1821:26:42[INFO]AcquiredRedisLock...2023-07-1821:26:42[INFO]Lockacquired.Executingfunction...2023-07-1821:26:42[INFO]TrytoAcquireRedisLock...2023-07-1821:26:42[INFO]TrytoAcquireRedisLock...2023-07-1821:26:42[INFO]Releasinglock...2023-07-18T20:26:42.6810302023-07-1821:26:42[INFO]TrytoAcquireRedisLock...2023-07-1821:26:42[INFO]Acquiredlock,blocking:True2023-07-1821:26:42[INFO]AcquiredRedisLock...2023-07-1821:26:42[INFO]Lockacquired.Executingfunction...2023-07-1821:26:42[INFO]Releasinglock...2023-07-18T20:26:42.6898192023-07-1821:26:42[INFO]Acquiredlock,blocking:True2023-07-1821:26:42[INFO]AcquiredRedisLock...2023-07-1821:26:42[INFO]Lockacquired.Executingfunction...2023-07-1821:26:42[INFO]Releasinglock...2023-07-18T20:26:42.7507812023-07-1821:26:42[INFO]TrytoAcquireRedisLock...2023-07-1821:26:42[INFO]Acquiredlock,blocking:True2023-07-1821:26:42[INFO]AcquiredRedisLock...2023-07-1821:26:42[INFO]Lockacquired.Executingfunction...2023-07-1821:26:42[INFO]Releasinglock...2023-07-18T20:26:42.7602802023-07-1821:26:42[INFO]Acquiredlock,blocking:True2023-07-1821:26:42[INFO]AcquiredRedisLock...2023-07-1821:26:42[INFO]Lockacquired.Executingfunction...2023-07-1821:26:42[INFO]Releasinglock...2023-07-18T20:26:42.8665342023-07-1821:26:42[INFO]TrytoAcquireRedisLock...2023-07-1821:26:42[INFO]Acquiredlock,blocking:True2023-07-1821:26:42[INFO]AcquiredRedisLock...2023-07-1821:26:42[INFO]Lockacquired.Executingfunction...2023-07-1821:26:42[INFO]Releasinglock...2023-07-18T20:26:42.8825192023-07-1821:26:42[INFO]Acquiredlock,blocking:True2023-07-1821:26:42[INFO]AcquiredRedisLock...2023-07-1821:26:42[INFO]Lockacquired.Executingfunction...2023-07-1821:26:42[INFO]Releasinglock...2023-07-18T20:26:42.9850082023-07-1821:26:42[INFO]TrytoAcquireRedisLock...2023-07-1821:26:42[INFO]Acquiredlock,blocking:True2023-07-1821:26:42[INFO]AcquiredRedisLock...2023-07-1821:26:42[INFO]Lockacquired.Executingfunction...2023-07-1821:26:43[INFO]Releasinglock...2023-07-18T20:26:43.000558id04132132Structuretest_run
└──_delta_log
│├──00000000000000000000.json
│├──00000000000000000001.json
│├──00000000000000000002.json
│├──00000000000000000003.json
│├──00000000000000000004.json
│├──00000000000000000005.json
│└──00000000000000000006.json
└──part-00001-a13ca1fe-0a52-44c2-b2ce-b7eb95704536-c000.zstd.parquet1directory,8filesThis can be executed with something like:seq2|xargs-I{}-P2poetryrunpythonrun.pySetup From ScratchRequirement^python3.9poetry 1.1.13make (GNU Make 3.81)Setupmakesetup-environmentUpdate packagemakeupdateTestexportPYTHONPATH="${PYTHONPATH}:src"maketesttype=unitDockerThe reasondockeris used in the source code here, is to be able to build up an encapsulated
environment of the codebase, and dounit/integration and load tests.makebuild-container-imageDOCKER_BUILD="buildx build --platform linux/amd64"CONTEXT=.makerun-container-teststype=unit