Package: kvdb-py
kvdb-pyKey-Value DB Python Client Abstraction built on top of RedisOverviewThis library builds upon the experience of my previous library,aiokeydb-pyand the learnings and take-aways from maintaining it and using it in production. Whileaiokeydbsupportsredispy< 5.0, this library only supportsredispy>= 5.0.Supports:RedisKeyDBDragonflyMemcachedDynamoDBtikvThe goals of this library are:Provide a simple abstraction over the Redis client that allows both async and sync code, and adding minor tweaks in underlying implementation to make it more performant, and dynamic.Provide a unified interface for interacting with various Key-Value DBs, such as Redis, Memcached, etc. Inspired bygokvEnable interacting with multiple Key-Value DBs at the same time, with the same APIEnable usage of Key-Value DBs in the same manner as a native Python dictionaryProvide a task queue implementation that can be used to distribute work across multiple workers.Capability to customize and override the underlying classes easily without requiring to manually patch objects.Why not useredispydirectly?Redispyis a great library, and this library itself is built on top of it. However,redispylacks certain quality of life features that make it ready to use out of the box without having to extend the library to support these features.kvdbalso provides a powerful in-flight serialization and deserialization mechanism that allows you to store almost any type of object in the Key-Value DB without having to worry about serializing and deserializing the data prior to storing it. This is done by storing the object's type and any additional metadata required to reconstruct the object. You can read more about this in theSerializationsection.Additionally,kvdbprovides a unified async and sync interface for interacting with the Key-Value DB, rather than having to use two separate clients and instantiate two separate connections, defined as aKVDBSession. Each underlyingsessioncan share the same connection pool, so long as theConnectionPoolconfiguration is the same, meaning spawning multiple sessions will not saturate the connection pool.kvdbutilizes singleton objects that manage instances of objects, such asKVDBSessionandCachify. This helps ensure performance and reduce memory usage by preventing the creation of multiple instances of the same object.Installationpipinstallkvdb-pyUsageCheck out theexamplesdirectory for examples on how to use this library.Quick ExamplefromkvdbimportKVDBClientfrompydanticimportBaseModel# Explicitly define the config. Otherwise it will use the defaults, which are determined via environment variables.# session = KVDBClient.get_session(name="default", url=os.environ.get("REDIS_URL|KVDB_URL|KEYDB_URL|MEMCACHED_URL"))session=KVDBClient.get_session(name="default",url="redis://localhost:6379/0",serializer='json')classMyModel(BaseModel):name:strage:int# This demonstrates how the pydantic model is automatically serialized and deserializednew_model=MyModel(name='John',age=30)session.set(new_model.name,new_model)src_model=session.get('John')assertsrc_model==new_model# Without explicitly defining the serailizer, the same would fail.session2=KVDBClient.get_session(name="test-1")# This will fail because the serializer is not definedsession2.set(new_model.name,new_model)Task QueuesThis is a client-side example of how to use kvdb-py tasks.By Default,taskswill usedb_id=3for the task queue to prevent collisions with other data. You can change this by passingdb_idto theregisterdecorator orcreate_contextfunction. If you do, ensure that the queue you are using also uses the samedb_id.This will be able to remotely execute the tasks fromexamples/queue/dummy_tasks.pyTo run this example, you will need to run the following commands:Terminal 1:# cwd: examples/queue$kvdb-task-i"dummy_tasks.initializer"-n1-q"test"-q"global"# Output:-[WorkerID]:63f0e517-e3bd-4d6c-xxx-xxxx[WorkerName]:global-local-0[NodeName]:computer.local-[Concurrency]:150/jobs,50/broadcasts
-[Queues]:
-[test]@redis://localhost:6379/3DB:3,1functions,0cronjobs[Functions]:`['my_task']`-[global]@redis://localhost:6379/3DB:3,2functions,0cronjobs[Functions]:`['TestWorker.task1','TestWorker.task2']Terminal 2:# cwd: examples/queue$pythonclient_side.py# dummy_task.pyimportabcfromkvdbimporttasksfromkvdb.utils.logsimportlogger# Create a new task context that will be used to register the tasks# Any params passed to create_context will be used as partials for the register decorator# that can subsequently be overriden by the kwargs passed to the register decorator# Since no `queue_name` is passed, it will default to `global`task_context=tasks.create_context(disable_patch=False)# This will be registered to the `test` queue@tasks.register(queue_name='test')asyncdefmy_task(*args,**kwargs):logger.info(f'Hello Worlddddddd{args}{kwargs}')returnNone# This will be registered to the `global` queue# This decorator is required when using an object and# registering the object's methods as tasks# so that the object can be patched with initializers# to add the registered functions to the task context@task_context.register_object()classTestWorker(abc.ABC):def__init__(self,*args,**kwargs):logger.info(f'initializing{args}{kwargs}')# Both of these should work# @tasks.register(queue_name='global')@task_context.register()asyncdeftask1(self,ctx,*args,**kwargs):# This function will be passed the task context as the first argument# since it has `ctx` as the first argument# The task context is a singleton object that is shared across all tasks# that contains `Job`, `TaskWorker`# and any contexts that are initialized.logger.info(f'task 1 hello{ctx}{args}{kwargs}{self}')return'hello_1'@task_context.register()asyncdeftask2(self,*args,**kwargs):logger.info(f'task 2 hello{args}{kwargs}{self}')return'hello_2'# Since we pass -i "dummy_tasks.initializer" to the kvdb-task command# it will run this function to initialize the task context. If the# object is not initialized, then the functions will not be registereddefinitializer():x=TestWorker()# client_side.pyimportsysimportasynciofrompathlibimportPathsys.path.append(str(Path(__file__).parent))importdummy_tasksfromkvdbimporttasksasyncdefentrypoint():# Initialize and register the tasksx=dummy_tasks.TestWorker()q1=tasks.get_task_queue(queue_name='global',debug_enabled=True)q2=tasks.get_task_queue(queue_name='test',debug_enabled=True)# We can spawn tasks one of two ways. Either by calling the function directly# since we have it patched with a decorator# or by calling the task queue's enqueue method# Task 1 requires ctx, so it can only be called via the task queueawaitq1.enqueue('TestWorker.task1')# await x.task1(blocking = False)awaitq1.enqueue('TestWorker.task2')# You can also enqueue a function directly by passing the functionawaitq2.enqueue(dummy_tasks.my_task)# Additionally, since the underlying function is patched with the register decorator# you can also call the function directly and it will be executed as a taskawaitdummy_tasks.my_task(blocking=False)awaitx.task2(blocking=False)awaitq1(x.task1,blocking=False)# If `blocking` is set to `True`, then the task will be executed and will wait for the result# If `blocking` is set to `False`, then the task will be executed and will not wait for the result, but instead return a `Job` objectresult_1=awaitx.task2(blocking=True)result_2=awaitq1(x.task1,blocking=True)logger.info(f'result_1:{result_1}')logger.info(f'result_2:{result_2}')assertresult_1=='hello_2'assertresult_2=='hello_1'if__name__=='__main__':asyncio.run(entrypoint())CachingBelow are two examples of how to use the caching functionality ofkvdb-py. The first example demonstrates how to use the caching functionality with any functions, and the second example demonstrates how to use the caching functionality with an object.See the following files for more examples:examples/cache/basic.pyfor an example of how to use caching with functionsexamples/cache/objects.pyfor an example of how to use caching with an objectexamples/cache/context.pyfor an example of how to use caching with a pre-defined contextimporttimeimportabcimportasynciofromlazyops.utils.timesimportTimerfromkvdb.ioimportcachifyfromkvdb.utils.logsimportloggerDEBUG_ENABLED=False@cachify.register(ttl=10,verbosity=2ifDEBUG_ENABLEDelseNone,cache_max_size=15)asyncdefasync_fibonacci(number:int):ifnumber==0:return0elifnumber==1:return1returnawaitasync_fibonacci(number-1)+awaitasync_fibonacci(number-2)@cachify.register(ttl=10,verbosity=2ifDEBUG_ENABLEDelseNone)deffibonacci(number:int):ifnumber==0:return0elifnumber==1:return1returnfibonacci(number-1)+fibonacci(number-2)# No Cache Versionsasyncdefasync_fibonacci_nc(number:int):ifnumber==0:return0elifnumber==1:return1returnawaitasync_fibonacci_nc(number-1)+awaitasync_fibonacci_nc(number-2)deffibonacci_nc(number:int):ifnumber==0:return0elifnumber==1:return1returnfibonacci_nc(number-1)+fibonacci_nc(number-2)## Object Caching@cachify.register_object()classTestObject(abc.ABC):def__init__(self,*args,**kwargs):logger.info('running init')@cachify.register(ttl=10,verbosity=2ifDEBUG_ENABLEDelseNone,cache_max_size=15)asyncdefasync_fibonacci(self,number:int):ifnumber==0:return0elifnumber==1:return1returnawaitself.async_fibonacci(number-1)+awaitself.async_fibonacci(number-2)@cachify.register(ttl=10,verbosity=2ifDEBUG_ENABLEDelseNone)deffibonacci(self,number:int):ifnumber==0:return0elifnumber==1:return1returnself.fibonacci(number-1)+self.fibonacci(number-2)# No Cache Versionsasyncdefasync_fibonacci_nc(self,number:int):ifnumber==0:return0elifnumber==1:return1returnawaitself.async_fibonacci_nc(number-1)+awaitself.async_fibonacci_nc(number-2)deffibonacci_nc(self,number:int):ifnumber==0:return0elifnumber==1:return1returnself.fibonacci_nc(number-1)+self.fibonacci_nc(number-2)PersistenceAn additional feature ofkvdb-pyis the ability to access aDict-like object that is persisted to the Key-Value DB. This is useful for storing data that needs to be accessed by multiple processes or machines. This is powered by the underlyingPersistentDictfromlazyops, withkvdb-pyproviding the backend to the Key-Value DB.importrandomfrompydanticimportBaseModel,FieldfromkvdbimportKVDBClientclassNewObject(BaseModel):x:int=Field(default_factory=lambda:random.randint(0,100))test_1=NewObject()test_2=NewObject(x=1)# Create a new Session that does not use a Serializer# and uses the default connection pools=KVDBClient.get_session(name='default',persistence_base_key='test',persistence_serializer='json',)# You can set the attributes directly# or call the underlying `persistence` objects['test_1']=test_1s['test_2']=test_2# s.persistence['test_1'] = test_1# s.persistence['test_2'] = test_2result_1=s['test_1']result_2=s['test_2']# result_1 = s.persistence['test_1']# result_2 = s.persistence['test_2']print(result_1,'type',type(result_1))print(result_2,'type',type(result_2))# Create a new Session that uses the JSON Serializer# natively within the connection pools_json=KVDBClient.get_session(name='s_json',serializer='json',persistence_base_key='test_json',)s_json['test_1']=test_1s_json['test_2']=test_2# s_json.persistence['test_1'] = test_1# s_json.persistence['test_2'] = test_2json_result_1=s_json['test_1']json_result_2=s_json['test_2']# json_result_1 = s_json.persistence['test_1']# json_result_2 = s_json.persistence['test_2']print(json_result_1,'type',type(json_result_1))print(json_result_2,'type',type(json_result_2))# Test assertionsassertresult_1==json_result_1assertresult_2==json_result_2# Delete the keysdels['test_1']dels['test_2']# s.delitem('test_1')# s.delitem('test_2')# s.persistence.pop('test_1')# s.persistence.pop('test_2')dels_json['test_1']dels_json['test_2']# s_json.persistence.pop('test_1')# s_json.persistence.pop('test_2')## Additionally, you can initialize a new persistence object directlyp=s_json.get_persistence(name='p',base_key='test_pickle',serializer='pickle',)p['test_1']=test_1p['test_2']=test_2CustomizationSince no objects/settings are initialized until they are first directly called, you can easily customize and define configurations for the underlying objects. For example, if you want to use a different serializer, you can do the following:fromkvdb.io.serializersimportset_default_serializer,set_default_serializer_lib,register_serializer# Set the default serializer to use# If `propogate=True` This will set the default serializer to pickle for all modules, such as cache, tasks, etc. Otherwise, it will only affect instances where `serializer=None`.set_default_serializer('pickle',propagate=True)# Set the default serializer library to useset_default_serializer_lib('cloudpickle')# You can also register a custom serializerregister_serializer('custom-serializer',MySerializerClass,set_as_default=True,propogate=True)# Alternatively, you can selectively modify the default serializer for certain modulesfromkvdb.configsimportset_default_serializer_for,set_default_kvdb_urlset_default_kvdb_url('redis://remote.redis.com:6379/0')set_default_serializer_for('custom-serializer','session','task')fromkvdbimportKVDBClientsession=KVDBClient.get_session()classMyModel(BaseModel):name:strage:int# This demonstrates how the pydantic model is automatically serialized and deserializednew_model=MyModel(name='John',age=30)session.set(new_model.name,new_model)src_model=session.get('John')# The session will use the custom serializer for the sessionassertsrc_model==new_model