Package: datatype-redis
Redis for builtin-datatypesThis library implements all builtin datatypes from python3 with a redis backend.StatusWork in progress, not usableUsagefromdatatype_redisimportStringfoo=String("bar")IntroductionAfter a lot of research to backup all my dict and lists to redis to use them in my microservice ecosystem, i was very disappointed about the current situation in the library environment for this purpose. There are a lot of libraries, which handle it, but no one includes everything i need. So i had to implement this by myself. You see here the results.InspirationThis library is heavily inspired byhot-redisandpy-redis-pubsub-dict.OptionsIn the following, the options of this library will be described.Set redis as clientAs default, the library use Redis fromredis-pyas implementation. If you want to use another implementation, you have to inherit from this class.If you do not set the client-argument as follows, the library assumes to use a local installation on port 6379.fromdatatype_redisimportStringfoo=String("bar",client={"host":"myremotehost","port":6379})You can initialize the redis client by yourself, too.fromdatatype_redisimportStringfromredisimportRedisr=Redis(host='myremotehost',port=6379,db=0)foo=String("bar",client=r)Beware: If the client is not compatible with this library, it will raise anValueError.If you want to use a redisCluster, you can do this with the client-parameter, too.fromredisclusterimportRedisClusterfromdatatype_redisimportString# Requires at least one node for cluster discovery. Multiple nodes is recommended.startup_nodes=[{"host":"myremotehost","port":"7000"}]foo=String("bar",client=RedisCluster,startup_nodes=startup_nodes)You can overwrite the default behaviour with the followingconfigurefunction.fromdatatype_redisimportconfigureconfigure(host='myremotehost',port=6380,db=0)Of course, you can use it to set your initialized client as default.fromdatatype_redisimportconfigurefromredisclusterimportRedisCluster# Requires at least one node for cluster discovery. Multiple nodes is recommended.configure(client=RedisCluster,startup_nodes=startup_nodes)Beware: If you set host and port and client at the same time, the library will take your client as default and ignore host and port.You can use any implementation as client you want and set all parameters to configure. The library initialize your implementation with the given parameters for you threadsafe, so you will get a client for each thread you initialize. This behaviour helps to get a better performance in webservers etc.Redis keyRedis is a key-value-store, so it needs a key to store the values. To reduce the typing, you can use the default behaviour of this library, which uses unique identifier as keys. You can get the key with theget_redis_key-method to any datatype.But if you want to use this library in a microservice, which should be duplicated over your cluster, this behaviour would be toxic to your implementation, so you can use thekey-parameter to set an identifier.fromdatatype_redisimportStringfoo_uuid=String("bar")foo_key=String("bar",key="foo")print(foo_uuid)# expect: "bar"print(foo_uuid.get_redis_key())# expect: similar to "12345678-1234-5678-1234-567812345678"print(foo_key)# expect: "bar"print(foo_key.get_redis_key())# expect: "foo"If you query the uuid as key in your redis installation, you will not get the serialization of your data as you would expected, because this library uses a prefix per default to work with a big redis installation and to not interfere with any other implementations. You can get the key with prefix withget_redis_key_full-method and set this prefix inconfigure. The default value for prefix isdatatype-redis.fromdatatype_redisimportconfigurefoo_uuid=String("bar",key="fooKey")print(foo_uuid.get_redis_key_full())# expect: "datatype-redis/fooKey"print(foo_uuid)# expect: "bar"configure(prefix="fooPrefix")print(foo_uuid.get_redis_key_full())# expect: "fooPrefix/fooKey"print(foo_uuid)# expect: "bar"Beware: If you change your prefix, while you initialized a datatype, the library will rename all keys automatically with a matching filter. This will take some time on larger redis installations.If you want to get the current prefix as string, you can call theget_prefix-method.fromdatatype_redisimportget_prefixprint(get_prefix())# expect default: "datatype-redis"Rename keyIf you want to store your value in another key, you can do that with therename-method.fromdatatype_redisimportconfigurefoo_uuid=String("bar",key="fooKey")print(foo_uuid.get_redis_key_full())# expect: "datatype-redis/fooKey"foo_uuid.rename("barKey")print(foo_uuid.get_redis_key_full())# expect: "datatype-redis/barKey"Msgpack as serializerIf you want to store complex datatypes (like dict or list) in redis, you need to use a serializer. This library usemsgpackfor this purpose. If you want to use another implementation, your implementation needs adumpsandloadsmethod.The constructor of any datatype take your implementation withserializer-key.Example (use json as serializer):fromdatatype_redisimportStringimportjsonfoo=String("foo",serializer=json)Beware: If you want to use nested datatypes with redis backend from this library, you should use uuid-keys. Otherwise you could overwrite existing keys and your implementation does not work as you would expect.PubSub BackendIf you need to get your results ASAP, but you change them rarely, you should take a look at the pubsub-cache implementation. This will reduce the latency for get-requests drastically, but you will get every problem with caches.If you want to set a cache to an atomic datatype like String or Int, you can use thePubSubCacheAtomicclass. You have to initialize a cache to store the values. Also, you have to use a specialized implementation of the corresponding datatype, calledPubSub*. See the following example.fromdatatype_redisimportPubSubCacheAtomic,PubSubStringfrompylruimportlrucachecache=lrucache(1)store=PubSubString("foo")redcache=PubSubCacheAtomic(store,cache)The following table shows the PubSub classes of each atomic datatype.atomicPubSubBoolPubSubBoolStringPubSubStringIntPubSubIntFloatPubSubFloatFor complex datatypes (called datastructure) like Dict or List, you have to use the correspondigPubSubCache*classes (Do not use thePubSubCacheAtomicclass for datastructures).datastructurePubSubCache*DictPubSubCacheDictListPubSubCacheListYou use them as follows.fromdatatype_redisimportPubSubCacheDict,PubSubDictfrompylruimportlrucachecache=lrucache(10)store=PubSubDict()redcache=PubSubCacheDict(store,cache)Examples and DemonstrationsText TypeStringsfromdatatype_redisimportStringfoo=String("bar")print(foo)# expect: "bar"foo+="foo"print(foo)# expect: "barfoo"Numeric TypesIntegerfromdatatype_redisimportIntfoo=Int(3)print(foo)# expect: "3"foo+=7print(foo)# expect: "10"foo-=11print(foo)# expect: "-1"Floatfromdatatype_redisimportFloatfoo=Float(3.4)print(foo)# expect: "3.4"foo+=7print(foo)# expect: "10.4"foo-=11print(foo)# expect: "-0.6"ComplexSequence TypesListfromdatatype_redisimportListfoo=List()print(foo)# expect: []foo.append("foo")print(foo)# expect: ["foo"]("foo"infoo)# expect: Truefoo=List(["bar"])print(foo)# expect: ["bar"]("foo"infoo)# expect: FalseTupleRangeMapping TypeDictfromdatatype_redisimportDictfoo=Dict()print(foo)# expect: {}foo["foo"]="barprint(foo["foo"])# expect: "bar"print(foo["bar"])# expect: raise KeyErrorprint(foo)# expect: {"foo":"bar"}print(foo.items())# expect: [("foo", "bar")]print(foo.values())# expect: [("bar")]print(foo.keys())# expect: [("foo")]foo["bar"]=[1,2,3]print(foo["bar"])# expect: [1,2,3]print(foo)# expect: {"foo":"bar", "bar":[1,2,3]}Set TypesSetFrozen SetBoolean TypeboolBinary TypesBytesBytearrayMemoryViewMicroservicesThe main purpose of this library is to use in a microservice ecosystem like a container in kubernetes. You have to run a redis instance in your cloud, configure your client withconfigure()for this library, so your microservice use redis as backend for your datatypes. This will make your values available on all your microservice implementations with same keys.For redis in kubernetes, you should use a helm chart e.g.redis-ha.