Package: jamboree
Jamboree: A Fast General Data Engineering Library.Jamboreeis extremely early, meaning it should be used with caution. There are plans to improve the system and many components are subject to change. If you look at the improvement plans linked at the bottom you'll be able to see it.The goal ofjamboreewas to manage the complexities of data engineering.InstallThe library requires andredisto operate for the time being.pipinstalljamboreeInstall RedisAll of the redis installation instructions arehere. Though because the current module setup uses redisearch and will likely use many other modules in the future. Because installing modules is a bit more complex than necessary right now it's best to use docker:$dockerrun\-p6379:6379\-v/home/{PUTNAMEHERE}/data:/data\redislabs/redismod\--dir/dataThe output should look like the following.1:C24Apr201921:46:40.382# oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo...1:M24Apr201921:46:40.474*Module'ai'loadedfrom/usr/lib/redis/modules/redisai.so1:M24Apr201921:46:40.474*<ft>RediSearchversion1.4.7(Git=)1:M24Apr201921:46:40.474*<ft>concurrency:ON,gc:ON,prefixminlength:2,prefixmaxexpansions:200,querytimeout(ms):500,timeoutpolicy:return,cursorreadsize:1000,cursormaxidle(ms):300000,maxdoctablesize:1000000,searchpoolsize:20,indexpoolsize:8,1:M24Apr201921:46:40.475*<ft>Initializedthreadpool!1:M24Apr201921:46:40.475*Module'ft'loadedfrom/usr/lib/redis/modules/redisearch.so1:M24Apr201921:46:40.476*<graph>Threadpoolcreated,using8threads.1:M24Apr201921:46:40.476*Module'graph'loadedfrom/usr/lib/redis/modules/redisgraph.so
loadeddefaultMAX_SAMPLE_PER_CHUNKpolicy:3601:M24Apr201921:46:40.476*Module'timeseries'loadedfrom/usr/lib/redis/modules/redistimeseries.so1:M24Apr201921:46:40.476# <ReJSON> JSON data type for Redis v1.0.4 [encver 0]1:M24Apr201921:46:40.476*Module'ReJSON'loadedfrom/usr/lib/redis/modules/rejson.so1:M24Apr201921:46:40.476*Module'bf'loadedfrom/usr/lib/redis/modules/rebloom.so1:M24Apr201921:46:40.477*<rg>RedisGearsversion0.2.1,git_sha=fb97ad757eb7238259de47035bdd582735b5c81b1:M24Apr201921:46:40.477*<rg>PythonHomeDir:/usr/lib/redis/modules/deps/cpython/1:M24Apr201921:46:40.477*<rg>MaxExecutions:10001:M24Apr201921:46:40.477*<rg>RedisAIapiloadedsuccessfully.1:M24Apr201921:46:40.477# <rg> RediSearch api loaded successfully.1:M24Apr201921:46:40.521*Module'rg'loadedfrom/usr/lib/redis/modules/redisgears.so1:M24Apr201921:46:40.521*ReadytoacceptconnectionsTo run it in the background and let it start when the computer does$dockerrun\-p6379:6379-d\--restart=always\-v/home/{PUTNAMEHERE}/data:/data\redislabs/redismod\--dir/dataWhat is Event State Carrying?State Carrying is a round about way of saying tracking information through their interactions oversp time more so than exact states. It helps us construct a story of all things that have happened in a system over time. It looks like the image below.State carrying is dragging the current state along over time.The ultimate result is that you'd have tracability in your system. This is great when you're trying to see how interactions happen through time.How The Library WorksThe Jamboree Library Is Split In Two Parts:Jamboree Event SourcingObject HandlerTheJamboreeobject is rather simple. It only saves, reads, and deletes records in bothredisandmongodb. Redis to give it fast read times, mongodb as backup to the data.Handlershave very explicit storage procedures that interact with the Jamboree object. A good example is the code below.The idea is straightforward:We create aJamboreeobject. The Jamboree object manages connections to databases at a high speed and low latency.After we create the Handler object, and set the limit (max number of records we want to look at), we start adding records until we stop. At the end, we get the amount of time it took to push the records.Periodically, we do a small calculation to older information prior to adding a record.Creating a HandlerclassSampleEnvHandler(DBHandler):"""Abstract handler that we use to keep track of information."""def__init__(self,**kwargs):super().__init__()self.entity="sample"self.required={"episode":str}self._balance=0self._limit=100@propertydeflimit(self):returnself._limit@limit.setterdeflimit(self,limit):self._limit=limit@propertydefbalance(self):""" Gets the sum of the last three values at set the value """returnself._balance@propertydeftransactions(self)->vaex.dataframe:""" Get the last 100 transactions """many_records=self.many(self.limit)ifisinstance(many_records,dict):frame=pd.DataFrame(many_records)transactions_frame=vaex.from_pandas(frame)returntransactions_frame.sort('timestamp',ascending=False)iflen(many_records)>0:frame=pd.DataFrame(many_records)transactions_frame=vaex.from_pandas(frame)returntransactions_frame.sort('timestamp',ascending=False)returnvaex.from_pandas(pd.DataFrame())defsave_update_recent(self,data:dict):transactions=self.transactionscount=transactions.count()new_value=data['value']+countdata['value']=int(new_value)super().save(data)defflip(n=0.02):ifn>=random.uniform(0,1):returnTruereturnFalseif__name__=="__main__":main()Timing The Handlerjambo=Jamboree()sample_env_handler=SampleEnvHandler()sample_env_handler.limit=250sample_env_handler.event=jambosample_env_handler['episode']=uuid.uuid1().hexwithtimecontext():foriinrange(10000):v1=randint(0,12)sample_env_handler.save({"value":v1})ifflip(0.05):sample_env_handler.save_update_recent({"value":v1})Improvement PlansJamboree currently has a list of improvements thathttps://trello.com/b/9vwpc5C6