Package: ttseries
High performance engine to store Time series data in Redis.Redis Version required: >=5.0TT-series is based on redis sorted sets to store the time-series data,Sorted setstore scores with
unique numbers under a single key, but it has a weakness to store records, only unique members are allowed
and trying to record a time-series entry with the same value as a previous will result in only updating the score.
So TT-series provide a solution to solve that problem.TT series normally can support redis version > 3.0, and will supportredis 5.0in the future.TipsMax Store series lengthFor 32 bit Redis on a 32 bit platform redis sorted sets can support maximum 2**32-1 members,
and for 64 bit redis on a 64 bit platform can support maximum 2**64-1 members.
But large amount of data would cause more CPU activity, so better keep a balance with length of records is
very important.Only Support Python 3.6Because python 3.6 changed the dictionary implementation for better performance,
so in Python 3.6 dictionaries are insertion ordered.
links:https://stackoverflow.com/questions/39980323/are-dictionaries-ordered-in-python-3-6Performance TipsWith hiredis-py which it’s targeted at speeding up parsing multi bulk replies from redis-server.
So with a large amount of bulk data insertion or getting from redis-server, it can improve a great performance improvement.InstallInstall python package from pip release:pip install ttseriesDocumentationFeaturesQuick inserts 100,000 records (1-2 sec) and get slice of 100,000 records (0.4-0.5 sec).Support Data Serializer, Default Enable with MessagePack.Support Data Compression.In-order to void update previous records, Support Redis Hashes Time-Series storage format.Support Numpy ndarray data type.Support max length to auto to trim records.UsageTT-series provide three implementation to support different kinds of time-series data type.RedisSimpleTimeSeries: Normally only base on Sorted sets to store records, previous records will impact the new inserting records which areNOTunique numbers.RedisHashTimeSeries: Use Redis Sorted sets with Hashes to store time-series data, User don’t need to consider the data repeatability with records, but sorted sets with hashes would take some extra memories to store the keys.RedisNumpyTimeSeries: Supportnumpy.ndarrayto store time-series records in redis sorted set.RedisPandasTimeSeries: Supportpandas.DataFrameto store time-series records in redis sorted set.Serializer DataTT-series useMsgPackto serializer data, because compare with other data serializer’s solutions,
MsgPack provide a better performance solution to serialize data. If user don’t want to use MsgPack to
serializer data, just inherit fromttseries.BaseSerializerclass to implement the supported
serializer class methods.ExamplesRedisSimpleTimeSeries&&RedisHashTimeSeries&&RedisNumpyTimeSeries&&RedisPandasTimeSeriesThree series data implementation provide the same functions and methods, in the usage will
provide the difference in the methods.Prepare data records:fromdatetimeimportdatetimefromredisimportStrictRedisnow=datetime.now()timestamp=now.timestamp()series_data=[]foriinrange(1000):series_data.append((timestamp+i,i))client=StrictRedis()# redis clientAdd recordsfromttseriesimportRedisSimpleTimeSeriessimple_series=RedisSimpleTimeSeries(client=client)key="TEST:SIMPLE"simple_series.add_many(key,series_data)Count records lengthGet the length of the records or need just get the length from timestamp span.# get the records lengthsimple_series.length(key)# result: ...: 1000# get the records length from start timestamp and end timestampsimple_series.count(key,from_timestamp=timestamp,end_timestamp=timestamp+10)# result: ...: 11trim recordsTrim the records as the ASC.simple_series.trim(key,10)# trim 10 length of recordsdelete timestamp spanDelete timestamp provide delete key or delete records from start timestamp to end timestamp.simple_series.delete(key)# delete key with all recordssimple_series.delete(key,start_timestamp=timestamp)# delete key form start timestampGet SliceGet slice form records provide start timestamp and end timestamp withASCorDESCordered.Default Order:ASCIf user want to get the timestamp great than (>) or less than (<) which not including the timestamp record.
just use(timestampwhich support<timestampor>timestampsign format like this.# get series data from start timestamp ordered as ASC.simple_series.get_slice(key,start_timestamp=timestamp,acs=True)# get series data from great than start timestamp order as ASCsimple_series.get_slice(key,start_timestamp="("+str(timestamp),asc=True)# get series data from start timestamp and limit the numbers with 500time_series.get_slice(key,start_timestamp=timestamp,limit=500)iteryield item from records.foriteminsimple_series.iter(key):print(item)RedisNumpyTimeSeriesNumpy array support providenumpy.dtypeor just arrays with data.Usenumpy.dtypeto create records. must providetimestamp_column_nameanddtypeparameters.importnumpyasnpfromttseriesimportRedisNumpyTimeSeriesdtype=[("timestamp","float64"),("value","i")]array=np.array(series_data,dtype=dtype)np_series=RedisNumpyTimeSeries(client=client,dtype=dtype,timestamp_column_name="timestamp")Or just numpy array without dtype, but must providetimestamp_column_indexparameter.array=np.array(series_data)np_series=RedisNumpyTimeSeries(client=client,timestamp_column_index=0)RedisPandasTimeSeriesPandas TimeSeries usepandas.DataFrameto store time-series in redis.
To initialize the class must providecolumnsanddtypesparameters.columnsparameter indicates the column names of thepandas.DataFrame.dtypesindicates the dtype of each column in DataFrame, for example:{"value1":"int64","value2":"float32"}reference link:http://pbpython.com/pandas_dtypes.htmlfromdatetimeimportdatetimekey="AA:MIN"now=datetime.now()columns=["value"]date_range=pandas.date_range(now,periods=10,freq="1min")data_frame=pandas.DataFrame([i+1foriinrange(len(date_range))],index=date_range,columns=columns)dtypes={"value":"int64"}pandas_ts=RedisPandasTimeSeries(client=client,columns=columns,dtypes=dtypes)AddAdd a time-series record to redis,seriesparameter indicatespandas.Seriesdata type.
and especial theseriesname value data type must be thepandas.DatetimeIndex.series_item=data_frame.iloc[0]pandas_ts.add(key,series_item)add_manyAdd large amount ofpandas.DataFrameinto redis, with thedataframeindex data type must be
thepandas.DatetimeIndex.
For better insert performance, just usechunks_sizeto split the dataframe into fixedchunks_sizerows of dataframes.pandas_ts.add_many(key,data_frame)iter & Getretrieve records from redis sorted set, both of methods returnpandas.Series.# yield all records data from redisforiteminpandas_ts.iter(key):print(item)# return one record with specific timestamppandas_ts.get(key,1536157765.464465)get_sliceretrieve records to slice withstart timestamporend timestamp, withlimitlength.
returnpandas.DataFrame# return records from start timestamp 1536157765.464465result_frame=pandas_ts.get_slice(key,start_timestamp=1536157765.464465)# return records from start timestamp 1536157765.464465 to end timestamp 1536157780.464465result2_frame=padas_ts.get_slice(key,start_timestamp=1536157765.464465,end_timestamp=1536157780.464465)Benchmarkjust runmakebenchmark-init, after then startmakebenchmark-test.Go to the benchmark directory there exist an example of the benchmark test reports.TODOSupport Redis 5.0Support Redis cluster.AuthorWinton WangDonateContactEmail:365504029@qq.comReferencelinks:https://www.infoq.com/articles/redis-time-series