Package: tshistory
TSHISTORYThis is a library to store/retrieve pandas timeseries to/from a
postgres database, tracking their successive versions.IntroductionPurposetshistoryis targetted at applications using time series wherebacktestingandcross-validationare an essential feature.It provides exhaustivity and efficiency of the storage, with a simple
Python api.It can be used as a building block for machine learning, model
optimization and validation, both for inputs and outputs.PrinciplesThere are many ways to represent timeseries in a relational database,
andtshistoryprovides two things:a base python API which abstracts away the underlying storagea postgres model, which emphasizes the compact storage of successive
states of seriesThe core idea of tshistory is to handle successive versions of
timeseries as they grow in time, allowing to get older states of any
series.Basic usageStarting with a fresh databaseYou need a postgresql database. You can create one like this:createdbmydbThen, initialize thetshistorytables, like this:tshinit-dbpostgresql://me:password@localhost/mydbFrom this you're ready to go !Creating a seriesHowever here's a simple example:>>>importpandasaspd>>>fromtshistory.apiimporttimeseries>>>>>>tsa=timeseries('postgres://me:password@localhost/mydb')>>>>>>series=pd.Series([1,2,3],...pd.date_range(start=pd.Timestamp(2017,1,1),...freq='D',periods=3))# db insertion>>>tsa.update('my_series',series,'babar@pythonian.fr')...2017-01-011.02017-01-022.02017-01-033.0Freq:D,Name:my_series,dtype:float64# note how our integers got turned into floats# (there are no provisions to handle integer series as of today)# retrieval>>>tsa.get('my_series')...2017-01-011.02017-01-022.02017-01-033.0Name:my_series,dtype:float64Note that we generally adopt the convention to name the time series
api objecttsa.Updating a seriesThis is good. Now, let's insert more:>>>series=pd.Series([2,7,8,9],...pd.date_range(start=pd.Timestamp(2017,1,2),...freq='D',periods=4))# db insertion>>>tsa.update('my_series',series,'babar@pythonian.fr')...2017-01-037.02017-01-048.02017-01-059.0Name:my_series,dtype:float64# you get back the *new information* you put inside# and this is why the `2` doesn't appear (it was already put# there in the first step)# db retrieval>>>tsa.get('my_series')...2017-01-011.02017-01-022.02017-01-037.02017-01-048.02017-01-059.0Name:my_series,dtype:float64It is important to note that the third value wasreplaced, and the two
last values were justappended. As noted the point at2017-1-2wasn't a new information so it was
just ignored.Retrieving historyWe can access the whole history (or parts of it) in one call:>>>history=tsa.history('my_series')...>>>>>>foridate,seriesinhistory.items():# it's a dict...print('insertion date:',idate)...print(series)...insertiondate:2018-09-2617:10:36.988920+02:002017-01-011.02017-01-022.02017-01-033.0Name:my_series,dtype:float64insertiondate:2018-09-2617:12:54.508252+02:002017-01-011.02017-01-022.02017-01-037.02017-01-048.02017-01-059.0Name:my_series,dtype:float64Note how this shows the full serie state for each insertion date. Also the insertion date is timzeone aware.Specific versions of a series can be retrieved individually using thegetmethod as follows:>>>tsa.get('my_series',revision_date=pd.Timestamp('2018-09-26 17:11+02:00'))...2017-01-011.02017-01-022.02017-01-033.0Name:my_series,dtype:float64>>>>>>tsa.get('my_series',revision_date=pd.Timestamp('2018-09-26 17:14+02:00'))...2017-01-011.02017-01-022.02017-01-037.02017-01-048.02017-01-059.0Name:my_series,dtype:float64It is possible to retrieve only the differences between successive insertions:>>>diffs=tsa.history('my_series',diffmode=True)...>>>foridate,seriesindiffs.items():...print('insertion date:',idate)...print(series)...insertiondate:2018-09-2617:10:36.988920+02:002017-01-011.02017-01-022.02017-01-033.0Name:my_series,dtype:float64insertiondate:2018-09-2617:12:54.508252+02:002017-01-037.02017-01-048.02017-01-059.0Name:my_series,dtype:float64You can see a series metadata:>>>tsa.update_metadata('series',{'foo':42})>>>tsa.metadata('series'){foo:42}Staircase seriesA staircase series can be defined as a series of which values originate from successive
revisions with a fixed time span between revision date and value date. This is
especially useful for backtesting.Basic staircaseLet us take an example assuming a series calleddaily_serieshas been created with
insertions given by the following table (considering row indices are value dates, and
columns indices are insertion dates):2020-01-0100:00+002020-01-0200:00+002020-01-0300:00+002020-01-011.12020-01-022.12.22020-01-033.13.23.32020-01-044.24.32020-01-055.3Supposing this series is a forecast published on a daily basis, we can for example
reconstruct the day-ahead forecast series, i.e. the values such that the time span
between revision date and value date is 1 day (or more) as follows:>>>tsa.staircase('daily_series',from_value_date=pd.Timestamp('2020-01-01'),to_value_date=pd.Timestamp('2020-01-07'),delta=pd.Timedelta(days=1))...2020-01-022.12020-01-033.22020-01-044.32020-01-055.3Name:daily_series,dtype:float64The name "staircase" refers to the way in which these values are picked from the history:2020-01-0100:00+002020-01-0200:00+002020-01-0300:00+002020-01-012020-01-022.12020-01-033.22020-01-044.32020-01-055.3Now if instead we consider an hourly forecast series, we may want to define day-ahead
forecast as a staircase series with a daily revision occurring at 9am, and link each
revision to the 24 hours of the next day. More generally we may want to reconstruct a
staircase series where successive revisions each relate to several value dates. Such
cases should instead be handled using theblock_staircasemethod described below.Block staircaseLet us take another example considering the serieshourly_serieswith following insertions:2020-01-0106:00+002020-01-0114:00+002020-01-0206:00+002020-01-0214:00+002020-01-01 00:00+001.11.22020-01-01 08:00+002.12.22020-01-01 16:00+003.13.22020-01-02 00:00+004.14.24.34.42020-01-02 08:00+005.15.25.35.42020-01-02 16:00+006.16.26.36.42020-01-03 00:00+007.17.27.37.42020-01-03 08:00+008.18.28.38.42020-01-03 16:00+009.19.29.39.42020-01-04 00:00+0010.310.42020-01-04 08:00+0011.311.42020-01-04 16:00+0012.312.4Then the day-ahead forecast with revisions at 9am can be computed as follows:>>>tsa.block_staircase('hourly_series',from_value_date=pd.Timestamp('2020-01-01',tz="utc"),to_value_date=pd.Timestamp('2020-01-05',tz="utc"),revision_freq={'days':1},revision_time={'hour':9},revision_tz='utc',maturity_offset={'days':1},maturity_time={'hour':0})...2020-01-0200:00:00+00:004.12020-01-0208:00:00+00:005.12020-01-0216:00:00+00:006.12020-01-0300:00:00+00:007.32020-01-0308:00:00+00:008.32020-01-0316:00:00+00:009.32020-01-0400:00:00+00:0010.42020-01-0408:00:00+00:0011.42020-01-0416:00:00+00:0012.4Name:hourly_series,dtype:float64Note that withrevision_time={'hour': 9}, the method ends up picking values from the
two 6am insertions (except for the values of 2020-01-04 when latest available revision
is 2020-01-02 14:00). Taking revision time after 2pm, sayrevision_time={'hour': 20}, would instead select values from the 2pm insertions only.In general, the arguments ofblock_staircaseshould be used as follows:from_value_dateandto_value_date: time range on which values are retrievedrevision_freq: revision frequency, as a dictionary of integers of which keys must be taken from["years", "months", "weeks", "bdays", "days", "hours", "minutes", "seconds"]revision_time: revision time, as a dictionary of integers of which keys should be
taken from["year", "month", "day", "weekday", "hour", "minute", "second"]. It is used
for revision date initialisation. The next revision dates are then obtained by
successively addingrevision_freq.revision_tz: time zone in which revision date and time are expressedmaturity_offset: time span between each revision date and start time
of related block of values, as dictionary of integers. Its keys must be taken from["years", "months", "weeks", "bdays", "days", "hours", "minutes", "seconds"]. No lag
is considered if it is not specified, i.e. the revision date is the block start datematurity_time: start time of each block, as a dictionary of integers of which keys
should be taken from["year", "month", "day", "hour", "minute", "second"]. The start
date of each block is thus obtained by addingmaturity_offsetto revision date and
then applyingmaturity_time. If not specified block start date is just the revision
date shifted bymaturity_offsetOther use casesTheblock_staircasemethod covers multiple use cases, such as week-ahead revisions or
revision by business day, as described in the following examples.Week-ahead staircaseConsider a series namedweekly_serieswith following insertions:2021-01-05(Tue)2021-01-07(Thu)2021-01-12(Tue)2021-01-14(Thu)2021-01-11 (Mon)1.11.22021-01-12 (Tue)2.12.22021-01-13 (Wed)3.13.22021-01-14 (Thu)4.14.22021-01-15 (Fri)5.15.22021-01-16 (Sat)6.16.22021-01-17 (Sun)7.17.22021-01-18 (Mon)8.18.28.38.42021-01-19 (Tue)9.19.29.39.42021-01-20 (Wed)10.110.210.310.42021-01-21 (Thu)11.311.42021-01-22 (Fri)12.312.42021-01-23 (Sat)13.313.42021-01-24 (Sun)14.314.42021-01-25 (Mon)15.315.42021-01-26 (Tue)16.316.42021-01-27 (Wed)17.317.4Then the week-ahead staircase with weekly revision on Friday can be retrieved as
follows:>>>tsa.block_staircase('weekly_series',from_value_date=pd.Timestamp('2021-01-10'),to_value_date=pd.Timestamp('2021-01-30'),revision_freq={'days':7},revision_time={'weekday':4},revision_tz='utc',maturity_offset={'days':3},maturity_time={'hour':0})...2021-01-111.22021-01-122.22021-01-133.22021-01-144.22021-01-155.22021-01-166.22021-01-177.22021-01-188.42021-01-199.42021-01-2010.42021-01-2111.42021-01-2212.42021-01-2313.42021-01-2414.42021-01-2515.42021-01-2616.42021-01-2717.4Name:weekly_series,dtype:float64It is also possible to retrieve a month-ahead staircase series taking insteadrevision_freq={'months': 1}and, for example,revision_time={'day': 15}to perform
monthly revision every 15th day of the month.Revision by business dayThe block_staircase method allows to express revision frequency and/or maturity time
span in business days. Consider a series namedbusiness_day_serieswith these
insertions:2021-01-13(Wed)2021-01-14(Thu)2021-01-15(Fri)2021-01-16(Sat)2021-01-17(Sun)2021-01-18(Mon)2021-01-13 (Wed)3.12021-01-14 (Thu)4.14.22021-01-15 (Fri)5.15.25.32021-01-16 (Sat)6.16.26.36.42021-01-17 (Sun)7.27.37.47.52021-01-18 (Mon)8.38.48.59.62021-01-19 (Tue)9.49.511.62021-01-20 (Wed)10.512.62021-01-21 (Thu)13.6Then we can retrieve a business-day-ahead staircase series with revision every business
day as follows:>>>tsa.block_staircase('business_day_series',from_value_date=pd.Timestamp('2021-01-13'),to_value_date=pd.Timestamp('2021-01-21'),revision_freq={'bdays':1},revision_tz='utc',maturity_offset={'bdays':1})...2021-01-144.12021-01-155.22021-01-166.22021-01-177.22021-01-188.32021-01-1911.62021-01-2012.62021-01-2113.6Name:weekly_series,dtype:float64Optimizing staircase computations with history cacheIt may be useful in some cases to compute multiple staircase series from the same source
series. For example, given a series named"my_forecast", we could reconstruct both the
one-day-ahead and two-days-ahead staircase series by doing>>>ts_1da=tsa.block_staircase('my_forecast',from_value_date=pd.Timestamp('2020-01-01',tz="utc"),to_value_date=pd.Timestamp('2022-01-01',tz="utc"),revision_freq={'days':1},revision_time={'hour':9},revision_tz='utc',maturity_offset={'days':1},maturity_time={'hour':0})>>>ts_2da=tsa.block_staircase('my_forecast',from_value_date=pd.Timestamp('2020-01-01',tz="utc"),to_value_date=pd.Timestamp('2022-01-01',tz="utc"),revision_freq={'days':1},revision_time={'hour':9},revision_tz='utc',maturity_offset={'days':2},maturity_time={'hour':0})However, theblock_staircasefunction makes use of thehistoryfunction to
reconstruct staircase series. For this reason, the response time may be a bit long
depending on the time span of the staircase series and the number of insertions.To optimize the total execution time, we could retrieve the history of"my_forecast"series, store it in memory and reconstruct the staircase series from it. This can be
done using thehistorycacheobject from thetshsitory.tsiomodule, as follows>>>fromtshistory.tsioimporthistorycache>>>>>>hist=tsa.history('my_forecast'from_value_date=pd.Timestamp('2020-01-01',tz="utc"),to_value_date=pd.Timestamp('2022-01-01',tz="utc"))>>>hcache=historycache('my_forecast_cache',hist=hist,tzaware=True)>>>>>>ts_1da=hcache.block_staircase(from_value_date=pd.Timestamp('2020-01-01',tz="utc"),to_value_date=pd.Timestamp('2022-01-01',tz="utc"),revision_freq={'days':1},revision_time={'hour':9},revision_tz='utc',maturity_offset={'days':1},maturity_time={'hour':0})>>>ts_2da=hcache.block_staircase(from_value_date=pd.Timestamp('2020-01-01',tz="utc"),to_value_date=pd.Timestamp('2022-01-01',tz="utc"),revision_freq={'days':1},revision_time={'hour':9},revision_tz='utc',maturity_offset={'days':2},maturity_time={'hour':0})In the example above, thehistoryfunction of tshistory API is called once. Then the
staircase series are computed using the history data stored in memory using thehistorycacheobject, which avoids one API call and reduce execution time.This example assumes the series"my_forecast"is timezone-aware. In the case of
timezone-naive series, thetzawareparameter ofhistorycacheshould be adapted
accordingly.Thehistorycacheclass also provides astaircasemethod, so this technique can also
be used for basic staircase computation.The API objectIn the few examples above we manipulate the time series through an
object that talks directly to the postgresql back end.It is possible to also talk to a rest api using the same api, like
shown below and proceed exactly like in the above code examples:>>>fromtshistory.apiimporttimeseries>>>>>>tsa=timeseries('http://my.timeseries.info/api')Using an HTTP/REST end pointFor the rest api, you need to build a smallflaskapp like
this (in anapp.pymodule):fromflaskimportFlaskfromtshistory.apiimporttimeseriesfromtshistory.http.serverimportblueprintasblueprintdefmake_app(dburi):app=Flask('my-timeseries-app')app.register_blueprint(blueprint(timeseries(dburi)),url_prefix='/api')returnappThen, you can start it in development mode like this:app=make_app('postgresql://me:password@localhost/mydb')app.run('192.168.1.1',8080)or just leave it to a wsgi container in e.g. awsgi.pymodule:frommy_series_app.appimportmake_appapp=make_app('postgresql://me:password@localhost/mydb')API surfaceFor now we only provide a list of supported methods.Information access (read methods)catalogexistsgethistoryintervalmetadatastaircaseblock_staircasetypeInformation update (write methods)updateupdate_metadatareplacerenamedeleteCommand lineBasic operationsA command line tool is provided, calledtsh. It provides its usage
guidelines:$tshUsage:tsh[OPTIONS]COMMAND[ARGS]...Options:--helpShowthismessageandexit.

Commands:checkcoherencechecksofthedbgetshowaserieinitscurrentstatehistoryshowaseriefullhistoryinfoshowglobalstatisticsoftherepositoryinit-dbinitializeannewdb.logshowrevisionhistoryofentirerepositoryor...viewvisualizetimeseriesthroughthewebInfoprovides an overview of the time series repository (number of
committed changes, number and series and their names).$tshinfopostgres://babar:babarpassword@dataserver:5432/banana_studieschangesetcount:209seriescount:144seriesnames:banana_spot_price,banana_trades,banana_turnoverLogprovides the full history of editions to time series in the
repository.$tshlogpostgres://babar:babar@dataserver:5432/banana_studies--limit3revision:206author:BABARdate:2017-06-0615:32:51.502507series:banana_spot_pricerevision:207author:BABARdate:2017-06-0615:32:51.676507series:banana_tradesrevision:209author:CELESTEdate:2017-06-0615:32:51.977507series:banana_turnoverAll options of all commands can be obtained by using the--helpswitch:$tshlog--helpUsage:tshlog[OPTIONS]DB_URIOptions:-l,--limitTEXT--show-diff-s,--serieTEXT--from-revTEXT--to-revTEXT--helpShowthismessageandexit.ExtensionsIt is possible to augment thetshcommand with new subcommands (or
augment, modify existing commands).Any program doing so must define a new command and declare a setup
tools entry point namedtshistory:subcommandas in e.g.:entry_points={'tshistory.subcommands':['view=tsview.command:view']}For instance, thetsviewpython package provides such aviewsubcommand for generic time series visualisation.StatusIt is currently consideredbetasoftware even though it has been in
production for two years. It is still evolving. Schema/Database
changes come with migration procedure using thetshutility.When it is good: if you do mostly appends (and occasional edits in the
past) it will store data in a very compact way.Reading any version of the series will always be the fastest (io-bound)
operation.Alternative backend storage and storage strategies will be considered
in the future.