Package: cs.timeseries
Efficient portable machine native columnar file storage of time
series data for double float and signed 64-bit integers.Latest release 20240201:
Release with "csts" script.The core purpose is to provide time series data storage; there
are assorted convenience methods to export arbitrary subsets
of the data for use by other libraries in common forms, such
as dataframes or series, numpy arrays and simple lists.
There are also some simple plot methods for plotting graphs.Three levels of storage are defined here:TimeSeriesFile: a single file containing a binary list of
float64 or signed int64 valuesTimeSeriesPartitioned: a directory containing multipleTimeSeriesFilefiles, each covering a separate time span
according to a supplied policy, for example a calendar monthTimeSeriesDataDir: a directory containing multipleTimeSeriesPartitionedsubdirectories, each for a different
time series, for example one subdirectory for grid voltage
and another for grid powerTogether these provide a hierarchy for finite sized files storing
unbounded time series data for multiple parameters.On a personal basis, I use this as efficient storage of time
series data from my solar inverter, which reports in a slightly
clunky time limited CSV format; I import those CSVs into
time series data directories which contain the overall accrued
data; see mycs.splinkmodule which is built on this module.Functionarray_byteswapped(ary)Context manager to byteswap thearray.arrayarytemporarily.ClassArrowBasedTimespanPolicy(TimespanPolicy, icontract._metaclass.DBC, HasEpochMixin, TimeStepsMixin, cs.deco.Promotable)ATimespanPolicybased on an Arrow format string.See theraw_edgesmethod for the specifics of how these are defined.Functionas_datetime64s(times, unit='s', utcoffset=0)Return a Numpy array ofdatetime64values
computed from an iterable ofint/floatUNIX timestamp values.The optionalunitparameter (default's') may be one of:'s': seconds'ms': milliseconds'us': microseconds'ns': nanoseconds
and represents the precision to preserve in the source time
when converting to adatetime64.
Less precision gives greater time range.Functiondatetime64_as_timestamp(dt64: numpy.datetime64)Return the UNIX timestamp for thedatetime64valuedt64.Functiondeduce_type_bigendianness(typecode: str) -> boolDeduce the native endianness fortypecode,
an array/struct typecode character.ClassEpoch(Epoch, builtins.tuple, TimeStepsMixin, cs.deco.Promotable)The basis of time references with a starting UNIX timestartand astepdefining the width of a time slot.Functionget_default_timezone_name()Return the default timezone name.ClassHasEpochMixin(TimeStepsMixin)ATimeStepsMixinwith.startand.stepderived fromself.epoch.Functionmain(argv=None)Run the command line tool forTimeSeriesdata.Functionplot_events(start, stop, events, value_func, *, utcoffset, figure=None, ax=None, **scatter_kw) -> matplotlib.axes._axes.AxesPlotevents, an iterable of objects with.unixtimeattributes such as anSQLTagSet.
Return theAxeson which the plot was made.Parameters:events: an iterable of objects with.unixtimeattributesvalue_func: a callable to compute the y-axis value from an eventstart: optional start UNIX time, used to crop the events plottedstop: optional stop UNIX time, used to crop the events plottedfigure,ax: optional arguments as forcs.mplutils.axesutcoffset: optional UTC offset for presentation
Other keyword parameters are passed toAxes.scatter.ClassPlotSeries(PlotSeries, builtins.tuple, cs.deco.Promotable)Information about a series to be plotted:label: the label for this seriesseries: aSeriesextra: adictof extra information such as plot stylingClassTimePartition(TimePartition, builtins.tuple, TimeStepsMixin)Anamedtuplefor a slice of time with the following attributes:epoch: the referenceEpochname: the name for this slicestart_offset: the epoch offset of the start timeend_offset: the epoch offset of the end timeThese are used byTimespanPolicyinstances to express the partitions
into which they divide time.Functiontimerange(*da, **dkw)A decorator intended for plotting functions or methods which
presents optionalstartandstopleading positional
parameters and optionaltzorutcoffsetkeyword parameters.
The decorated function will be called with leadingstartandstoppositional parameters and a specificutcoffsetkeyword parameter.The as-decorated function is called with the following parameters:start: an optional UNIX timestamp positional for the
start of the range; if omitted the default isself.start;
this is a required parameter if the decorator hasneeds_start=Truestop: an optional UNIX timestamp positional parameter for the end
of the range; if omitted the default isself.stop;
this is a required parameter if the decorator hasneeds_stop=Truetz: optional timezonedatetime.tzinfoobject or
specification as fortzfor();
this is used to infer a UTC offset in secondsutcoffset: an optional offset from UTC time in seconds
Other parameters are passed through to the deocrated function.A decoratedmethodis then called as:method(self, start, stop, *a, utcoffset=utcoffset, **kw)where*aand**kware the additional positional and keyword
parameters respectively, if any.A decoratedfunctionis called as:function(start, stop, *a, utcoffset=utcoffset, **kw)Theutcoffsetis an offset to apply to UTC-based time data
forpresentationon the graph, largely because the plotting
functions useDataFrame.plotwhich broadly ignores attempts
to set locators or formatters because it supplies its own.
The plotting function would shift the values of theDataFrameindex using this value.If neitherutcoffsetortzis supplied by the caller, theutcoffsetis0.0.
A specifiedutcoffsetis passed through.
Atzis promoted to atzinfoinstance via thetzfor()function and applied to thestoptimestamp to obtain adatetimefrom which theutcoffsetwill be derived.
It is an error to specify bothutcoffsetandtz.ClassTimeSeries(cs.resources.MultiOpenMixin, cs.context.ContextManagerMixin, HasEpochMixin, TimeStepsMixin)Common base class of any time series.Functiontimeseries_from_path(tspath: str, epoch: Optional[cs.timeseries.Epoch] = None, typecode=None)Turn a time series filesystem path into a time series:a file: aTimeSeriesFilea directory holding.cstsfiles: aTimeSeriesPartitioneda directory: aTimeSeriesDataDirClassTimeSeriesBaseCommand(cs.cmdutils.BaseCommand)Abstract base class for command line interfaces toTimeSeriesdata files.Command line usage:Usage: timeseriesbase subcommand [...]
  Subcommands:
    fetch ...
      Fetch raw data files from the primary source to a local spool.
      To be implemented in subclasses.
    help [-l] [subcommand-names...]
      Print the full help for the named subcommands,
      or for all subcommands if no names are specified.
      -l  Long help even if no subcommand-names provided.
    import ...
      Import data into the time series.
      To be implemented in subclasses.
    info
      Report information.
    plot [-f] [-o imgpath.png] [--show] [--tz tzspec] start-time [stop-time] [{glob|fields}...]
      Plot the data from specified fields for the specified time range.
      Options:
        --bare          Strip axes and padding from the plot.
        -f              Force. -o will overwrite an existing image file.
        -o imgpath.png  File system path to which to save the plot.
        --show          Show the image in the GUI.
        --tz tzspec     Skew the UTC times presented on the graph
                        The default skew is 0 i.e. UTC.
                        to emulate the timezone specified by tzspec.
        --stacked       Stack the plot lines/areas.
        start-time      An integer number of days before the current time
                        or any datetime specification recognised by
                        dateutil.parser.parse.
        stop-time       Optional stop time, default now.
                        An integer number of days before the current time
                        or any datetime specification recognised by
                        dateutil.parser.parse.
        glob|fields     If glob is supplied, constrain the keys of
                        a TimeSeriesDataDir by the glob.
    shell
      Run a command prompt via cmd.Cmd using this command's subcommands.ClassTimeSeriesCommand(TimeSeriesBaseCommand, cs.cmdutils.BaseCommand)Command line interface toTimeSeriesdata files.Command line usage:Usage: timeseries [-s ts-step] tspath subcommand...
    -s ts-step  Specify the UNIX time step for the time series,
                used if the time series is new and checked otherwise.
    tspath      The filesystem path to the time series;
                this may refer to a single .csts TimeSeriesFile, a
                TimeSeriesPartitioned directory of such files, or
                a TimeSeriesDataDir containing partitions for
                multiple keys.
  Subcommands:
    dump
      Dump the contents of tspath.
    fetch ...
      Fetch raw data files from the primary source to a local spool.
      To be implemented in subclasses.
    help [-l] [subcommand-names...]
      Print the full help for the named subcommands,
      or for all subcommands if no names are specified.
      -l  Long help even if no subcommand-names provided.
    import csvpath datecol[:conv] [import_columns...]
      Import data into the time series.
      csvpath   The CSV file to import.
      datecol[:conv]
                Specify the timestamp column and optional
                conversion function.
                "datecol" can be either the column header name
                or a numeric column index counting from 0.
                If "conv" is omitted, the column should contain
                a UNIX seconds timestamp.  Otherwise "conv"
                should be either an identifier naming one of
                the known conversion functions or an "arrow.get"
                compatible time format string.
      import_columns
                An optional list of column names or their derived
                attribute names. The default is to import every
                numeric column except for the datecol.
    info
      Report infomation about the time series stored at tspath.
    plot [-f] [-o imgpath.png] [--show] [--tz tzspec] start-time [stop-time] [{glob|fields}...]
      Plot the data from specified fields for the specified time range.
      Options:
        --bare          Strip axes and padding from the plot.
        -f              Force. -o will overwrite an existing image file.
        -o imgpath.png  File system path to which to save the plot.
        --show          Show the image in the GUI.
        --tz tzspec     Skew the UTC times presented on the graph
                        The default skew is 0 i.e. UTC.
                        to emulate the timezone specified by tzspec.
        --stacked       Stack the plot lines/areas.
        start-time      An integer number of days before the current time
                        or any datetime specification recognised by
                        dateutil.parser.parse.
        stop-time       Optional stop time, default now.
                        An integer number of days before the current time
                        or any datetime specification recognised by
                        dateutil.parser.parse.
        glob|fields     If glob is supplied, constrain the keys of
                        a TimeSeriesDataDir by the glob.
    shell
      Run a command prompt via cmd.Cmd using this command's subcommands.
    test [testnames...]
      Run some tests of functionality.ClassTimeSeriesDataDir(TimeSeriesMapping, builtins.dict, cs.resources.MultiOpenMixin, cs.context.ContextManagerMixin, cs.fs.HasFSPath, cs.configutils.HasConfigIni, HasEpochMixin, TimeStepsMixin)A directory containing a collection ofTimeSeriesPartitionedsubdirectories.ClassTimeSeriesFile(TimeSeries, cs.resources.MultiOpenMixin, cs.context.ContextManagerMixin, HasEpochMixin, TimeStepsMixin, cs.fs.HasFSPath)A file containing a single time series for a single data field.This provides easy access to a time series data file.
The instance can be indexed by UNIX time stamp for time based access
or its.arrayproperty can be accessed for the raw data.The data file itself has a header indicating the file data big endianness,
the datum type and the time type (botharray.arraytype codes).
Following these are the start and step sizes in the time type format.
This is automatically honoured on load and save.A new file will use the native endianness, but files of other
endianness are correctly handled, making aTimeSeriesFileportable between architectures.Read only users can just instantiate an instance and access
its.arrayproperty, or use thepeekandpeek_offsetmethods.Read/write users should use the instance as a context manager,
which will automatically update the file with the array data
on exit:with TimeSeriesFile(fspath) as ts:
    ... work with ts here ...Note that the save-on-close is done withTimeSeries.flush()which only saves ifself.modified.
Use of the__setitem__orpad_tomethods set this flag automatically.
Direct access via the.arraywill not set it,
so users working that way for performance should update the flag themselves.ATimeSeriesFilehas two underlying modes of operation:
in-memoryarray.arraymode and direct-to-filemmapmode.The in-memory mode reads the whole file into anarray.arrayinstance,
and all updates then modify the in-memoryarray.
The file is saved when the context manager exits or when.save()is called.
This maximises efficiency when many accesses are done.Themmapmode maps the file into memory, and accesses operate
directly against the file contents.
This is more efficient for just a few accesses,
but every "write" access (setting a datum) will make the mmapped page dirty,
causing the OS to queue it for disc.
This mode is recommended for small accesses
such as updating a single datum, eg from polling a data source.Presently the mode used is triggered by the access method.
Using thepeekandpokemethods usesmmapby default.
Other accesses default to use the in-memory mode.
Access to the.arrayproperty forces use of thearraymode.
Poll/update operations should usually choose to usepeek/poke.MethodTimeSeriesFile.__init__(self, fspath: str, typecode: Optional[cs.timeseries.TypeCode] = None, *, epoch: Optional[cs.timeseries.Epoch] = None, fill=None, fstags: cs.fstags.FSTags):
Prepare a new time series stored in the file atfspathcontaining machine native data for the time series values.Parameters:fspath: the filename of the data filetypecodeoptional expectedarray.typecodevalue of the data;
if specified and the data file exists, they must match;
if not specified then the data file must exist
and thetypecodewill be obtained from its headerepoch: optionalEpochspecifying the start time and
step size for the time series data in the file;
if specified and the data file exists, they must match;
if not specified then the data file must exist
and theepochwill be obtained from its headerfill: optional default fill values forpad_to;
if unspecified, fill with0for'q'andfloat('nan')for'd'ClassTimeSeriesFileHeader(cs.binary.SimpleBinary, types.SimpleNamespace, cs.binary.AbstractBinary, cs.binary.BinaryMixin, HasEpochMixin, TimeStepsMixin)The binary data structure of theTimeSeriesFilefile header.This is 24 bytes long and consists of:the 4 byte magic number,b'csts'the file bigendian marker, astructbyte order indicator
with a value ofb'>'for big endian data
orb'<'for little endian datathe datum typecode,b'd'for double float
orb'q'for signed 64 bit integerthe time typecode,b'd'for double float
orb'q'for signed 64 bit integera pad byte, valueb'_'the start UNIX time, a double float or signed 64 bit integer
according to the time typecode and bigendian flagthe step size, a double float or signed 64 bit integer
according to the time typecode and bigendian flagIn addition to the header values tnd methods this also presents:datum_type: aBinarySingleStructfor the binary form of a data valuetime_type:  aBinarySingleStructfor the binary form of a time valueClassTimeSeriesMapping(builtins.dict, cs.resources.MultiOpenMixin, cs.context.ContextManagerMixin, HasEpochMixin, TimeStepsMixin)A group of namedTimeSeriesinstances, indexed by a key.This is the basis forTimeSeriesDataDir.ClassTimeSeriesPartitioned(TimeSeries, cs.resources.MultiOpenMixin, cs.context.ContextManagerMixin, HasEpochMixin, TimeStepsMixin, cs.fs.HasFSPath)A collection ofTimeSeriesfiles in a subdirectory.
We have one of these for eachTimeSeriesDataDirkey.This class manages a collection of files
named by the partition from aTimespanPolicy,
which dictates which partition holds the datum for a UNIX time.MethodTimeSeriesPartitioned.__init__(self, dirpath: str, typecode: Optional[cs.timeseries.TypeCode] = None, *, epoch: Optional[cs.timeseries.Epoch] = None, policy, fstags: cs.fstags.FSTags):
Initialise theTimeSeriesPartitionedinstance.Parameters:dirpath: the directory filesystem path,
known as.fspathwithin the instancetypecode: thearraytype code for the dataepoch: the time seriesEpochpolicy: the partitioningTimespanPolicyThe instance requires a reference epoch
because thepolicystart times will almost always
not fall on exact multiples ofepoch.step.
The reference allows for reliable placement of times
which fall withinepoch.stepof a partition boundary.
For example, ifepoch.start==0andepoch.step==6and a
partition boundary came at19due to some calendar based
policy then a time of20would fall in the partion left
of the boundary because it belongs to the time slot commencing
at18.Ifepochortypecodeare omitted the file's
fstags will be consulted for their values.
Thestartparameter will further fall back to0.
This class does not set these tags (that would presume write
access to the parent directory or its.fstagsfile)
when aTimeSeriesPartitionedis made by aTimeSeriesDataDirinstance it sets these flags.ClassTimespanPolicy(icontract._metaclass.DBC, HasEpochMixin, TimeStepsMixin, cs.deco.Promotable)A class implementing a policy allocating times to named time spans.TheTimeSeriesPartitioneduses these policies
to partition data among multipleTimeSeriesdata files.Probably the most important methods are:span_for_time: return aTimePartitionfrom a UNIX timespan_for_name: return aTimePartitionfrom a partition nameMethodTimespanPolicy.__init__(self, epoch: cs.timeseries.Epoch):
Initialise the policy.ClassTimespanPolicyAnnual(ArrowBasedTimespanPolicy, TimespanPolicy, icontract._metaclass.DBC, HasEpochMixin, TimeStepsMixin, cs.deco.Promotable)A annual time policy.
PARTITION_FORMAT = 'YYYY'
ARROW_SHIFT_PARAMS = {'years': 1}ClassTimespanPolicyDaily(ArrowBasedTimespanPolicy, TimespanPolicy, icontract._metaclass.DBC, HasEpochMixin, TimeStepsMixin, cs.deco.Promotable)A daily time policy.
PARTITION_FORMAT = 'YYYY-MM-DD'
ARROW_SHIFT_PARAMS = {'days': 1}ClassTimespanPolicyMonthly(ArrowBasedTimespanPolicy, TimespanPolicy, icontract._metaclass.DBC, HasEpochMixin, TimeStepsMixin, cs.deco.Promotable)A monthly time policy.
PARTITION_FORMAT = 'YYYY-MM'
ARROW_SHIFT_PARAMS = {'months': 1}ClassTimespanPolicyWeekly(ArrowBasedTimespanPolicy, TimespanPolicy, icontract._metaclass.DBC, HasEpochMixin, TimeStepsMixin, cs.deco.Promotable)A weekly time policy.
PARTITION_FORMAT = 'W'
ARROW_SHIFT_PARAMS = {'weeks': 1}ClassTimespanPolicyYearly(ArrowBasedTimespanPolicy, TimespanPolicy, icontract._metaclass.DBC, HasEpochMixin, TimeStepsMixin, cs.deco.Promotable)A annual time policy.
PARTITION_FORMAT = 'YYYY'
ARROW_SHIFT_PARAMS = {'years': 1}ClassTimeStepsMixinMethods for an object withstartandstepattributes.ClassTypeCode(builtins.str, cs.deco.Promotable)A validarraytypecode with convenience methods.MethodTypeCode.__new__(cls, t):
Return a newTypeCodeinstance fromt, which may be:astr: expected to be anarraytype codeint:arraytype codeq(signed 64 bit)float:arraytype coded(double float)Functiontzfor(tzspec: Union[str, datetime.tzinfo, NoneType] = None) -> datetime.tzinfoPromote the timezone specificationtzspecto atzinfoinstance.
Iftzspecis an instance oftzinfoit is returned unchanged.
Iftzspecis omitted or the string'local'this returnsdateutil.tz.gettz(), the local system timezone.
Otherwise it returnsdateutil.tz.gettz(tzspec).Release LogRelease 20240201:
Release with "csts" script.Release 20230612:Epoch.promote: do not special case None, let Optional[Epoch] type annoations handle that.Mark PlotSeries.promote as incomplete (raises RuntimeError).TimespanPolicy.promote: use cls.from_name() instead of TimespanPolicy.from_name().Assorted other small updates.Release 20230217:TimeSeriesFile.save_to: use atomic_filename() to create the updated file.Other small fixes and updates.Release 20220918:TimeSeriesMapping.as_pd_dataframe: renamekeystodf_data, and accept either a time series key or a(key,series)tuple.TimeSeriesMapping.as_pd_dataframe: defaultkey_map: annotate columns with their original CSV headers if present.TimeSeriesMapping.plot: renamekeystoplot_dataas foras_pd_dataframe, addstackedandkindparameters so that we can derivekindfromstacked.as_datetime64s: apply optional utcoffset timeshift.Plumb optional pad=False option through data, data2, as_pd_series.New PlotSeries namedtuple holding a label, a series and an extra dict as common carrier for data which will get plotted.Release 20220805:Rename @plotrange to @timerange since it is not inherently associated with plotting, support both methods and functions.print_figure, save_figure and saved_figure now moved to cs.mplutils.plot_events: use the utcoffset parameter.TimeSeriesBaseCommand.cmd_plot: new --bare option for unadorned plots.Release 20220626:New TypeCode(str) representing an array type code with associated properties and methods.New TimeSeriesMapping.read_csv wrapper for pandas.read_csv to import a CSV file into a TimeSeriesMapping.TimeSeriesFile.save,save_to: open the file for overwrite, not truncate, by default.TimeSeriesFile: new setitems(whens,values) method for fast batch updates.as_datetime64s: accept optional units parameter to trade off range versus precision.@plotrange: accept new optional tz/utcoffset parameters and pass the resulting utcoffset to the wrapped function along with a huge disclaimer about timezones and plots.New tzfor(tzspec) to return a tzinfo object from dateutil.tz.gettz, accepts 'local' for the system local default timezone.TimeSeriesMapping.as_pd_dataframe: accept optional utcoffset to skew the index for the DataFrame, used for time presentation in plots.New TimeSeriesMapping.to_csv(start,stop,f) method to write CSV data between start and stop to a file via DataFrame.to_csv.TimeSeriesBaseCommand: new parsetime and poptime methods, cmd_plot: update to expect start-time and optional stop-time.Release 20220606:
Initial PyPI release.