Package: historian-query
Historian QueryThis Python library helps query regularized time series based on raw historian data with
irregular time intervals (due to compression, deadband, outages etc.) on Spark. It is particularly
suited for compressed manufacturing time series coming from a historian database, and it imitates the
resampling functionality as offered by some historians.Installationhistorian-querycan be installed from pip using the following command:pip install historian-queryUseful LinksHistorian Query referenceDatabricks Tempo documentationExample usageAn input table / dataframe with columns["tag_name", "ts", "value_double", "quality"]is required.
This is then loaded and resampled as follows:fromhistorian_queryimportHistorianQuerytable_name="ot_database.historian_timeseries"tag_dict={"PR03.229HIC0384_SPEED_PV":"belt_speed","PR03.730KIC4985_FLOW_PV":"water_flow","PR03.849ZMC3525_PRESS_PV":"pressure",}HQ=HistorianQuery(table_name,tag_list=list(tag_dict.keys()),sample_freq="10 seconds",ff_timeout="15 minutes",keep_quality=3,)df=HQ.resample("2023-08-01 00:00:00","2023-08-02 00:00:00")Note that instead oftable_nameandtag_listyou can alternatively pass a Spark dataframedf,
which gives more flexibility for your data source (see thereference).
The output is a resampled Spark dataframe in long format:tag_nametsvalue_doublequalityorig_tsPR03.229HIC0384_SPEED_PV2023-08-01 00:00:00.0003564.92041032023-07-31 23:59:53.000PR03.730KIC4985_FLOW_PV2023-08-01 00:00:00.00053.21819632023-07-31 23:59:58.000PR03.849ZMC3525_PRESS_PV2023-08-01 00:00:00.0007.48543232023-07-31 23:59:58.000PR03.229HIC0384_SPEED_PV2023-08-01 00:00:10.0003565.10400432023-08-01 00:00:03.000PR03.730KIC4985_FLOW_PV2023-08-01 00:00:10.00053.21819632023-07-31 23:59:58.000PR03.849ZMC3525_PRESS_PV2023-08-01 00:00:10.0008.02875532023-08-01 00:00:08.000PR03.229HIC0384_SPEED_PV2023-08-01 00:00:20.0003565.10400432023-08-01 00:00:03.000PR03.730KIC4985_FLOW_PV2023-08-01 00:00:20.00053.21819632023-07-31 23:59:58.000PR03.849ZMC3525_PRESS_PV2023-08-01 00:00:20.0007.37116932023-08-01 00:00:18.000...Each tag now has observations every 10 seconds. Note that the original time stampsorig_tsare
slightly earlier, and some have been filled forward several times (no newer observation is
available).We can now use standard Spark tools for aggregation and feature engineering, e.g. average over 5
minute windows:fromitertoolsimportchainfrompyspark.sql.functionsimportwindow,avg,create_map,lit,col# aggregate over 5 minute windowsdf=df.groupBy("tag_name",window("ts","5 minutes").end.alias("ts")).agg(avg("value_double").alias("avg_val"))# map tag namestag_map=create_map([lit(x)forxinchain(*tag_dict.items())])df=df.withColumn("tag_descr",tag_map[col("tag_name")])# pivot to get tags as columnsdf.toPandas().pivot(index="ts",columns="tag_descr",values="avg_val")The output is now in a suitable format for using in reports or ML models:tsbelt_speedpressurewater_flow2023-08-01 00:05:003564.9501227.77432653.0105702023-08-01 00:10:003565.0531587.81265653.0429732023-08-01 00:15:003564.6262457.86969252.9816312023-08-01 00:20:003564.9342697.82822852.985406...Note that we use the end-point timestamp for each aggregation interval, so that each record is only
based on data that is available at the specified timestamp.In summary, we recommend a two-step approach:resample to a fine time granularity,aggregate to the desired time windows.Step 1 ensures that the potentially irregular observations from raw historian data do not bias
the aggregation (more weight on volatile periods) or lead to skipped/null intervals (due to gaps).How this worksTheHistorianQueryclass is a wrapper around theTSDFclass in the DatabricksTempolibrary. It adds some functionality
that is helpful for querying data from a historian in manufacturing context:Returns records from start to end time (instead of just between first and last record for each
tag).Interval timestamp is rounded up instead of down, reporting the last known value at the given
time point.The original timestamp of the observation is also reported.Timeout functionality - forward fill only up to a specified time interval to avoid stale values.Filter by quality flag - historians often capture an indication of how reliable each observation
is. This is to remind users to keep only good quality observations.Keep or ignore nulls - a null value recorded by a historian can be an indication that the last
known value should no longer be forward filled.While Tempo offers other options, here the resampling configuration has been fixed to the following :Interpolation:ffill- forwards fillAggregation:ceil- returns the latest value by timestamp.Forward filling (along with rounding up interval timestamps) ensures that observations are only
based on past data. This is important for e.g. Machine Learning use-cases, where the training
samples must contain only information that is available at inference time during production.Taking the last known value instead ofmeanensures that the value makes sense for any tag, e.g.
also for 0/1 indicators. For custom aggregation, follow the approach described in sectionExample Usage.