Package: pf-dre-database-client
Meter Management System ClientThe python implementation of the pf-dre-database repo provides a client
for all Data interactions required with the Meter Management System:Relational tables (Read Only)Timescale DB (Read/Write - No insertion or deletion)JSON SchemaNarrow Data Format SchemaThis python implementation is to be built and deployed to PyPI for use
across all python subsystems of the Demand Response Engine.Real-Time Input Data FormatWhen issuing calls to the MMS which require a time series based DataFrame to be
passed the format of the schema should be followed with the following general
rules.Timestamps are to be generated in string format following the ISO 8601
standard and to follow simple conventions should be kept in UTC format.Any columns within the data structure which are a JSON datatype are to be
created in serialized string format, not as a pure python dictionary.importjson# Correct Formatcorrect_jsonb_value=json.dumps({'A':'Dictionary','B':'to','C':'Send'})# Incorrect Formatincorrect_jsonb_value={'A':'Dictionary','B':'to','C':'Send'}Example Data Frame for a narrow column format schemameasurement_datedevice_iddevice_metric_type_idvalue2020-01-01T12:00:00.0001P1.02020-01-01T12:01:00.0001P2.02020-01-01T12:00:00.0001Q-1.02020-01-01T12:01:00.0001Q-2.02020-01-01T12:00:00.0002P10.02020-01-01T12:01:00.0002P20.02020-01-01T12:00:00.0002Q-10.02020-01-01T12:01:00.0002Q-20.0object (str)int64object (str)float64Example Data Frame for a JSON schemameasurement_datedevice_idmetrics2020-01-01T12:00:00.0001{"P": 1.0, "Q": -1.0, "S": 'NaN'}2020-01-01T12:00:00.0002{"P": 2.0, "Q": -2.0}2020-01-01T12:01:00.0001{"P": 10.0, "Q": -10.0}2020-01-01T12:01:00.0002{"P": 20.0, "Q": -20.0}object (str)int64object (str)Real-Time Standardized Output DataFrame FormatWhen issuing calls to the MMS which return a time series DataFrame, the client,
regardless of schema will be constructed to return in a standardized format.
This makes the reading and manipulation of data consistent.device_iddevice_metric_type_idmeasurement_datevalue1P2020-01-01T12:00:00.0001001.02020-01-01T12:01:00.0001012.0Q2020-01-01T12:00:00.00012.1322020-01-01T12:01:00.000-2.1322P2020-01-01T12:00:00.0002001.02020-01-01T12:01:00.0002012.0Q2020-01-01T12:00:00.00022.1322020-01-01T12:01:00.000-3.132int64object (str)object (str)float64The client also has the option of returing the data frame results in a raw,
un-standardised format. In this case, the dataframe will be returned in the
format of the underlying database schema without any alteration.Forecast Input Data FormatWhen issuing calls to the MMS which require a forecast time series based
DataFrame to be passed, the format of the schema should be followed with the
following general rules.Timestamps are to be generated in string format following the ISO 8601
standard and to follow simple conventions should be kept in UTC format.Any columns within the data structure which are a JSON datatype are to be
created in serialized string format, not as a pure python dictionary.Example Data Frame for a narrow column format schemareceived_datedevice_iddevice_metric_type_idmeasurement_datevalue2020-01-01T12:00:00.0001P2020-01-01T12:00:00.0001.02020-01-01T12:00:00.0001P2020-01-01T12:01:00.0002.02020-01-01T12:00:00.0001P2020-01-01T12:02:00.0003.02020-01-01T12:00:00.0001Q2020-01-01T12:00:00.000-1.02020-01-01T12:00:00.0001Q2020-01-01T12:01:00.000-2.02020-01-01T12:00:00.0001Q2020-01-01T12:02:00.000-3.02020-01-01T12:01:00.0001P2020-01-01T12:01:00.0002.02020-01-01T12:01:00.0001P2020-01-01T12:02:00.0003.02020-01-01T12:01:00.0001P2020-01-01T12:03:00.0004.02020-01-01T12:01:00.0001Q2020-01-01T12:01:00.000-2.02020-01-01T12:01:00.0001Q2020-01-01T12:02:00.000-3.02020-01-01T12:01:00.0001Q2020-01-01T12:03:00.000-4.0object (str)int64object (str)objectfloat64Example Forecast Data Frame for a JSON schemareceived_datedevice_idmetrics2020-01-01T12:00:00.0001{ "P": {"2020-01-01T12:00:00+00:00": 1.0, "2020-01-01T12:01:00+00:00": 2.0, "2020-01-01T12:02:00+00:00": 3.0}, "Q": {"2020-01-01T12:00:00+00:00": -1.0, "2020-01-01T12:01:00+00:00": -2.0, "2020-01-01T12:02:00+00:00": -3.0, "2020-01-01T12:03:00+00:00": 'NaN'}}2020-01-01T12:01:00.0001{ "P": {"2020-01-01T12:01:00+00:00": 2.0, "2020-01-01T12:02:00+00:00": 3.0, "2020-01-01T12:03:00+00:00": 4.0}, "Q": {"2020-01-01T12:01:00+00:00": -2.0, "2020-01-01T12:02:00+00:00": -3.0, "2020-01-01T12:03:00+00:00": -4.0}}object (str)int64objectForecast Standardized Output DataFrame FormatWhen issuing calls to the MMS which return a time series DataFrame, the client,
regardless of schema will be constructed to return in a standardized format.
This makes the reading and manipulation of data consistent.received_datedevice_iddevice_metric_type_idmeasurement_datevalue2020-01-01T12:00:00.0001P2020-01-01T12:00:00.0001.02020-01-01T12:01:00.0002.02020-01-01T12:02:00.0003.0Q2020-01-01T12:00:00.000-1.02020-01-01T12:01:00.000-2.02020-01-01T12:02:00.000-3.02020-01-01T12:01:00.0001P2020-01-01T12:01:00.0002.02020-01-01T12:02:00.0003.02020-01-01T12:03:00.0004.0Q2020-01-01T12:01:00.000-2.02020-01-01T12:02:00.000-3.02020-01-01T12:03:00.000-4.0object (str)int64object (str)object (str)float64The client also has the option of returning the data frame results in a raw,
un-standardised format by initializing the client with the argument:standardized=FalseIn this case, the data frame will be returned in the
format of the underlying database schema without any alteration.Accessing Standardized DataFrame ContentsReal-time DataFramespython # Accessing Device Metric Timeseries time_series = df.loc[(1, 'P'), 'value'] time_series_times = df.loc[(1, 'P'), 'value'].index.values.tolist()) time_series_values = df.loc[(1, 'P'), 'value'].values.tolist()) # Accessing Most Recent Device Metric  # (Note the standardized format is sorted in ascending order) latest_P = df.loc[(1, 'P'), 'value'].iloc[0]Forecast DataFramesPrerequisitesPython 3.7.0+SetupThe following environment variables are required in order to make use
of the client.PGDATABASE: The name of the MMS Database instance.PGUSER: MMS Database user.PGPASSWORD: MMS Database password.PGHOST: MMS Database host.PGPORT: MMS Database port (read/write permissions required).