Package: pyaware
pyAWAREPython middleware for AWARE Gateway.
This is to be installed on an edge device for reading data from non-iot devices, aggregating the data and then publishes
the data to either the cloud or to a local network via MQTT.There are sections of this readme which will diverge between legacy and current implementation. See below devices that
require our legacy steps vs the current steps,CurrentMOXA UC-2100MOXA V2403LegacySiemens IOT2040ContentsInstallationBuilding new releasesField Production InstallationProduction Image InstallationDev InstallationOnline InstallationOffline InstallationService InstallationUsageFolder StructureGateway Config (gateway.yaml)MQTT Connection Config (connection.yaml)Translation Config (translation.yaml)Deploying GatewayCreating new devicesSourcesTransformationsParametersInstallationInstall is recommended on a virtual environment on python 3.7 or above.pip install virtualenvvirtualenv /path/to/new/env -p /path/to/pythonActivate the environment to do commands such as pip/path/to/new/env/Scripts/activateor/path/to/ENV/bin/activateOr run the python directly/path/to/new/env/Scripts/pythonor/path/to/ENV/bin/pythonRunning the test suiteTo run the test suite (with an activated venv)pip install -e .
pip install -r requirements.txt
pip install -r requirements-test.txt
pytestIt should take ~30 seconds for all tests to run.Building new releasesIn order to do a new release follow the following checklist:Create a release branchSet the current working directory to the top level folder of pyAWAREUpdate ReleaseNotes.md to include the major changes in this revision (review git logs)Update__version__string in__init__.pyin the top level folderRunpythonsetup.pybdist_wheelThe output .whl folder should be in the dist/* folder (empty this folder before running as there may be issues with
other whl files with the same name in this directory)Open it with 7zip or similar and check that the python files, non-python files and scripts are included as expected.Test the features and fixes that have been introduced in this revision. With this test the core functionality of
pyAWARE i.e. modbus comms, MQTT comms, cloud comms etc.Create, approve and merge into both the 'develop' branch and the 'production' branchField Production InstallationNOTE: If unit doesn't have access to the internet, follow the offline installation instructions for getting the
dependencies installed.Current InstallFor rolling out updates. If it is a pure python release update, copy the wheel file via SCP to /usr/share/aware
then usepip install pyaware-<version>-py3-non-any.whlLegacy InstallFor rolling out updates. If it is a pure python release update, copy the wheel file via SCP to /etc/AWARE
then usepip install pyaware-<version>-py3-non-any.whlProduction Image InstallationRetrieve the current system image fromSubversionfor your device.CurrentInstall the image onto a USB Drive.Enter uBoot and install the image to the IPC.Update the pyAWARE install and dependencies.LegacyBurn this to an SD card to be installed onto your device.Install the SD card into your device and perform an initial boot.Update the pyAWARE install and dependencies.Dev InstallationHave python 3.7.X installedCheck out the pyAWARE projectOpen terminal or cmd prompt and navigate to the checked out projectpip install -e .pip install -r requirements.txtThis will connect to the internet and download the necessary packagesOnline InstallationHave python 3.7.X installedCheck out the pyaware projectOpen terminal or cmd prompt and navigate to the checked out projectpip install .pip install -r requirements.txtThis will connect to the internet and download the necessary packagesOffline InstallationInstalling can take a long time on embedded devices such as the IOT 2040
To speed up this process do the following,Have python 3.7 or above installedCheck out the pyAWARE projectOpen terminal or cmd prompt and navigate to the checked out projectDownload the required packages by doing the following (Note: The platform and index will change based on the architecture )
For the moxa we are using the piwheels cache which has prebuilt wheels for the arm architecture.
On a desktop (tested on python 3.9) we need to get other packages to support the install that pip doesn't download on the other architecture.pip download pyaware wheel pip importlib_metadata ruamel.yaml.clib zipp --extra-index-url https://www.piwheels.org/simple --platform linux_armv7l --python-version 3.7 --only-binary=:all: -d pkgsOn moxa device
Activate a python 3.7 environment. This is required as hbmqtt does not have a distribution release for 3.7 so specifying 3.7 doesn't work.
By using the natural 3.7 environment we get the 3.4-3.5 package which is still compatible.pip download pyaware pip wheel -d pkgs --extra-index-url https://www.piwheels.org/simple --only-binary=:all:Copy the pkgs directory to the target deviceActivate the virtual environmentInstall the packagespip install --upgrade pip wheel --no-index --find-links="pkgs" && pip install --upgrade pyaware --no-index --find-links="pkgs"Download and install only source files and platform agnostic files.
Do this on the device so that it will not download x86 binaries.pip download setuptools wheel pip pyaware cython -d pkgs --no-cacheOffline install from sourcepip install --upgrade pip wheel --no-index --find-links="pkgs" && pip install --upgrade pyaware --no-index --find-links="pkgs"Service InstallationCurrentNOTE: This only is to be performed if pyaware.service is not found at this location -> /etc/systemd/systemCreate a service file calledpyaware.servicewhich contains the below,[unit]
Description=pyaware
After=network.target

[Service]
Type=simple
Restart=always
RestartSec=1
ExecStart=/usr/share/aware/venv/bin/python -m pyaware /etc/aware/

[Install]WantedBy=multi-user.targetPut this file in/etc/systemd/systemThis service should start on boot and will restart if it errors out. To explicitly start/stop/restart this service use
the commands below,sudo systemctl [cmd] pyawarewhere cmd can be -> status, start, stop, restart etc.LegacyNOTE: This only is to be performed if inittab does not contain the lines below.Append to the file at/etc/inittabthe following lines#StarttheawareprocessAWAR:2345:respawn:/usr/bin/env AWAREPATH=/etc/AWARE python3 /etc/AWARE/main.pyExample Configuration:pyAWARE has example configuration files available to use through deployment. In the event of an installation of pyAWARE,
pyAWARE will place the configuration files at the base of your python deployment in a folder calledpyaware_example_config.UsagepyAWARE uses config files to take inputs into the program. These are read in on pyAWARE boot and will require a restart
to the program in order for any changes to occur. The config files are written in yaml and are required to be in a
specific folder structure. All config files should be placed in /etc/AWARE (on Linux).Folder StructureA generic folder structure is shown below that details the path each config file should be found in. Starting at the
AWAREPATH i.e./etc/AWARE(on Linux)..├── config|   ├── connection.yaml|   ├── gateway.yaml|   └── translation.yaml└── credentials├── google_roots.pem└── rsa_private.pemGateway Config (gateway.yaml)Gateway config is the heart of pyAWARE and is the config file responsible for using pyAWARE's core features. An example
gateway.yaml file is shown below,aware_version:manualscheduled_restart:hour:14database:max_size:500MBmemory:true#  relative_directory: db#  absolute_directory: "C:\dev\Ampcontrol IOT\Device Management\Deployments\tempdb"backlog_enable:truecommunication:-name:...type:...version:1params:...devices:[...]Optionsaware_version- Specifies what pyAWARE version to use either put in the exact version required or one of the
options below. Defaults tolatest. NOTE: To update the system requires internet access.
-manual- pyAWARE will not attempt to update.
-latest- pyAWARE will attempt to update to the latest version.scheduled_restart- Specifies if pyAWARE should restart after a period of time. Defaults to never restart.hour- Integer specifying the time in hours when pyAWARE should restart after starting up.database- Specifies a database to backup unsent MQTT data. Defaults to initialising a 500MB database using in
memory SQLite. Will only use one of the memory options below with the following prioritymemory->absolute_directory->relative_directory.max_size- Maximum size of the database in the formatXXXMB.memory- Specifies pyAWARE to use in memory SQLite for intialising the database. Value is to betrueorfalseabsolute_directory- Specifies the absolute directory.relative_directory- Specifies the relative directory. Identifies the folder structure from the AWAREPATH.backlog_enable- Specifies whether or not to use the backlog to retain MQTT messages if the broker can not be
reached. This defaults tofalse. Value is to betrueorfalse.communication- Specifies the communication options required in the current deployment.
-name- Name of the communication. This is a config only reference to be used throughout the config file.
-type- Type of pyAWARE supported communication.
-version- Version of the communication method to be used.
-params- Parameters to be passed to the communication. These are communication specific and are detailed inCommunication Parameters.devices- Specifies the devices that are currently in use by communication reference name. This is written in
config as an array of names. This is currently only required by the cloud MQTT.Communication TypesCommunication types describe servers, clients and devices that make up the core of pyAWARE. The types of communication
and their configparameterswill be specified here.modbus_rtu- Pymodbus' Modbus RTU Serial Client.Required:port- Name of the serial port to use. i.e. COM# (for windows) OR /dev/ttyS# (for linux), where # is the number
of the port.stopbits- Number of stop bits.parity- Type of parity. e.g. 'N' for None, 'E' for Even Or 'O' for Odd.baudrate- The baudrate of the communication.Optional:bytesize- The number of bits in a byte of serial data (defaults to 8).timeout- The timeout for loss of communication in seconds (defaults to 3 s).modbus_rtu2- AioModbus' Modbus RTU Serial Client.Required:port- Name of the serial port to use. i.e. COM# (for windows) OR /dev/ttyS# (for linux), where # is the number
of the port.stopbits- Number of stop bits.parity- Type of parity. e.g. 'N' for None, 'E' for Even Or 'O' for Odd.baudrate- The baudrate of the communication.Optional:bytesize- The number of bits in a byte of serial data (defaults to 8).default_unit_idThe modbus unit id to use if there is no unit id present on the child device to this client
(defaults to 0).modbus_tcp- Pymodbus' Modbus TCP Client.Required:host- Hostname / IP address of the client.Optional:port- Port to set up the client connection (defaults to 502).client_port- Port of the connected device (defaults to 0 or listen on all ports).modbus_tcp2- AioModbus' Modbus TCP Client.Required:host- Hostname / IP address of the client.Optional:port- Port to set up the client connection (defaults to 502).client_port- Port of the connected device (defaults to 0 or listen on all ports).auto_reconnect_after- Time to automatically reconnect to the client on a disconnect event in seconds
(defaults to None i.e. does not try to reconnect).default_timeout- Timeout from a client to disconnect in seconds (defaults to 0.2).max_active_request- Maximum number of requests on which to disconnect (defaults to None).default_unit_id- The modbus unit id to use if there is no unit id present on the child device to this client
(defaults to 0).modbus_tcp_server- Pymodbus' Modbus TCP Server.Required:host- Modbus TCP Server Hostname / IP Address.server_id- ID of the server as a string. This is to be used to map devices into the server.Optional:port- Port to set up the server connection (defaults to 502).coil_register_blocks- The contiguous blocks of coil registers to intialise the server as arrays of ints written
in the form [start, end], (defaults to None i.e. initialises 0-65535 with zeros).discrete_register_blocks- The contiguous blocks of discrete registers to intialise the server as arrays of ints
written in the form [start, end], (defaults to None i.e. initialises 0-65535 with zeros).holding_register_blocks- The contiguous blocks of holding registers to intialise the server as arrays of ints
written in the form [start, end], (defaults to None i.e. initialises 0-65535 with zeros).input_register_blocks- The contiguous blocks of input registers to intialise the server as arrays of ints
written in the form [start, end], (defaults to None i.e. initialises 0-65535 with zeros).sp_pro- SP PRO Serial Client using AioSPPRO. Communicates with Selectronic devices that speak the SPPRO Protocol.Required:port- Name of the serial port to use. i.e. COM# (for windows) OR /dev/ttyS# (for linux), where # is the number
of the port.stopbits- Number of stop bits.parity- Type of parity. e.g. 'N' for None, 'E' for Even Or 'O' for Odd.baudrate- The baudrate of the communication.Optional:bytesize- The number of bits in a byte of serial data (defaults to 8).default_timeout- The timeout for loss of communication in seconds (defaults to 0.2 s).password- The SP PRO password to use to log into the unit (defaults to Selectronic SP PRO).auto_reconnect_after- Time to automatically reconnect to the client on a disconnect event in seconds
(defaults to None i.e. does not try to reconnect).retry_failed_login- Flag to retry a failed login attempt on the SP PRO (defaults to True).login_attempts- The number of attempts to try and login to the SP PRO (defaults to 3).imac2_auto_detect- Establishes the IMAC 2 Protocol and begins the auto detection process. Also performs topology
functionality.Required:client_ser- Modbus RTU Serial Client that the IMAC 2 is using to make connection.client_eth- Modbus TCP Client that the IMAC 2 is using to make connection.Optional:unit- Modbus unit number of the iMAC (defaults to 1).translator- The translator controller. This translates data from / to devices from / to servers. Uses a separatetranslation config fileto determine mappings.Required:config- Translation configuration file.devices- List of device ID's to be used.servers- List of server ID's to be used.modbus_device- A generic modbus device definition.Required:client- Modbus Client as defined in previous communication. i.e. Either modbus_rtu. modbus_rtu2, modbus_tcp
modbus_tcp2.device_id- Device Identifier to use throughout the program. This has to be the same name as the cloud id for
cloud implementation.config- Device Config file. This can either be a file defined in pyAWARE (located in src/pyaware/devices) or a
self defined config file.Optional:unit- Modbus Unit ID (defaults to 0).address_shift- Integer value to shift all device parameter addresses from the ones defined in config
(defaults to 0).poll_intervals- Sets the interval of the poll in seconds to read data from the device. The device config file
may have different polls, so the interval has to be defined as poll specific in the form {"poll1": 1} (defaults to
None).comap_device- A ComAP device that is inherited from Modbus device and has all the parameters defined within that
along with the additional functionality to execute ComAp device.Required:Refer to required parameters of modbus_device.Optional:Refer to optional parameters of modbus_device.sp_pro_device- A Selectronic, SP PRO device.Required:client- SP PRO serial client as defined in previous communication i.e. sp_prodevice_id- Device Identifier to use throughout the program. This has to be the same name as the cloud id for
cloud implementation.config- Device Config file. This can either be a file defined in pyAWARE (located in src/pyaware/devices) or
a self defined config file.Optional:poll_intervals- Sets the interval of the poll in seconds to read data from the device. The device config file
may have different polls, so the interval has to be defined as poll specific in the form {"poll1": 1} (defaults to
None).solar_cube- Solar Cube deployment. Establishes internal heartbeat and internal mqtt watchdogs for cloud mqtt.Required:device_id- Device Identifier to use throughout the program. This has to be the same name as the cloud id for
cloud implementation.Optional:cloud_broker- Instance of the cloud MQTT broker for the watchdogs to monitor.Communication ParametersEach communication type has a list of parameters. These parameters are specific to the communication option chosen
(e.g. A device may need a device_id as an input and will error until you define this in config).
Each parameter has a type and value. Parameter types tell PyAWARE how to interpret the parameter value and the
parameter value is what you would like to set as that parameter. As you will see this can be either a specified value,
function or reference. Example config can be seen below detailing how to write this in config,communication:-name:ref_comms_name# Comms protocol e.g. modbus or similar...-name:ref_dev_name# Device defintion e.g. power-meter...-name:...type:...# This is an example communication option which makes use of each different communication parameter typeparams:param1:type:valuevalue:"Thisismyparametervalue"param2:type:ref_commsvalue:ref_comms_name# This is the reference name of a comms protocol in the config fileparam3:type:ref_pathvalue:ref/path/from/AWAREPATH# This is the reference path from AWAREPATHparam4:type:ref_devicevalue:ref_dev_name# This is the reference name of a device in the config fileparam5:type:ref_comms_paramvalue:ref_comms_namekey:ref_comms_param_name# Parameter name from the reference device objectparam6:type:ref_translation_configvalue:translation/location# Location of the translation.yaml file from AWAREPATHparam7:type:ref_translationvalue:[...]# List of reference config names to pass through to translationparam8:type:ref_mac_addressvalue:ethx# Adapter name to get MAC Address fromvalue- The most common option. This is just a value that is interpreted raw. It can be a string, number, etc.ref_comms- Reference to a communication protocol elsewhere in the config file. e.g. If you have a modbus TCP
client defined with the name 'modbus_client' this parameter will take the object instantiated during that process as
the reference parameter.ref_path- This parameter is interpreted as a path from the AWAREPATH. e.g. If you defined a path 'location/here'
the path this parameter becomes is AWAREPATH/location/here.ref_device- Reference to a device definition elsewhere in the config file. e.g. If you have a modbus device
defined with the name 'modbus_device_1' this parameter will take the object instantiated during that process as the
reference parameter.ref_comms_param- Reference to a parameter belonging to a communication protocol elsewhere in the config file.
e.g. If you have a modbus TCP client defined with the name 'modbus_client' and it has a parameter called 'client_id'
this parameter will take the object instantiated during that process as the reference object and retrieve the
parameter with the name defined inkeywhich in this case lets say 'client_id' and store it as that parameter on the
current item.ref_translation_config- Reference to the location of the translation config file (translation.yaml). e.g. If
you defined a path 'location/here' the path this parameter becomes is AWAREPATH/config/location/here.ref_translation- List of object references in the current config file to put into a parameter.ref_mac_address- The adapter name that you wish to retrieve the parameter mac address from. This has to be the
name your platform has given the adapter and has to be a valid adapter on the machine.MQTT Connection Config (connection.yaml)Connection.yaml specifies the all the requirements for connecting to an appropriate mqtt broker. This is separated from
gateway.yaml in order to provide robustness to failure when a user uploads an invalid configuration,
pyaware will still connect in order to receive a new configuration.
An example connection.yaml file is shown
below,defaults:'gcp'# feature_flags:#   feature_mqtt: truedevice-id:'gateway-id'project_id:'project-id'registry_id:'registry-id'cloud_region:'gcp-region'Optionsdefaults- Chooses which base configuration defaults to usegcp- Use the Google Cloud Defaultslocal- Use the local on-premises mqtt broker defaultfeature_flags- Set of core pyAWARE features to enable or disable.feature_mqtt- Enables or disables the use of an mqtt broker. Defaults totrue.All options can be provided in the dynamic form as can be specifiedCommunication Parameters..
Note: Any parameters that rely on information retrieved from devices in gateway.yaml will delay the final connection to
the broker until all of gateway.yaml is processed.
Any invalid gateway.yaml will then cause connection.yaml to not be processed and pyaware never connecting to the broker.Local ConfigurationsRequireddevice_id- Device ID of the device that is required to be connected to.Optionalgateway_id- Gateway ID to publish / subscribe to via MQTT (defaults to "").serial_number- Serial Number to publish / subscribe to via MQTT (defaults to "").authentication_required- Determines whether authentication is required to connect to the Local Broker (defaults
to False).host- Hostname or IP address of the local MQTT broker (defaults to 127.0.0.1).port- The port number of the Local MQTT broker (defaults to 1883).keepalive- The number of seconds to keep the MQTT connection alive (defaults to 60).bind_address- The IP address of a local network interface to bind this client to, assuming multiple interfaces
exist (defaults to "").clean_session- A boolean that determines the client type. If True, the broker will remove all information about
this client when it disconnects. If False, the client is a durable client and subscription information and queued
messages will be retained when the client disconnects (defaults to False).parsers- A dictionary containing additional MQTT parsers for the broker. A full list of the current parsers are
found in src\pyaware\mqtt\config as default_local_parsers() (defaults to the list of local parsers). e.g.
{"topic": "my/MQTT/topic/{variable_name}/here"}disable_parsers- A list containing the keys representing the MQTT parsers to remove on the broker. A full list
of the current parsers are found in src\pyaware\mqtt\config as default_local_parsers().subscribe_qos- Topic subscription qos (defaults to 1).publish_qos- Topic publish qos (defaults to 1).token_life- Life of an MQTT token in seconds (defaults to 0).batch- Batch MQTT messages enable (defaults to True).batch_hold_off- Time window to batch MQTT messages in seconds (defaults to 5).max_message_queue_size- Maximum number of messages to have in the queue (defaults to 100).max_in_flight_messages- Maximum number of in-flight MQTT messages (defaults to 100).backlog_enable- Enable to backlog mqtt messages if not sent to MQTT broker (defaults to false).parsers_default- Select the topic parser defaults that can be overridden by parsers. (defaults to 'local')GCP ConfigurationsRequireddevice_id- Device ID of the device as configured in GCP IOT CoreOptionalproject_id- Project ID on google cloud platform (defaults to aware-iot).registry_id- Registry ID on google cloud platform (defaults to aware-iot).token_life- Life of an MQTT token in seconds (defaults to 60).cloud_region- Cloud region on google cloud platform (defaults to asia-east1).host- Hostname or IP address of the GCP broker (defaults to mqtt.googleapis.com).port- The port number of the Local MQTT broker (defaults to 8883).keepalive- The number of seconds to keep the MQTT connection alive (defaults to 60).bind_address- The IP address of a local network interface to bind this client to, assuming multiple interfaces
exist (defaults to "").clean_session- A boolean that determines the client type. If True, the broker will remove all information about
this client when it disconnects. If False, the client is a durable client and subscription information and queued
messages will be retained when the client disconnects (defaults to False).parsers- A dictionary containing additional MQTT parsers for the broker. A full list of the current parsers are
found in src\pyaware\mqtt\config as default_gcp_parsers() (defaults to the list of local parsers). e.g.
{"topic": "my/MQTT/topic/{variable_name}/here"}disable_parsers- A list containing the keys representing the MQTT parsers to remove on the broker. A full list
of the current parsers are found in src\pyaware\mqtt\config as default_gcp_parsers().private_key_path- Path to the private key file "rsa_private.pem" (defaults to
AWAREPATH/credentials/rsa_private.pem).ca_cert_path- Path to the Google Cloud Platform ca certificate file "google_roots.pem" (defaults to
AWAREPATH/credentials/google_roots.pem).authentication_required- Determines whether authentication is required to connect to the Local Broker (defaults
to False).subscribe_qos- Topic subscription qos (defaults to 1).publish_qos- Topic publish qos (defaults to 1).batch- Batch MQTT messages enable (defaults to True).batch_hold_off- Time window to batch MQTT messages in seconds (defaults to 5).max_message_queue_size- Maximum number of messages to have in the queue (defaults to 100).max_in_flight_messages- Maximum number of in-flight MQTT messages (defaults to 100).backlog_enable- Enable to backlog mqtt messages if not sent to MQTT broker (defaults to false).parsers_default- Select the topic parser defaults that can be overridden by parsers. (defaults to 'gcp')Translation Config (translation.yaml)PyAWARE can also be used to host data in a server. In order to pass data from / to devices from / to servers, pyAWARE
uses this translation configuration to determine how the data is to be mapped from point A to point B and vice versa.
An example translation.yaml is shown below. This shows how a server 'server_1' can pass data to / from a device
'device_1'.server-translation:server_1:devices:device_1:transformations:-rename_keys:definition:reg-0:zeroreg-1:onereg-2:tworeg-3:threereg-4:fourreg-5:fivereg-6:sixreg-7:sevenreg-8:eightreg-9:nineregisters:holding:zero:form:type:ParamCTypeaddress:1000idx:zerodata_type:ushortone:form:type:ParamCTypeaddress:1001idx:onedata_type:ushorttwo:form:type:ParamCTypeaddress:1002idx:twodata_type:ushortthree:form:type:ParamCTypeaddress:1003idx:threedata_type:ushortfour:form:type:ParamCTypeaddress:1004idx:fourdata_type:ushortfive:form:type:ParamCTypeaddress:1005idx:fivedata_type:ushortsix:form:type:ParamCTypeaddress:1006idx:sixdata_type:ushortseven:form:type:ParamCTypeaddress:1007idx:sevendata_type:ushorteight:form:type:ParamCTypeaddress:1008idx:eightdata_type:ushortnine:form:type:ParamCTypeaddress:1009idx:ninedata_type:ushortdevice-translation:device_1:servers:server_1:transformations:-rename_keys:definition:zero:reg-0one:reg-1two:reg-2three:reg-3four:reg-4five:reg-5six:reg-6seven:reg-7eight:reg-8nine:reg-9There are two key parts to this config file. We have a server translation that maps how data is recieved from the device
and placed into the server. We also have a device translation which does the opposite i.e. takes data from a server and
maps it back to the device. This relationship can be from any number of servers to/from any number of devices.RegistersThis defines how the selected server will layout its registers and map each of its parameters to an address.NOTE:
This is similar how devices layout their parameters. An example is below,...registers:holding:zero:form:type:ParamCTypeaddress:1000idx:zerodata_type:ushortone:form:type:ParamCTypeaddress:1001idx:onedata_type:ushorttwo:form:type:ParamCTypeaddress:1002idx:twodata_type:ushortthree:form:type:ParamCTypeaddress:1003idx:threedata_type:ushortfour:form:type:ParamCTypeaddress:1004idx:fourdata_type:ushortfive:form:type:ParamCTypeaddress:1005idx:fivedata_type:ushortsix:form:type:ParamCTypeaddress:1006idx:sixdata_type:ushortseven:form:type:ParamCTypeaddress:1007idx:sevendata_type:ushorteight:form:type:ParamCTypeaddress:1008idx:eightdata_type:ushortnine:form:type:ParamCTypeaddress:1009idx:ninedata_type:ushort...The parameter directly under 'registers' is the type of register to be defined, there are four registers in a Modbus
registers to be defined with the names below. Under each register type here is a list of server parameters with their
form which includes the type of parameter, address and data type.NOTE: Each register must have been defined already
under the server in gateway.yaml.holding- Modbus Holding Registerinput- Modbus Input Registercoil- Modbus Coil Registerdiscrete- Modbus Discrete RegisterTransformationsThis describes how the data is to be transformed before entering either the server or the device depending on if it is
defined under the server or device. The list of transformations is described below,rename_keys- Takes a dictionary of key mapping to map parameters by....transformations:...- rename_keys:definitions:device_or_server_param_name:new_server_or_device_param_name...remove_keys- Takes a set of keys to be removed from the param pool....transformations:...- remove_keys:definitions:-device_or_server_param_name_to_remove...scale_values- Takes a dictionary of parameter-scale value mapping. Returns a dictionary of scaled parameters....transformations:...- scale_values:definitions:device_or_server_param_name_to_scale:scale_factor...multiply_params- Takes a dictionary of parameter-parameter mapping. Returns a dictionary of multiplied
parameters....transformations:...- multiply_params:definitions:device_or_server_param_to_multiply:[device_or_server_param_to_multiply]# We can also multiply a param by itself by belowdevice_param1:[device_param1]# Or between multiple params and itselfdevice_param2:[device_param2,device_param1,device_param3]...group_params- Takes a dictionary of the parameter name to map to and a list of param names to group....transformations:...- group_params:definitions:device_or_server_param_to_map_top_level:[device_or_server_param1,device_or_server_param2,...,device_or_server_paramN]...System commandsThere are system commands that target the gateway instead of the attached devices.
They can be sent by using the usual command topic under the subtopic system.
Eg.
/devices/gateway1/events/commands/system/...Live Debugging /system/debugpyAWARE supports live debug logging.
The purpose of this is to increase the amount of logging detail on an already running instance.
This is managed through the MQTT connection through commands.The content of the message will be a JSON list of values. Any values that are not present will be disabled.
Eg. sending [] payload will turn off runtime logging. If any logging option is enabled then log level DEBUG will be enabled throughout the program.
The options available in the message areall : Turns on all available loggingcore: Will turn the global debugging level to DEBUG instead of INFO.
This option is enabled by default when any other option is set.
This will update std out and the AWARE.log file with the new debug messages.modbus : Turns on modbus logging. This will be present in separate files from the AWARE.logmqtt: Turns on mqtt logging. This will be present in separate files from the AWARE.logtriggers : Turns on the triggers logging. Provides logging for each run of the triggers against the new data.state_changes: Turns on the state changes. This will provide logging for all changes to the current state, event state and send state of each device. This will increase the performance impact as it tries to form
copies of the current state and new state in order to provide more insightful logs.Stop /system/stopStops the pyaware process. This will result in a restart of pyaware if it is an enabled linux service.
The payload is ignored.Deploying GatewayFor any new deployment on an embedded device, an image needs to be generated. All images for deployment can be found
onSubversion.After the device is set up with all the correct dependencies (as per offline installation),
an image should be stored for reuse. This can be done by doing the following.Generating reusable SD card image.Setting up an new SD card with the image.Once the image is loaded, test your new deployment by,Inserting the new SD card / Booting the device with the new image.Installing the latest pyAWARE as per installation instructions.Copying over the new config files.Generating credentials for Google Cloud and putting them in /etc/AWARE/credentialsValidate it works by running python -m pyaware /etc/AWARERestart IOT DeviceCreating new devicesDevice configs are stored in src/pyaware/devices.
An example device config can be seen below,sources:poll:type:pollhandle:holdingpoll_interval:0blocks:-[0,10]poll_input:type:pollhandle:inputpoll_interval:5blocks:-[0,10]parameters:reg-0:source:pollform:type:Paramaddress:0idx:reg-0triggers:process:send:-[state]reg-1:source:pollform:type:Paramaddress:1idx:reg-1triggers:process:send:-[state]reg-2:source:pollform:type:Paramaddress:2idx:reg-2triggers:process:send:-[value,50]reg-3:source:pollform:type:Paramaddress:3idx:reg-3triggers:process:send:-[time,5]reg-4:source:pollform:type:Paramaddress:4idx:reg-4triggers:process:send:-[state]-[time,60]reg-5:source:pollform:type:Paramaddress:5idx:reg-5triggers:process:send:-[state]-[time,60]reg-6:source:pollform:type:Paramaddress:6idx:reg-6triggers:process:send:-[state]-[time,60]reg-7:source:pollform:type:Paramaddress:7idx:reg-7triggers:process:send:-[state]-[time,60]reg-8:source:pollform:type:Paramaddress:8idx:reg-8triggers:process:send:-[state]-[time,60]reg-9:source:pollform:type:Paramaddress:9idx:reg-9triggers:process:send:-[state]-[time,60]reg-input-0:source:poll_inputform:type:Paramaddress:0idx:reg-input-0triggers:process:send:-[state]-[time,60]reg-input-1:source:poll_inputform:type:Paramaddress:1idx:reg-input-1triggers:process:store:-[always]send:-[state]-[time,60]reg-input-2:source:poll_inputform:type:Paramaddress:2idx:reg-input-2triggers:process:store:-[always]send:-[state]-[time,60]reg-input-3:source:poll_inputform:type:Paramaddress:3idx:reg-input-3triggers:process:store:-[always]send:-[state]-[time,60]reg-input-4:source:poll_inputform:type:Paramaddress:4idx:reg-input-4triggers:process:store:-[always]send:-[state]-[time,60]reg-input-5:source:poll_inputform:type:Paramaddress:5idx:reg-input-5triggers:process:store:-[always]send:-[state]-[time,60]reg-input-6:source:poll_inputform:type:Paramaddress:6idx:reg-input-6triggers:process:store:-[always]send:-[state]-[time,60]reg-input-7:source:poll_inputform:type:Paramaddress:7idx:reg-input-7triggers:process:store:-[always]send:-[state]-[time,60]reg-input-8:source:poll_inputform:type:Paramaddress:8idx:reg-input-8triggers:process:store:-[always]send:-[state]-[time,60]reg-input-9:source:poll_inputform:type:Paramaddress:9idx:reg-input-9triggers:process:store:-[always]send:-[state]-[time,60]transformations:-scale_values:definition:reg-input-9:3.052e-5There are a few key areas of device config, sources, transformations & parameters.SourcesSources represent all of the device blocks to be polled along side the names of each poll. Under each poll we have the
following parameters to define,Requiredblocks- This defines each contiguous block of addresses to read for each device in the form
[start_address, end_address].Optional:poll_interval- The poll interval in seconds in which to read this data (Defaults to 5).TransformationsThese are identical to transformations defined in translation.yaml except they only apply to the device and the result
will be sent directly to MQTT. See translation.yaml for more details.ParametersThese are the core of each device, they detail the parameters of each device, what data types the parameters are made
up of and what address(es) they belong to.Conditionally Required:source- The poll this parameter currently belongs to. Required on at least the parent parameter.form- What the parameter looks like. These are defined inparameter types.
Required on at least the children parameters.Optional:children- Parameters that are related to the parent parameter. Seeparameter hierarchyParameter TypesParam- Raw parameter value taken from a single address. Scaling and rounding to the significant figures is
provided.Required:address- The start address of the parameter.NOTE: This is required to be within the source address blocks.idx- The actual name of the parameter.Optional:scale- Factor to multiply the parameter value with (Defaults to 1).significant_figures- Number of significant figures to round to (defaults to None).ParamStatic- This is a 'virtual' constant parameter where the user defines its value and it is not read from the
device.Required:value- Value of the parameter.idx- The actual name of the parameter.ParamBoolArray- Converts the list of addresses to an array of integer values either 1 or 0. Gives the parameter
name 'idx' to all these values.Required:address- The addresses of the parameter as a block.NOTE: This is required to be within the source address
blocks.idx- The actual name of the parameter.length- Length of the array.ParamEnumBoolArray- The table initialises all parameters in it to False. If a parameter name is found in the list
of addresses it then sets this to false until a terminator is reached (if one is provided) or it ends the collection.Required:address- The addresses of the parameter as a block.NOTE: This is required to be within the source address
blocks.table- Dictionary of each number with its corresponding value.Optional:terminator- Character to stop reading the addresses on (Defaults to None).ParamText- Converts parameter to text.Required:address- The start address the parameter is located.idx- The actual name of the parameter.length- Length of the text to read from the start address.Optional:padding- Padding that the text is padded with from the addresses read (Defaults to b"\x00").swap_bytes- Enable to indicate if the bytes are to be swapped (Defaults to False)strip_leading- Text to strip from the start of the string (Defaults to "").strip_lagging- Text to strip from the end of the string (Defaults to "").ParamDict- Parameter to be decoded as a json object.Required:key- Parameter key to fetch from the json object.idx- Name to rename the json object.Optional:table- Table representing the json structure (Defaults to an empty dict)ParamBits- Parameter bitmask register.
NOTE: This is legacy and should use the hierarchical pattern for new devices that implement the new parameter pattern.Required:address- The start address the parameter is located.bitmask- A dictionary presenting the bitmask of the parameter written as {0: "stop", 1:"start", ...}Optional:idx- Name of the total bitmask register (Defaults to None).ParamMask- Applies a mask to the parameter after read.Required:address- The start address the parameter is located.idx- The actual name of the parameter.Optional:mask- Mask to apply to the parameter (Defaults to 0xFFFF).rshift- Shifts the bits to the right by the int provided (Defaults to 0).ParamOffset- Treats the Parameter as a ParamMask but also provides an offset.Optional:offset- Offset to apply to the parameter (Defaults to 0).ParamMaskBool- Treats the Parameter as a ParamMask but returns all the values as bools instead of ints.ParamMaskScale-  Applies a mask to the parameter after read and scales the result.Required:address- The start address the parameter is located.idx- The actual name of the parameter.Optional:mask- Mask to apply to the parameter (Defaults to 0xFFFF).rshift- Shifts the bits to the right by the int provided (Defaults to 0).scale- Scale factor to scale the parameter by (Defaults to 1).significant_figures- Number of significant figures to round to (defaults to None).ParamLookup- For the value read from the address will apply the result to a table and return the result similar to
and Enum.Requiredaddress- The start address the parameter is located.idx- The actual name of the parameter.table- Dictionary of each number with its corresponding value.table- Inversed dictionary of each number with its corresponding value.Optional:mask- Mask to apply to the parameter (Defaults to 0xFFFF).rshift- Shifts the bits to the right by the int provided (Defaults to 0).ParamCType- This is used for most Parameters, this enforces a parameter to be a particular data type from the
registers provided and returns an exception if the data binding cannot be performed.Required:type- ParamCTypeaddress- The start address of the parameter.NOTE: This is required to be within the source address blocks.idx- The actual name of the parameter.Optional:data_type- The data type of the parameter (Defaults to ushort). The data types are detailedhere. These types are written as below,charscharboolshortushortintuintlongulonglonglongulonglongfloatdoublechar[]byte_order- Order of each byte (Defaults to >) e.g. '>' big endian, '<' little endian.word_order- Order of each word (Defaults to >) e.g. '>' big endian, '<' little endian.ParamCTypeScale- Treats the parameter as a ParamCType but applies scaling to the parameter after decode and removes
scaling during the encode.Optional:scale- Scale factor to scale the parameter by (Defaults to 1).ParamCTypeScaleModulus- Treats the parameter as a ParamCType but applies scaling and modulus to the parameter after
decode and removes scaling and modulus operations during the encode.Optional:scale- Scale factor to scale the parameter by (Defaults to 1).modulus- Modulus to apply (Defaults to 65535).invert_on_overflow- Enable to invert the parameter sign on overflow (Defaults to False).ParamArray- Treats the parameter as an array of uint16 values from a starting address for a specified countRequiredaddress- The start address the parameter is located.idx- The actual name of the parameter.count- The number of addresses starting ataddressto include in the array.Parameter HierarchyParameters can be defined in a hierarchical pattern. This allows you to define parameters that need to be related for
purposes of managing reads/writes by parameter name. The hierarchy is defined by setting the children on the parent
parameter. If there are children on the parameter then the form becomes optional. Caution must be made when not
specifying the form on the parent as writes to the children parameters will ignore the current state of any
bits/registers that are not explicitly defined in the heirarchy.For examplereg-3:source:pollform:type:Paramaddress:3idx:reg-3children:reg-3-bit-0:form:type:ParamBitaddress:3idx:reg-3-bit-0bit:0triggers:process:send:-[always]write:validators:-trig_type:writablereg-3-bit-1:form:type:ParamBitaddress:3idx:reg-3-bit-1bit:1triggers:write:validators:-trig_type:writablereg-3-bit-2:form:type:ParamBitaddress:3idx:reg-3-bit-2bit:2triggers:write:validators:-trig_type:writabletriggers:process:#        send:#          - [time, 5]send:-[always]If the reg-3 initial state is 0xFFFF and a write parameters to reg-3-bit-1 is False, then the register's final state
would be 0xFFFD. However, if reg-3 did not have the form specified (capturing the whole register) then the final value
would be 0x0005.TriggersTriggers are applied to parameter data to determine different operations that are required to trigger to certain inputs.
An example of triggers can be seen in the sample device parameter file below,...parameters:reg-0:source:pollform:type:Paramaddress:0idx:reg-0triggers:process:store:-[always]event:-[state]send:-[state]-[time,30]write:validators:-trig_type:rangemin:0max:255-trig_type:comparisonref_name:current_stateref_key:modecomp_type:ecomp_value:1-trig_type:in_listlst:-0-1collect:read:-[deadline,3600]...There are three trigger types,process,writeandcollect.ProcessProcess triggers are triggers which when called process the parameter data. They can have 3 different states
(send, event, store). Each process trigger states have different types. Process types describe when these events will
trigger.Send- A send state will send the data on this trigger.Event- An event state will trigger an event.Store- A store state will store the data on this trigger.Process Trigger Typesstate- When the parameter data changes state (i.e. value), this will trigger.time- When a specified period of time has passed (in seconds) this will trigger.delta- When the parameter data has changed by a specified amount (delta) this will trigger.always- This will always trigger when the triggers are fired.and- This will perform an and operation on the data. Fires when the result of this operation is true.comparison- Compares this parameter with another parameter on the same device. Triggers when the result is true.
Current supported operations are, greater than (g), greater than equal to (ge), less than (l), less than equal to
(le), equal (e).WriteWrite triggers are triggers which when called write the parameter data to a device. They can have 1 state(validators).
Each process trigger states have different types. Write types describe when these events will trigger.Validators- Validators will validate the parameter write trigger based on the trigger types passed to it.Write Trigger Typesrange- Triggers when the data to write is within the range specified.in_list- Triggers when the data is in the list of values provided.comparison- Compares this parameter with another parameter on the same device. Triggers when the result is true.
Current supported operations are, greater than (g), greater than equal to (ge), less than (l), less than equal to
(le), equal (e).CollectCollect triggers are triggers which when called collect the parameter data. They can have 1 state
(read). This can have one trigger type.Read- A read state will collect the parameter data on a read operation.Collect Trigger Typesdeadline- When a deadline is reached this will trigger.SequencingAllows for sequencing operations for parameter operationsWrite SequencingParameters that need to have a special way to be written can use write sequencing.
Write sequencing is called after the validators are passed.
This allows a small subset of operations that can be done every time the write_parameters call is done either through
code or MQTT Commands such as "set-parameters"system-reset:#...sequences:# Negative pulse trigger (>100ms)write:-type:write_valuevalue:1-type:write_valuevalue:0-type:waitseconds:0.2-type:write_valuevalue:1Write Sequencing Typeswrite- Writes the value as specified in the payload. This is the default operation if no sequence is specifiedwrite_value- Writes a specified value to the parameter overriding the value set in the parameter payload.value- The value to set to the parameterwait- Waits the specified number of seconds before moving to the next stepseconds- The seconds to waitwrite_parameter_value- Writes a specified value to another parameter by nameparameter- The parameter name to write tovalue- The value to write to the parameterwrite_parameter- Writes the value from this parameter to another parameter by nameparameter- The parameter name to write toAggregationsAggregates can be applied to parameter data before being sent onwards to its destination. An example use of aggregation
can be seen in the sample device parameter file below,...parameters:reg-0:source:pollform:type:Paramaddress:0idx:reg-0triggers:process:send:-[state]-[time,30]aggregations:-latest-samples-min-max-sum...Aggregations is a list passed from the parameter to pyAWARE to inform pyAWARE that these aggregations are to be
performed.Currently supported aggregations (NOTE: If aggregations are not defined, the default aggregation operations
performed are samples and latest),samples- Returns the total number of samples taken (poll intervals) for the current sending operation as defined
in triggers.latest- Returns the latest sample for the current sending operation as defined in triggers.sum- Returns the sumation of all samples for the current sending operation as defined in triggers.min- Returns the minimum of all samples for the current sending operation as defined in triggers.max- Returns the maximum of all samples for the current sending operation as defined in triggers.Release NotesRelease 7.0.0AWARE-1836Added REX640 templateAWARE-2542 - Add hardware type to the pyaware gateway state for seimans boxAdded config path to state objectAdded executable path to state objectAdded service type to state objectAdded maintenance helper functions from standalone updater scriptAWARE-2722 - Substation ComissioningFixed issue with blocks hitting illegal data addressAWARE-2723 - iMAC doesn't roll call gen 2Updated iMAC roll call to force completion of roll call before doing any module critical parameter discovery. This will have impact on start up speed but should resolve most potential future issues with roll callUpdated roll call system to prioritise getting gg2 and aim online for safe-gas as fast as possible as a bulk operationFixed bug where address-bypass is not correctly renamed throughout pyawareFixed tests for discover at addressUpdating imac modules to hit the deadlines for poll data to be sent at maximum 5 minutes and block data sent at maximum < day (85000 seconds)Updating roll call to stop other roll calls occurring while resolving all new modules and critical parameters. Should also show a banner on the front endFixing command spec set for gg2 testingUpdating roll call to stop other roll calls occurring while resolving all new modules and critical parameters. Should also show a banner on the front endUpdating pipeline to use latest pip dependency for bundle-dependenciesAWARE-2806 - New GGL Moxa Image for ProductionAdds SWUpdate generation tools and adds swupdate installation status to gateway heartbeatAdded support for building pyz with arm wheel filesMoved docker run command into run_pyaware.shShell script fixAdded some support for cross platform buildsAdded native armv7 pyz build processRemoved tox and bitbucket arm build as it can't buildAdded separate builds for 3.7 and 3.12Added multi stage builds for pyaware outputsStandardised build process for any version of pythonFixed deprecated safe load function to be compatible with latest versionAdded missing requirement for requirements.txtAdded compiled requirements.txtchanged run_pyaware script to check if container called pyaware exists at all rather than just for a running containerUpdated yaml dump to new apiUpdated requirements-test.txt to include method tools for weakref handlesUpdate yaml config parsing as new version doesn't guarantee round trip consistancyRemoved 8080 port mapping as it is not needed for modbus developmentfix swupdate scripts to account for new build methodAWARE-2809 - PIM Module parameter and displayRemoved misnamed fileUpdated ParamLookup to optionally have a reversed table as well as adding default values for decodingUpdated pim to correctly send pressure values when not in faultAdded faults lookup for pim when in specified fault rangesUpdated pim module fault to use int enum output instead of text enum output for better storageFixed issue with pim module fault triggersUpdated to simplify pim output. Relying on server changes for module variants instead of enforcing on pyawareUpdated pressure to not conditionally send and instead do on backend processing.Updated ParamLookup to correctly return a black dictionary when erroringUpdated commission3.py to workAWARE-2810 - Generate pyz on all pyaware branchesUpdated build pipelines to output pyz file for swupdateAWARE-2824 - sw-versions generated incorrectlychanged regex in install.lua to the versions file generates correctlyinstall.luaAWARE-2856 - PIM not sending errors-softFixed PIM definition to correctly send errors-softAWARE-2873 - AIM Configuration ValuesAdded write validator for hysteresis-setting-powerRelease 6.2.2 - Hpb hotfixAWARE-2808 HPB config hotfixFixing issue where hpb doesn't apply scaling to voltage and currents from the config xin, xunRelease 6.2.1 - Cheese Factory Road SAPS AdditionsAWARE-2474 Fix Influxdb schema issuesChanged gasguard2 address-bypass parameter name due to type clashes.AWARE-2502 Add ComAP InteliLite 4 AMF-25Added Comap InteliLite AMF-25 to devices.Added parameters to sp proAWARE-2503 Add AERL Solar ChargerAdded AERL Coolmax SRX-600 to devices.Implemented a work-around for pyaware events being garbage collected. Adjusted a parameter type on SP PRO. Returned the translator object to stop it from being garbage collected.Updated selectronic read blocks with new parameters added.AWARE-2373 Pyaware pipelines are brokenFixed issue where twine was grabbing the latest urllib3 even though version 2.0 brought breaking changes for SSL complianceAWARE-2567 Add ABB Ekip Com TmaxAdded ABB EKIP COM TMAX to devices. Updated AMF modbus addresses.Added the controller mode to separate digital outputs.AWARE-2504 Commission SAPSUpdated translator to do cross device transformations. Added a sum function to transformations. Updated solar charger modbus parameters.Release 6.1.1 - Hotfix Pinned requirementsHotfix to fix an issue with a sub-dependency breaking modbus reads and timeouts.Frozen all requirementsRelease 6.1.0 - Auto Updater SeparationAWARE-2178 Allow for sending ntpd restart commands to connected devicesAdded ntp restart command to pyawareAdded dev versioning for special case of porting devices to clearblade where the ntp server is out of syncAWARE-2186 Aim modules reconnect logicUpdated the update_devices method to remove a device if it doesn't match the new roll call module type before adding it again. Will not revert a known device to unknown.AWARE-2260 Separate pyaware update scriptCreated pyaware updater standalone scriptIncreased updater verbosityIncluded updater resilienceAdded cron job scriptFixed crontab being disabledAdded check to not run updater if update already runningAdded back ability of pyaware to update itself even if auto updater is installedUpdated hardware detection to service detection to be more robust and generic and actually run the updater on the service that pyaware is going to run against.Release 6.0.0 - Store and ForwardAWARE-1316 Store and Forward data separationBacklog implementationUpdated error message that should no longer be reachedAdded docker compose dev environmentUpdated backlog manager to more efficiently package messagesAdded artificial delay to message publishing to avoid clogging gglserverAdded random delays to mqtt backlog manager to ensure that multiple moxas starting at the same time will not flood the serverAWARE-1836Added rex640.jinja2AWARE-1837Added data type support for PM180Added pm180.yamlAWARE-1838Added hpb.jinja2AWARE-1839Added ret615.jinja2AWARE-1941 GCP to Azure MigrationCleaner error handling and loggingSync twin is now properly called on connectionIncreased robustness around missing configuration files and azure connect logicAdded gateway stateFixed issue where multiple topic types in backlog would only do the first one.AWARE-1941Updated payload to include topic type for azure message filteringImplemented store and forward for azureAdded confirmation for gateway configuration for digital twinReduced client side throttlingFixed issue where child parameters would not correctly execute read-parametersAdded azure device commandsFixed bug in database message ack'ingFixed batch ack's to avoid maximum sql parameterisation of 999Created new azure exampleAWARE-1960 Support changes to connection.yaml from gateway.yamlUpdated ability for gateway.yaml to update connection.yamlAdded connection.yaml backup when updatedRestores connection.yaml from backup if badly formed or doesn't get a connection within 10 minutesDeletes backup connection.yaml if the current connection.yaml is validated to connect within 10 minutesUpdated gcp example to use production test sandbox using current naming schemesUpdated config manipulation to be in a more nice reading yaml formatAWARE-2008 Store and Forward roll outChanged backlog_enable to feature_backfill to fit other feature flagsAdded gglive example with backfill so that the config can be shipped with pyawareAWARE-2015Fixes for RET615Release 5.4.0 - Integration Service UpdateAWARE-1895Updated the mqtt receive message loop to sleep for 1sec if the client could not receive feedback from the broker. This stops a fast tight loop and spamming the logs with error messages.AWARE-1891Updated the raw integration data to include registers 0 - 1697Extended the modbus TCP read blocks to include a continuous range from 0 to 0x6a2 so that raw integration data doesn't have gapsFixed modbus ranges on the Imac Master ControllerUpdated pipelines to conform to PEP440 naming conventions that are now enforced by setuptoolsRelease 5.3.0 - Memory leak fixesAWARE-1802 - HISAPS comissioningUpdated device specs for Solar Qube devices as per testing on site.Updated Sungrow SH-5K addresses to align with datasheet.AWARE-1828 - Remote telemetry tests timing outAdded tracking of parent object to subscriptions so parent object can unsubscribe itself explicitly.Added explicit unsubscription of events on imac modules when they are disconnected.Fixed issue with unsubscribe events not matching correctly on idAWARE-1846Various issues inhttps://github.com/Yakifo/amqtt/pull/127Using fork amqtt-pazzarpj which incorporates this fixUpdating mqtt client to use fork with fixesUpdating requirements for websockets as there is conflict with amqttPacking version pinned as there is an api breakAWARE 1848 - Duplicate triggers on long running systemAdded weak reference implementation to stop event holding onto event callbacks when the module is removed.Fixed references to imac modules still staying after being "removed"Updated documentation to describe the various module references in code to ensure new updates do not interfere.Fixed testsFixed address reference for linearity-test-last-pointsUpdated event triggers to reset priority of parameters instead of directly reading themAWARE 1852 - Detector type unknown does not stickUpdated detector-type to be a virtual parameter so that data read from detector will first check if the data is valid before returning the detector-type read from the gg2Fixed issue with packaging new update not accepting arbitrary versionsFixed issue where it was not collecting the detector-type-raw on detector head being re-attachedUpdated the deadline for the collect triggers to give opportunity for other parameters to push themselves even higher than the first reads for other parametersRelease 5.2.0AWARE-1788 Refresh data is failing on GG2Removed parameter_poll_reader and replaced with removing the send_state so that the parameter will trigger next read.Updated parameter commands to not validate as now it is not guaranteed that the parameters will be returned in the case of poll parameters.Updated read parameters to not error with missing parameter handlerFixed issue with shutdown code that would error on Modbus Serial clientFixed tests to match the new functionalityAdded IO Error failure if no parameters are returned.Release 5.1.0AWARE-1786Updated the topology request loop to explicitly request the topology each interval instead of sending the last topologyChanged interval topology to only log new topologyChanged interval topology to only send one messageRelease 5.0.1AWARE-1782Reverted touchpro to only support 12 channels as the base firmware will error if modbus registers are out of range.Release 5.0.0AWARE-749Updated requirements.txt to include known working versions of dependencies of hbmqtt 0.9.6 as per commit b2d75d7 for issue AWARE-742.Updated client.py to include memory leak fix for hbmqtt. As per commit c3cd85f for issue AWARE-730.Updated ipc.py to include gateway heartbeat fix. As per commit 41c13c7 as part of issue AWARE-693.Updated release notes and bumped version. Updated formatting.Updated formatting.AWARE-853Changed Selectroniv alarm values to fix firmware update issues with Selectronic alarming.Updated versioning and release notes.Added remote start/stop procedure for Comap Intelilite MRS-16. Updated release notes.Updated formatting black.Code CleanupUpdated release notes.AWARE-891Added a solar cube example config.Updated version number and release notes.AWARE-926Fixed issue with parameters not being updated from collect schedule when a related parameter was read.Added test to ensure correct calls are made when checking which parameter reads failed and which are newAWARE-1474 Update pyaware to include additional solar inverterAdded Second Solar Inverter to Selectronic device specification and updated scaling.Updated release notes.Updated formatting.AWARE-1482 Live Pyaware DebuggingUpdated to cleanly remove a module as it was staying in the subcommand memory when removedAdded command to turn on debugging on/off an already running pyaware instanceUpdated modbus library to support loggingUpdated debug logger to handle modbus loggingAdded a logging class to clean up the different logging optionsAdded state logger to monitor updates to the state dictionaryFixed import errorsUpdated logger initialisation to use existing logger instead of new logger to fileAdded helper functions for the logger to determine the state objectsUpdated state objects to use a wrappable dictionary as the default cannot be overridden.Added debug statement for poll loop timeUpdated debug command based on feedbackUpdated readme to describe commandFixed issue where logging tried to log triggers for device_id on imac modules when it doesn't existAWARE-1487 Customer Factory Acceptance TestingUpdated the comap engine state and breaker state enums.Updated release notes and version numberUpdated test frameworkAWARE-1591 Produce versioned ARM artifactsUpdate pipelines to support arm builds for each branch type to preserve versioning within the packageAWARE-1612 Detector Latched BitAdded Configs for Feature Bits + Latched TripAdded handler for clearing LatchAdded test for configuration filesAdded feature flags to RTSFixed comment block on clear_latch_at_addressAWARE-1614 iMAC address bypass off by oneFixing off-by-one error in address-bypassaddress-bypass is addresses 1-40AWARE-1621 Add Sungrow inverterAdded Sungrow SH5K-30 modbus spec to pyaware devices.Removed store triggers as they are already triggerred on send. Removed daily parameters as it depends on the clock of the device not the clock of the MOXA unit. Removed co2 reduction as it is to be calculated by aware.AWARE-1624 Bleeder Fan TripAdded configuration for bleeder fanAdded command to RTS to set Bleeder on/offFixed addressesFixed off by one error from latched tripChanged set_bleeder implementationAWARE-1638 Calibration Cup data not updated on user calibrationAdded event trigger to read calibration data whenever the cal cup is removed. Adds 20 second delay, just in case GG2 hasn't processed yetAWARE-1651 Devices not correctly being removedAdded check connected method on imac modules. It uses block 0 to check if a module still exists.Added check connected calls for missing modules on system roll call and roll call addressAdded watchdog to parameter block read or write to run the check connected method on exception with a 100000 second timeout.Removed test for delete missing modules at address as the functionality has changedAWARE-1652 Devices sending telemetry when not connected to the busFixed issue where devices were sending telemetry when not connected to the bus.AWARE-1673 Solar Qube - Alarms not clearing when occurring in quick successionUpdated timing of comap alarm clearing check to ensure it is always cleared in the event the alarms are fired in quick succession.Updated version number and release notes.AWARE-1675 RTS is sending topology as an iMACRemoved topology message from RTS ControllersAWARE-1683 Check connected erroring on RTS moduleFixed issue where RTS module was erroring on check_connected methodAWARE-1684 Migrate Solar Qube release branch into develop branch of pyawareUpdated release notes.Updated solar qube names to reflect branding changes.Updated solar qube configuration to align with current pyaware version.AWARE-1694 Update pyaware to support 16 channels for Touch ProAdded jinja templates to allow for dynamic devices such as touchpro amount of channelsChanged so all yaml parsing goes through pyaware.config.load_yaml as it is cached and works with templatingAdded tests for parsing touchpro templatesUpdated tests to auto include a loop cleanup fixture so the tests don't hang or bleed into other testsFixed modbus holding register range to support up to 48 channels of input valuesUpdated manifest to explicitly include the jinja templatesAWARE-1736 Add shunt resistance to iMAC screenAdded shunt resistance to spec file.AWARE-1752 RTS incorrectly being removed from topology messagesRemoved module watchdog setup (block read) for modules of type "imac-controller-rts" as they do not have block reads.Release 4.8.1AWARE-1610 iMAC failed to read block errorsChanged set_bit used in read and write parameters to skip the checked bit as it is a race condition against the iMACRelease 4.8.0AWARE-1324Added timeouts to SNMP functions. Wrapped SNMP blocking functions so that they do not block the event loop.Removed unused import. Updated formatting black.Updated version number of release. Commented out tests to be fixed.Fixed tests. Updated snmp to use partial functions over positional arguments.Updated the system flags to send the data correctlyFixed tests. Updated version number to align with develop.Reformatted black.AWARE-1447Added DO4-4 Parameter Spec and module.Updated DO4-4 Parameter SpecRemoved unnecessary DO4-4 comments from spec. Updated roll-callable and online address for DO4-4.Updated DO4-4 Parameter spec.Updated DO4-4 online and roll-callable parameter names.Updated DO4-4 parameter spec to remove status from names.AWARE-1459Updated imac master commands to include a clear error counts, a clear offline count and clear clash count for an imac address.Reformatted.Updated the clear all error counts command to clear all error registers.Removed address 0 from imac master specific commands. You no longer can clear address 0, or clear counts on address 0.AWARE-1464Added new GasGuard command to set the front panel lockout.Updated set-front-panel-lockout to use SetParameters command procedure to set the command rather than write and validate.Changed name from front-panel-lockout to ui-lockout. Updated with respect to customer comments.Reformatted.AWARE-1556Updated triggers to match requirementsAWARE-1577Reading the detector type when detector data invalid triggers to False will no longer block until the detector type is read.Default gas value for safegas set to -2000Release 4.7.0AWARE-1310Added optional parameter to set the topology intervalSet the default topology interval to 60 secondsAWARE-1313Add timestamp to gateway stateAWARE-1315Updated iMAC input modules to send polled data at most once per hourRelease 4.6.0AWARE-996 Specify eth adapter for GGLiveUpdated gateway_id in gglive example config to always be the external facing ethernet adapter mac address.AWARE-997 Connection.yaml to fully resolve before attempting to initialise devicesChanged start up logic to remove a bunch of async functions that can be synchronousChanged the device state message to not be cached as it was waiting for the DB to be initialised before sending.Changed the device state message to send on each connection in case it gets lost due to cache not storing the message.Fixed issue where state was sent twice at start upFixed issue where waiting for remote config would still happen even after receiving a blank configChanged error reporting on MQTT setup tasks cancellation as it was not compatible with python 3.7Run Modbus Device creation in a thread to prevent event loop locking up. Particularly for siemens boxes.Fixed deferred gateway topics update referenceRelease 4.5.0AWARE-949 Update documentation for offline preparation and installationAdded instructions for downloading the python installation various waysAdded bitbucket custom pipeline to generate bundled packagesRenamed package build to armv7 to indicate architectureAdded package build to production pipelineAWARE-952 Update bypass logicSend all bypass status every minuteAWARE-962 Pyaware start-up needs to subscribe to config topic before waiting for remote configurationFixed a bug where remote configuration was never being used in GCP applications.Fixed a bug where remote configuration would not restart pyaware.Updated tests.Updated formatting.Added logging for missing configuration files. Updated local remote configuration timeout to 2 seconds so that the remote config would update gateway.yaml before the local was parsed. Small timing fixes on start up to ensure the database was correctly initiated before actioned on. Updated mqtt config handling to take in the correct gateway config topic depending on gcp or local mqtt.Removed logging configuration in gateway.yaml. This is to be revisited if it is required at some stage. Reverted change to restart pyaware on remote configuration received.Reintroduced a pyaware stop when we receive new configuration from a config topic after config parsing.Added setup waits to disk storage to ensure it is initialised before runningAWARE-973 Pim IntegrationAdded PIM moduleAdded collect triggers for pimAWARE-980 Change Schneider Power Meters to IEC Power FactorChanged from 4Q_FP_PF to the IEC Float versionAWARE-988 Gateway id from mac address to be deterministicPut a check in to ensure that the configuration parameter for gateway_id was used over the default setting. If none is set then the program will attempt to be deterministic by selecting the last mac address that is not a loopback adapter to use.Added a solution for an error state where gateway_id is never set.AWARE-992Fixed issue where data files weren't included on linuxAWARE-994Added error check to parsing mac address from config so that it doesn't crash pyaware when processing gateway.yaml with invalid configuration.Release 4.4.0AWARE-619 - RTGM - RTS Auto restart if Fieldbus 0Moved gateway state and heartbeat from gateway_ipc into mqtt client. These will be started by default when pyaware is started.Added connection.yaml feature flag functionality.Updated the mqtt client setup to send the network details to topology when state is changed.Moved topology to gateway configuration. Moved GatewayIPC to Solar Cube dependant functionality as gateway configuration. Removed gateway IPC.Fixed timing of sending topology messages. Further decoupled the topology messages from gateway IPC.Gateway Heartbeat and State messages made default.Moved Topology into imac auto_detect functionality.Changed the gateway_id to the last mac address read from the device.Fixed sequencing of mqtt storage database so that gateway state messages can be sent on startup.AWARE-880 - Potential pyaware defaults can cause issues with the defaults not being processedUpdated README with the correct database config information.Added the ability to disable MQTT parsers. Made the ability to update MQTT parsers more explicit. Added documentation to the README for the above changes.Re-enabled the ability for users to change the logging configuration through gateway yaml. Removed the unnecessary log_config dictionary of logging config.Fixed comms params parsing to account for NoneTypes on read. This assumes that NoneTypes are not valid types to set to comms params.Fixed an error where if a key error was present in parse_comms_params this did not always indicate a deferred parameter.Fixed database defaults parsing issue if no config was present. Changed the object default database size from unlimited to a hard 500MB limit.AWARE-885 - PyAware - Safegas MQTT Messages - Add AIM as Sensor typeUpdated process_safe_gas function documentation.Added AIM module support for safegas data.AWARE-888 - RTGM - Missing MQTT messages from PyawareAdded device_message flag to mqtt topics to allow for certain topics to be sent even though gateway config parsing is not complete.Added warning to gateway config parsing in the event the commands topic is never established.Increased mqtt timeout to 60 seconds due to this config parsing change.Moved mqtt setup functions outside of setup loop. Kept mqtt network setup inside of setup loop. Improved error handling.Added error messages for gateway command topic parsing errors. In the event of an error a topic subscription is still setup.Added device_message flag to GCP state message so that it is sent on startup before device mqtt setup.Removed state message from setup tasks as it is not a setup task.AWARE-891 - Pyaware - Add Config FilesAdded the ability for config files to be added to the wheel file.Added local mqtt example config. Added README for local mqtt config.Added gcp example. Added README for gcp example.Changed recursive-include to recursive-exclude for .git files. Added a GGL example in example configs.AWARE-898 - RTGM - How do we manage old version of pyaware sitting on spares shelf?Removed unused MQTT models and topics. Added version strings to all MQTT messages. Made a note to migrate old version of MQTT models into new models when appropriate.Repositioned__version__to align with PEP8 module level dunder names. Provided__version__on the state message mqtt model. Fixed mqtt model imports to align with new names.Added version numbers to command request to ensure that there will be no validation errors with GCP. In theory, API should send this, overriding this field.Removed snake-casing from MQTT models for camel casing. Updated tests.AWARE-904 - Pyaware has a dictionary modified while it is being iterated. InvestigateFixed issue where the processing of the controller data would fail if a new module was added during processing.AWARE-917 - RTGM - AIM module address change causes data to stop sendingAdded logging for block and roll call lock acquire and release.Forced all address roll calls to happen in sequence with no other block reads allowed.Broke imac addresses out into two states, online and rollcallable. This logical breakout is to solve iMAC modules going offline when there are only one of its addresses going offline for multi-address modules.Release 4.3.0This release removes the requirement for a gateway.yaml on startup and moves to connection.yaml as the primary way to
set up the mqtt connection.AWARE-448 - Remove pymodbus clientRemoved pymodbus and associated legacy codeAWARE-750 - Move mqtt details out of gateway.yamlAdded parse_mqtt to config which will run before aware updatesAdded connection.yaml as a base config for awareAdded gateway_config_received event to indicate when the configuration has been received.Updated configuration parsing to more closely match the functionUpdated MQTT to use configuration as rawImproved error reportingChanged logging order so it will log the mqtt connection informationRemoved logging parameters from config as it is not loaded at this pointFixed a bug where disk_storage would import None if the mqtt client was initialised before the databaseChanged the mqtt local config topic to replace the device_id with gateway_id for the gateway. This allows us to target the gateway with configuration updatesAdded default id for client_id for local mqttAdded build_connection_yaml to build the new connection configuration from available dataAdded a new parameter to mqtt config to set how long to wait for a remote gateway config before loading from diskFixed existing tests around configuration parsingAdded tests for auto generating connection.yamlAdded ability to change the default parsers more simply in connection.yamlAdded backwards compatibility to ensure that gateway_id can still be populated when only device_id is provided by legacy cloud.yamlAdded error checking for missing configuration fileAWARE-784 - Command write parameters errors on invalid handlesChanged the parameter handles to have a default of holding register when not present in the config fileChanged the write handle to not error when there is no existing handleFixed issues with Async mockingAWARE-789 - Substation PLC Configuration UpdatesUpdated configuration file to correctly distinguish between coil and input registersUpdated address_shift to work with parameters with blank addressesAdded missing address for outlet-5-eflo-tripChanged uint to float typeAdded commission3.py to be compatible with async configuration loadingUpdated requirements for aiomodbus fixed on coil readsChanged the byte ordering based on hst feedbackChanged the variable names to line up with requests from HSTAWARE-798 - Local/Remote Bypass indication not workingFixed a bug with config parsing where mqtt config parameters that rely on information in gateway.yaml were not being parsed.Fixed a bug where the bypass status was not correctly set when the unit was in local bypass.Changed the subscription topics so that mqtt commands are done only if there is enough information to do so. General code cleanup.Added an option to disable MQTT for instances of pyAWARE without it. This is contained in connection.yaml and is an explicit parameter called disable_mqtt.Removed disable_mqtt if set to False as its not used in comms param parsing.Updated formatting black. Removed duplicate import from requirements-test.txtChanged the local and remote bypass bits to their correct values. Moved connection yaml parsing outside of mqtt_parse. Added type hinting. Changed update function on mqtt config dataclass to check that keys exist before updating.Fixed a bug where commands topic was not being subscribed to if a connection.yaml was present that did not depend on gateway.yamlUpdated a test that required parse_mqtt to have parameter added to its import.AWARE-804 - RTGM third party integrationAdded raw imac data to safegas messageUpdated safegas to use 60 detectorsGrouped iMAC testsBasic test to check that parameters exist as expectedAWARE-811 - Fixes to connection.yaml topic parsersFixed issue where adding parsers to config would override all default parsersUpdated the config parsing flow so it only evaluates parameters it was unable to last passRelease 4.2.0AWARE-690 Write sequencing support for commandsAdded controller base class to implement common parameter operations such as "set-parameters" and "get-parameters"Added run_write_sequence which skips parameters that are written in a groupAdded first pass Parameter implementation for generic read/write control across all parametersConfig parsing for the parametersAdded support for parameter validationChanged parameter read/write handles to be properties that the controller has to setUpdated the Modbus Controller to parse the read/write handlersWrite parameters will look to the parent parameter to set itAdded get-parameters command to modbus controllerFixed bugs found by test failuresWrites will now use current state for grouped parameters to ensure that they are not overwrittenWrite sequences now workWrite parameter sequence now respects the target parameters write sequenceAdded ParamBitParent/children parameters addedDocumentation updates on TCP connections for modbusFixed bug where address masking was resulting in bits not clearing from the appropriate addressChanged parameter current state to refer to the same current state as the parentChanged the current state to update every decode cycle as the parameters are not explicitly read when called from a block poll of the controller.Test on ParamBit more able to pick up bitwise issues in encode/decodeAdded multiple addresses return for datatypes that have multiple addressesAdded better data validation for missing parameters in set-parametersAdded ParamBool for coil registersAdded austech touchpro device configuration fileFixed issue where the addresses were returned incorrectly for ParamCTypeUpdated documentation on parameter hierarchy and write sequencingRemoving dead code from controller sequences as they are implemented in the new parameter modelFixed issues with trigger and aggregation parsing not working on children parametersFixed issues channel setup addressesAdded assertion to enforce contiguous addresses for the current implementation [No ParamBoolArray like in imac)Black formattedMade commands more robust to invalid payloadsChanged command ordering so that ack is sent if the message makes it though before trying to process it.Updated command request model to have multiple data typesUpdated CommandRequest model for cloud based to have the optional list type for parameter parsingFixed ParamText to return the appropriate amount of addressesFixed GetParameters to work correctly with dict and listTrimmed touchpro mapping to 4 channelsAWARE-713 SNMP ImplementationRefactored data types into its own module.Added namespace for modbus to modbus parametersMoved ParamDict to common as it is not a modbus parameterInitial design idea. Added snmp to list of controllers. Added puresnmp to list of required packages. Added autocard.yaml.Initial design idea. Added snmp to list of controllers. Added puresnmp to list of required packages. Added autocard.yaml.Moved ParamStatic to Common data types.Added SNMP datatypes integer32 and string. Added tests for these datatypes. Added these datatypes to parameter types list. Refactored shelfkit.yaml.Initial shelfkit implementation for autocard 1.Updated the SNMP datatypes to use the paramform base class. Implemented a bare bones snmp read controller.Added controller methods. Added get_parameter method. Added a test method for this get_parameter method.Added test for snmp controller read_parameters method. This resulted in a fix of the device config as it was incorrectly written. Fixed doc strings for read_parameters method on Modbus and SNMP controller.Refactored naming on snmp decode encode methods.Refactored naming of read_multi.Added tests for read_multiple on SNMP device.Added a write multiple parameters method to SNMP. Changed the encoding/decoding of SNMP to use puresnmp types as when encoded the data needs to use puresnmp types.Implemented a write parameters and a set parameters command for snmp.Implemented the get_parameters method, set_parameters method & set_parameter method on snmp device controller.Implemented a test for the get_parameters method on the snmp controller.Added tests for the snmp set parameter and set parameters methods.Added test for write parameters on snmp device. Made the encoder factory optional for the parameter writer to remove necessary AddressMapUint16 encoder from global default.Added documentation & typing for the snmp controller and snmp datatype.Added documentation & typing for the snmp controller and snmp datatype.Formatted black.Added the rest of the nine autocards to shelfkit.yaml.Added the rest of the 5 autocards to shelfkit.yaml.Updated the send intervals in shelfkit.yamlUpdated the name of the shelf kitAdded link to the shelf kit MIB file in shelf_kit.yamlFormatted black.Changing patch to patch object to potentially fix a failed pipeline testPut device inside test_get_parameter_not_strict_success method. To achieve isolation.Changing patch to patch object to potentially fix a failed pipeline testAttempt to fix test pipeline.Reformatted black.Removed autocard.yaml.Removed AsyncMock from test_snmp_controller.py as it is not used.AWARE-745 Update Safegas ImplementationAdded Scaled gas array for GG2 (not AIM)Added address status array based on ModuleStatus EnumAdded ParamArray to have an array type for the address-dataUpdated iMAC controller to process address-dataUpdated safe-gas to use the address-data as a base for the analog valuesUpdated default flags register to 0xffffChanged address-data to start at address 0 to match the address-status arrayReworked the poll loop so that process_safe_gas happens in the same thread to prevent a race condition when looping over the modules. This could cause an error during roll call if a module was discovered at the same time that safe gas was processing.Updated the comments to more accurately reflect the payload contentsUpdated documentation for ParamArrayUpdate aware safe gas array to use analog registers instead of flag registersVersion 4.1.1This is release fixes the slow memory leak in the mqtt clientAWARE-730Improved memory profile snapshotsAdded fix for a memory leak on the hbmqtt clientA more correct implementation of the mqtt fix so that cleanup on shutdown is successfulAWARE-731Changed the ordering of watchdog bindings so that if the protocol_lost or protocol_connected is called before it gets to add the watchdog it will not fail.Tries to read 50 times to overcome issue with iMAC in recently discovered failure state which causes pyaware to never be able to read from it. Should restart in 1 minute if not available to read.AWARE-740CommandRequestV1 now has data as an optional field to be complaint with commands that don't use the fieldAWARE-742Updated requirements.txt to include known working versions of dependencies of hbmqtt 0.9.6AWARE-751Changed imac_heartbeat_poll to still send mqtt messages even if the modbus reads are failingAdded a new parameter to watchdog.watch to allow the watchdog to be fed on entry to the function instead of just successfully completing the functionVersion 4.1.0This is the minimum required version for Real-time Gas Monitoring (RTGM).AWARE-693 RTGM - Gateway Heartbeat not Validating through PydanticFixed a bug where the ip address was never being updated on network changes.Stopped the gateway state message from being sent twice on startup. Now only sent when valid IP Address is received.AWARE-691 Pyaware - Safegas array logicAdded enforcing of gas value index for safegas to ensure that gas values are within the 40 address range.Added AIM module to safegas message. AIM module in the address range 41-80 should have a Gasguard 1 detector connected. This is not enforced through code, only through documentation.Added raw data flags to the AIM module so that they can be read in and sent via safegas.AWARE-685 Add set-parameters command to modbus deviceAdd set-parameters to modbus deviceAdded command input validationChanged sub topics to be a max of 3 topics as GCP seems to ignore topics in subscription messages after the fourth subscription.Commands will get sent to the appropriate device based on command topic nameAdded command response. Tested against aware-development pubsub topicsChanged created_at in model from string to datetimeFixed a bug where the subscriptions wouldn't happen if anything other than the device_id needed to be populated into the topic nameVersion 4.0.1AWARE-672 Update with PLC DeviceAdded hst substation plc device.Updated PLC flag sending time to 5 minutes or change of state.AWARE-671 Update HST config for faster device dataRemoved raw_values from TelemetryV1 and TelemetryV3 as it was causing a bug that prevented any aggregation other than samples and latest to be sent. Added a test for this specific case.Version 4.0.0AWARE-664 GG2 Telemetry Test Causing other GG2 detectors values to changeChanged Telemetry Test With timestamp to without due to an error with asyncio locks. The lock was released too soon resulting in the imac and associated modules being put in an unknown state. This resulted in random data being written to modules on the bus.AWARE-605 Review BugsChange of fieldbus address breaks RTSFixed issue where serial number changes would stop parameter triggers workingChanged rts schema matching to ensure that the address is inside the schema range before testing if all the addresses are onlineForce RTS to restart pyaware if fieldbus address is 0AWARE-624 Bypass BehaviourAdded event for address-flags changingRemoved faulty code in update_specs for setting the remote and local bypassAdded parameter trigger for updating the remote and local bypass on address-flags changeAdded meta field to parameters to house data specific to the implementation.Imac modules now check the meta field of a parameter to check if the ignore-status field is set for parsing data in offline addresses.bypass-local has the ignore-status set so that when a di4 moves addresses, the local bypass will clear when the module goes offline at the previous address.AWARE-612 Retain Messages ReviewChanged all retain mqtt messages to not retainedAWARE-647 Roll call on offline and remove devices on failure to respondAdded any address online to ImacModule to determine if the module needs to be disconnected on failed roll callNow updates topology on removal of deviceNow roll calls offline addresses to make sure that they will remove devices that are no longer connectedAdded delete_missing_modules_at_address to force roll callAdded tests for roll call module deletionAWARE-648 Schedule roll call of system addresses and remove devices on failure to respondRemoved duplicate topology message from IPCAdded scheduled daily read of all system owned addressesChanged it to only call remove_devices if a device is to be removed so that topology is not spammedAWARE-630 'Remote Bypass' Address clearAdded Address-Bypass addresses from 0-40 on the imac controller master.Changed the address-bypass values from True/False to 1/0 for ease of reading.Changed the range of addresses read so that the array length is definitely 40 with 0 always false.AWARE-655 'SCADA' Integration using EIP CIP LibraryAdded a custom device topic to local MQTT parsers. This is to allow devices to publish specific topics dependent on their operation. Hence this use case is for writing gas values to safe gas.Pointed process_safe_gas to an appropriate MQTT custom topic.Added data-flags register to safe-gas mqtt messages.Included the bypass flag at the end of the 16 bit value as a combination of local and remote bypass flags.Added int wrappers around address values in modules.Added comments to safegas function and data-flags initialisation so it is more easily picked up again if required.Enforced address to be an int.Added a way for local MQTT parsers to be defined in config. Removed the custom device topic from MQTT parsers.Ensured that the default local MQTT parsers are overwritten with config defined parsers if config defined parsers are available.Version 3.6.4.solarcubeAWARE-1673AWARE-1673 Updated timing of comap alarm clearing check to ensure it is always cleared in the event the alarms are fired in quick succession.Version 3.6.3.solarcubeAWARE-1487AWARE-1487 Updated the comap engine state and breaker state enums.Version 3.6.2.solarcubeAWARE-1474AWARE-874 Added Second Solar Inverter to Selectronic device specification and updated scaling.Version 3.6.1.solarcubeAWARE-891AWARE-891 Added a solar cube example config.Version 3.6.0.solarcubeAWARE-853AWARE-853 Added remote start/stop procedure for Comap Intelilite MRS-16.AWARE-853 Changed Selectronic alarm values to fix firmware update issues with Selectronic alarming.Version 3.5.9.solarcubeHOTFIX-AWARE-749AWARE-749 Updated requirements.txt to include known working versions of dependencies of hbmqtt 0.9.6 as per commit b2d75d7 for issue AWARE-742.AWARE-749 Updated client.py to include memory leak fix for hbmqtt. As per commit c3cd85f for issue AWARE-730.AWARE-749 Updated ipc.py to include gateway heartbeat fix. As per commit 41c13c7 as part of issue AWARE-693.Version 3.5.8AWARE-378 Modbus TCP Server CreationCreated Modbus TCP Server and allowed it to run successfully in isolation.Updated the modbus TCP server with dummy data, basic scaling and enum converting.Implemented the encode method for ParamCType and a test case for the method.Added support for Address maps in renaming keys. Created a test for this case. Created an eq method tied to Address maps that compares that the buffer objects are equal.Added a test case for ParamCType Decode method to prove that it will support one and two length modbus addresses.Added an overwrite option to the Address Map. This allows for current addresses to be overwritten if required else will raise a ValueError if there is overlap. Overwrite is set to False by default.Updated typehinting on transformation "rename_keys".Updated encode method for ParamCType to ensure the bytes and words are in the correct order.Added an update server function in modbus device to update the server based on a mapping.Removed overwrite option from AddressMap merge and added update functionality to the AddressMap.Updated the modbus protocol with a server translator to separate the modbus server from all devices.Added server translator to communications parsing.Fixed comments on address map merge.Reworked server and device translator.Reworked rename keys transformations to apply to individual devices.Added support for NemoD4Le Power Meter on Modbus TCP Server. Reformatted the file.Updated ParamCType encode method to work with data types that are not multiples of 2.Added an ethernet comms status for modbus devices which can be enabled through config. Enabled comms status for nemo-d4 Le power meter.Updated the nemo powermeter to update on change of state. Moved read handle to cope with disconnect and reconnect events.Allowed the device translator to set a poll interval for each device. Allowed the modbus client to tell all of its devices its connection state. Captured an error where the modbus device never recovered upon disconnection by giving it 1 second to attempt reconnect before next read. Renamed the horizon test power meter to match its cloud id.Moved poll_pipeline to outside of Attribute Error exception. This ensures we still capture Attribute errors of poll_pipeline in logs.Ensured integer is encoded on server translator.Added a parambits test for encoding and initialised the input address map with the parameter address so that bitwise operations can be performed.Added a grouping for parameters as a transform. This was specifically done to create a list of parameters to put into a bitmask. Added a test to test this function as well.Added an error message to the ParamCType encode method to improve debugging. Fixed an issue with the group params transformation where the list of parameters would not reset with each grouping and instead continue growing for each parameter required to be grouped.Added IPC heartbeat to modbus server. Heartbeat is stored as EPOCH time.Added the generator controller for horizon into device config.Added the generator controller to the modbus server.Added a test for ParamLookup.Added an option for bytes to be input into ParamLookup encode function. This was to allow ParamCType chars to be encoded into an ENUM without breaking the way it currently works. This is proven through the test.Added a comms status flag for the ipc comms with google cloud mqtt.Fixed a bug where if two instances of telemetry were running, validation checks would fail and not send data.Added comms status to mdobus devices, ipc cloud mqtt and sp pro devices.Updated modbus server table.Added a new datatype. This is to take an array of addresses and interpret each address as the same enum. If the enum is in the data set that parameter to true else return false for that parameter. Returns enum as a bool dict.Updated the SP PRO protocol to include service reason codes. Added these changes to the modbus server.Moved modbus server address initialisation to config. Built to mirror device address map definitions. Updated NotImplementedError for ParamEnumBoolArray.Moved server translator definitions to config.Updated selectronic scaling.Updated storage on sppro devices.Fixed a bug where the cloud never connected due to an ondisconnect event.Updated data type tests.Updated ParamCType encode method to ensure no errors arise with unsigned and signed chars.Updated storage triggers for Nemo D4-Le, SP Pro 2i, Intelilite MRS-16. Removed event trigger for serial comms status as it is not being used.Removed cloud comms status awaiting introduction of watchdogs.Added events enable to ModbusDevice.Made comms parsing functions that are being used async. This is to reflect changes made in previous revisions of pyaware.Updated poll interval for sp pro device and ModbusDevice. Set numeric type on modbus device and sp pro device by removing it from the if statement (was false on startup). Add read handle to modbus device with sleep time based on disconnect.Preserved the ModbusTCPClient protocol if it had been intialised already. Moved the read handle in Modbus device to account for this and client disconnect event in the read loop.Removed read handles. Replaced with read_registers, with a register type received from config. Added a client disconnect check while reading the modbus device, so that logs are not constantly generated from failed reads. Waits for restablished comms before attempting to read again.Added watchdog to monitor ethernet comms status of all modbus devices. Added watchdog to monitor serial comms status of SP PRO devices. Added watchdog to monitor cloud communications status with gatewayipc only available if a reference to the cloud broker is added in config.Added watchdog to monitor ethernet comms status on Modbus Device.AWARE-380 Add Selectronic Addresses to PyAWAREAdded selectronic devices to the modbus server. BMS parameters are the only parameters being read in at this stage.Added Solar Inverter to selectronic Modbus Server Table.Added the battery inverter addresses to the modbus server.AWARE-311 PyAWARE Slave Start Up TasksFirst pass at separating the definition between controllers (Which actuate the protocol) and protocol (Which is the line actuator)Recreated the imac2 protocol to be more protocol specific and used it in the imac2 controller logic.Updated the config file names and locations to work with the new setupRenaming the Imac2Protocol to Imac2MasterControllerDisabled RTS config parsing until it is fixedWait for the protocol to connect before doing a readCreated ImacControllerDevice for a stripped down imac functions.Updated imac parameter spec file to match the code version and testedFixed imac protocol testsAdded tests for rest dataImproved auto detect so it returns the imac2 master controller if the SLP version matchesAdded pytest mock to test requirementsAWARE-312 RTS/iMAC update telemetry topic names - SlaveRemoving the device_id from trigger_send and putting it as a parameter.Now can format topic with all **kwargs (including device_id)Command response also uses device_id as a parameter instead of a topic nameTopic formatting will now raise an error if it is missing a parameterAdded telemetry_serial which puts the serial number in the topic pathFixed tests so they don't break the event loop on run causing it to hangFixed issue where swapped bytes could leave a null character in the stringFixed a test where the serial number was not parsed to the controller correctlyAdded new reference parameter so that the gateway can pull parameters from another parsed communication device. Eg get the serial number from an imac controllerFixed the IPC to be able to send topics based on referenced devices with optional serial numbers in the topic namesRemoved set fieldbus address from masterRemoved start up fieldbus set from config for master.pyRemoved RTS dynamic load and ip address configAdded RTS controller deviceFixed an issue where the client host configuration files were writing up too many folder levels now that the architecture has changedChanged master-fieldbus-address location as we are no longer writing to it. This also keeps the same location between both the master and the rtsAdded update_topology to RTSUpdated IPC to store the recent topology messages for each unique top level device (ones that send topology messages).Added event message to ask for all the topologies for devices that are active in pyaware. This should allow us to invalidate old devices if needed.Added rts migration testsMigrating legacy imac into the tests to give permananceSome fixes from the testsAdded RTS module in tests so we can separate the RTS that will live on the imac bus vs the RTS reporting directly via a moxa unitSome fixes from the testsAdded recursive form parsing for deeply embedded parameters. Eg. bitmask parametersAdded recursive parameter parsing for listsRTS Config now matches the original polling dataUpdated the roll call to return the new RTS moduleSeparated controller logic from the device logicUpdated RTS triggers to send relay state more often and controller temperature lessFixed issues related to RTS roll call not having enough parametersReverted change by delta on controller temperature back to change of stateFixed issue where roll call address could be greater than 255 due to some imac quirkCommands now have an optional destination so that you can address imac controllers and rts controllers by serial numberAdded init method to all controllers and updated the _main method to call these on start-up. This allows us to sequence the start-up more sequentially allowing things like having the imac controller auto detect but not having messages missed because it read data and "sent" it before the mqtt broker was set upFixed issue with command destination not working for top level controllerChanging parameters so that the module type and controller type of RTS will reuse the same parameters.Fixed issue where roll call would fail if the bit cleared before checking the bit was written correctlyAll parameters on the imac module are now prefixed with l1 to indicate that the data is coming from the l1 busMoved imac controller device logic back into the RTSAdded static data polling to RTS so that parameters such as config addresses are publishedMoved commands for remote bypass and remote trip from RTS Controller to RTS Module as they cannot be actuated on the RTS commsAdded last trip parametersChanged all rts controller parameters that were prefix with l1 to postfix with l1 to avoid naming confusion with the parameters on the rts moduleAdded PLC heartbeat to RTSUpdated RTS device id to lower case to match topic namesAdded resolve_static_data_types so that things like rts-config-address can resolve before the rest of the config that may depend on this addresAdded static data processing to imac moduleAll rts parameters now coming through MQTTFixed the tests around RTS parameters to match what was agreed to for the renaming of parameter from before the migration.AWARE-313 RTS Slave listen for relevant commandsFixed issue where the trip reset was not correctly being executedChanged the command destination to match the logical name of the RTS. This is so the same destination can be used whether it goes via the imac controller via l1 bus or directly to the moxa on the rtsFixed a bug where porting the command from the rts controller to module did not fix the logical number not being part of an imac module.Fixed a bug where the switch status was looking at the dip switches instead of the input switchesAWARE-308 Changes as result of updated architectureAdded scaling to aim module power set pointsFixing rebase issues with config and storageAdded dev tag to the pipelines buildFixing rebase issues related to retain messages not being sentFixing rebase issues related to retain messages not being sentAdded batchable parameter to force topics to not batch even when batching is set to trueAdded write_by_serial_number_no_checkFixed testsFixed issues with refactoringAdded client id to storageFixed issues with batchable topicsFixed issues with batchable topics testsRemoved aiosppro driver from requirementsAdded exception count to starve on exceptionChanged watchdogs to more reasonable times for serial and ethernetFixed bitbucket-pipelines.yml develop build to have a more semantic versioningAWARE-411 Update comms status to push into heartbeatAdding watchdog structure to set up watchdogs for communication statusesWIP aggregated watchdogFixed issue with warnings being shown that the event wasn't scheduledStopping development and tests on aggregated watchdogsAdded watchdog managerAdded watch and watch_starve decorators to attach to a functionAdded string formatting for watched dogs to match based on instance variablesAdded watchdog call backs to be asyncAdded more tests for watchdog on classesImplemented watchdogs on imac controller and rts controllerUpdated tests to be less flaky for watch dog fed testsAWARE-414 Gateway state should be retained due to low frequency of messages on that topicChanged form_message to support retained messagesChanged state and state_serial topology messages to retainAdded mqtt connected status that will send gateways/{device_id}/mqtt with {"connected": true} once setup and {"connected": false} once disconnected using the last will and testament mqtt connection packetAWARE-415 Gasguard sensor type sometimes displays as unknownRemoved detector-gas-analog as per requestChanged behaviour so that gasguard 2 doesn't send detector symbol units etc. Unless explicitly having read the detector type firstChanged start-up behaviour to read detector-type on startup so that gas values can be pushed through asapAWARE-416 IO errors during roll call can stop it completing or retryingAdded find-system command which will do a full system roll call like what is done at start upCatches any error during system roll call and reschedules another system roll callEach failed address based roll call will now reschedule that address to be roll calledAWARE-417 Mqtt status message doesn't match the serial number type from other gateway messagesUpdated mqtt broker to be able to use serial number as well as the logical device_id. Prefers serial numberFixed temperature, humidity scaling on GG2Fixed power set point alarms on aimAWARE-425 Feedback for Release 1 - pyAWAREDetector gas value now sent on change of state or every 6 minutes. Only send if valid sensor head connected.Added address-offline-count and address-clash-countChanged address-resistance to have 0 resistance at address 0 instead of EOL resistance as it is misleadingChanged l1-data-block-just-complete to send every 60 secondsAdded write triggers for hysteresis configFixing an issue where sometimes the iMAC will report some bits in the high register of the address field causing the address to display wrong and it error on reads.Fixed issue with rtd1 where address-flags and address-temp were the same addressFixed issue where SIMP would not send the addressAdd write validator triggers to RO4 to fix issue where the energize mask wouldn't writeProperly shift the relay timersFixed issue where relay timers could not be disabled (set to 0)Fixed SIM-P command for updating parametersChanged t90 test result to scale to secondsFixed issue where the detector-gas-value was returning as a list of valuesRemoved legacy RTSScaled last-t90-test-resultAWARE-438 Slave Testing FeedbackFixed a bug in non-batched mode which didn't have the appropriate topic parserTurn off batching by default in local config as telemetry test results were delayedAWARE-441 Store and forward topic retrieval is pushing topic type back instead of topic nameFixed issue where the backlog was storing the topic type instead of the topic in the topic fieldAWARE-440 Exception count watchdog addition doesn't seem to workDiscovered there is no issue with the starve on exception calling code. Added a testAWARE-439 Aiomodbus doesn't work after rebaseAdded aiomodbus functionality to gglive just need to specify the client port in the config fileAWARE-442 Add ability to mark certain topics to not be cached on disk such as heartbeatsAdded optional cache parameter to configAdded feature to allow some messages to not get inserted into the databaseAWARE-432 Make rotary switch changes on RTS controller kill the pyaware processAdded rotary-sw event to kill pyaware if the switch position changesAWARE-444 Timestamp no longer encoding in latest developUpdated ctype encode method to work with multiple parametersUpdated postbox reader to correctly return the parameters readAWARE-443 Add ability to only batch up responses from backlog and not from normal telemetry per topicAdded new parameter to mqtt config "batchable_backlog"Changed the backlog and batch message manager to discriminate between backlog messages and normal messages for the purpose of batching.AWARE-446 Memory leak on slave when MQTT disconnectedFixed issue where telemetry was v3 for local telemetry as uid are not providedFixed issue where memory leak happened when the mqtt broker was disconnected because it would block on adding it to the message manager queue and basically all messages remained in memory until the mqtt was clearedAdded basic memory line profiler to command line argsChanged the memory profile directory to match the cpu profile directoryOptimised form message by removing "remove_keys" and delegating it to the pydantic library. Reduces cpu usage from form_message for telemetry by 66%Remove deepcopy for memory storage element for increased performanceImproved run_triggers performance by remoing iteration on every data item if it doesn't exist in the triggersParsing status was consuming significant CPU cycles. By moving to a lookup table we use slightly more RAM to gain big CPU improvementsYanked network cables can cause cancelled errors to bubble up from underlying libraries. Catching this for purposes of cleanup can cause half shutdowns. Now relies fully on the stop event being set to clean up.AWARE-451 Changing detector type head result in very delayed updates and false representation of dataFixed bug where imac modules would subscribe to parameter triggers multiple times if they were roll called more than once.Added update detector method to GG2 to force scans of detector type when a hardware fault is clearedUpdated the detector type to unknown during a hardware faultAWARE-472 DI4 and RO4 are inconsistent in the relay representationFixed issue where relay status was inverting when RO4 does not actuate the line.AWARE-482 RTD Module causing Phantom RTS to appearAdded tests for roll call by addressAdded failing case that must pass to change the RTS detection to rely on all addresses being online in the schema.Added multiple address lookups for rts to improve robustnessAWARE-489 Topology Message is being sent to the cloudUpdated the release notes. Removed topology message from gcp parsers.AWARE-454 Feedback for testing round 27-10-2020Added retry to set parameters for the mismatched parameters for if the parameters in the module are not correctly set before being read backAdded state parameters to process_static_data so that current state objects can be sent on a poll loop.Change logical_number to a state parameter so it doesn't change during the lifetime of the processUpdated gasguard triggers to reduce storage usageAdded error for when there is no module as the indicated addressAdded test for bound method watchdog callAdded gas-value-invalid as it can be set without the hardware-fault being set when removing the gas head.Added starve on exception for imac_eth_status and imac_ser_statusImproved response time for serial communication status to come back online. Reduced heartbeat 30->10 secondsOnly create RTSs if no modules are roll called and the address is onlineAdded check in system roll call to ensure that the online address in the RTS address range is actually an RTSChanged from gas-value-invalid to detector-data-invalid for detection of change of headChanged imac_module triggers to resolve all event triggers and modify the device data based on the return values. This allows things like changing the detector type to 0 to propagate into the current data processing.Fixed issue where rts was returning as upper case instead of lower case making commands not workFixed issue set parameters were returning more parameters than were set was raising a mismatch errorAWARE-430 Add write commands to Modbus TCP ServerFixed missed error handling where if there was an attribute error in a comms object it would pass. Now it will be raised and the program will exit.Added process write to modbus.pyRemoved unused imports.Moved translator to translator.py and reworked translator config.Fixed encoding method inputs in translator and modbus writes.Moved SP Pro device and translator to controllers. Updated translator publish and subscribe events to reference the id of their objects rather than their device/server ids given in config. This was to ensure uniqueness regardless of config.Updated comap intelilite addresses from commissioning data. Added the ability for modbus devices to read discrete addresses from commissioning discoveries involving the binary input addresses on the comap intelilite controller.Updated comap intelilite addresses from commissioning data. Fixed the discrete registers for pymodbus as they are read in as bits not addresses. Changed wording of "modbus-eth-status" to "modbus-comms-status" as serial devices use the same connection.Tidied up read handle function on ModbusDevice. Added a TODO to remove pymodbus read functions in the future and exclusively use aiomodbus.Fixed a spelling mistake with the pymodbus async wrapper for reading coils.Added the ability for the ModbusDevice Client to use transformations. If no transforms exist in config the transformation pipeline will just pass the data through unchanged.Fixed a bug where ParamCType's were not being encoded if data was a char.Added support to aiomodbus for writing to addresses on ModbusDevice and integrated change with pymodbus in the form of wrapper functions. Tidied up translator write & modbusdevice process_write code.Updated SP PRO device with same device process_write changes as the ModbusDevice and reformatted with black.Updated write on sp pro device with comments.Updated Selectronic and ComAp Gen Controller from commissioning.Updated ParamBits encode method to work with previous pyAWARE changes.Added a ParamBits encode change to ensure that server can read and write to these params. Changed the ParamBits test to include a decode aswell testing full functionality of the parameter. Fixed a bug where the blocks to write to where unordered causing errors in writing to modbus addresses.Renamed pymodbus wrapper functions to indicate that they are private functions and part of pymodbus. Removed comments surrounding these.Updated translator to use pyaware.transformations.transform.Updated SP PRO config to include bms current param multiplication.Hooked into Pymodbus set values method, to take advantage of server wirtes.Added write addresses to both the Intelilite MRS16 and sppro 2i.Fixed a bug on the ParamCType encode method where chars were not encoded back to integers for sending. This might need to be revised to determine whether unsigned or signed chars are best in this case. Unsigned chars have been chosen for now.Added a new config reader to transformations.py to allow transformations to be built from a reference dictionary.Added a reference for objects associated with translator.Added modbus server writes to the translator, modbus devices and sp pro devices. Allowed for different modbus server register types.AWARE-473 AIM power set points command isn't scaled correctlyChanged aim power set point write triggers to match scaling on commands receivedFixed issue where scaling has to be done with integersFixed issue with ParamLookup datatype erroring if the parameter is not presentAWARE-493 RTGM - Test RTD and LIM ModulesUpdated rtd parameter specs to match datasheetCleaned up RTD module to use the imac module defaultsChanged comp_value in RTD1 to comp_ref_value to directly compare the valueFixed issue where voltage-l1 was reading as address-flagsFixed issue write validator was using the wrong comparison valueFixed voltage-l1 scale for rtd1Fixed ParamCTypeScale for encodingAdded some data_type testsAWARE-496 RTS module in imac are all responding to the remote bypass command. Make sure that the remote bypass is targetedFixed issue where remote trip and remote bypass were sending the command to all RTSs including phantom onesAdded l1 scan debug parameters. Sending on change of state as they only update at the end of the polling cycle.Fixed topic nameAWARE-497 Data returned from devices not on the bus after being disconnected physicallyAdded "remove-devices" command that can remove all references to the device from the polling and autodiscoverAWARE-530 Add Support for Bender NGRM700Add Device Bender NGRMAWARE-552 Industrial PC controller (IPC) causing cannot allocate memory error when getting IP addressSetting close_fds to True in case it isn't the default in order to potentially fix a file descriptor leak.AWARE-460 Horizon Solar Cube CommissioningAdded parameters to IME Nemo power meter for horizon's complete modbus table.Updated intelilite MRS16 with complete horizon modbus table parameters.Added complete horizon modbus table parameters to the sp pro battery inverter.Removed send triggers from SP PRO parameters that are required locally but not for the cloud.Removed send trigger from nemo power meter parameter as it is not required for the cloud, only locally.Removed send triggers from intelilite_mrs16 parameters that are not required for cloud implementation only local.Removed unnecessary parameters from intelilite_mrs16.yaml and sppro2i_spmc482_au.yaml.Removed change of state triggers on parameters that did not need to be monitored continuously. These changes were made on intelilite_mrs16.yaml, nemo-d4Le.yaml and sppro2i_spmc482_au.yaml.Updated intelite MRS-16 config file to correct formatting errors and remove address blocks that are not required.Removed unused address blocks from sppro2i_spmc482_au.yamlFixed address blocks and data types for parameters in intelilite_mrs16.yaml and sppro2i_spmc482_au.yamlRenamed watchdog ethernet to watchdog comms on ModbusDevice as comms encompasses the serial or tcp nature of the ModbusDevices.Added a new datatype for decoding due to changes with the generator controller. Takes a list of string block addresses and determines whether an identifier is in them to set the parameter true or false.Removed ParamTextToBool as it is no longer required.Added alarms list to intelilite_mrs16.yaml config file so we know what alarms to trigger the fault reset alarm on. Added the fault reset button to acknowledge triggered alarms. Changed discrete poll to coils poll so they can be written to if required.Changed the process_write function in ModbusDevice to allow writing from different registers. It will now use the handle given to the device in config to determine which register type to write to instead of using the register type it is given via events.publish. This way it is always guaranteed to be written to the right address.Added a comap specific device to trigger on alarm events to acknowledge their occurance.Updated the nemo-d4Le and intelilite_mrs16 to allow comms status to be retrieved via GCP.Updated the intelilite_mrs16 to allow fault reset.Added intelilite_mrs16.yaml fault reset command procedure to acknowledge alarms.Added sp pro parameters that are being used on the CPS test device. This is to mainly maintain consistency.Added warning logs for the Comap Alarm Acknowledgement process.Moved fault reset command to the device class.Added a metadata field to parameters. This is parsed separately from other parameter information. At this stage parameter kind is all that is stored here.Moved ComApInteliliteMRS16 password definition to gateway.yaml as this is dependant upon user information.Move the comap acknowledge alarm function into two logical pieces. The command execution and the alarm acknowledgement. Added function comments to comap commands.Removed subprocesses from gateway heartbeat in replacement of ifaddr package. This is compatible with Linux and Windows.AWARE-555 On Fieldbus value change reboot SlaveAdded stop on change of state for rts: fieldbus, serial_number, logical_numberAdded stop on change of state for imac: fieldbus, serial_numberRemoved telemetry for rts: fieldbus, serial_number, logical_numberRemoved telemetry for imac: fieldbus, serial_numberAWARE-562 Update to HeartbeatsAdded mac address param to configChanged mqtt config to behave like a dictionary when unpacking (**self.config)Changed the topic names to include the gateway_id in local topic namesSub topics and form message now can use all the parameters in the mqtt configRefactored mac address to not use subprocess. Non-linux ignores the interface parameter and bases it off the primary network adapterAdded break to interfaces so it doesn't go through all the interfacesRemoved mqtt retained status messages as per requestFixed issue with watchdog where an exception on callback would cause pyaware to get stuck in an infinite loopChanged the default callback function in watchdog to be a callable that does nothingAdded an Empty Payload model which returns a 0 length string no matter the message contentAdded device heartbeats with data sourceAdded device watchdog heartbeats to imac and rts controllersFixed heartbeat not working on imac masterFixed devices heartbeat topic prefix from gateways to devicesChanged the command topics to use the logical name with serial instead of the gateway id as the front end doesn't know where it is targeting.AWARE-583 Gateway IP Address not validating correctlyEnsured that the gateway fetch_ip address function returns the IP address as a string, not the IP object.AWARE-526 Test Fault Reset on Gen ControllerFixed a bug where await was called outside of an asynchronous function.Stopped modbus reads for write only addresses.Updated fault reset addresses and removed the need for a password.Updated comap fault reset functions.Updated comap alarms.Updated comap alarms to include global alarm acknowledgement parameter.Updated the alarm acknowledgement parameters.AWARE-615 Test 3.5.8 releaseAdded tests for newly added transformation functions.Added config read tests.Added tests for comap commands. This is currently implemented in the Horizon Power Solar Cube.Version 3.5.7HOTFIX-AWARE-574Added a config flag to disable initialising the backlog manager.Version 3.5.6HOTFIX-AWARE-544Corrected insulation-fail mapping in IPC Deviceadded outlet-run in IPC DeviceVersion 3.5.5HOTFIX-AWARE-540Replaced the pip search with raw html parsing of pypi.org/simple/pyawareAdded beautifulsoup to requirements to parse the raw htmlAdded index_url optional parameter to config file to set where to search and update pyawareAdded try_except block around updates to ensure that a failed update will still run pyawareFixed tests to not lock up when calling init_dbChanged regex to use raw stringsVersion 3.5.4HOTFIX-AWARE-488Added a DbVersionManager to do database migrations.Made the default migration to delete the current database and start fresh.Removed the WAL files as well to ensure that transactions in the WAL are not committed to db.Version 3.5.3HOTFIX-AWARE-489Deletes store and forward db on startup to clear old messagesVersion 3.5.2HOTFIX-3.5.2Removed topology messages from default gcp parsers.Version 3.5.1HOTFIX-3.5.1Remove aiosppro from requirements.txtUpdated aiomodbus in requirements.txtVersion 3.5.0AWARE-286 Store and forwardRenamed storage to memory_storage to better indicate that it isn't on-disk storage- Moved memory storage to it's own file- Preliminary skeleton structure for on disk management- Added delayed commits to prevent every message being pushed to disk- Added UUID to the disk storage- Added UUID to the disk storageUpdated acking to delete the reference in the dbAdded UUID as unique constraint to the DB and ignored duplicate entries as already being ready for recall in the database (allows for new attempts at messages to not need to touch the files)- Moved most of the publish disk storage logic to the send methodSeparated batch send vs single send methods (batch send yet to be implemented)Now correctly sends single messages and puts into DB when delayed and clears once sent- Batching and cache to disk now works for standard mqtt send commands- Changed RuntimeError to StopException to stop catching RuntimeErrors that were not anticipated- Added max queue size for batched messagesAdded task_done calls to the queue for better clean up in the future- Renamed hbmqtt to client as we are only running one client variant nowMoved the batch and single message handling into their own classes to make it easier to test and reason about- Added id back in as a primary key for easier sortingGet all unsent now has batched queries and returns a dictionary- Changed from sqlalchemy to aiosqlite improves performance by order of magnitudeMemory usage is optimised for get_all_unsent 1000 batch_size is about 5MB extra memory see Jira issue comments for benchmarks- Changed default message retrieval from 100 messages to 1000Added unique constraint on UUID- Implemented the backlog managerRefactored the batch processing so it can do batch hold off for each topicAdded max in flight messages limit to prevent ever exploding RAM when doing the backlog processingAdded configurable path locations from configAdded disk storage managerSQLite set to autocommit mode so that simultaneous queries will not all join into the same transaction.Database Size Manager can hard limit the size of the databaseDatabase Size Manager will delete a portion of the old messages as set by clean_up_portionFailed inserts that hit the hard limit will wait for the clean up and then retryAdded clarifying commentsAdding storage testsAdded pytest mock to test requirementsAdded test and fixed bug with page_count calculationsAdded test for database clean upAdded storage clean up testsAdded mqtt insert testMoved from global reference to disk storage to explicit function reference parsed into the classAdded aiosqlite to requirementsAdded in flight messages limit to handle single messagesAdded aiosqlite to requirementsAdded in flight messages limit to handle single messagesMoving more of the publish logic to the message manageClearing in flight messages on disconnect so they get resent after the device setup is completeReworked setup event to work in message handlerAdded disconnect callback to client to do clean upFixed bug where I accidentally had disconnect code in the connect code meaning that it would never connect properlyBacklog manager now waits for the mqtt device setup to be on before queueing messages to sendImproved logging for backlog manager to show how many messages it queuedFixed a bug where the scheduled disconnect would conflict with the on_disconnect tasksRefactored message managers to use a base classChanged modbus timeouts to not log tracebacks as it was polluting the logsAdded ability for hbmqtt gcp to parse the config file parametersFixed issue with pyaware not correctly shutting down due to excessive exception capturing. Look to removing all BaseException captures when the errors are known.Added batching to GCPAttempt to fix issue with windows not getting the clean up signalsRemoved sqlalchemy from requirements as now using exclusively aiosqliteUpdated black formattingAdded database cleanup so it doesn't hang on shutdownAdded default config to database initialisation to store in memory and limit it to 500MBFixed bug in store and forward where the CancelledError was raised at the top level not allowing the poll loops to continue after a cancelled failure. This was introduced trying to help the clean up functions.Added better error handling to attempt to fix issue related to backlog messages not being sent on errorAWARE-348 Telemetry Test CommandAdded command response model and timestampFixed strange bug where imac address would have a bit in the high byte set causing the whole roll call to not work. Needs to be fixed on the iMACAdded reset to idleAdded unsafe parameter writing to get a more accurate start time for telemetry testAdded timestamp to telemetry test the moment the data is set to start the testAdded checks to make sure actions don't take place when another action is ongoingFixed tests to account for timestamp in commandsAWARE-377 Add Power Meter Nemo D4-LeAdded Power Meter Nemo D4-Le config file.Fixed addressing and parameter naming conventions.Removed scaling from nemo-d4Le config file. Scaling for this is moved to the front-end.AWARE-379 Add Selectronic Protocol SP-PROAdded an SP Pro protocol device to device protocols. Updated config to parse this device from config.Added a mock SP PRO device which is imported in place of pyserial. This is synchronous for the time being but replies with the expected responses. NOTE: This has been tested to work in place of an SP Pro device.Added a test for the crc calculate function and read request creator function for the SP Pro protocol.Added a read response validator for SP PRO protocol.Updated the SP PRO device reference. Added a parse in config for the SP PRO client. Added aiosppro to list of dependencies.Added Selectronic SP PRO 2i Battery Inverter SPMC48-AU config file.Updated selectronic sp pro config file to fix formatting errors.Added a reading poll interval to the SP PRO device mirrored from modbus device.Added additional error checking for SP PRO read.Added identify method to the SP PRO device so that it will appear on Topology call.Updated trigger_send to prevent errors.Updated the sppro config file to fix missing parameters and allow transformations.Added transformations to allow pipeline functions to eb performed on each device.Added a transformation pipeline on the SP PRO device. This allows for scaling, multiplying and removal of keys.Removed SP PRO tests from test files as they are included in aiosppro.Updated formatting.Updated config file to remove the remove keys stage from transformation pipeline and added in change of state.Added comments to transformation functions.Version 3.4.0AWARE-136 Telemetry TestFixed a bug in action-remote-telemetry-test where there would be an error if the payload did not include data.hbmqtt porttopic name filters and fixesTopology now working in topic separationCommands and command responses now working in topic separationAWARE-230 iMAC module additionsAdded ability to parse config file format into parameters grouped by data sourceAdded spec files for iMAC to approprite directory (Not moved yet as still moving over parsing the configs)Added Parameter form to imac_module_parameter_spec mirroring the generic modbus parameter specRefactored block_specs into parameters["block"]Updated default iMAC module to parse and store the config in the deviceUpdated default iMAC module to parse the param data types from the config file when presentRefactored imac controller and modules to use the config files in devices/ampcontrol/imacFixed issue in events where if there was no futures it would error when calling first or allAdded Lim parameter spec fileFixed issue where ValidateParams would not validate due to block_specs no longer existingAdded proof of concept trigger for more trigger usesTransitioning triggers to more powerful setup to accommodate the LIMMoving triggers to legacy by default until new triggers implementedRefactored triggers to return bools and update process_triggers to produce the desired outputNew triggers with legacy definition working with modbus protocol test deviceAdded comparison and and triggersUpdated Lim to use comparison triggers so that digital and analog parameters are only sent in the appropriate modeMoving imac to new triggersFixed triggers for imac modulesAdded ability to add references to other parameters for the parameter form argumentsUpdated the base ImacModule to parse the parameter specs on the update_specs() call so that address information can be encoded into the parametersRemoved event trigger from lim mode as the event methods are no longer neededAdded Sim-P using new address based parameter config parsingAdded missing meta field for modbus devicesFixed issue with default imac module referencing data outside the modbus block rangesFixed some issues with config file data_type parsingSIMG is now implemented in configAdded DO4Added AIM ModuleFixed issues with detectors being mislabeled as infra-red when they were actually electrochemicalRemoving legacy codeRefactored triggers into separate filesRemoved legacy triggersFixed issue where process triggers were missingAdded write validator triggers so that the set-parameters command can use these triggers instead of custom code per deviceAdded write triggers for set-parameters command on imac modulesUpdate set parameters command to update_specsFixed config file where process triggers were blankFixed issue where write triggers were not being parsed correctlyChanged scheduled reads to retry once on failure and then schedule for 10 minutes in the future if it still fails. This ignores the configured deadlines in the config filesFixes for parameter scheduling updatesAdded 10 minute deadline to failed readsCollect triggers refactorFixed an issue where the trigger_heartbeat did not stop when pyaware was meant to stop causing the program close to hangMade auto discover more reliable by logging warnings instead of stopping the process when modules do not find the missing starting data. This can prevent issues where a module misbehaves for the missing data (eg. a GG2 without a sensor head)Updated scheduled reads and collect triggers to work togetherChanged write triggers so it can process multiple triggers in a listChanged the validation error to InvalidCommandData so it raises the appropriate error code for the commandFixed issue where it would error if the command was already processed (edge case of repeated command sends)Code lintingAdded read only parameter blocks for set point and display format configuration for SIM-GFixed process comparison function namesAdded write comparison triggerUpdated comparison trigger to be able to trigger against set values instead of just the command valuesUpdated the LIM to have write validatorsUpdated the SIM-G to clarify that it is a modbus slave address andAdded write triggers to SIM-GChanged range_int to range to closer represent that it now supports stepsUpdated RO4 to use new parameter spec fileRemoved RO4 code that is now handled by configFixed issues parsing by comparison found during testingFixed RO4 definitions to match datasheetFixed scaling range when scale was NoneFixed misnamed parameters in sim-gAdded flag parameter triggersRemoved store triggers when the send trigger was alwaysAdded better feedback for failed lint checksFixed indentation issue with black linting that didn't appear on windowsMoxa only has openssl 1.0.1 installed. Cryptography 3.0 requires 1.1.1. Reverting until we can compile latest openssl for moxa boxAWARE-258 Bugfix Gasguard Timestamp mismatchConverted from ampcontrol timestamps to using integers packed into 32 bit registers for the timestampRemoved unused importsConverted from ampcontrol timestamps to using integers packed into 32 bit registers for the timestampFixed issue where the timestamp didn't subtract the diff but just added the current time to the stampCode lintingAWARE-267 Add pyaware linterReformatting with blackAdded code format checks to the build processAWARE-270 Bugfix Linearity test time due no longer worksRemoved linearity-test-time-due parameter. We can’t send through these values with the Moxa / Server architecture we will have. So we will remove this telemetry value (which was originally a helper function for the front end). The front end will now have to compensate it with the timestamp parsing logic that was done as part of AWARE-258AWARE-271 Bugfix topics missing slash when moving to hbmqttAdded leading slashes backAWARE-272 Bugfix roll call system gets stuck in some hardware arrangementsFixed issue with roll calling that could cause the imac2 to get in an infinite roll call loop. There is now a conditional lock during roll calls, enforces block reads only from the current generation in the roll call sequenceAWARE-273 Bugfix Commands with hbmqtt don't workFixed the commands topic to include the device_idAWARE-276 Cleanup unused code and packagesFirst pass at deleting unused codeRemoved sync vs async where it would swap based on the execution pathUpdating requirements to remove all the additional packages and update the versions in order to test compatibility breaksRemoved some dead scriptsReverted the removal of pymodbus config parsing as it is used in imac2 currentlyAdded cryptography back into requirementsCode lintingAWARE-286 Store and forwardAdded write validators to GG2 for set-parameters commandFixed issue with gg2 config errorAdded generic get parameters commandAdded address-status-raw for unparsed address status readsChanged the module status parser to be more explicit and only look at L1 lineL1 Mask removed as hard coded binary number and added as bitwise combinationChanged parse_status to match the logic in the javascript implementation on the imac controllerChanged parse_status to match the logic in the javascript implementation on the imac controllerUpdated parse_status to cover all test cases and changed the logic to matchAWARE-293 Remove leading slashes for topic namesRemoved leading slashes in local topic definitionsAdded send gateway heartbeat function that operates under imac2 protocol for now. Sends IP address (hard-coded for now) and timestamp every 30s via MQTT.Error handling required for the function in future commits.Function to be moved to gateway protocol in future commits.Added error handling & logging around gateway heartbeatAdded ip address fetcher to fetch ip address from device command line. This is currently only supported on linux systems, in future to be supported on windows systems.Updated the Topology MQTT message to include a top level Gateway device with the devices attached to it. Added support for generic gateway_ipc Added fetch function for gateway mac address. The gateway heartbeat in IMAC2 is to be removed, but is there for the sake of testing.Removed gateway heartbeat from IMAC protocol into GatewayIPC. Updated device id of gateway to be the set id OR the id of the device it is connected to. Use Case: RTGM. Updated the topology to be updated with the IP-Address of the gateway.Identify gateway during post init to remove invalid unconfigured mqtt messages through heartbeat.Updated formatting -> blackUpdated MQTT GCP topics to include state of device/gateway that is to be sent on startup or on change of ip address.Added topics topology and gateway heartbeat to gcp mqtt to reduce errors. These can be removed if required.Added device state to MQTT local parsers.AWARE-304 Slave ip address reportingRemove slashes from topicsAWARE-329 Bugfix AIM module not sending through analog dataSend through data-analog and data-powerAWARE-330 Remove Slash from State topicRemoved reference to a dictionary where the data expected is an array.Removed the gateway heartbeat from GCP Parsers as this is not required in these applications.Added a check in topology factory to ensure that devices are in a List.Refactored the topology message to ensure that the message is formatted correctly through the use of a pydantic model.Updated checking if device has any data.Updated identify method on modbus to work with current topology frameworkUpdated formattingChanged topic name to gatewaysAWARE-343 Update the module status to match the iMAC 2 webpageFixed bug where numbers bigger than 8 bit would cause an errorAWARE-346 Add build pipeline for develop and PRsAdd build steps for develop and all PRs (no pyz builds)Added branch name to the pyaware version stringAWARE-350 Bugfix IPM modbus mapping trip status 1 and 2Updated trip status 1 & 2 to the correct trip bits.Reformmatted code for pipelineAWARE-351 Bugfix Add Relay Running Status to IPMAdded Relay Running Status to IPMAWARE-361 Bugfix DI4 edit address not workingSimplified DI4 to more closely match RO4 configuration.Fixed issue with edit address not workingUpdated black formattingAWARE-374 Bugfix Gateway state message not sending to cloudFixed a bug where the state message was not sending via GCP.Fixed a bug where if a message had a topic that wasn't allowed for, it would attempt to send anyway.Fixed formatting.Version 3.3.0AWARE-164Fixed issue where adding a topic during a publish would cause a RuntimeError due to the dictionary changing size during the publishAdded system wide roll call on start-upDelete various sync implementations (we only use async) to clean up codeAdded special lock synchronisation object that will publish internal topics when acquired and releaseAdded roll-call-active to this new lockFixed issue where some parameters were returning 0-1 instead of False-TrueFixed issue ParamMaskBool would fail if the decoded data didn't existAWARE-229Added new fault statesSeparated safe-gas integration as its own parameterRemoved detector-analog-diagnostic-state as it is a duplication of gas-value-invalidFixed issue with logged error raising it's own errorChanged find_missing_addresses to find_missing_starting_data.Added base method to iMAC module for find_missing_starting_data that uses starting params so that each module doesn't need to implement the method individuallyAWARE-230Added B30 configOptimised ranges to read less dataRemoved current-in-thd as per ben requestVersion 3.2.0AWARE-161Refactored code to allow for multiple sources to schedule readsAdded postbox read to gasguard 2Added postbox triggers for reading and sendingRefactored timestamp commands to use the new read parametersRemoved sync read/write versions from modules.py as they are no longer used and were clogging up the fileRenamed telemetry-time to telemetry-test-time to be clearer on the parameter namingRenamed calibration-time to linearity-test-time to be clearer on the parameter namingFixed test due to changed command nameAdded linearity test due parameter which is 6 months from the last linearity testRefactored modbus config parsing to be able to parse multiple sources and specify which modbus read type to useChanged from partial functions to a factory functions as partial functions were showing warnings during shutdown that they were not scriptable (This issue is fixed in python 3.8)AWARE-206Cleaned up modbus read handler so it doesn't need to be grabbed each loopAdded EMAX2 config fileAdded ekipelsig.yamlFixed some scaling factors to match datasheetAdded m2m.yamlFixed modbus blocks as there are empty registers in m2m data tableParses through config parameters so you can point to a networked network brokerAdded frequency to m2m as it is in cloud but not in confluenceFixed word order and frequency scalingRemoved scaling in order to do it on the front endAdded the correct block read for total positive energyRemoved scaling from EKIPELSIG to move to front end scalingAdded little endian word order to EKIPELSIGRemoved scaling factors from ABB M2MAWARE-212First draft of IPD.yamlFixed issue where sum could cause an error when summing a stringUpdated IPD addressing to match datasheetFixed references from IPB to IPDApparently IPSIs are weird and ignore your starting address. Eg. reading from address 258 as your start address will actually return address 257 but reading from address 257 will return 257. I've adjusted the block start to address 1 so that the modbus addresses line up the data sheet.Updating IPB and IPC to use the numerical over current curve instead of stringsRemoving sum aggregate as it should not be done here but should use aggregation frameworkUpdated over-current-curve to Param as it is now parsed on the cloud as an enumFixed IP relay mappings for IPSIAWARE-214Fixes for unreliable reads from the m2m locking up the asyncio taskAWARE-217Added voltage-3-phase and phase to phase currents for m2mAdded phase to phase currents for i33Added phase to phase currents for i35Added PM710.yamlFixes for PM710.yaml based on testingAdded new datatype to support the signed power factor unsigned intAWARE-220Fixed issue with imac protocol not using the correct aggregation methodFixed issues with imac modules not parsing the config file for aggregationsFixed issue where I was assuming that all params had an idx. ParamBits did not. Changing it so it doesn't rely on that parameter.Added frequencyFixed modbus blocks to stop illegal address errors due to devices not having modbus blocks throughout the whole address spaceAWARE-227Updated to use modbus library with fixes to timeoutsUpdated modbus to parse through address_shift to fix issue with IPDs not readingRTGM-232Added aggregations to build from configAdded aggregations to devices before sendingUpdated the MQTT models and factories to accommodate the changesChanged the default aggregations to the test device to check all the aggregation typesAdded some config testsUpdated dev_gcp config to use manual aware_versionAdded the aggregation testsModified the all aggregation for more performanceUpdated config files to always store data for substation monitoring devicesAdded more tests for auto update so that dev versions don't get updated if they are a higher version but do update if it is the same versionUpdated the storage triggers on most imac modules to take advantage of the send aggregationsUpdated config files to no longer do unnecessary storage triggers as the send trigger will now correctly store the current data.Version 3.1.0Cloud and modbus reliability fixes and optimisations.Auto restart added.AWARE-103IPM Config fixesToken disconnection and qos fixes for hbmqttAWARE-119Dev setup to test the mqtt disconnect logicCreated an auto disconnection after a specified interval from configFixed an issue where the performance degraded over time. Every time it reconnected to the server, a new virtual device was created. This meant that each reconnect would send more and more messages, breaking the devices over timeHides the error that happens when the client schedules a disconnect before the jwt expires.Added token life parameter so that the token expiry can be set via configUpdated dev config to match the local and cloud brokerBackported the virtual device accumulation fix to paho clientImproved logging showing resolved messages. Still seems to duplicate logsIncreased token life back to 60 minutesAWARE-142Fixed issues with modbus race conditions (new aiomodbus)Fixed issue with subscriptions from GCP errorsChanged the modbus device poll to only sleep the poll interval after the first round of readsChanged logging config to print to console instead of file due to unresolved issues with loggingUpdated H035 to have config files store always to fit with new dev version of send aggregatesUpdated aiomodbus requirementsRefactored to optimise imports and hopefully speed up start-up time on Siemens boxesOptimised imports so pymodbus isn't imported unless directly requiredOptimised mqtt client to no longer need virtual devicesRemoved issue where jwt token would generate twiceOptimised subscriptions into one packet instead of 3 per deviceAdded logging for the config initialisationRevert version bump commitAdded optional profiler to aware to debug H035 issuesAdded optional profiler to requirementsAdded profiler run for H035 for debugging purposesUpdated config handling to cache the configurations read until a configuration save happensUpdated hbmqtt config handling to do string comparison instead of element by element to save CPU timeRemoved json prettier formatting for output logs to optimise CPU usageFixed bug where the send data was sending empty payloadsFixed bug where lru_cache was returning the exact same object dictionary which would modify it across all devices resulting in devices not having parameters parsed correctlyAdded triggers config load to the use config loaderChanged json encoder and decoder to use orjsonFixed issue with copying lru cache not having ability to clear the cacheReplacing orjson with rapidjson as orjson doesn't have a precompiled 32 bit wheel and requires rust installed to compile.Changed rapidjson to an optional import as the performance benefits didn't play out for our current workloadsAdded profiler dumps for H-035Updated devices to always store so that the send aggregations reflect the samples takenAWARE-158Fixed naming for current-thd parametersAWARE-175Fixed modbus block range to include the full total-positive-energy registerAdded phase to phase voltagesAWARE-172Removed deployments folder and moved tohttp://prwivsvn0.primatec.local/svn/automation/AmpcontrolIOT/Device Management/DeploymentsAWARE-174Removing deployment data from H035 merged requestAWARE-175Added pm5110.yamlAWARE-191Fixed issue with logger that did not correctly override the root logger settings to log to file and consoleFixed issue in hbmqtt where the resolved mqtt message was not showing in loggingAdded config file logging parameters to be able to disable certain logging functions logging: to_file: true # Setting to false will stop the AWARE.log default logging to happen to_stdout: true # Setting to false will stop it showing in the console or journalctl mqtt_log: true # Setting to false will stop the mqtt messages logging to individual topic files and will be hidden from the other loggersAdded scheduled restartRefactored auto update into maintenanceFixed auto update testsAdded hour parameter for future proofing more refined scheduled restartsRTGM-200Added ability for command data to be parsed to the topic task. Allows for responding with data and pulling out specific command metadataAdded command response success on completion of discover process with data payload determining if the module was found or not; as per comments on RTGM-200 issue.Fixed a bug where if the send data trigger was not stored beforehand, the send trigger would happen repeatedlyAdded the send_data to the payload even if it had not triggered a storageRTGM-203Fixed DI4 and RO4 relay and switch status telemetryUpdated tests for storage to check for patching itemsUpdated triggers to use latest sample in the reference dictionary instead of the first in order to be able to more simply store the values in send state and store stateUpdate generic modbus device to store the data and pull from storage when triggering sendUpdated test device to store alwaysUpdated imac controller and modules to use storage data for triggeringRTGM-224Fixed Param lookup to check against scaled values in the lookup instead of rawStreamlined ParamLookup decoderRTGM-230Added the ability to push the topic to subscribers (helps with wildcard subscriptions)Added peek to get storage results non-destructivelyAdded arbitrary nested dictionary storageVersion 3.0.4Cloud reconnect fixesAWARE-103Updating H335 standard config to use integrated IPM config instead of local IPM configUpdating H335 standard config to include Outlet4 I33Updating remaining IPM config based devices to use the installed IPM configAdded credentials for H334Added credentials for H314Added credentials for H330Added credentials for H332Added credentials and config for SS002Added IPC config fileAdded H270 Config fileAdded H316 CredentialsRemoving H270 IPC devices until added to the library and cloudAWARE-119Dev setup to test the mqtt disconnect logicCreated an auto disconnection after a specified interval from configFixed an issue where the performance degraded over time. Every time it reconnected to the server, a new virtual device was created. This meant that each reconnect would send more and more messages, breaking the devices over timeHides the error that happens when the client schedules a disconnect before the jwt expires.Added token life parameter so that the token expiry can be set via configUpdated dev config to match the local and cloud brokerBackported the virtual device accumulation fix to paho clientImproved logging showing resolved messages. Still seems to duplicate logsIncreased token life back to 60 minutesRTGM-203Fixed DI4 and RO4 relay and switch status telemetryRTGM-224Fixed Param lookup to check against scaled values in the lookup instead of rawStreamlined ParamLookup decoderVersion 3.0.3IPM Config FixesAWARE-103IPM Config fixesAdded missing I33 for H324Version 3.0.2WCX3 ConfigAWARE-103Added IPM Config fileAdded Config for WCX3 Underground Substations and DCBsVersion 3.0.1Fixes for auto updateAWARE-37H-266 update to configAWARE-114Added wheel to the pip install requirements as pip 20.1 needs wheel installed to build some packagesAdded aware_version: "manual" which will skip auto updates no matter what is installedAdded better error handling for pyaware.stop() to hbmqtt so it can escape during setup.Added better logging for pyaware.stop() for hbmqtt clientpyaware.stop() will now raise an error the currently running task as well. This ensures that the process doesn't have to wait for an await signal to start cleaning up.Added a _stopping variable to ensure that only one shutdown method attempts to happen at once. A second stop can happen after the first shutdown fails to complete.Catches RuntimeError to prevent attempting to stop multiple timesFixed Hbmqtt error regarding downloading config from GCPVersion 3.0.0Major release for auto update feature.AWARE-7test 035 with best case connection settingsAdded configurable QOS, keep alive, and clean session for cloud.yaml to overrideAdded keep alive to hbmqtt.pyAdded example on how to override parameters for mqtt clientsFixed reference to correct config for qos parametersAWARE-18Added H-546 commission filesAdd Main.py for HST H-546Added RSA key for H-546H-546 Remove Main.py update credentials update configAWARE-107H-233 added gateway.yaml and Commission fileUpdated H-234 ConfigAWARE-108Added subscriptions to system/reboot which will kill the pyaware process for gateway devicesAdded example configuration with local MQTT and simple modbus device for easy testingFixed hbmqtt so it can now receive commandsAdded error handling on main startup so if invalid config is present it doesn't lock up pyawareAdded topic wildcard matching for hbmqtt subscription callbacksNow ignores commands in the default command handler with no payload as there is no correlation ID to work with.Renaming reboot to stop to reflect the actual action taking place and leaving reboot available for OS rebootAWARE-110Fixed issue with total-positive-energy scaling incorrectlyAWARE-113Created awaresupport accounts to pypiUpdated bitbucket-pipelines.yaml to include deployment to pypiTested integration to pypi and test pypiAdded Secured API tokens to bitbucketTried and removed atlassian/pypi-publish:0.2.13 integration as it doesn't workTested builds deploy to test pypiAWARE-114Added version checker auto update using pip and pypi.orgAuto update uses the version logic as specified in AWARE-114 issue descriptionAdded unit tests to check the auto update logicTested the auto update logic manually and confirmed to workRTGM-205Changed name of L1 Short circuit to include statusVersion 2.4.0AWARE-7Setup now retries if an exception is raised (Does not resend message on timeout as that is not in MQT3.1.1 spec so I'm going to trust it)Added file logging to AWARE.log for main entry point for config file based setupsUpdated aiomodbus requirementsAdded delay for failed setupAWARE-19updated gateway.yamlAWARE-37Added Config file for IPBAdded no-authentication mode to hbmqtt for local debuggingaiomodbus rtu config parsingAdded address shifting to generic modbus device to support IPSI addressingUpdated IPB config to include processing triggers on the status registerAdded Gateway config for H266Updated ParamLookup to not return if the lookup value is invalidUpdated config to have separate comms parameter parsingChanged the logging of modbus_device pipeline to only log on start and not each iterationBumped the aiomodbus versionAdded commission2.py which does a full raw read based on the config filesChanged the cloud broker to hbmqttFixed issue where changing the broker uri wasn't enough to get a clean session to GCP. Now has a fresh session each connectAWARE-44Update/Create all commission files for HST SS003AWARE-55Added main.py for H-270Credentials addedUpdates to H-270 deployment filesAWARE-68Added config parsers for local and cloud mqtt brokers for hbmqtt paho and gmqtt.pyAdded on_connect and overrode connection method to allow for generating the uri username password for each reconnectMoved paho client into it's own module WIP GMQTT basic implementation but does not work with GCP in current formAdded paho import for legacy entry pointReplaced the connected event with the one linked with the hbmqtt client.Added error handling to the hbmqtt receive loopRTGM-200Added find module by serial command methodFixed issue with connect not checking the is_set correctly as it was left over from using it as a flag instead of an eventAdded a clearer error message when the module doesn't appear to existRTGM-203add Raw DI valuesAdded switch-status and relay-status to reflect the switch and relay state without the inversion present from the invert bitAdded explicit read for invert status on module discoveryThe processed relay-status and switch-status now only complete if invert-status is knownRTGM-221Added group validators for set-parametersAdded WriteAndValidateParams to do the writing, reading and validating parameters is one stepUpdated the modules to use Group validators and WriteAndValidateParams for set-parameters commandFixed the validators to work with 0 values and true/falseVersion 2.3.0Added GCP config parsingImproved logging throughoutFixed issue with triggers always firingAdded HBMqtt as an alternate clientAdded aiomodbus as an alternate clientAdded configuration file device definition for substation monitoringRefactored config and mqtt codeVersion 2.2.0Address 0 Status will always be System OwnedSet-Parameters no longer throws error for undefined parametersVersion 2.0.0Ensham first releaseRTGM setup for phase 1 deploymentHuge changes overall, maintains backward compatibility with Substation Monitoring through legacy API (main.py)Builds available for zip app deploymentVersion 1.2Added GCP clientAdded iMAC protocolSkeleton structures
and implementation in preparation for Architecture 2.0Added more parameters to I35Added HpbSubPlc to devicesCloud loading of configurationCleaner shutdownsSaves and shutdown after new gateway configurationVersion 1.1.1Updated device data_info to new naming conventionUpdated script to update database from configuration fileH363 specific deployment informationVersion 1.1This release is aimed at targeting the Real Time Gas Monitoring demo and will reflect the state of the library of the demo as it is deployed.Added measurement scaling to SI unitsAdded Modbus device write/updateAdded ModbusBitMask to break down registers into labelled individual bitsAdded configuration files to eventually substitute main.pyAdded ModbusTCPAdded better error handling and cleanup to the schedulerChanged the default byte and word order of modbus to fit industry standardCreated script for updating mongodb from deployment configurationAdded the median aggregation