Package: serial-jobs
A tool for bidirectional communication between serial devices and MQTT brokers.1Features2How it works2.1Reading from serial devices2.2Writing to serial devices3Installation4Usage5Configuration5.1Configuration data structures5.1.1Terminology5.1.2Top-level configuration structures5.1.2.1Common5.1.2.2Related to reading from devices5.1.2.3Related to writing to devices5.1.3Lower-level configuration structures1FeaturesData flow is fully defined in a configuration file.Built-in support for theModbusprotocol.Adding support for custom serial protocols possible by extending code.Ability to send user-specific initialization MQTT messages on startup.NoteThe initialization MQTT messages can be used
to automatically configure the consumers of the value-carrying MQTT messages,
likeHome Assistant, to handle them appropriately.One of the design goals is to enable regular users
to get desired data from any Modbus device
merely by editing a configuration file.2How it works2.1Reading from serial devicesConfigured serial devices are periodically polled for data.These data are then converted to usable values according to the configured rules.The obtained values are then sent to the configured MQTT brokers.2.2Writing to serial devicesSubscribtions to the configured MQTT topics are created.The configured handlers are run when a matching MQTT message is received.Handlers extract values from the incoming MQTT messages
and convert them to protocol-specific data according to the configured rules.The obtained data are then written to the configured serial devices.3Installation$pipinstallserial-jobs4UsageEdit the provided YAML configuration file stubs in theconfig_stubsdirectory
or create new ones as desired.Merge the relevant YAML configuration file stubs into a single file.$merge_configconfig_stubs/*INFO serial_jobs.config loading stub configuration file config_stubs/epever-charge-controller.yaml
INFO serial_jobs.config loading stub configuration file config_stubs/jbd-battery-management-system.yaml
INFO serial_jobs.config loading stub configuration file config_stubs/mqtt-brokers.yaml
INFO serial_jobs.config loading stub configuration file config_stubs/orno-energy-meter.yaml
Overwrite configuration file ./configuration.yaml? (y/n) y
INFO serial_jobs.config saving configuration to file ./configuration.yaml
INFO serial_jobs.config configuration file savedConvert the configuration file from YAML format to JSON format
for faster parsing at runtime.$convert_configINFO serial_jobs.config loading configuration file ./configuration.yaml
INFO serial_jobs.config configuration file loaded
Overwrite configuration file ./configuration.json? (y/n) y
INFO serial_jobs.config saving configuration to file ./configuration.json
INFO serial_jobs.config configuration file savedRun the application.$serial_jobsINFO serial_jobs.config loading configuration file ./configuration.json
INFO serial_jobs.config configuration file loaded
INFO serial_jobs.device.base creating device lock /dev/ttyUSB0
...5ConfigurationThe application is configured via a configuration file in either YAML or JSON format.Below is an example of a simple configuration file which uses the YAML format.mqtt_brokers:-id:localhost:127.0.0.1port:1883username:password:devices:-id:orno-or-we-514name:ORNO OR-WE-514 single phase energy meterserial:port:/dev/ttyUSB0baud_rate:9600data_bits:8stop_bits:1parity:Etimeout:0.1protocol:modbus_address:0x13tasks:-id:o-active-powername:active power in Wdevice:orno-or-we-514mqtt_topic:orno-or-we-514/active-powervalue:data:-signed_long:register_type:holdingregister_count:2address:0x140jobs:-id:power-metermqtt_messages:-homeassistant/sensor/default/power-meter-active-power/config:device:name:ORNO energy metermanufacturer:ORNOmodel:OR-WE-514identifiers:orno-or-we-514device_class:powername:Active Powerstate_class:measurementstate_topic:orno-or-we-514/active-powerobject_id:orno-or-we-514-active-powerunique_id:orno-or-we-514-active-powerunit_of_measurement:Wsleep:10tasks:-o-active-powerIt instructsserial-jobsto poll an energy meter
for the current active power value every 10 seconds
and publish it to an MQTT broker.Configuration files for some real devices and real use cases
are available in theconfig_stubsdirectory.The data structures used within the configuration file are described below.5.1Configuration data structuresValid configuration data structures are: a string, a number, a boolean, a sequence or an object.NoteAll numeric formats recognized by Python are supported.
For instance, values which represent hardware addresses of any kind
might be specified as hexadecimal numbers, i.e. numbers prefixed with0x.Asequenceis a list-like data structure.
It must contain items of the same type.Anobjectis a dictionary-like data structure.
It consists offields.Afieldis represented by akeyand avalue.
Akeyis a string.
Avaluecan be any valid configuration data structure,
i.e. a string, a number, a boolean, a sequence or an object.There can be at most one field with a particular key within a specific object.5.1.1TerminologyThis section explains the terminology used within this documentation,
the configuration files and the source code.data partPartially meaningful piece of information
that can be converted to protocol-specific sequence of bytes
suitable for a particular serial device.valueA meaningful piece of information represented bydata partson a serial device.taskA routine which retrieves a particularvaluefrom a particular serial device
and sends it within an MQTT message.jobA group oftasksperformed periodically.handlerA routine which extracts a particularvaluefrom an incoming MQTT message
with a particular MQTT topic and writes it to a particular serial device.serviceA group ofhandlersto run upon receiving messages from a particular MQTT broker.5.1.2Top-level configuration structuresThis section describes configuration data structures
that might be present at the top-level of the configuration file.5.1.2.1Commonmqtt_brokersA part of configuration which specifies how to communicate with MQTT brokers.It consists of asequenceof MQTT broker specifications.
Each specification might contain the fields defined below.id:Unique ID of the defined MQTT broker.
It is used for referring to a particular MQTT broker within this configuration.name:(optional)Human-readable name of the defined MQTT broker.
It might be used to make the configuration file less ambiguous.host:Hostname of the defined MQTT broker.port:Port of the defined MQTT broker.username:Username for connecting to the defined MQTT broker.password:Password for the defined username.Example:mqtt_brokers:-id:localhost:127.0.0.1port:1883username:password:devicesA part of configuration which specifies how to communicate with serial devices.It consists of asequenceof serial device specifications.
Each specification might contain fields defined below.id:Unique ID of the defined serial device.
It is used for referring to a particular serial device within this configuration.name:(optional)Human-readable name of the defined serial device.
It might be used to make the configuration file less ambiguous.type:(optional)Type of the defined serial device.
Defaults toModbusDevice.Available device types:ModbusDevicefor devices which communicate
over theModbusprotocol.BMSDevicefor devices which communicate
over a protocol that is used by the battery management systems
from manufacturers such asJBD.NoteIt is possible to define a custom device type
by creating a subclass ofserial_jobs.device.Deviceand importing it in thesrc/serial_jobs/device/__init__.pyfile.serial:Specificationof the parameters for serial communication with the defined device.protocol:(optional)Device-type-specific protocol details needed for communicating with the device.It might contain fields defined below.modbus_address:(optional)Modbus device ID (or Modbus deviceaddress)
used for communicating with the defined Modbus device.In case ofModbusDevicedevice type, this field ismandatory.Example:devices:-id:orno-or-we-514name:ORNO OR-WE-514 single phase energy meterserial:port:/dev/ttyUSB0baud_rate:9600data_bits:8stop_bits:1parity:Etimeout:0.1protocol:modbus_address:0x135.1.2.2Related to reading from devicestasksA part of configuration which specifies
how to retrievevaluesfromdevicesand also where and how to send them within MQTT messages.It consists of asequenceof task specifications.
Each specification might contain fields defined below.id:Unique ID of the defined task.
It is used for referring to a particular task within this configuration.name:(optional)Human-readable name of the defined task.
It might be used to make the configuration file less ambiguous.device:(optional)ID of the configured serial device which is used by this task.
If there is only one configured device,
then this field might be omitted
and the only configured device will be used.mqtt_broker:(optional)ID of the configured MQTT broker which is used by this task.
If there is only one configured MQTT broker,
then this field might be omitted
and the only configured MQTT broker will be used.mqtt_topic:MQTT topic to which the MQTT messages with the obtained values will be sent.value:Specificationof how to obtain the value
for sending to the configured MQTT broker from the configured serial device.Example:tasks:-id:o-active-powername:active power in Wdevice:orno-or-we-514mqtt_topic:orno-or-we-514/active-powervalue:data:-signed_long:register_type:holdingregister_count:2address:0x140jobsA part of configuration which specifies how often to perform particulartasks.It consists of asequenceof job specifications.
Each specification might contain fields defined below.id:Unique ID of the defined job.
It is used for referring to a particular job within this configuration.name:(optional)Human-readable name of the defined job.
It might be used to make the configuration file less ambiguous.enabled:(optional)Boolean flag indicating whether to run this job or not.
Defaults totrue.mqtt_broker:(optional)ID of the configured MQTT broker
which is used to send initialization MQTT messages by this job.
If there is only one configured MQTT broker,
then this field might be omitted
and the only configured MQTT broker will be used.mqtt_messages:(optional)Asequenceof specifications
of initialization MQTT messages
to be sent before running this job for the first time.Each specification consists of anobjectwith a single field.
Its key denotes the MQTTtopicto which the message will be sent.
Its value denotes the message content.NoteThe initialization MQTT messages are sent with theRETAINflag.sleep:Number of seconds to wait between consecutive runs of this job.tasks:A sequence of IDs oftaskswhich should be performed by this job.Example:jobs:-id:power-metermqtt_messages:-homeassistant/sensor/default/power-meter-active-power/config:device:name:ORNO energy metermanufacturer:ORNOmodel:OR-WE-514identifiers:orno-or-we-514device_class:powername:Active Powerstate_class:measurementstate_topic:orno-or-we-514/active-powerobject_id:orno-or-we-514-active-powerunique_id:orno-or-we-514-active-powerunit_of_measurement:Wsleep:10tasks:-o-active-power5.1.2.3Related to writing to deviceshandlersA part of configuration which specifies
how to extract particularvaluesfrom incoming MQTT messages with particular MQTT topics
and how to write them to particulardevices.It consists of asequenceof handler specifications.
Each specification might contain fields defined below.id:Unique ID of the defined handler.
It is used for referring to a particular handler within this configuration.name:(optional)Human-readable name of the defined handler.
It might be used to make the configuration file less ambiguous.device:(optional)ID of the configured serial device which is used by this handler.
If there is only one configured device,
then this field might be omitted
and the only configured device will be used.mqtt_topic:MQTT topic of MQTT messages
from whose content this handler will extract the values.value:Specificationof how to extract the value
from the incoming MQTT messages
and how to write it to the configured serial device.Example:handlers:-id:e-battery-float-voltage-setdevice:epever-xtra-4415nmqtt_topic:epever-xtra-4415n/battery-float-voltage/setvalue:type:floatdata:-short:register_type:holdingwritable_block:start_address:0x9003stop_address:0x900Faddress:0x9008scale_factor:100servicesA part of configuration which specifies whichhandlersto run
upon receiving messages from particular MQTT brokers.It consists of asequenceof service specifications.
Each specification might contain fields defined below.id:Unique ID of the defined service.
It is used for referring to a particular service within this configuration.name:(optional)Human-readable name of the defined service.
It might be used to make the configuration file less ambiguous.enabled:(optional)Boolean flag indicating
whether this service should be provided or not.
Defaults totrue.mqtt_broker:(optional)ID of the configured MQTT broker which is used by this service.
If there is only one configured MQTT broker,
then this field might be omitted
and the only configured MQTT broker will be used.mqtt_messages:(optional)Asequenceof specifications
of initialization MQTT messages
to be sent before starting to provide this service.Each specification consists of anobjectwith a single field.
Its key denotes the MQTTtopicto which the message will be sent.
Its value denotes the message content.NoteThe initialization MQTT messages are sent with theRETAINflag.handlers:A sequence of IDs ofhandlerswhich should be dispatched by this service.Example:services:-id:charge-controller-float-voltagemqtt_messages:-homeassistant/number/default/charge-controller-battery-float-voltage/config:device:name:EPEVER solar charge controllermanufacturer:EPEVERmodel:XTRA 4415Nidentifiers:epever-xtra-4415nentity_category:configname:Battery Float Voltagemin:12max:14.6step:0.01device_class:voltagestate_class:measurementcommand_topic:epever-xtra-4415n/battery-float-voltage/setstate_topic:epever-xtra-4415n/battery-float-voltageobject_id:epever-xtra-4415n-battery-float-voltageunique_id:epever-xtra-4415n-battery-float-voltageunit_of_measurement:Vhandlers:-e-battery-float-voltage-set5.1.3Lower-level configuration structuresThis section describes configuration data structures
that might only be present within certain other configuration data structures.serialSpecification of the parameters for serial communication with the defined device.The specified values must be accepted by theserial.Serialclass
from thepyserialmodule.
It might contain fields defined below.port:Name of the hardware device (or port)
used for serial communication with the defined device.baud_rate:Baud rate used for serial communication with the defined device.data_bits:Number of data bits used for serial communication
with the defined device.stop_bits:Number of stop bits used for serial communication
with the defined device.parity:Parity used for serial communication with the defined device.timeout:Timeout in seconds used for serial communication
with the defined device.Example:serial:port:/dev/ttyUSB0baud_rate:9600data_bits:8stop_bits:1parity:Etimeout:0.1value(when used withintasksas MQTT output value)A part of configuration which specifies
how to map simpledataparts obtained from serialdevicesto a serializable value that can be used
when communicating with an MQTT broker.It consists of an object which specifies how to obtain the value
for sending to the configured MQTT broker from the configured serial device.It might contain fields defined below.type:(optional)Python data type to which to convert the obtained data
before serializing it to string for sending to the configured MQTT broker.Available values are:float,int,str,date,datetimeandtime.mapping:(optional)An object containing string-to-string mapping
applied to the obtained databeforeconverting it to the final value type.Example:mapping:0:normal1:high temperature warning2:low temperature warningdata:Asequenceof data partspecifications.
They specify how to convert the obtained data parts
to Python values which could be serialized as an MQTT message.Example:value:mapping:0:normal1:overvoltage2:undervoltage3:low voltage disconnect4:faultdata:-short:address:0x3200bitmask:0b1111value(when used withinhandlersas MQTT input value)A part of configuration which specifies
how to extract value from an incoming MQTT message
and how to write it to a particular serialdevice.It consists of an object which specifies how to obtain the value
for writing to the configured serial device from the incoming MQTT messages.It might contain fields defined below.mapping:(optional)An object containing string-to-string mapping
applied to the obtained MQTT message contentbeforeconverting it to the final value type.Example:mapping:false:0true:1type:(optional)Python data type to which to convert
the string content of the obtained MQTT message
before transforming it into data parts
which will then be written to the configured serial device.Available values are:float,int,str,date,datetimeandtime.NoteParsing of MQTT message payload
into thedate,datetimeandtimetypes
is implemented via their respectivefromisoformatmethods.The parsed date or datetime or time value is then split
into tuples of the individual numeric fields
which can be written to the desired device registers.data:Asequenceof data partspecifications.
They specify how to convert the obtained Python value to data parts
which could be written to the configured serial device.Example:value:type:floatdata:-short:register_type:holdingwritable_block:start_address:0x9003stop_address:0x900Faddress:0x9008scale_factor:100dataA part of configuration which specifies
the mapping between raw bytes fromdevicesand simpledata parts.It consists of asequenceof data part specifications.A data part specification is an object
which defines how to map an individual data part to raw device bytes.
It consists of a singlefieldwhose name determines thedata typeof the data part
and whose value determines the deviceregisterswhich contain the bytes for the data part.Available data types are:string: string of byte-sized charactersbyte: one-byte unsigned integersigned_byte: one-byte signed integershort: two-byte unsigned integersigned_short: two-byte signed integerlong: four-byte unsigned integersigned_long: four-byte signed integerfloat: four-byte floating point numberdouble: eight-byte floating point numberAll multi-byte numeric data types are by default expected to use
big-endian byte order (i.e. the most significant byte has the smallest memory address).The value of this field, representing the device registers,
might contain the following fields:register_type:(optional)Type of the register to read from.
Defaults todefault.Available values are:default: The device-specific default register type.
For devices of typeModbusDevicethe default register type
is theinputregister type.coil: A readable and writable register type which holds one bit of data.
Available only for devices of typeModbusDevice.discrete: A read-only register type which holds one bit of data.
Available only for devices of typeModbusDevice.holding: A readable and writable register type which holds two bytes of data.
Available only for devices of typeModbusDevice.input: A read-only register type which holds two bytes of data.
Available only for devices of typeModbusDevice.writable_block:(optional)Specification of the block of registers
which need to be written to the device at the same time
when writing the data to this particular register block.It might contain fields defined below.start_address: Start address (inclusive) of the register block to write.stop_address: Stop address (exclusive) of the register block to write.NoteRegisters in the defined writable block
which are unaffected by the operations on this data part
are at first read from the serial device
and then written back unchanged.register_count:(optional)Number of consecutive registers to use. Defaults to 1.address:Start address (inclusive) of the register block to use.byte_order:(optional)Sequence of zero-based byte indices
determining how to order the bytes from this register block
into the resulting data part.NoteThis field can be used to change the byte order.
For example, converting 4 bytes long little-endian value from a serial device
to 4 bytes long big-endian data part can be done like this:byte_order:[3,2,1,0]byte_offset:(optional)Number of bytes from this register block to skip
before creating the resulting data part.byte_count:(optional)number of bytes from this register block
starting atbyte_offsetto use for creating the resulting data part.byte_index:(optional)Index of a single byte within this register block
to use for creating the resulting data part.
If defined, it overridesbyte_offsetandbyte_count.bitmask:(optional)Binary integer (i.e. a number prefixed with0b)
determining the bitmask applied to the sequence of bytes
extracted from this register block.Example:0b10001100bitshift:(optional)Number of bits to shift the sequence of bytes
extracted from this register block.
If positive, shift to the right (i.e. divide by a power of two).
If negative, shift to the left (i.e. multiply by a power of two).scale_factor:(optional)Number by which to divide the bit-shifted data value.increase_by:(optional)Number which is added to the scaled data value.NoteThe order and meaning of the data operations mentioned above applies
when converting raw device bytes to data parts withintasks.This order isreversedwhen converting data parts to raw device bytes withinhandlers.For instance, the operation which is performedlastwithintasks,
i.e. increasing of a value by a constant, is performedfirstwithinhandlers.In addition, instead of performing the operation in its original form,
itsreverseis performed,
i.e. the value of the data part is decreased by the same constant.Example:data:-long:register_count:2address:0x3102byte_order:[2,3,0,1]scale_factor:100In this example, the data flow when reading from a serial device
starts at the device registers0x3102and0x3103.
Reading those registers results in four obtained bytes.
Those four bytes are then reordered using the defined permutation.
Then they are converted to unsigned long integer.
And then this value is divided by 100.
The outcome is then processed further as a data part.The data flow when writing to a serial device is reversed.
At first the unsigned long integer is multiplied by 100.
Then its bytes are reordered using the inverse of the defined permutation.
And then the resulting bytes are written
to device registers0x3102and0x3103.