Package: brokkr
BrokkrA client for data ingest/logging/uplink, remote management and autonomous & central control of scientific IoT sensors as part of the Mjolnir system.
Originally developed to receive science data and status information from a local HAMMA2 lightning sensor and a Sunsaver MPPT-15L charge controller, store it locally, and transmit it back to a central server (generally, but not necessarily one running the Sindri package), but is now being evolved to be extensible and configurable for a wide variety of applications, from low-cost, low-power arrays of hundreds of environmental monitoring sensors, to sophisticated instruments mounted on aircraft and UAVs.
Further, it can maintain a reverse SSH tunnel to an accessible server for remote access, and receive and execute power, processing system and sensor control commands forwarded as TCP packets over said connection.The goal of Project Mjolnir is to make it easy for PIs or students without a coding background to easily get started gathering and analyzing data from low-cost sensors right away, and allowing those with basic programming experience to easily develop, test and share their own plugins to work with new types of sensors, outputs and more with little or no extra work over a "quick and dirty" approach but major long-term benefits.
The long-term vision is to create an ecosystem of open-source presets, plugins, examples and more for a wide variety of low-cost scientific IoT sensors.Key FeaturesSupport for SPI, I2C, GPIO, Analog, SMBus, UART, Modbus, TCP, UDP, and more as inputs, and print/pretty print, file and system logging, CSV, and TCP packets as output built into the coreEasy to use, powerful plugin framework with a simple API and minimal boilterplate for input, processing and output steps; can be either simple Python files placed in a specific directory, or proper Python packagesPlugins for alerts/triggered actions, Slack notifications, REST/web APIs and much moreBuilt-in support for dozens of different sensors including ECH2O EC-5, EC-10, EC-20, DS18S20, DS1822, DS18B20, DS28EA00, DS1825, MAX31850K, HIH6130, Si7021, SHT31D, MPL3115A2, MLX90614, HTU21D, DHT11, DHT22, BMP280, BME280, MPPT-15L, HAMMA2, ADS1015, ADS1115, and generic switch, counter, GPIO and analog anemometer dir and speed, plus time, runtime, ping, and moreDrop-in, declarative preset system for supporting new sensors, devices and protocols; presets can be enabled with as little as 1 line in the master config file, and extensively customized via config optionsRobust error handling, status logging, automatic installation, service configuration and multiprocess management infrastructureHierarchical configuration system, allowing for multiple levels of settings and overridesSystem-independent and fully multi-system capable; all metadata, config, plugins and presets stored within VCS-trackable self-contained package for easy managementLicenseCopyright (c) 2019-2022 C.A.M. Gerlach, the UAH HAMMA Group and the Project Mjolnir ContributorsThis project is distributed under the terms of the MIT (Expat) License; see theLICENSE.txtfor more details.Installation and SetupBrokkr is built and tested under Python 3.6-3.10, with a relatively minimal set of lightweight, pure-Python core dependencies.
It works best on Linux, but is tested to be fully functional (aside from service features) on Windows (andshouldwork equally on macOS) using the Anaconda distribution.Standard installOn Linux (or other platforms, for experienced users), Brokkr can be installed like any other Python package, viapipinto avenvvirtual environment.For example, with the venv created insideENV_DIRin the current working directory, and installing theEXTRApackages needed to support specific sensor types (e.g.modbus,serial,adafruit, etc, orallfor all of them) as desired:python3-mvenvENV_DIRsourceENV_DIR/bin/activate
pipinstallbrokkr[EXTRA1,EXTRA2...]On Windows and Mac, use of Anaconda/Miniconda is recommended, substituting conda environments for venvs.
While these platforms are supported for development, some functionality specific to running Brokkr in production may be unavailable.For information on installing a development version, see theContributing Guide.Then, you need to take a few more steps to get your environment set up: clone the system config package(s) you want to use with Brokkr (replace the examplemjolnir-config-templatepath with yours), register them, and set up your config and unit information.SYSTEM_SHORTNAMEis whatever name you want to register the system with in the system file, andUNIT_NUMBERis the integer number (arbitrary, but should be unique) you want to designate the device you're installing on.gitclonehttps://github.com/project-mjolnir/mjolnir-config-template.git
brokkrconfigure-systemSYSTEM_SHORTNAME/path/to/system/mjolnir-config-template
brokkrconfigure-init
brokkrconfigure-unitUNIT_NUMBERFinally, you can run the post-installation setup steps as needed for your system.
First, you'll want to install any system-specific dependencies,brokkrinstall-dependenciesThen, to just install the Systemd service unit to run Brokkr on startup, run:sudo/PATH/TO/ENV_DIR/bin/python-mbrokkrinstall-service``brokkrinstall-dependenciesYou can set a specific account, install path and startup arguments if desired; seebrokkr install-service --helpfor more usage and option information.For a full install of all post-setup tasks, including the config files, scripts, system-specific dependencies, firewall access, and (on Linux) serial port access, Brokkr systemd service, and SSH/AutoSSH service and configuration, you can instead run:sudo/PATH/TO/ENV_DIR/bin/python-mbrokkrinstall-all
brokkrinstall-dependenciesFinally, you can check that Brokkr is installed and set up correctly withbrokkr --version,brokkr statusand the other commands detailed inbrokkr --help.
Simply reboot to automatically complete setup and start thebrokkrservice, or on all platforms you can manually execute it on the command line immediately withbrokkr start.Automated clean install (under development)For setup on typical IoT devices (i.e. single-board computers like the Raspberry Pi) running a clean copy of a modern Linux-based operating system, Brokkr features a comprehensive setup routine that can bootstrap all key aspects of a factory-fresh system to be ready for deployment in the field.
Simply declare the configuration files you want copied, packages and services you want installed/enabled/disabled/removed, firewall ports you want open closed, and other custom actions (move files, sed scripts, commands run, etc) for each phase of the install as part of the system config package, and on your command, brokkr will do the rest.A typical semi-automated install flow might look like the followingFlash SD card with OS imagePerform basic raspi-config, Fedora, etc. setup; change username if desiredCreate and activate venv,pip install brokkr --no-dependenciesfrom offline sdist and copy system config dir and any keyfilesRunbrokkr configure-system <systemname> <systempath>to set the system config dir pathRunbrokkr install --phase 1to perform the necessary steps to enable InternetUpdate all packages to latest (apt update && apt full-upgrade && apt autoremove) and reinstall brokkr with all packages (pip uninstall brokkr && pip install brokkr)Runbrokkr install --phase 2to install remaining itemsRunbrokkr setup-deviceto trigger device-specific setup actionsCreate venv for Sindri andpip install sindriit (optional)Once on-site, perform unit configuration (see below)A sample bash script will be provided that runs steps 3-9 of this workflow automatically, and can be customized to the needs of a specific system.Flashing Brokkr onto a prepared cardIf a card is already prepared via the steps mentioned in the "Automated clean install" section (minus thebrokkr setup-devicestep), flashing it onto another device and preparing it for deployment is simple.After flashing the Pi and activating the appropriate venv, runbrokkr setup-deviceto regenerate the harnesses device-specific items (password, hashes, SSH keys, etc). You’ll need to enter the Pi’s current and desired password at the interactive prompt.Finally, on site, once the final unit configuration is set (or after it is changed in the future), perform on-site setup as belowOn-site setupOn site, you'll want to take a couple additional actions to pair a specific device with a specific site, and test connectivity.Runbrokkr configure-unit <unit-number> --network-interface <network-interface>to set up the basic unit configurationRunbrokkr setup-unitto perform final per-unit on-site setup, register and test the link to the sensor, and verify connectivity to the upstream serverPower off the device, connect it to all desired hardware and rebootUsageOverviewSeebrokkr --helpandbrokkr <SUBCOMMAND> --helpfor detailed documentation of Brokkr's CLI, invocation, options and subcommands.
The following is a brief, high-level summary.For a quick check of Brokkr, its version and that of the current system (if configured), usebrokkr --version.
Run thebrokkr statuscommand to get a snapshot of the monitoring data, and thebrokkr monitorcommand to get a pretty-printed display of all the main monitoring variables, updated in real time (every 1 s by default) as you watch.brokkr startis the main entrypoint for Brokkr's core functionality, loading and executing the configured data acquisition, processing and output pipelines, and in normal usage is run though the Brokkr service.Thebrokkr install-*commands perform installation functions and thebrokkr configure-*functionality helps set up a new or updatedbrokkrinstall.
On Linux, thebrokkr-SYSTEMNAMEsystemd service can be interacted with via the standard systemd commands, e.g.sudo systemd {start, stop, enable, disable} brokkr-SYSTEMNAME,systemd status brokkr-SYSTEMNAME,journalctl -u brokkr-SYSTEMNAME, etc, and the same forautossh-brokkrwhich controls remote SSH connectivity.Interactive Use (Foreground)First, activate the appropriate Python virtual environment (e.g.source ENV_DIR/bin/activate).Then, you have a few options:Main foreground start command, for testing:brokkr startOneshot status output:brokkr statusLightweight realtime monitoring (prints to screen, can also write to file):brokkr monitorRunning Brokkr as a Service (Background)Generate, install and enable service automatically:sudo /home/pi/path/to/ENV_DIR/bin/python -m brokkr install-serviceStart/stop:sudo systemctl start brokkr-SYSTEMNAMEsudo systemctl stop brokkr-SYSTEMNAMEEnable/disable running on startup:sudo systemctl enable brokkr-SYSTEMNAMEsudo systemctl disable brokkr-SYSTEMNAMEBasic status check and latest log output:systemctl status brokkr-SYSTEMNAMEFull log output (also logged to text file~/brokkr/hamma/brokkr_hamma_NNN.log)journalctl -xe -u brokkr-SYSTEMNAMEConfigurationA major design goal of Brokkr and the Mjolnir system is extensive, flexible and straightforward reconfiguration for different sensor networks and changing needs.
For example, with the UAH HAMMA2 system, all the system configuration is normally handled through theMjolnir-HAMMA system config packagein the standard Mjolnir config schema developed for this system, aside from a few high-level elements specific to each unit which all have interactive configuration commands as below.However, if local customization is needed beyond the high-level options specified here, instead of modifying the version-control-tracked system config package directly, the config system built for this is fully hierarchical and all settings can be fully overridden via the corresponding local config in~/.config/brokkr/SYSTEM_NAME.
Brokkr fully supports configuration, logging, operation and output of any number of Mjolnir systems simultaneously, all on the same Pi.Configuration files are located under the XDG-standard~/.config/brokkrdirectory in the ini-likeTOMLformat; they can be generated by runningbrokkr configure-init(which will not overwrite them if they already exist), and reset to defaults withbrokkr configure-reset.High-level local setting configurationRegister, update and remove systemsRegister a Mjolnir system:brokkrconfigure-system<SYSTEM-NAME></PATH/TO/SYSTEM/CONFIG/DIR>(e.g.brokkr configure-system hamma /home/pi/dev/mjolnir-hamma)You can also use this command to remove, update, verify and set default systems with the appropriate arguments; seebrokkr configure-system --helpGenerate local config filesGenerate empty local per-system (i.e. override) config files if not already present:brokkrconfigure-initSet per-unit configurationbrokkrconfigure-unit<UNIT_NUMBER>--network-interface<INTERFACE>(e.g.brokkr configure-unit 1 --network-interface wlan0)Reset configurationReset unit and local override configuration (optionally minus the system registry):brokkrconfigure-reset