Package: senor-octopus
They say there are only 2 kinds of work: you either move information from one place to another, or you move mass from one place to another.Señor Octopus is an application that moves data around. It reads a YAML configuration file that describes how to connectnodes. For example, you might want to measure your internet speed every hour and store it in a database:speedtest:plugin:source.speedtestflow:-> dbschedule:@hourlydb:plugin:sink.db.postgresqlflow:speedtest ->user:alicepassword:XXXhost:localhostport:5432dbname:defaultNodes are connected by theflowattribute. Thespeedtestnode is connected to thedbnode because it points to it:speedtest:flow:-> dbThedbnode, on the other hand, listens to events from thespeedtestnode:db:flow:speedtest ->We can also use*as a wildcard, if we want a node to connect to all other nodes, or specify a list of nodes:speedtest:flow:-> db, logdb:flow:"*->"Note that in YAML we need to quote attributes that start with an asterisk.Running Señor OctopusYou can save the configuration above to a file calledspeedtest.yamland run:$pipinstallsenor-octopus$sroctospeedtest.yamlEvery hour thespeedtestsourcenode will run, and the results will be sent to thedbsinknode, which writes them to a Postgres database.How to these results look like?EventsSeñor Octopus uses a very simple but flexible data model to move data around. We have nodes calledsourcesthat create a stream of events, each one like this:classEvent(TypedDict):timestamp:datetimename:strvalue:AnyAn event has atimestampassociated with it, aname, and avalue. Note that the value can be anything!Asourcewill produce a stream of events. In the example above, once per hour thespeedtestsource will produce events like these:[{'timestamp':datetime.datetime(2021,5,11,22,16,26,812083,tzinfo=datetime.timezone.utc),'name':'hub.speedtest.download','value':16568200.018792046,},{'timestamp':datetime.datetime(2021,5,11,22,16,26,812966,tzinfo=datetime.timezone.utc),'name':'hub.speedtest.upload','value':5449607.159468643,},{'timestamp':datetime.datetime(2021,5,11,22,16,26,820369,tzinfo=datetime.timezone.utc),'name':'hub.speedtest.client','value':{'ip':'173.211.12.32','lat':'37.751','lon':'-97.822','isp':'Colocation America Corporation','isprating':'3.7','rating':'0','ispdlavg':'0','ispulavg':'0','loggedin':'0','country':'US',}},...]The events are sent tosinks, which consume the stream. In this example, thedbsink will receive the events and store them in a Postgres database.Event-driven sourcesIn the previous example we configured thespeedtestsource to run hourly. Not all sources need to be scheduled, though. We can have a source that listens to a given topic inMQTT, eg:mqtt:plugin:source.mqttflow:-> dbtopics:-"srocto/feeds/#"host:localhostport:1883username:bobpassword:XXXmessage_is_json:trueThe source above will immediately send an event to thedbnode every time a new message shows up in the topic wildcardsrocto/feeds/#, so it can be written to the database — a super easy way of persisting a message queue to disk!Batching eventsThe example above is not super efficient, since it writes to the database every time an event arrives. Instead, we can easilybatchthe events so that they’re accumulated in a queue and processed every, say, 5 minutes:db:plugin:sink.db.postgreslflow:speedtest, mqtt ->batch:5 minutesuser:alicepassword:XXXhost:localhostport:5432dbname:defaultWith thebatchparameter any incoming events are stored in a queue for the configured time, and processed by the sink together. Any pending events in the queue will still be processed ifsroctoterminates gracefully (eg, withctrl+C).Filtering eventsMuch of the flexibility of Señor Octopus comes from a third type of node, thefilter. Filters can be used to not only filter data, but also format it. For example, let’s say we want to turn on some lights at sunset. Thesunsource will send events with a value of “sunset” or “sunrise” every time one occurs:{'timestamp':...,'name':'hub.sun','value':'sunset',}Thetuyasink can be used to control a smart switch, but in order to turn it on it expects an event that looks like this:{'timestamp':...,'name':...,'value':'on',}We can use thejinjafilter to ignore “sunrise” events, and to convert the “sunset” value into “on”:sun:plugin:source.sunflow:-> sunsetlatitude:38.3longitude:-123.0sunset:plugin:filter.jinjaflow:sun -> lightstemplate:>{% if event['value'] == 'sunset' %}on{% endif %}lights:plugin:sink.tuyaflow:sunset ->device:"Porchlights"email:charlie@example.compassword:XXXcountry:"1"application:smart_lifeWith this configuration thesunsetfilter will drop any events that don’t have a value of “sunset”. And for those events that have, the value will be replaced by the string “on” so it can activate the lights in thelightsnode.Throttling eventsSometimes we want to limit the number of events being consumed by a sink. For example, imagine that we want to use Señor Octopus to monitor air quality using anAwair Element, sending us an SMS when the score is below a given threshold. We would like the SMS to be sent at most once every 30 minutes, and only between 8am and 10pm.Here’s how we can do that:awair:plugin:source.awairflow:-> bad_airschedule:0/10 * * * *access_token:XXXdevice_type:awair-elementdevice_id:12345bad_air:plugin:filter.jinjaflow:awair -> smstemplate:>{% ifevent['timestamp'].astimezone().hour >= 8 andevent['timestamp'].astimezone().hour <= 21 andevent['name'] == 'hub.awair.score' andevent['value'] < 80%}Air quality score is low: {{ event['value'] }}{% endif %}sms:plugin:sink.smsflow:bad_air ->throttle:30 minutesaccount_sid:XXXauth_token:XXXfrom:"+18002738255"to:"+15558675309"In the example above, theawairsource will fetch air quality data every 10 minutes, and send it tobad_air. The filter checks for the hour, to prevent sending an SMS from 10pm to 8am, and checks the air quality score — if it’s lower than 80 it will reformat the value of the event to a nice message, eg:“Air quality score is low: 70”This is then sent to thesmssink, which has athrottleof 30 minutes. The throttle configuration will prevent the sink from running more than once every 30 minutes, to avoid spamming us with messages in case the score remains low.PluginsSeñor Octopus supports an increasing list of plugins, and it’s straightforward to add new ones. Each plugin is simply a function that produces, processes, or consumes a stream.Here’s therandomsource, which produces random numbers:asyncdefrand(events:int=10,prefix:str="hub.random")->Stream:for_inrange(events):yield{"timestamp":datetime.now(timezone.utc),"name":prefix,"value":random.random(),}This is the full source code for thejinjafilter:asyncdefjinja(stream:Stream,template:str)->Stream:_logger.debug("Applying template to events")tmpl=Template(template)asyncforeventinstream:value=tmpl.render(event=event)ifvalue:yield{"timestamp":event["timestamp"],"name":event["name"],"value":value,}And this is thesmssink:asyncdefsms(stream:Stream,account_sid:str,auth_token:str,to:str,**kwargs:str)->None:from_=kwargs["from"]client=Client(account_sid,auth_token)asyncforeventinstream:_logger.debug(event)_logger.info("Sending SMS")client.messages.create(body=str(event["value"]).strip(),from_=from_,to=to)As you can see, a source is an async generator that yields events. A filter receives the stream with additional configuration parameters, and also returns a stream. And a sink receives a stream with additional parameters, and returns nothing.SourcesThe current plugins for sources are:source.awair: Fetch air quality data from Awair Element monitor.source.crypto: Fetch price of cryptocurrencies from cryptocompare.com.source.mqtt: Subscribe to messages on one or more MQTT topics.source.rand: Generate random numbers between 0 and 1.source.speed: Measure internet speed.source.sqla: Read data from database.source.static: Generate static events.source.stock: Fetch stock price form Yahoo! Finance.source.sun: Send events on sunrise and sunset.source.udp: Listens to UDP messages on a given port.source.weatherapi: Fetch weather forecast data from weatherapi.com.source.whistle: Fetch device information and location for a Whistle pet tracker.FiltersThe existing filters are very similar, the main difference being how you configure them:filter.format: Format an event stream based using Python string formatting.filter.jinja: Apply a Jinja2 template to events.filter.jsonpath: Filter event stream based on a JSON path.SinksThese are the current sinks:sink.log: Send events to a logger.sink.mqtt: Send events as messages to an MQTT topic.sink.pushover: Send events to the Pushover mobile app.sink.slack: Send messages to a Slack channel.sink.sms: Send SMS via Twilio.sink.tuya: Send commands to a Tuya/Smart Life device.