Upgraded DIY home monitoring/automation system - Page 1
Forum section: electronics / projects

author: paulca
author_rank: Super Contributor
timestamp: August 13, 2020, 01:03:08 pm
content: So my little rasberry pi running my home automation system failed. Killed it's SDCard... total loss. While I could pull the code/files back out of gitlab the data and configuration was lost.So I spend a few evenings and a few afternoons at the weekend restoring operation.Instead of reuse the previous "RRDTool + Cacti" monitoring and data logging components which took dozens of hours to configure, I replaced them with something modern. InfluxDB and Grafana.I certainly like the results.The architecture, very briefly is...ESP8266/32s with sensors, like temperature, air pressure humidity or reading from an RS485 bus on the solar controller for example; send their data via UDP to a hub.The hub holds only the current data in memory making it available to automations etc.Once a minute a scraper grabs all this data and pushes it to an InfluxDB.Grafana is then configured to graph and otherwise visualise this data.Behind this there is the heating control system which is my next task .... visualise it's innards such as demands, schedules, target temps, overrides, profiles etc.The ESP8266/32's are remote flashed using a manager application i wrote. I install a basic base image on the device, plug it in and when it connects to the Wifi it broadcasts to the whole network that it is there and what firmware it is running. The manager application looks it up in a table and if it's not running the right firmware, flashes it and reboots it. The firmware is build in VisualCode using PlatformIO. I have yet to automate the release process direct to the manager application to allow direct CI deploy.The backend stack (Hub, Scheduler, demand processor, temperature manager, heating/lighting controllers, data scraper, InfluxDB, grafana etc. etc.) all run in individual microservice style dockers and the whole thing is brought up and orchestrated by docker-compose. Currently running on one host, but there is not stipulation all of these could not run on different hosts.A prominent feature is "zero-config" for as many things as possible. Right now if I install a new sensor which emits something in "*C" it will propegate through the whole stack and even appear on that "Current Temperatures" graph.... without touching a line of config, except in the sensor itself. If I install a new sensor that emits something else, like Watts or Pixies, they will not end up on the temperature graphs, but they will be available in the hub and they will get pushed into InfluxDB. The sensor data is self describing. it's just that the temp graph queries based on the data "Units" tag.The next thing to monitor will be my electricity usage using a smart meter with an RS485 bus or similar.

author: aiq25
author_rank: Regular Contributor
timestamp: August 13, 2020, 02:21:59 pm
content: Nice project. I'm looking into something similar but I would like to add couple of camera's for video and maybe just save pictures to a server. I don't need live video feed. Not sure if I want to use a Raspberry Pi or ESP32 for the video/picture capturing.

author: paulca
author_rank: Super Contributor
timestamp: August 13, 2020, 03:10:31 pm
content: Quote from: aiq25 on August 13, 2020, 02:21:59 pmGrab yourself an ESP32Cam off Amazon. It's pretty neat. Once set up it only requires a 3.3V/5V to run. There are "sketches" to download that will do an HTTP POST for each screen grab. I believe there are other sketches that allow UDP/RTSP streaming of video too.I am only starting to look into it, but I got an ESP32 set up and presenting it's own web interface allowing viewing of video and/or stills.Performance wise at 640x480 you get about 10 FPS. At full 1080p HD however it's more like 1 FPS. I believe the largest res is higher again, but it's more a series of still shots by that stage.The ESP32Cam has a built in SDCard slot, so you could keep the last 24 hours of images on it as a security backup, in case you are worried a burglar could use a Wifi signal jammer to blind your cams.Anyway, if you are interested I keep all my stuff here:https://gitlab.com/paulcam/home_heatingI have yet to commit the DockerCompose/InfluxDB/Grafana changes, but they will be along shortly.

author: Renate
author_rank: Super Contributor
timestamp: August 13, 2020, 03:32:00 pm
content: Hmm, interesting.I have my own system for my RV, temp, heating, tanks, solar, tilt, motion, GPS...Are you using an EP Solar, EP Ever or rebranded solar controller?Your barometer reminds me that that would be useful.I also have an almanac based on current GPS with dawn, sunrise, noon, etc and angles.

author: paulca
author_rank: Super Contributor
timestamp: August 13, 2020, 03:47:45 pm
content: Quote from: Renate on August 13, 2020, 03:32:00 pmI'm using an EP Ever TrackerA 10 amp I think. Been bullet proof for 3 years now. Touch wood. Nice data on the RS485 bus which is how I monitor it. (There is a thread of mine with build details, code on github)I think the barometer is on a BME280 sensor. I tried the more expensive one with the air quality sensor, but couldn't get it to work as my breakout board didn't match the docs (chinesium). It would be a consideration in an RV I'd imagine. The air quality sensor measures certain organic chemicals and hydrocarbons in the air (as far as I understand it).

author: Renate
author_rank: Super Contributor
timestamp: August 13, 2020, 04:20:48 pm
content: Yeah, I had a Tracer A 20 Amp.At one point it blew the 5V regulator that powers the RS-485.I don't know how that happened because I used a three conductor cable that was only crimped in three positions on the RJ-45 connector.I wasn't touching the 5V that they put on one of the other pins.It was an SMD regulator, I didn't have one so I threw a TO-92 78L05 on it.It worked fine until I upgraded to a Tracer B 40 Amp.I got the model without any front panel.https://www.amazon.com/EPEVER-Controller-Identifying-Intelligent-Regulator/dp/B081GSFSV7/(If there is one thing that I hate it is playing button hockey on stupid panels, "Hold the enter key down and hit up arrow six times...")Still, the units are built fairly well.It's cloudy today.

author: dondon
author_rank: Newbie
timestamp: July 04, 2022, 09:12:13 pm
content: Hi paulca, Thanks for the thread and Info, I know it's a bit old, but it is exactly what I am after, can you tell me what the best way would be to do this without a Raspberry Pi?I plan on hooking up a thermal sensor to a cooling water pipe and having the data sent to grafana for a real-time reading/alerting if the temperature gets too hot?There is a router on at the remote location and any data should be routed to a remote Grafana server.I love the idea of docker/docker-compose as well and maybe Kubernetes to orchestrate?The sensor would need to be RS485/Modbus compatible I assume.what would be the best route to take with this, please?Cheers

author: paulca
author_rank: Super Contributor
timestamp: July 27, 2022, 09:10:43 pm
content: Quote from: dondon on July 04, 2022, 09:12:13 pmNot that old. It's improved a bit since though.Easiest way to monitor a remote temperature with grafana. Hmmm.I assume you have a Wifi network on that router? If so, an ESP8266 based sensor. You can make one for about $10 + a USB charger to power it.Alternatively, sourcing "Tasmota compatible / fashable or pre-flashed" smart devices that have temp sensors built in or pluggable. These exist and it's best to start your search on the Tasmota website for supported devices and work backwards to a source in your locale. An example might be several of the SOnOff Wifi power relays which have optional temp sensor ports (3.5mm jack).Tasmota is generic firmware for the ESP8266. Saves writing your own. It supports a bunch of things out of the box, including half a dozen different temperature sensors. It has a feature where it will send the data every N seconds to an MQTT server.You can place the MQTT server at your remote location. I use a very simple python script to receive the MQTT messages and publish them to InfluxDB which is the backend for Grafana.Technically, I think you could write direct to InfluxDB from the ESP8266, IIRC the HTTP API is just JSON POST. But Tasmota->MQTT->Influx is a more flexible option.No need for kubernetes. There isn't enough complexity in the hierarchy and demand profiles. Docker-Compose is sufficient as all you are trying to get is a contained, managed collection of small microservices. One for InfuxDB, one for your MQTT server, one for Gafana and half a dozen for scripts to automate. Kubernetes and docker-swarm are more catered for heavy loads and scaling horizontally, rolling deployments etc.docker-compose up --build --daemon # when you made a change to the system. Takes 10 seconds.and the occasional:docker-compose restartWhen you just can't be bothered working what fell over to fix it properly.