Visualizing your home temperature with MCU8266, MQTT and AWS
By: Dan Siwiec
Tags: Terraform, Python, AWS, IoT, Arduino

Let’s build an IoT, cloud-connected thermostat! We will go through a step-by-step process of building a thermostat and connecting it over MQTT to AWS cloud for processing and visualization. We will cover all aspects of this project — the circuit design, the application running on the board, the MQTT protocol, as well as all the AWS cloud components. In the end, we will produce a graph of temperature over time like the one below. Can you tell when I took the device to my bedroom downstairs? (You might think the readings are off, as the low 50s is a little chilly, but that’s a story for a different occasion 😊).

Temperature graph in AWS

Solution Overview

There are two parts to this project— the device with the temperature sensor and the cloud platform processing the data. An application that runs on the device, periodically (every 5 minutes) sends temperature readings to the cloud, where it is stored, processed, and plotted on a diagram for visual analysis.

High level architecture

Device design

Parts

No soldering is required in this project and all the parts add up to ~30$ with plenty of spares for future projects.

ESP8266 DevKit board (example)

10k Ohm NTC Thermistor (example)

10k Ohm Resistor (example)

Breadboard. A mini 170 board works great (example)

1 x M-M jumper wire. They only sell packs, like this one. They’ll come in handy for future projects, though.

(optional) USB Battery Pack (example)

The centerpiece of the device is the ESP8266 NodeMCU DevKit board. ES8266 is a low-cost (~$4) compact WiFi-equipped microcontroller with low-energy consumption properties and a mini USB port. It is perfect for simple, battery-powered applications like this one.

ESP8266 NodeMCU DevKit board

Circuit

Temperature calculation

The temperature measurement will rely on the Steinhart-Hart equation, which models the relationship between temperature and resistance in a semiconductor, such as a thermistor. A thermistor is a resistor with variable resistance, heavily changing with temperature. To measure the thermistor’s resistance, we will combine it in series with a standard resistor, between the ground and a reference voltage. We will then use ESP8266’s built-in analog-digital converter (ADC) to measure the voltage on the thermistor. The circuit below depicts this configuration.

Thermistor/resistor schematic

There are 3 equations describing this circuit.

Circuit equations

The first one defines the relationship between voltage and resistance of the resistors in series, based on Ohm’s Law. The second one reflects the total voltage on the resistors, given they’re configured between the ground and the reference voltage of 3.3V, based on Kirchhoff’s circuit low. The last one defines the relationship between the ADC reading A and the voltage on the pin, which is the voltage of the thermistor. You can read more about the ADC calculation here. Combining these 3 equations, we get the following formula for the thermistor’s resistance:

Thermistor’s resistance formula

We can apply the Steinhart-Hart equation to calculate the temperature, which is:

Steinhart-Hart equation for thermistor temperature

Where T₀ is a reference temperature (25C⁰) at which the thermistor has a reference resistance of R₀ (10k Ohm). B is known as the thermistor’s coefficient and is a constant value of 3950.

Circuit build-out

We will replicate the schematic above on a breadboard. For now, don’t worry about the connection between the D0 and RST pins — we will cover this part later on.

A complete breadboard setup looks like this:

Let’s now work on the AWS side of the project, as some output parameters from this process will be necessary for configuring the application running on the device.

AWS Cloud services

The cloud architecture is composed of 3 main components: IoT Core, DynamoDB, and Sagemaker. The IoT Core service plays the role of a gateway receiving the stream of temperature readings from the device, DynamoDB provides a way to store them, and Sagemaker — to analyze them.

AWS cloud architecture

We will cover the important pieces of the AWS infrastructure and some of the configuration items, but don’t worry just yet about recreating them in the AWS console by hand, as we will later use a tool named terraform to spin it all up with a single command.

MQTT

Our device will connect to AWS over a protocol called MQTT. MQTT, a protocol originally developed by IBM in 1999 to monitor an oil pipeline in the desert, has been a go-to for IoT applications and is now a standard officially sanctioned by the Organization for Advancement of Structured Information (OASIS). This lightweight TCP/IP-based protocol intended for IoT communication is built on a pub/sub architecture and is characterized by a small client footprint and low bandwidth consumption thanks to its very little packet overhead, compared to HTTP.

MQTT Architecture (image from mqtt.org)

In the MQTT architecture, clients subscribe with the broker to a particular message topic that other clients can publish messages to. In our case, the device will be the client publishing readings to a topic and AWS will provide the broker.

AWS IoT Core

AWS IoT Core is a set of services dedicated to the integration of IoT devices to AWS cloud. The platform provides an MQTT broker, to which our device will connect and publish messages.

Setting up IoT Core involves creating an IoT Thing Type, an IoT Thing, and an IoT Rule. The Thing represents a single device, with a dedicated certificate that our client will use to connect to the broker. The certificate is attached to an IAM (Identity Access Management) policy, that defines permission the client will have, allowing it to connect and publish to specific topics. Here are the important parts of the policy:

{

"Action": [

"iot:Publish"

],

"Resource": "...:topic/tempReading/*"

},

{

"Action": "iot:Connect",

"Resource": "...:client/esp8266*"

}

In a nutshell, this policy allows any client starting with the name ‘esp8266’ to connect and publish messages to topics starting with the name ‘tempReading/’.

AWS IoT Rule and DynamoDB

The next component in our architecture is the IoT Rule. This element defines actions for the incoming messages — effectively routing them to other AWS services. Our rule will forward the message for persistence in a DynamoDB table — a simple NoSQL document database by AWS.

IoT Rule configuration

Notice the timestamp() in the SQL statement — this will enrich all incoming messages with a field holding the current time.

DynamoDB table

AWS Sagemaker

The final component of the cloud infrastructure is an instance of AWS Sagemaker. Sagemaker is an advanced service intended for Machine Learning purposes. We will use the Jupyter Notebook functionality to execute code analyzing and plotting the temperature readings.

AWS Sagemaker Notebook instance

We will use this instance to create a visualization of the temperature later in this article.

Terraform

We will use a configuration management tool called terraform to provision and configure all the above components and some miscellaneous elements like IAM roles and policies. Terraform allows for automated provisioning of resources across dozens of services like AWS through a practice called Infrastructure as Code, which enables configuration to be expressed through a high-level syntax. We will assume the reader is familiar with terraform basics. To get up to speed on those I recommend this quick tutorial or the official docs.

First, let’s clone the git repository holding all the configuration and code needed for this project.



Cloning into 'esp8266-mqtt-aws-iot-temperature'...

...

Resolving deltas: 100% (22/22), done. ~ git clone https://github.com/danielsiwiec/esp8266-mqtt-aws-iot-temperature Cloning into 'esp8266-mqtt-aws-iot-temperature'......Resolving deltas: 100% (22/22), done. ~

Next, make sure your AWS access key and secret key are configured and saved in the ~/.aws/credentials file, as terraform will need these. You can follow this guide to get it all set up.

It is also a good idea to tag the AWS resources you’re creating, for reference purposes. Edit the terraform/terraform.tfvars file with your custom tags, or remove the entries in it altogether leaving common_tag={} if you’re not interested in tagging your resources.

We can now start the provisioning. Open your terminal in the folder you cloned and change to the terraform folder. The first time you run terraform, you need to initialize it.

~ cd terraform ~ terraform init Initializing the backend... Initializing provider plugins...

- Finding latest version of hashicorp/aws...

- Installing hashicorp/aws v3.22.0...

- Installed hashicorp/aws v3.22.0 (signed by HashiCorp) ... Terraform has been successfully initialized! ... ~

This will take a few seconds, where terraform will download the AWS plugin. All you have to do now is kick off the provisioning with the apply command. When prompted, answer yes to accept the resources being provisioned. This operation will take about 3 minutes. Do you think you could beat that with manual provisioning through the AWS console? 😎

~ terraform apply ... Plan: 14 to add, 0 to change, 0 to destroy. ... Enter a value: yes Apply complete! Resources: 7 added, 1 changed, 0 destroyed. Outputs: cert_pem = -----BEGIN CERTIFICATE-----

MIIDWTCCAkGgAwIBAgIUTHFMBieJZAGgCgrD5RGX/xyPIRkwDQYJKoZIhvcNAQEL

...

AcvMAdMIv56kfxLPQBSHG1i4NYX6+UmFZ+hZGfIUYRcDAxcdrSmK9+QOEOTD

-----END CERTIFICATE----- iot_endpoint = a2XXXXXXX.iot.us-east-1.amazonaws.com private_key = -----BEGIN RSA PRIVATE KEY-----

MIIEpAIBAAKCAQEAy2gRySw4BIpGGuKBkAJmlNpZ4s6lUEH/hWglu/0/RTQWYgIp

...

/lc1GMA+el2VqHSYu/ji8VLe/9NKP1jU+D7DBSmKeeEFY/vSiRdfRQ==

-----END RSA PRIVATE KEY-----

Let’s analyze the output of this command. 14 resources were provisioned (many of them being IAM resources, like roles, policies, and attachments). Among them — IoT Thing, DynamoDB, and Sagemaker. The creation of the last one was the longest and took about 3 minutes. We also have some outputs listed. These values are specifically requested in the terraform scripts to be printed out. Take note of their existence (iot_endpoint, cert_pem, and private_key) as we will use them soon, but don’t worry about saving them, we will always have access to them through the terraform output command.

Note: Some of the AWS services, most notably the AWS Sagemaker are not cheap (around 6¢/h or $45 monthly). Once you’re done with this project, destroy all your assets with terraform destroy . You can always spin them back up with terraform apply .

Device Application

Now that our device build-out and cloud services are complete, we’ll switch our focus to the application running on the device. The default ESP8266 firmware allows the device to be programmed with the Arduino language, however, we will pick a more interesting alternative — MicroPython. MicroPython is a slimmed-down version of the Python 3 interpreter and runtime, which is optimized to run on bare metal. To use this, however, we will need to flash our board with special MicroPython firmware.

Flashing the board

The ESP8266 dev kit has a built-in micro USB port that we can use to connect the device to a computer. To flash the board we will use a Python tool by Espressif named esptool.py, which you can install with pip install esptool. First, let’s create a Python virtual environment, using Python’s built-in venv package, to separate this and other Python packages we will use from your system-wide packages. You will need Python 3.x for this.

~ python -m venv .venv ~ source .venv/bin/activate ~ python -m pip install --upgrade pip ~ pip install esptool

Collecting esptool

...

...

Successfully installed bitstring-3.1.7 cffi-1.14.4 cryptography-3.3.1 ecdsa-0.16.1 esptool-3.0 pycparser-2.20 pyserial-3.5 reedsolo-1.5.4 six-1.15.0 ~ esptool.py version

esptool.py v3.0

3.0

If you can run the version command at the bottom, you’re good to go.

Now, on to flashing! First, download the MicroPython firmware for the board from this location. Unless you’re feeling adventurous, I suggest you pick the recent stable version of the firmware, which at the time of writing this article, was v1.13. Download the .bin file and save it in a known location. The esptool will require two parameters: port and baud rate. The port parameter specifies the location where the device is mounted on your file system. In my case, it is /dev/cu.usbserial-0001. You can list the available serial ports in the following way:

~ ls -p /dev/cu* /dev/cu.Bluetooth-Incoming-Port /dev/cu.DanSiwiecsAirPods-Wirel /dev/cu.DansQC35-SPPDev /dev/cu.usbserial-0001

The second parameter, baud rate, defines the data transfer rate. If the rate is too high, the device won’t be able to ‘keep up’ and the transfer will get (quietly) corrupted. For the ESP8266, 921600 is a safe value. Let’s now erase the current firmware on the board, to make room for the new one we just downloaded.

Note: For the remainder of this section, you will need to disconnect the D0 <-> RST jumper cable (red on the images above). Otherwise, esptool won’t be able to connect to the board.

~ esptool.py --port /dev/cu.usbserial-0001 erase_flash esptool.py v3.0

Serial port /dev/cu.usbserial-0001

Connecting....

Detecting chip type... ESP8266

Chip is ESP8266EX

Features: WiFi

Crystal is 26MHz

MAC: 40:f5:20:2e:1e:5f

Uploading stub...

Running stub...

Stub running...

Erasing flash (this may take a while)...

Chip erase completed successfully in 12.5s

Hard resetting via RTS pin...

Next, we will write the new firmware into the board:

~ esptool.py --port /dev/cu.usbserial-0001 --baud 921600 write_flash --flash_size=detect 0 esp8266-20200911-v1.13.bin esptool.py v3.0

Serial port /dev/cu.usbserial-0001

Connecting....

Detecting chip type... ESP8266

Chip is ESP8266EX

Features: WiFi

Crystal is 26MHz

MAC: 40:f5:20:2e:1e:5f

Uploading stub...

Running stub...

Stub running...

Changing baud rate to 921600

Changed.

Configuring flash size...

Auto-detected Flash size: 4MB

Flash params set to 0x0040

Compressed 638928 bytes to 419659...

Wrote 638928 bytes (419659 compressed) at 0x00000000 in 6.1 seconds (effective 837.4 kbit/s)...

Hash of data verified. Leaving...

Hard resetting via RTS pin...

Done! We can confirm our installation was successful using a remote MicroPython shell called rshell, which we can install with pip install rshell.

~ rshell -p /dev/cu.usbserial-0001 repl "~ print(\"hello world\")~" Using buffer-size of 32

Connecting to /dev/cu.usbserial-0001 (buffer-size 32)...

Trying to connect to REPL connected

Testing if ubinascii.unhexlify exists ... Y

Retrieving root directories ... /boot.py/

Setting time ... Dec 20, 2020 10:32:50

Evaluating board_name ... pyboard

Retrieving time epoch ... Jan 01, 2000

/Users/daniel.siwiec/workspace/private/projects/iot/esp8266-thermometer> Entering REPL. Use Control-X to exit.

>

MicroPython v1.13 on 2020-09-11; ESP module with ESP8266

Type "help()" for more information.

>>>

>>> print("hello world")

hello world ~

The command above opens a Python REPL on the board, runs a command, in our case a ‘hello world’ print, and exits. If you see the above output, your firmware has been successfully written to the board. If there are errors, it’s possible the baud rate was too high, in which case you need to erase the flash again and try writing the firmware with a lower baud rate, like 115200.

The application

Let’s now dive into the MicroPython application running on the device. The overall structure of the app is very simple — there’s some setup code, which in our case is mostly establishing the WiFi connection and then there is the main loop, which executes the main functions of the application. We will walk through the important pieces of the code, which is available in full form here.

Device Application structure

Main Loop

The main loop in this app measures the temperature and sends it over MQTT to AWS IoT Core.

Main Loop code

Temperature measurement

As explained in the circuit section before, the temperature measurement is based on the thermistor’s resistance calculation and the Steinhart-Hart equation for semiconductor’s temperature. The code below is a direct implementation of the equations explained in the Circuit section earlier:

Thermistor temperature calculation

Deep Sleep

Additionally, since this is a battery-powered device, it’s wise to utilize the ESP8266’s deep sleep feature which drastically reduces the power consumption by shutting down most of the device’s functions. In this mode, the ESP8266 draws around 20 uA, which would let it run for over 12 years on a 2200mAh battery I’m using. This is where the mysterious D0 <-> RST connection comes in — according to the ESP8266 pin-out, pin D0 provides a wake functionality, which sends a signal at a predefined time and by connecting it to the RST pin — allows us to bring the board out of the deep sleep mode.

ESP8266 Deep Sleep & wake-up code

MQTT

MicroPython comes with a simple MQTT client. There are a few important parameters that need to be passed, so let’s walk through them. Below is a simplified code snippet that connects to the AWS MQTT broker and sends a message:

MQTT connection code

The important parts are connection parameters — client_id and mqtt_topic which need to reflect the IAM policy we created before, mqtt_host which AWS provisioned for us, and ssl_params that configure the certificate-based authentication required by the AWS IoT Core. These parameters will get automatically injected later on.

Let’s now get the code ready to be uploaded to the device. All the necessary files are in the same repository we used for the AWS terraform setup. There are a few steps in this process.

Fill out WiFi credentials

The board needs internet connectivity to send messages to AWS. Open the app/props.json.tmpl file and fill out WiFi SID (network name) and password fields. Inject AWS IoT Core attributes

The application will authenticate with the AWS IoT MQTT broker using a certificate. Remember those terraform outputs? This is where they come in handy. The output contained the certificate and private key needed for authentication. Both the certificate and private key need to be converted from PEM to DER format and saved to a file. Run make props in the main folder to fetch these values, convert them to DER format, and save them to files. This command will also inject the MQTT broker host endpoint into the properties file. Upload files to the board

We’re now ready to upload the files to the board. We will use rshell (the same tool we used earlier to run REPL). Run this command: rshell -p /dev/cu.usbserial-001 cp app/* /pyboard/ . Replace the port with the one you used earlier.

Let’s now confirm the application is running correctly on the device. The app makes a few print outs throughout its lifecycle. All these prints can be read from the board’s serial port. There are a few ways to read this data — Mac and Linux come with a tool called screen which will let us do this. To connect to the serial port execute screen /dev/cu.usbserial-001 115200 , substituting the port for yours, as earlier. Now press the RST button on the board, next to the USB port, previously disconnecting the RST<->D0 wire. You should see an output similar to the screen below. Don’t worry about the gibberish at the start of the output.

Note: The ESP8266 is a very simple device, with an 80MHz CPU and 128kBytes of RAM. The SSL handshake involved in the AWS connection pushes the device to its limits and takes about a minute to complete.

You can now reconnect the wire (it’ll be needed for the device wake-up) and exit the screen pressing Ctrl+A and typing :quit . It is important to exit the screen explicitly, rather than closing the terminal window, as otherwise it’ll keep the connection with the serial port open and will prevent you from reconnecting to it.

Finally, let’s confirm the MQTT messages are landing in AWS. Go to the AWS console and open the IoT Core service. Select Test from the sidebar menu, enter the name of the topic (tempReading/bedroom) and click Subscribe to topic. Since our device sleeps for a few minutes after each message, you might need to wait a bit for the message to arrive. When it does, it should look like the screenshot below.

MQTT message in AWS IoT test client

You can also see the messages being persisted in the DynamoDB table:

DynamoDB table populated with temperature readings

When you get here, you now have an assembled device, with a temperature measuring code running on it and an AWS infrastructure to receive and persist the messages. The last element in the puzzle is visualizing the data. You can unplug the device from your laptop and swap in a USB battery pack. The one I used is an old 2200mAh and it lasted around 3 days.

Jupyter Notebook

Jupyter Notebook is an open-source project that provides a browser-based environment to run Python code, including visualizations. As AWS provides a ‘manged’ (created by AWS) Jupyter Notebook service, it is ideal for our use case to connect to the database containing the temperature reads and visualize it.

Go to the Sagemaker service in the AWS console and select Notebook Instances in the left pane. Your window should look similar to this:

AWS Sagemaker Notebook instance

Open the temp-vis instance and click on Open Jupyter in the upper right corner. This should bring you to this screen:

Now, click the upload button, select the assets/Temp Visualization.ipynb file and open it once it’s uploaded. Your screen should look like this:

Now, select Cell->Run All from the menu at the top, and you should see the temperature graph over time at the bottom of the screen. Of course, your graph will start taking shape as more datapoint flow into the cloud.

Congratulations! You now have a cloud-integrated IoT thermostat.

Update: ESP32

I ended up upgrading the board from ESP8266 to ESP32. It’s a very similar chip with a more powerful processor. This helps reduce the SSL handshake from ~45 seconds down to 2. Additionally, the ESP32 has an internal wake connection, which removed the need for RST<->D0 connection. The few code changes need for ESP32 are at the same repository, in the esp32 branch. The final, soldered device looks like this:

Front

Back

Note: All the code for this article can be found in my GitHub repository here.

Thanks to Clayton Gibb and Amber Houle for providing feedback on this article.