TensorFlow, Meet The ESP32. How to Set Up TensorFlow Lite for ESP32... | by Wezley Sherman | Towards Data Science

If I had to choose a favorite field of computing, I would choose Embedded Systems. I’m a fan of the challenge that optimizing code to run in a resource-constrained environment presents.
When I realized that TensorFlow Lite supports microcontrollers, I flipped my desk — with excitement.
I have a couple of the ESP32-CAM modules that I’m wanting to use with a home security system. My idea is to deploy a model that recognizes people and starts recording as soon as the camera picks up on a person. I don’t want to use motion-sensing here because I have a dog who would just trip the sensors.
I see TensorFlow Lite as being a great tool for this use-case. I can train a model to recognize people on my desktop, and then deploy it to my ESP32-CAM modules.
After reading through the TensorFlow Lite documentation it became apparent that using PlatformIO wasn’t going to be as easy as calling:
platformio lib install tfmicro
Unless I want to use an Adafruit library (spoiler: I don’t).
With some careful trial and error, I was able to get TensorFlow Lite to play nice with PlatformIO on the ESP32.
This guide will go through how I got the “Hello World” example compiling and uploaded to an ESP32-CAM module using PlatformIO with Arduino-ESP32 support.
The first thing you’ll want to do is install PlatformIO. To do so open up a terminal and type:
pip install -U platformio
Now, create your project’s root directory. This directory should also contain sub-directories for src, lib, and include.
Within your project’s root directory, create a file named platformio.ini. This file will contain all of the information needed for PlatformIO to initialize your development environment. Mine looks like:
Next, you’ll need to create a file named custom.csv. This is the partition information for the ESP32’s flash. You can format the partition table based on your application’s needs and ESP32’s flash size. More information on ESP32 partition tables can be found here. Below is how my custom.csv file is formatted:
With all of that set up, go ahead and clone the TensorFlow repository (TensorFlow Lite is included). You can download this as a .zip and extract to a directory of your choosing, or clone using git:
git clone https://github.com/tensorflow/tensorflow.git
Once you have the TensorFlow repository downloaded, generate one of the sample ESP32 projects from the TensorFlow Lite folder. We want to generate a sample project so we can grab the tfmicro library that is generated and the sample model. To generate the sample project, navigate to the root ‘tensorflow’ folder and run:
sudo make -f tensorflow/lite/experimental/micro/tools/make/Makefile TARGET=esp generate_hello_world_esp_project
This will generate a sample project that’ll be located in:
tensorflow/lite/experimental/micro/tools/make/gen/esp_xtensa-esp32/prj/hello_world
Navigate to the hello_world/esp-idf directory and copy the tfmicro folder from components into the lib folder that you created earlier. If you plan on running the sample program to ensure you set up your environment correctly, then copy over sin_model_data.cc and sine_model_data.h from the main folder to your src and include folders.
Your file structure should now look something like:
Pretty easy so far, huh?
You are almost done! You just need to tweak a few things in the tfmicro library folder so PlatformIO can see all the third-party libraries TensorFlow Lite requires.
Navigate into the tfmicro folder. You should see two folders: tensorflow and third_party.
Go into third_party/flatbuffers/include and copy the flatbuffers sub-directory into the tfmicro root.
Next, go into third_party/gemmlowp and copy both the fixedpoint and internal directories into the tfmicro root directory.
Finally, copy over the kissfft directory from third_party into the tfmicro root directory.
At this point, you can go ahead and delete the third_party directory.
By moving all of the third-party libraries into the tfmicro root, PlatformIO can recognize and use them.
The final structure of your project should look like:
With the file structure finished, navigate to lib/tfmicro/flatbuffers and open base.h.Within base.h, change line 34 from:
#if defined(ARDUINO) && !defined(ARDUINOSTL_M_H)  #include <utility.h>#else  #include <utility>#endif
to:
#include <utility>
You are now done! The only thing that’s left to do is import and use TensorFlow Lite in a project.
To test that TensorFlow Lite was working correctly, I adapted the “Hello World!” sine model to be used in an Arduino-ESP32 sketch.
I’ll guide you through what I did below!
The first thing I did was import all of the libraries the project will use.
The libraries are as follows:
Arduino .h— Arduino support! :)
math.h — Specifically used here for definition of M_PI (Will discuss later).
all_ops_resolver.h — Defines operations used to run the model.
micro_error_reporter.h — Used for error reporting.
micro_interpreter.h — Runs the model.
sine_model_data.h — The sample model we are using.
Next, I set my variables:
The first global variable I defined was the memory pool to store the arrays generated by the model. TensorFlow’s documentation states that you may have to come up with the pool size from experimentation for different models. For this, I just went what was in the sample code that TensorFlow provided for running the sine model.
Then I defined the model, interpreter, and input/output nodes.
With global variables defined, it’s time to set up the environment.
Within the ‘setup()’ function, I started serial at 115200 baud and imported the sine model.
Then I instantiated the TensorFlow Lite Micro Interpreter
Finally, I allocated the model’s tensors into the memory pool that was defined as a global and I set the input and output variables to their corresponding nodes.
That concludes the ‘setup()’ function. In a gist: I imported the model, created the interpreter, and loaded the model into memory.
With the ‘setup()’ function finished, I started implementing the control loop logic.
For the control loop, I waited for user input from serial and converted it to a float. Then I checked to make sure the user input was within the model’s params (0–2*PI):
I then set the input node for the model to parsed user input, invoked the interpreter, and then printed out the result:
With the code finished, you can deploy to the ESP32 using PlatformIO with:
platformio run -t upload --upload-port /dev/ttyUSB0
You can then give the program input through serial by using:
screen /dev/ttyUSB0 115200
That’s it! I’m genuinely surprised at how easy TensorFlow makes it to deploy models to microcontrollers. My next article will be on how to create a model using TensorFlow and how to convert it to something that can be run by TensorFlow Lite!
GitHub Project Repo: https://github.com/wezleysherman/ESP32-TensorFlow-Lite-Sample