Detecting Pokemon on an Arduino using TinyML and TensorFlow
By: Juan De Dios Santos
Tags: Editors Pick, Programming, Arduino, Towards Data Science, Machine Learning

After training, I predicted using the test dataset. Then, I converted the outcome to 0 or 1, depending on whether the likelihoods are smaller or greater than the 0.50 threshold. Last, I used scikit-learn’s classification report to assess the model’s performance. Figure 11 shows the numbers.

Figure 11: Test dataset’s classification report.

With 0.97+ values all over the place, we can assume the model works as intended.

Converting the model

We can’t use this model on the Arduino as it is. First, we must convert it to a TensorFlow Lite model and then encode it as a byte array in a C++ header file. Like this (add the following lines to the training script):

# Convert the model

converter = tf.lite.TFLiteConverter.from_keras_model(model)

tflite_model = converter.convert() # Save the model to disk

model_path = "model/pika_bulba{}.tflite".format(ts)

open(model_path, "wb").write(tflite_model) print("Model is {} bytes".format(os.path.getsize(model_path)))

These lines convert the model into a TensorFlow Lite model and save it to disk. After exporting it, we check the model’s size (in bytes) to ensure it is small — mine is 984 bytes. Then, from the terminal, run xxd -i model_path > model.h to encode the model as an array of bytes in a C++ header file named model.h. This file is the model we will load in our Arduino program.

It should look like this (I removed most of the values for simplicity):

unsigned char model[] = {

0x1c, 0x00, 0x00, 0x00, 0x54, 0x46, 0x4c, 0x33, 0x14, 0x00, 0x20, 0x00,

0x04, 0x00, 0x08, 0x00, 0x0c, 0x00, 0x10, 0x00, 0x14, 0x00, 0x00, 0x00,

0x18, 0x00, 0x1c, 0x00, 0x14, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00,

...

};

unsigned int model_pika_bulba20210108_190240_tflite_len = 984;

The Arduino’s classifier Sketch

Now comes the part of the tutorial we have been waiting for: writing the Arduino’s classifier Sketch. As the data gathering Sketch, I used an official example as a base and adapted it to my case. To explain it better, I’ll split the code into three parts: the includes statements and variable initialization, and the setup and loop functions.

In Arduino’s IDE, go to File -> New to start a new Sketch. Create a new file, name it classifier.ino, and add:

/*

Object classifier by color

-------------------------- Uses RGB color sensor input to Neural Network to classify objects

Outputs object class to serial using unicode emojis Note: The direct use of C/C++ pointers, namespaces, and dynamic memory is generally

discouraged in Arduino examples, and in the future the TensorFlowLite library

might change to make the sketch simpler. Hardware: Arduino Nano 33 BLE Sense board. Created by Don Coleman, Sandeep Mistry

Adapted by Dominic Pajak This example code is in the public domain.

https://create.arduino.cc/editor/TensorFlowExamples/8508c70f-5155-4e3b-b982-c5f6bd36ea5c/preview.

I've modified it to suit my use case.

*/ Example inspired byI've modified it to suit my use case.*/ #include <TensorFlowLite.h> #include "tensorflow/lite/micro/all_ops_resolver.h"

#include "tensorflow/lite/micro/micro_error_reporter.h"

#include "tensorflow/lite/micro/micro_interpreter.h"

#include "tensorflow/lite/schema/schema_generated.h"

#include "tensorflow/lite/version.h"

#include "Arduino_APDS9960.h"

#include "model.h" // Set up logging

tflite::MicroErrorReporter tflErrorReporter; // Add all the TensorFlow Lite Micro operations

tflite::AllOpsResolver tflOpsResolver; const tflite::Model *tflModel = nullptr;

tflite::MicroInterpreter *tflInterpreter = nullptr;

TfLiteTensor *tflInputTensor = nullptr;

TfLiteTensor *tflOutputTensor = nullptr; // Create a static memory buffer for TFLM, the size may need to

// be adjusted based on the model you are using.

constexpr int tensorArenaSize = 8 * 1024;

byte tensorArena[tensorArenaSize];

This code has the include statements that load the libraries and initializes several variables. The first of them, tflErrorReporter , is a logger that's passed to the interpreter to write logs. Following it is tflOpsResolver , an instance of AllOpsResolver , a class that loads all the TensorFlow Lite Micro library operations. Then we have the model, the interpreter (responsible for predicting), the tensors that hold the model's input and output, and memory allocation for the input, output, and intermediate sensors.

Before continuing, create a new file named model.h and copy/paste the encoded model we created in the previous section.

Next is the setup() function (add this portion of the code after the byte tensorArena[tensorArenaSize] line).

void setup()

{

Serial.begin(9600);

while (!Serial)

{

}; Serial.println("Pokemon classification using RGB color sensor");

Serial.println("--------------------------------------------");

Serial.println("Arduino Nano 33 BLE Sense running TensorFlow Lite Micro");

Serial.println(""); if (!APDS.begin())

{

Serial.println("Error initializing APDS9960 sensor.");

} // Initialize the led's.

pinMode(LEDR, OUTPUT);

pinMode(LEDG, OUTPUT); // Ensure it is off by default

// On the Arduino NANO 33 BLE Sense, HIGH is off.

digitalWrite(LEDR, HIGH);

digitalWrite(LEDG, HIGH); // get the TFL representation of the model byte array

tflModel = tflite::GetModel(model);

if (tflModel->version() != TFLITE_SCHEMA_VERSION)

{

Serial.println("Model schema mismatch!");

while (1)

;

} // Create an interpreter to run the model

tflInterpreter = new tflite::MicroInterpreter(tflModel, tflOpsResolver, tensorArena, tensorArenaSize, &tflErrorReporter); // Allocate memory for the model's input and output tensors

tflInterpreter->AllocateTensors(); // Get pointers for the model's input and output tensors

tflInputTensor = tflInterpreter->input(0);

tflOutputTensor = tflInterpreter->output(0);

}

It starts by setting up the serial, printing some lines, and waiting until the sensor is ready. Next, it initialize the red and green LEDs and turn them off. Then, it load the model, initialize the interpreter (note the arguments are the variables declared earlier), allocate memory, and get the pointers to the model’s input and output. Now comes the loop() function (write it after setup() ):

void loop()

{

// a is the ambient light intensity.

int r, g, b, a, p;

float sum; // Check if both color and proximity data sample is available.

while (!APDS.colorAvailable() || !APDS.proximityAvailable())

{

} // Read the color and proximity sensor.

APDS.readColor(r, g, b, a);

p = APDS.readProximity();

sum = r + g + b; // Check if there's an object close and well illuminated enough.

if (p == 0 && a > 10 && sum > 0)

{ // Normalize the values.

float redRatio = r / sum;

float greenRatio = g / sum;

float blueRatio = b / sum; // Input sensor data to the loaded model.

tflInputTensor->data.f[0] = redRatio;

tflInputTensor->data.f[1] = greenRatio;

tflInputTensor->data.f[2] = blueRatio;

// This is a great guide explaining the process

//

TfLiteStatus invokeStatus = tflInterpreter->Invoke();

if (invokeStatus != kTfLiteOk)

{

Serial.println("Invoke failed!");

while (1)

;

return;

} // Invoke the inference.// This is a great guide explaining the process// https://www.tensorflow.org/lite/guide/inference TfLiteStatus invokeStatus = tflInterpreter->Invoke();if (invokeStatus != kTfLiteOk)Serial.println("Invoke failed!");while (1)return; // 0.50 is my threshold

if (tflOutputTensor->data.f[0] < 0.50)

{

Serial.print("Pikachu: ");

// Turn on the red and green LEDs to get yellow.

digitalWrite(LEDR, LOW);

digitalWrite(LEDG, LOW);

}

else

{

Serial.print("Bulbasaur: ");

// Turn on the green LED.

digitalWrite(LEDG, LOW);

} Serial.print(float(tflOutputTensor->data.f[0]), 5);

Serial.println(); // Wait until the sensor does not detect the object.

while (!APDS.proximityAvailable() || (APDS.readProximity() == 0))

{

} // Turn off the LEDs.

digitalWrite(LEDR, HIGH);

digitalWrite(LEDG, HIGH);

}

}

The loop() function first lines are the same from the color sampler Sketch; it declares the colors, light intensity, proximity, and sum variables and checks if there are color and proximity data available. If so, we read the color values and check if the object is near and the scene well-illuminated. Inside the if we normalize the values and use them as the model's input in the tfInputTensor variable. Then, we use the interpreter to invoke the inference (equivalent to Python's TensorFlow's predict). Unlike other ML libraries, invoking the interpreter does not return a prediction vector; instead, it returns the prediction's status (whether it succeeds or failed).

To get the predicted value, we must use the output tensors declared in the setup function (the interpreter passes by reference the predictions to this tensor). While reading the prediction, we check if the value is below the 0.50 threshold. If true, then the predicted value is Pikachu, so we print “Pikachu” and turn on the red and green LEDs — red + green = yellow and the Arduino only has a red, green, and blue LED. Else, we print “Bulbasaur” and turn on the green LED. Last, we print the prediction, wait until the sensor does not detect the object, and reset the LEDs.

And that’s it! Happy inferring :)

Here are some examples (note the Arduino’s LED color):

Figure 12: Pikachu detected!

Figure 13: Bulbsaur detected!

Recap and conclusion

TinyML is the area of machine learning concerning models suitable for low-powered and embedded devices. In this article, I’ve presented an example of how to use one of these, an Arduino Nano 33 BLE Sense, alongside the TensorFlow Lite for Microcontrollers library to create a program that classifies my Pokemon plushies into Pikachu or Bulbasaur. During the tutorial, I explained how you could collect data using the Arduino, presented a data analysis that explored the dataset’s colors, trained a logistic regression model in TensorFlow, exported said model to TensorFlow Lite, and wrote a Sketch that loads the model to the Arduino so we can predict from it.

For more information about the Arduino and TensorFlow Lite for Microcontrollers library, see the following links:

You can find the complete code (including my datasets and model) at:

Thanks for reading!