Building a Network Controlled Robot With Arduino and Raspberry Pi
By: Jason Bowling
Tags: DIY, Raspberry Pi, Electronics, Robotics, Arduino

Power

One of my primary discoveries on this robot was just how useful it is to have an umbilical cable running from a bench power supply that can power the robot during testing on the bench. You don’t want to have to sweat battery charge state during those long debug sessions. It’s also a good idea to put the robot up on blocks so that it can’t drive off the bench at full throttle if you make a mistake. It saved my bot from unintended carnage more than once.

Any power source over 7.2V can be used. If you are going to use a voltage higher than the motors are rated for, it’s a good idea to limit the max PWM rate to prevent over-driving the motors. My code doesn’t currently do this — I’d rather live on the edge and have more power available, but it will surely reduce the life of the motors running at high power.

Note that lithium polymer batteries without protector circuits (RC batteries) carry significant risk due to their energy density. You should familiarize yourself with them if you choose to use them.

Top view

Cover

The chassis cover is foamboard. I like making the first iteration of a bot from easy to work materials to allow quick prototyping. Maybe the next one will be aluminum.

The Raspberry Pi camera is mounted in a 3D printed enclosure

Software Overview

Starting from the bottom up, we have the code running on the Arduino, listening for commands on the serial port from the Raspberry Pi. It turns these commands into outputs to the motor controller, and gets the battery voltage. It response with a serial acknowledgement string containing the battery voltage.

The little Python server running on the Pi opens a network socket and listens for connections. It also opens a serial connection to the Arduino. Commands received on the network socket are sent to the Arduino. Additionally, it gathers telemetry data from the Pi and inserts it into the reply string along with the Arduino’s reply, and sends it back to the control computer over the network socket.

The PC client program gets input from the user in the form of gamepad positions. It turns these into motor speeds, packages those speeds into the expected command string format, and sends it off to the Raspberry Pi’s control server. It displays the sensor values for the user to read.

Arduino Software Design

The Arduino has a simple but critical job. It has to listen for serial commands from the Pi and perform actions like driving the motors, checking battery voltage, and turning on and off the lights. Most importantly, if it doesn’t get a command from the Pi, it needs to STOP THE ROBOT, quickly. If your control program crashes or you drive out of range, you don’t want it to go tearing off at full speed. The Arduino code is set up to stop the motors within a second if it doesn’t get a valid command string from the Pi.

I chose to use a string with a human readable format, without trying to make it super efficient. The bandwidth used for control is quite small compared to the video stream, so I wasn’t worried about that. There are a number of ways this can be tightened up if you have a more constrained link. (A LORAWAN link to a semi-autonomous bot would be Mars-rover cool.)

The Arduino looks for strings in this format. It’s easy to parse with a colon delimiter.

C:80:80:1:1:E

C indicates start of command

first 2 digits: motor speed percentages, positive or negative to set direction

third: should I return sensor values, 0 or 1

fourth: 1= lights on, 0 = lights off

The command string ends with E, which is easier than detecting newline.

Low Latency Video Streaming

There are at least a half dozen ways to stream video from the Raspberry Pi’s camera to a remote computer. I’ve tried the majority of them, and they all have pros and cons. My decision on how to do it was driven purely by performance — I wanted the best possible video feed for remotely controlling a robot, and that meant I needed the least delay. Driving a vehicle with even a half-second delay is difficult and not a lot of fun.

The method that won out is great quality and with very low latency. It streams to a dedicated video viewing program on the client computer.

The output of the Raspberry Pi camera is sent to a Linux program called Netcat, which opens a server on the Pi and lets a network client directly connect.

On the Pi, the following is included in a script that starts with the robot. all one command:

raspivid -n -pf low -t 0 -b 2500000 -w 800 -h 600 -ih -fps 20 -o — | netcat -lkv4 2222 > /dev/null 2>&1

The arguments to raspivid set the bitrate, framerate, and resolution — you can adjust those to your liking. The -o argument tells raspivid to send it to the stdout pipe, which would normally print to your console. Instead, we pipe that to netcat and use it to open up a TCP server on port 2222. Finally, we drop any output from netcat. Alternatively, you could log that output, which might be useful for debugging.

On the client computer, we open up mplayer and tell it to get it’s input from a network stream, like so:

mplayer -geometry 50%:0% -fps 200 -demuxer h264es -noautosub -noass -nosound -framedrop -nocorrect-pts ffmpeg://tcp://192.168.1.214:2222

Note that the address of my robot in this example is 192.168.1.214 — that changes depending on what network you are on. An easy way to find it on your local network is to use nmap to look for machines listening on port 2222.

Control Protocol

The command and sensor stream are pretty simple. The PC control client grabs the position of the gamepad and makes the command string that the Arduino expects.

C:100:100:0:0:E

This example commands both motors full speed ahead, request no sensors, and headlights off. Because nothing else is requested, the robot server replies:

OK:

When sensor readings are requested, the command would be:

C:100:100:1:0:E

and the robot’s server replies:

OK:batteryVoltage:signalStrengh:

For example:

OK:12.54:-20:

I only poll the sensors every 20 or so command strings, because there is a little overhead associated with polling the sensors, and I don’t need super-frequent updates on them. They just don’t change that fast. That might change depending on what you were measuring — accelerometers, for example, might need a faster polling.

Raspberry Pi Control Server

With the video stream being handled by raspivid and netcat, the control server on the Pi really only has a couple functions:

Open a TCP port and accept incoming connections Pass the incoming command strings from the network client to the Arduino via serial. Gather the desired telemetry from the Pi itself and send it to the client. I chose to only send back wireless signal strength, to avoid driving the robot out of wireless range, but you could easily send back temperature, CPU utilization, memory use, system logs….

The client is written in Python 2 because it was a spinoff of an old project, and I have not yet gotten it working correctly in Python 3 yet. That’s a project for another day…

In addition to getting battery voltage back from the Arduino, the control server makes a system call on the Pi to get the wireless signal strength and parses the output.

PC Control Client

As mentioned previously, the video display is handled nicely by mplayer, so the client must do the following:

Get input from the gamepad — stick and button positions Translate the stick movements into relative motor speeds to produce the desired motion. Construct the serial command string that the Arduino expects and send it to the control server on the robot. Parse the reply containing acknowledgement and/or sensor values and display them for the user.

There are a bunch of ways to do this, but for ease of gamepad use and graphics display, I chose Pygame. The client requires Python3.

You can install the prerequisites with:

sudo apt-get install python3 python3-pip joystick python3-pygame mplayer

The most interesting part of the control client was the process of turning stick movements into motor speeds. It looks simple at first, but with a differential drive like a tracked vehicle, it requires some thought. For example, if you turn the stick right or left, with no forward motion, you probably want the robot to rotate in place. That’s easy — you run one motor forward and one in reverse, at the same speed.

If, however, you are pushing forward on the stick, and move the stick left or right as well, you probably want the robot to steer slightly while rolling forward. For that to happen, both motors are moving forward, but at slightly different speeds, so you have to check for that condition and compute the motor speeds differently. It’s fun, and enormously satisfying to get working.

If you want to record the client screen while on a drive, Simple Screen Recorder works great in Ubuntu.

Gamepad — note the LED pattern

Getting the Gamepad Working In Ubuntu

I used a generic Xbox-style gamepad. It took a bit of research to get it to work, but the answer turned out to be pretty simple. This controller (and likely others like it) can operate in different modes, and the default one doesn’t work with Ubuntu. If you change modes to match this LED pattern, by pressing the Select button repeatedly, it works great. I only had to do this once — it retains the setting the next time you plug it in.

To install the required software for the gamepad on your client computer, do:

apt-get install joystick jstest-gtk

The jstest-gtk utility is excellent for confirming that your gamepad works and buttons are mapped the way you expect. This is confirmed working for Ubuntu versions 18.04 through 20.04.

I built the client program around a modified version of this demo program, which is well documented and easy to understand.

Results, Ideas For Improvement and Next Steps

The robot works very well. Gamepad control is smooth and intuitive, and video is smooth. The auto-stop when network connectivity is lost works fine.

This serves as a fine jumping off point for more advanced control. Here are some of the ideas I’ve come up with.

More robust error checking in the robot’s control server. Right now it’s just a network to serial pass-through. If you put a cell modem on board the robot and had Pi VPN back to a home base, I think you could drive it from anywhere! How cool is that? A printed circuit board would be a lot cleaner than point to point wiring, once you solidified your design. Security. Right now there isn’t any. At all. Authentication, firewalling.. Better power management. Right now, if the robot’s battery gets too low and a human can’t get to it and power it down, it will continue to draw power just sitting there. That will harm the battery. A low voltage cutoff would likely be better. Watchdogs on the Raspberry Pi’s video and control servers. If those crash, you really want them to automatically restart so you can drive the robot back to you. There is still a bug that causes the client program to hang sometimes when the robot is driven to the outer edges of the network and connectivity gets sporadic.

Conclusion

I’ve had a blast. I hope that this write-up saves you some time or sparks ideas for a robot of your own. Thanks for reading! I’d love to hear about your projects — feel free to start a discussion in the comments!