White Paper

DevOps for Embedded:
Automating Embedded Software Development

ByHari Menon
Anurag J K
Adhil Xavier
31 August 2021

White Paper

DevOps for Embedded: Automating Embedded Software Development

INTRODUCTION
Traditionally, embedded software is often written for a chosen application-specifichardware platform because of the limited computing capabilities and memory
availability. Hence embedded software implementation is mainly dependent on thetarget hardware platform. The speed of embedded software development is slower
than other development domains like web and mobile applications. Though theembedded platforms have become more powerful, the development process has
not changed much from the past. In this era of digitization and rapid change where
IoT is a critical component, improving the pace of development will be greatly
beneficial to reduce the time to market.
Embedded software is slowly becoming more independent from the hardware they
reside in, as the computing platforms are becoming powerful. Embedded softwaredevelopment is unique, but the essential principles of DevOps can be applied withthe right tools and the right approach. The concept of virtualization can significantlyhelp the developers to extract the available infrastructure for software development.
Software for configuration management can be used with automatic verification andvalidation. Configuration tools that support development and testing will help
accelerate the product cycle time. This will boost the confidence level for the initial
deployment of feature rollouts from production.

SCOPE
The scope of our study and experiment was to establish how best DevOps practices
can be adopted for increasing the pace of embedded software development. The
biggest constraint is that often testing must be performed in the target hardware
along with the peripheral devices and testing equipment, locally at the developers’
office. If the tests could be controlled and supervised over a network, avoiding theneed for a physical presence, that would be a leap jump. We found that starting withversion control and working towards automated testing, it’s possible to continuously
improve the embedded software delivery pipeline without the need for large-scale
reorganizations.

THE CHALLENGING SCENARIO
DevOps in an IT enterprise will have a generic development and execution
environment. Contrary to this, the embedded development environment is oftenvarying. The execution environment is usually different than the one used fordevelopment. Hence, after building the software, the deployment to target devices
will be specific for each hardware, and the test devices might be at remote locations.
The other concerns that need to be addressed are different target hardware crosscompilation, cross-debugging, memory footprints, and security issues.
Page 1

Trenser Technology Solutions (P) Ltd.

White Paper

DevOps for Embedded: Automating Embedded Software Development

The DevOps environment is a continuous feedback loop which is easy when thedevelopment is on servers that are always online and under the user’s control. On theother hand, embedded devices are likely to be remotely distributed, and they may
not always be online. So, various such issues come into play when considering
traditional DevOps for deployment.
The Ops part of DevOps for embedded is a challenge because, in a traditional
DevOps environment for the cloud, the Ops is standard for developers running awebsite or developing an application with a cloud interface. However, when dealing
with embedded, it is about devices in the field. The deployment (OTA) is not a
standard process and varies for different manufacturers, and no common approachis available. Hence it needs assistance from the device manufacturers and cannot be
a completely “closed-loop” situation, like in the case of a cloud-based enterpriseproduct.

OUR APPROACH
Unlike DevOps in cloud applications, the approach for Embedded needs to bedifferent. The key here is how far we can bring in automation. Most of the “Dev” partin DevOps can be reused by adjusting existing proven solutions. But when it comes to“Ops”, things become a bit complicated. There is no ready-to-use solution available.
Traditionally manual operation works best with a physical hardware environment.
However, those are also not far from automating. The potential approach here is a
device that can act as a broker in between the target hardware, which can performthe physical tasks in an automated way.
We are presenting an approach with a custom OS image that can be used todevelop, build, and unit test the embedded software, which will be deployed to thetarget hardware via a sandbox server running in a single-board computer likeRaspberry Pi or Beagle Bone. A generic web application in the sandbox server shallbe used for communicating with the target hardware. The figure below shows thehigh-level concept.

Page 2Trenser Technology Solutions (P) Ltd.

White Paper

DevOps for Embedded: Automating Embedded Software Development

Figure I: Overview

The technologies can vary based on the target platform to the skill set of thedevelopers. The best suited mix are:Development Phase

Automation Approach/ Tools

Dev

Customized OS Image in Docker or as VM in Virtual boxor VMWare.

Source Code Repository

GitHub, Gitlab or Azure

CI/CD Pipeline

GitHub Actions, Gitlab CI or Azure Pipelines

Deployment

Raspberry Pi or BeagleBone

Monitoring

NodeJS/ Python based web app viewed from Chromeor Firefox.

SOLUTION DETAILSThe figure below shows the step-by-step workflow of the proposed approach taken
by us in the specific case we experimented.

Page 3

Trenser Technology Solutions (P) Ltd.

White Paper

DevOps for Embedded: Automating Embedded Software Development

1. Prepare Base Image:First things, first. For every project, the initial challenge lies in setting up a working
environment that includes a test setup and a development environment. When it
comes to setting up a development environment for embedded, you need to set up
the toolchain, compiler, linker, etc., before starting. The ideal solution to avoid a delay
in starting is to prepare a common development environment that includes the basehost image installed with dependent libraries, toolchains, compiler, etc. This baseimage can be converted to a virtual machine image or as a docker container. Herewe chose Docker containers for ease of development.
2. Chose Git Repository Management Tool:
There is no confusion on choosing Git as the source code repository. But whenselecting a management tool, we have options like GitHub, Gitlab, Azure, etc. All ofthem have advantages and disadvantages. To get started quickly, we think GitHub isthe best choice.
3. Create CI/CD Pipeline:
Based on the repository management tool selected, the pipeline also differs. Here wewill make use of GitHub Actions as our CI/CD pipeline. The pipeline shall contain theworkflow jobs to pull off the development image, clone the repository, build andexecute unit tests. The result of the pipeline shall be the executables for the target
hardware.
4. Setup Webhook and Deployment:
In most cases, the real hardware may not have direct access to the internet bydefault. So, a broker device that will be running preferably on a Single Board
Page 4

Trenser Technology Solutions (P) Ltd.

White Paper

DevOps for Embedded: Automating Embedded Software Development

Computer like Raspberry Pi/ Arduino can be used as an interlink, which gets the build
image from the pipeline and transfers the same locally to the target hardware.
Considering the features and community support, we propose Raspberry Pi here. TheRaspberry Pi and the pipeline must be connected through a webhook notification
event. Whenever a new build is available at the cloud server, a notification will bereceived in the web application running in Raspberry Pi, and further deployment canbe initiated.
5. Device Setup
Peripheral devices are interfaced to the sandbox via a communication bus (say, via
I2C bus) and various hardware interfaces required by the target hardware and test
equipment involved. The target device also will need special software hooks tointeract with the sandbox server to act and respond to the hardware level interactions
via communication bus and various peripheral devices and test equipment. Thesehooks will also include redirecting the debug logs and hardware responses back tothe sandbox server via the communication interface.
6. Monitor/ Test/ Debug:
A NodeJS or Python web server will act as a “bridge” for the user who wants to perform
some actions on the actual hardware. The web application shall have the option to
select the interfaces to which inputs to be provided, and once the input is processed,
the output can be monitored as well. This application can be enhanced for real-time
debugging and performing functional tests by inputting more sophisticated scripts
with improved functionality.

A TYPICAL WORKING SETUP
In developing embedded systems, testing on the target device is mandatory tocapture bugs that may show up only in the real hardware. Figure 3 shows a typicalworking setup. For functional verification, manual testing is done by developers when
new features are implemented. This can be time-consuming and error-prone.
There should be a test branch that is properly named and maintained always to havethe latest source code (main branch). After implementing a feature, the developer
will push changes to the same branch. The compiling, building, and unit testing canbe done as part of CI/CD pipeline. Unit testing is also performed at this stage.
Developers need to update the scripts to perform unit tests. If the unit test fails, thedeveloper will get notifications regarding the status and will stop other processes
involved.

Page 5

Trenser Technology Solutions (P) Ltd.

White Paper

DevOps for Embedded: Automating Embedded Software Development

Sandbox server will get a webhook notification event whenever a new build isavailable from the pipeline task. Upon receiving notification, the Sandbox server will
fetch the latest binary and make use of the programming hardware for flashing thebinary to the test device or directly installing it on hardware.
Developers need to create a Flash script that contains the commands for flashing ortransferring the executable to the test device. The programming hardware may be
JTAG, serial, or any other interface. Sandbox server must be installed with proper
drivers and software required for the programming hardware. In many cases, theprogramming hardware might not be necessary, with the availability of in-circuit
programming capability of the platform, and hence a simple SSH file transfer will be
all that is needed.
Once the binary file is flashed successfully, the total functionality can be tested.
Functionality testing is done using an F-test script. This script will utilize special
commands for interacting with the peripheral devices. Necessary drivers forsupporting the peripherals shall be readily installed in the Sandbox server. Peripheraldevices are connected to sandbox via a communication bus (say, via I2C bus) andcan be used for (but not limited to) the following:-GPIOs / Relays-UART-SPI-

I2C

-CAN

Page 6Trenser Technology Solutions (P) Ltd.

White Paper

DevOps for Embedded: Automating Embedded Software Development

If any F-test script or flash script fails, an email will be sent with required details, andcorresponding logs of the failed script will help determine the cause of failure. The test statuswill be sent to the concerned personnel upon completion of the test as well.

Figure IV: Accessing the Raspberry Pi terminal through which the user can get
access to the device if required.

Figure V: A CAN peripheral page where user can view and send CAN
messages in device

Page 7

Trenser Technology Solutions (P) Ltd.

White Paper

DevOps for Embedded: Automating Embedded Software Development

A web application will allow developers to opt for the required setup to perform thetest. It will provide an interface where the developers can interact with the real
hardware. They could even manage their scripts used for building (Build script),
flashing (flash script), or even for the total functionality test (F-test script). Thedevelopers are given a chance to customize the test by opting for the peripherals
required to perform the module-level testing.
With consistency in the integration process in place, teams are likely to commit codechanges more frequently, which leads to better collaboration and software quality.
Continuous Delivery picks up where continuous integration ends. Continuous Delivery
is established by automating quality gates and building confidence in the code tomove from development to production. Continuous Deployment starts with manualapprovals and identifies how to establish and then automate the quality gates.

CONCLUSION
The practices and tools described in this paper can be effectively utilized toimplement a DevOps-like system in embedded software development. Bare metalsystems are, however, not considered. This approach can be adopted for moderately
powerful computing platforms, especially platforms that run any RTOS or Embedded
flavors of Linux. Our approach features development, testing, and deployment (untilthe product testing stage) in the Embedded platform. Creation of the base image
shall be a one-time process during the start, and whenever a new developer joins theteam, he can be productive from day one; thus, the developer can adapt to a newproject with minimum effort and time.
As we are using an automated system for continuous testing, which is a timeconsuming and repetitive task, developers will be relieved to work on moreproductive and fruitful tasks. Identifying, tracking, and documenting bugs now
becomes seamless as proper logs and reports are generated.
The automated deployment process can be further extended to move your softwarefrom testing and production stages to field deployments by automating devicespecific OTA processes. However, this needs several considerations of the installed
environments and application criticality, as indicated in the challenges section. If
implemented, it creates a repeatable deployment process across the entire softwaredelivery cycle. This helps you release new features and applications more quickly andfrequently, reducing the human interventions in the deployment.

Page 8Trenser Technology Solutions (P) Ltd.