Unit testing for embedded system - Page 1
Forum section: electronics / general

author: Kittu20
author_rank: Regular Contributor
timestamp: February 10, 2024, 04:14:24 pm
content: I've been working with the PIC18F45K80 microcontroller using MPLAB XC8, C language and recently I came across the concept of unit testing. From what I gather, it involves checking individual components of a program, such as functions or modules, to verify their correctness. I think there are two possible methods: utilizing a third-party library or developing our own custom libraryHowever, I'm still unclear about its practical utility and how to implement it effectively in embedded systems I'd appreciate hearing about your experiences with unit testing in embedded systems. Do you incorporate unit testing into your development process?

author: PlainName
author_rank: Super Contributor
timestamp: February 10, 2024, 04:57:30 pm
content: The idea with unit testing is that you can verify that some code works as expected (and not as not expected). Useful when you are writing the code but also later when you make changes to a system. Essentially, you design the specification of something (perhaps binary to hex character) and then write a test that would prove an implementation completely works (or not! - give it max and min values, edge cases, data you know is problematic, etc), and then you write the actual implementation which is exercised by your test in all those ways. Your project would have many of these tests and the passing of them all would show that some change you've made hasn't screwed things up.At least, that's the blurb. It's a useful means for writing code for hardware that doesn't exist (yet, perhaps), although that can be problematic if your virtual hardware test assumes it works in a way it actually doesn't.Anyway, it can be hard to get into and it can be a pain to keep working to, but I think it is a plus overall. Since you write in C you're already limiting available options, but the one I use isCppUTestwhich is available on github. It is C++ based, but the target system is C so you don't need to either learn C++ (much) or change your language.Ultimately, even if you decide it's not for you, realising how to better partition you code, how to reduce dependencies and better modularise it, should be a win anyway.

author: Jeroen3
author_rank: Super Contributor
timestamp: February 10, 2024, 05:08:41 pm
content: Yes, testing is great! However, if all you've done now is interacting with compilers and code through an IDE like MPLAB you're in for a steep learning curve.You use unit testing to validate your software units. Eg: functions or libraries.Most people run this automatically on one of the git hosters and a cloud or their own runner.C makes unit testing tricky, because in order to test it must compile. And in embedded it's often very messy with dependencies. Especially when using vendor stuff.And since you don't want to emulate the entire chip, you substitute what your code depend on, simulate/mock or stub it.Simply put: you substitute what #include imports for test emulations.This is the hardest part, because it exposes your software design flaws. And correcting this is often a lot of work.Another tricky part is that you need to compile it to run in Linux/Windows, so you need to substitue compiler specific syntaxes.You can also make it run on pic but then you need a host board for it and write scripting to compile/load/run and view the results. People do this, but it's for the advanced user.A very easy start would be to download visual studio with c/c++ support and create a mstest/googletest project and include the files you want to test.The frameworksgoogletest/mstest offer various tools for running tests. Pay attention, most testing suites assume you run Linux.ci/cdAutomating stuff like this can be "as simple" as using GitLab pipeline default linux runner compiling it with your makefile and run the tests. Of course this requires you know how to compile your code from scratch with makefiles because there is no IDE....Or, create a VM with gitlab runner, put it on a pc in the corner and run your own ci/cd tests. Basically every git commit will then be sent to the runner and your automations will run, eg: run the visual studio googletest project. And a nice checkmark will show next to your commitsAnd it will yell at anyone who commits something that fails tests.exampleA unit test for a library such as an eeprom wear leveling storage driver you can verify if it writes to the right addressed and converts datatypes correctly.You can mock the HAL for eeprom access to access a big array and since you also have access to the big array you can check that the test function writes the right data to the right place.

author: Kittu20
author_rank: Regular Contributor
timestamp: February 10, 2024, 06:34:28 pm
content: Thank you for the detailed explanation. I appreciate the clarity you provided.If you've had experience with unit testing on the ARM platform, particularly with STM32 devices, I'm curious to know whether you utilized a third-party library or developed your own custom library for unit testing?

author: nctnico
author_rank: Super Contributor
timestamp: February 10, 2024, 06:54:11 pm
content: A first step towards testing inside / outside an embedded platform is to use the same tools for the embedded platform and on the PC platform you use for testing. This is much easier to do for ARM while using gcc / clang compared to proprietary compilers where it comes to accuracy of the results. But you can still get bitten by unexpected results if your coding style is sloppy. Or put differently: your code is open to interpretation by the compiler. Mapping variables or structs onto byte buffers is one example where you can run into nasty surprises.

author: ejeffrey
author_rank: Super Contributor
timestamp: February 10, 2024, 09:08:42 pm
content: It really helps if you design your code with testing in mind. In particular making sure that hardware specific code is compartmentalized. Then you can sub those out with code that e.g. logs the transactions or provides fake data that you can then verify is handled correctly. This is of course good practice anyway for portability and debugging but doubly so when you want to run tests. A random write directly to a GPIO peripheral will crash in the test environment if it doesn't get stubbed out because it bypasses your hardware abstraction layer.It's also possible to do full system tests in an emulator like qemu. Depending on how deep you want to go you can design virtual drivers for all your peripherals and run the unmodified binary in the emulator. That's usually more work than it's worth but i have used it when I wanted to test that my program wrote to specific IO memory address in a certain order.

author: PlainName
author_rank: Super Contributor
timestamp: February 10, 2024, 09:43:13 pm
content: QuoteActually, I think it's better to have different compilers for that case, to catch out compiler-specific code. It would be fine on the current product but what about if/when you port to a bigger and better processor (or smaller, cheaper)? Of course, one should be aware of that kind of thing and code appropriately anyway, but you'd never know until the fan starts spreading if you've actually managed that.

author: hans
author_rank: Super Contributor
timestamp: February 10, 2024, 11:20:52 pm
content: Yes I use unit testing extensively.TDD (Test Driven Development) is a paradigm where you first write a test, and then implement it. This forces you to think about how the code should behave externally.However, do mind a issues with unit testing on embedded, as there a few additional criteria to regular software:- Code may rely on hardware peripherals- Code may rely on vendor libraries- A project may be sensitive to time deadlines (realtime)I want to emphasize 2 distinct approaches to unit testing: you could do some kind of structured/automated testing on hardware itself, or you could crosscompile your C code to another platform (x86 on your PC) and run the code there. The latter is just a behavioural verification.Its usually quite a big step to properly set-up unit tests on a target platform itself. Its only useful for hardware tests, and if you want to test your hardware properly, you will need some oscilloscope or other external gear that can be remotely managed to exercise your firmware. It will require a specific setup which not every developer may have. So it has some use cases.. but for most projects I "manually" debug.It can be useful for qualification purposes.. e.g. you make some automated tests for battery life measurements on daily builds of your firmware.Everything that is above hardware is unit tested on PC though. Sometimes I may even go to the effort to make an emulation/simulation environment so I can actually run parts of the embedded application on my PC. For example for networked devices, it can be useful to test them with more powerful PC tools.Unit testing does not help in verifying if code is "always" correct though. For example, you can still write 'bad' code that in isolation works, but starts to break when timing deadlines are missed. Simiarily, code can also break on race conditions that are not reproducible on PC. Code can also break on non-portable code, which may work on 64-bit x86, but breaks on any other platform.

author: Jeroen3
author_rank: Super Contributor
timestamp: February 12, 2024, 01:50:52 pm
content: Unit testing is especially valuable in giving your a sense of trust in your code building blocks. In a way that you will not be hunting bugs in communication protocols cause by a glitch in the CRC function.The CRC function is an excellent example of code that is easily tested.

author: wek
author_rank: Frequent Contributor
timestamp: February 12, 2024, 02:13:11 pm
content: Give me your test, and I will write you a glitch.JW

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 12, 2024, 08:21:15 pm
content: Quote from: wek on February 12, 2024, 02:13:11 pm

author: djsb
author_rank: Super Contributor
timestamp: February 12, 2024, 08:44:48 pm
content: So would something as simple as flashing an LED or using a serial output for each function entered or exited be classed as a unit test? Or is this something much more complicated?

author: Cavhat
author_rank: Contributor
timestamp: February 13, 2024, 01:23:17 am
content: James Grenning wrote an excellent book on the topic:Test Driven Development for Embedded C(https://books.google.com/books/about/Test_Driven_Development_for_Embedded_C.html?id=dA9QDwAAQBAJ&source=kp_book_description). I found it very helpful in my efforts.

author: ejeffrey
author_rank: Super Contributor
timestamp: February 13, 2024, 02:29:55 am
content: Quote from: djsb on February 12, 2024, 08:44:48 pmUnit testing means to take a unit (a piece of the whole with well defined boundaries and behavior), make a request, and check that the response is what the specification says. That can be "call the sqrt function with many values and assert that the return values are correct" or "ask to read a sensor and verify that the correct SPI register reads are performed."Just flashing an LED when you enter a function could be part of a unittest, but its not really a test by itself. At least you need more description of what behavior you are testing and how flashing the LED accomplished it. Just showing that a function runs is usually not a unit test although it can be if the thing you are testing is an event handling framework.

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 13, 2024, 03:38:22 am
content: The best part of unit testing in an agile environment is the green light icons when all tests pass. In some settings, that's a shining sun. Invaluable.

author: dobsonr741
author_rank: Frequent Contributor
timestamp: February 13, 2024, 05:25:02 am
content: Take a look at this example:https://admantium.com/blog/pico03_testing_with_cmocka/

author: hans
author_rank: Super Contributor
timestamp: February 13, 2024, 11:53:51 am
content: Quote from: djsb on February 12, 2024, 08:44:48 pmNo it doesn't have to be. Most unit tests are just calling functions and checking whether it computes the right values. This can be a "pure" function like CRC, data goes in, checksum goes out, and there is only 1 correct answer.But it could also be a whole module with statemachines that is checked and exercised.Just checking the calling of functions is only a subset of what an unit test can be. It doesn't tell the whole story, although it is sometimes used in mocking. Mocking is a method to isolate your unit test code from other modules. You then "mock" whatever other code would be called. Many test frameworks allows you to set this up conveniently, and also test whether the mocked calls are correct (e.g. correct number of calls, correct arguments, etctera).For example: say you write a piece of code that stores and retrieves configuration parameters for your application. The settings are stored in EEPROM that is handled by some other code. If you want to test your configuration code in isolation, you would mock the EEPROM module You could then write unit tests for reading settings... e.g. you mock the EEPROM read function and in the test handcode some values that correspond to some expected configuration state. Likewise for writing you could pass some configuration state, and then mock the EEPROM write function.. at which you check whether the correct bytes are written to memory.These trivial examples may not be that interesting to test; the nice thing starts when you need to handle exceptional cases which are hard to exercise by hand or don't occur frequently. In manual testing these tend to be less well tested. Unit tests are just an automation of those tests. For example, migration of parameters between firmware upgrades might be one of those things you want to test. Etc.It just so happens a LED blinky would be quite hard to test becauseA) It interfaces directly to hardware code.. and unit tests on your workstation won't capture that unless you would go through the effort to setup a whole emulation environment etc. Or perhaps test on real hardware and have equipment that externally verify.. but that is more often only worth it as integration test. Unit tests should aid development and quality, not create a bunch of useless bureaucracy before you can start writing code.B) The requirements for a "correct" blinky should include timing, and that is one the hardest parts of unit testing.. Just the function calls themselves don't reveal anything about how long something will take, and whether that has any interactions with the rest of your program. Maybe you could write a test where you mock e.g. delay_ms.. but that assumes that delay_ms on your platform works OK.It does lead to another issue in writing "solid" code: programs that depend on timing are in my experience the worst to test and verify. However, this can be an essential part of embedded, where we have to deal with real-time deadlines for something as simple as an UART. Unit tests won't reveal whether your code will miss interrupts or have race conditions on the target platform.At best, you can test what happens in some emulated failure cases. E.g for an UART Rx interrupt.. getting the byte from hardware has a hard deadline. However, if that byte is pushed into a FIFO for the application to read, then that can still fail (e.g. your main loop is stalled for CPU time). You could write an unit test that checks what happens in cae of a FIFO overrun (e.g. you emulate the arrival of many UART Rx events). You could verify that your application won't hang, whether partial packets are discarded, etc. This can help in some robustness, but it won't magically make all problems go away.

author: Doctorandus_P
author_rank: Super Contributor
timestamp: February 13, 2024, 12:37:54 pm
content: C is a pretty much standardized and universal language. When I want to write a program for a microcontroller, I often start by writing some commandline PC programs first. Often quite big sections of a program for a microcontroller can be developed and debugged on a PC. Such as for example a loop for a PID controller, formatting of packets (inclusive checksum) for some network stack, string operations, calculations etc.There are also some differences. A PC does not have your uC hardware. You can create the (special function) registers as simple variables. This lets you at least compile the code code without complaints. Once the code is compiled, it's easy to write some test routines (put them in another file) to exercise the library code you have written in the first file. And with a little bit of extra care, your test routines are unit tests too. And you can use them if later a bug is found in your uC code, or just to verify the code still works after you have made modifications later.TDD (Test driven development) is good, but it is not a catch-all method, and it also does not always make sense to use it. Browse around and experiment with some other (more or less formal) methods of code design and writing, and when you have knowledge of a bunch of different development techniques, you can use the method that best fits (the part of) a program you are working on.

author: std
author_rank: Contributor
timestamp: February 14, 2024, 03:13:58 am
content: Quote from: Cavhat on February 13, 2024, 01:23:17 amI have read it, still hoping to see solution offered for hardware testing. Nothing like this. This is software testing. The word "embedded" is misleading here, author does not concern hardware. His only example with “Led Driver” is very artificial. So, +/- informative book about software testing, nothing for embedders.

author: PlainName
author_rank: Super Contributor
timestamp: February 14, 2024, 10:17:12 am
content: QuoteHow would you deal with hardware in the context of unit testing?

author: AVI-crak
author_rank: Regular Contributor
timestamp: February 14, 2024, 10:20:35 am
content: Quote from: std on February 14, 2024, 03:13:58 amThe hardware test for embedded systems is a separate one-time firmware test. At the assembly stage it is necessary to find out the serviceability of all electronic components on the PCB. However, the test itself requires reaction to all system states, it is a lot of lines of code, too many. In addition, the speed of execution of the algorithms is critically slow. You will have a hard time explaining the merits of your product to a customer, even harder to sell.Besides, error detection in a serial product does not make sense. Let's say a sensor fails, does the customer have to re-solder it himself?For a serial product there are two states available: working / requires repair in a service center. For this purpose it is possible to leave a very shortened test in the serial firmware, without going through all the states. Literally perform one variant of the test, and in case of failure de-energize the device (while there is still something to do).Translated with DeepL.com (free version)

author: std
author_rank: Contributor
timestamp: February 14, 2024, 11:36:30 am
content: Quote from: AVI-crak on February 14, 2024, 10:20:35 amAnalog poweron self-test - is the primary reason, why you see this relays in our product.The leds also idicating health / problems, including self-test failure code.

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 14, 2024, 07:18:34 pm
content: That's a lot of relays. But I like the green LEDs.

author: std
author_rank: Contributor
timestamp: February 16, 2024, 07:44:11 am
content: Quote from: SiliconWizard on February 14, 2024, 07:18:34 pmI exaggerated for the sake of sayingits not "primary" reason. We need all this relays because in power-off state realys switched all signals to go-through (and in the same time circuitry loopbacks for self-test). It's actually quite an old design, I just had it handy on my desk, newer one has small smd relays like small IC's.

author: hans
author_rank: Super Contributor
timestamp: February 16, 2024, 09:14:43 am
content: Quote from: std on February 14, 2024, 03:13:58 amThat's true.. but there are a few points that align with embedded:I would argue that a majority of modern firmware is already at the "software domain". We do have some specific requirements like no malloc, mindful of codesize, etc. Only a minority of the code I write actually interfaces with RTL.Thus by utilizing unit testing and other modern software development techniques, I'm able to remove the highest necessity of debugging code on MCUs with slow debug cables and minimize time spent in lab while occupying expensive test equipment.I see unit tests as SPICE is for analog.. you run the design with 1st order models to get 90% of the result, and the last 10% requires manual tweaking later. You could go the extra mile with 2nd, 3rd order models to get to 99% or 99.9%, but it becomes harder to do.Even for firmware tailored towards a PIC16, I can see some benefits. But I think these would be mostly theoretical, since the PICs (and some other cores) use ancient compilers which cannot tremendously deal well with abstractions. Those abstractions would be needed for e.g. a BSP layer to support mocking for unit tests. If we value devtime over anything else, obviously we wouldn't use a PIC16, and it also frees up a path to a workflow with (atleast initially) more overhead.Automated tests with hardware-in-the-loop is certainly possible.. but its a question of what is worth you investment. You could hook up a MCU to a 16, 32, 64ch logic analyzer (or AWG, scope, etc.), write scripts for all interfaces you may ever want to exercise, write DUT code you want to test in isolation, and then verify all BSP-level functionality works as expected. But do you really want to occupy potentially k$'s worth of equipment for that? Or would be more specific setups in production be a better bet?Another issue with such hardware-in-the-loop tests, is that they will be relatively slow. Crosscompiling, uploading, equipment control.. its all very slow. Unit testsshould befast. This is not a technological requirement at all, rather a psychological/process requirement: by keeping them fast, we as devs will keep on running them for every change we make. This keeps the loop tight at which we can catch accidently broken stuff. One compromise would be to move slow tests to a continuous integration server.. so for hardware-in-the-loop setup that would certainly be an option. But still.. those setups are expensive, and I don't think it's worth it to test some very basic SPI or UART code.In contrast, testing more product-level properties may be much more interesting. E.g. something like this:https://medium.com/tado-product-development-blog/building-an-iot-product-continuous-battery-lifetime-testing-3245b6cdfa40

author: NorthGuy
author_rank: Super Contributor
timestamp: February 16, 2024, 09:29:04 pm
content: Quote from: hans on February 16, 2024, 09:14:43 amThis is just amazing. A simple thought of measuring motor energy consumption separately from the electronics didn't even cross their mind.BTW: If they wanted less consumption in the stepper motor, then instead of thinking about "atomic operators to lock the threads in our operating system", they would be much better off developing a custom energy-saving driver for their stepper motors.

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 16, 2024, 09:36:20 pm
content: One of the neverending stream of examples of the "law of the instrument".

author: hans
author_rank: Super Contributor
timestamp: February 18, 2024, 07:57:34 am
content: Yep.. when stuck on a problem..A hardware engineer will take out the DMM/scope and doubts the software lastA software engineer will take out the debugger and doubts the hardware lastPoor hardware engineering choices aside, for illustration purposes, I think the utility of a CI/CD pipeline is quite neat for embedded. But I doubt whether thats a good idea for any embedded device, considering some are worked on only intermittently. Many software products may have an infinite release cycle, while many hardware products certainly don't. No code change=no need to rerun tests=no need to keep such a hardware setup alive, unless you have enough money in which you can tie down (expensive) measurement equipment forever.

author: Siwastaja
author_rank: Super Contributor
timestamp: February 18, 2024, 09:36:03 am
content: Quote from: hans on February 18, 2024, 07:57:34 amThis is why I want full responsibility for both hardware and firmware. Obviously doesn't work for massive megaprojects, but with suitable KISS attitude, for something surprisingly big (medium complexity projects). Drawing the line between the firmware and cloud, or between firmware and UI, is much simpler IMHO, and forces you to write good abstractions for working in teams. On the other hand, hardware-firmware interactions are sometimes weird and ugly so one person needs to understand both.

author: PlainName
author_rank: Super Contributor
timestamp: February 18, 2024, 11:20:46 am
content: QuoteActually, I think the software engineer will suspect the hardware. If the code was a problem it'd've been fixed, after all. Similarly, the hardware engineer will suspect the software is doing weird things with the decent hardware.Were that not the case we'd all be able to spot our own bugs as we put them in the code, not much later when a colleague (or worse!) points out the simple and obvious typo.

author: AVI-crak
author_rank: Regular Contributor
timestamp: February 18, 2024, 08:17:36 pm
content: The scenario and application area are not defined and therefore:Military and space equipment performs mutual testing of modules and units.In the case when one part of the complex is missing or critically malfunctioning - the report is performed by the remaining parts of the system, including for the malfunctioning part. And after that, one of the many emergency state scenarios is performed.It looks like this - you will die after the task is completed, but now work.

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 18, 2024, 10:02:54 pm
content: Quote from: PlainName on February 18, 2024, 11:20:46 amYes, that's rather common too. It just all depends on how humble said engineer is. Young engineers tend to suspect anything but their own work, the exception being a minority IME.

author: Siwastaja
author_rank: Super Contributor
timestamp: February 19, 2024, 09:48:52 am
content: Quote from: SiliconWizard on February 18, 2024, 10:02:54 pmI have learned to always suspect both my work and that of others are broken. Usually this is the case. The issue with code/hardware of others is that it's usually impossible or harder to fix and needs to be worked around, which is more work than fixing my own work. Therefore, having as much in my hands is helpful - I still do mistakes but can fix them faster - unless it's too much to handle within realistic timeframe.

author: Nominal Animal
author_rank: Super Contributor
timestamp: February 19, 2024, 10:07:49 am
content: I do more unit testing in the design phase than in the implementation phase.I also usually create several test "firmwares" for the target hardware, using the algorithms and approaches verified in the design phase, to discover the behaviour and real world implications of the choices (mostly in isolation, thus "unit" testing). Sometimes this yields a "no, that's not going to work", so I have to go back to the design phase.Thus, the implementation phase for me is mostly about integration, and testing combined features and their interactions; much more integration and user testing than unit testing.

author: hans
author_rank: Super Contributor
timestamp: February 19, 2024, 03:24:08 pm
content: Quote from: PlainName on February 18, 2024, 11:20:46 amBut as a first hypothesis? Code doesn't get magically fixed neither, so one would have to investigate that first. I wrote "last" in that sentence on purpose, as a final conclusion if all "own" aspects of a problem have been exhausted. E.g. some MCU firmware that occasionally locks up could also be caused by a failed (extended) memtest to external RAM. It's very hard to fix that with unit tests!I think what younger engineers typically do, is they don't exhaust all their own options before pointing at something else. Ignorance is usually present for both, but I think one tilts more to arrogance ("Mount Stupid" of Dunning-Krieger curve in full swing), and the other is simply inexperience e.g. seeing your own mistakes is harder if you don't know about failure modes XYZ or what exact behaviour you're looking for.

author: PlainName
author_rank: Super Contributor
timestamp: February 19, 2024, 08:49:02 pm
content: Quote from: hans on February 19, 2024, 03:24:08 pmI wrote it as a bit tongue in cheek. Another way of looking at it is the classic man with a hammer way - the software guy is going to think in software terms; the hardware guy in hardware terms.However, in reality I think it depends on where it is in the development cycle, ranging from first blood to "2m with customers". If you're actually writing the code then you will likely think that some problem is a result of what you're writing. And even if eventually it turns out it isn't - that is, it's a hardware issue - you'd probably still be thinking in terms of a software fix because the hardware is what it is and a change would range from 'not easy' to 'never happen'.