How do you Unit Test your embedded C/C++ code? - Page 1
Forum section: electronics / general

author: obiwanjacobi
author_rank: Super Contributor
timestamp: March 19, 2017, 07:41:00 am
content: I am looking for information on the kinds of problems you encounter when trying to unit test your code and what solutions you use to manage it.* How do you organize/design your code to make testing possible?* Do you test hardware-related code (like HALs) on the specific device or in a simulator?* What Unit Testing framework do you use and why?* What makes unit testing embedded code different from 'normal' (PC-based) code?Please include any (support) languages and tools you use.

author: filssavi
author_rank: Frequent Contributor
timestamp: March 19, 2017, 07:53:33 am
content: The gold is that you can unit test the high level code with the usual suspects (cUnit and such) for correctness, if your HAL is good.The bad is that the HAL by definition has to be tester in hardware, as I'm not aware of any emulation system that let's you configure I/OThe ugly is that the United tests are almost useless because you for a test to be truly meaningfull you need to run It with the cirrect timings.Another thing that makes the embedded firmware dificult to unit test is that it almost invariably is interrupt heavyYou might want to take a look at static analisys and formal methods instead

author: andersm
author_rank: Super Contributor
timestamp: March 19, 2017, 10:42:03 am
content: Two decent books on the subject:Test-Driven Development for Embedded C, andModern C++ Programming with Test-Driven Development. The latter isn't about embedded code specifically, but does cover writing testable code.My personal opinion is not to spend too much effort on testing low-level hardware code, because once it's been written, it's usually very stable over the lifetime of a project, and is rarely a source of bugs. I would also recommend doing as much testing as possible on the desktop, just for the productivity boost.

author: nctnico
author_rank: Super Contributor
timestamp: March 19, 2017, 01:00:36 pm
content: Quote from: andersm on March 19, 2017, 10:42:03 amYes, testing as much code on a PC as possible makes development easier. What also helps is adding lots of boundary checks to the code and avoid using pointers. Boundary checks will prevent errors from silently corrupting memory or doing damage and overall will help to catch a bug early because the code doesn't work as expected immediatly.

author: andyturk
author_rank: Frequent Contributor
timestamp: March 20, 2017, 04:06:01 am
content: Quote from: obiwanjacobi on March 19, 2017, 07:41:00 amWe work hard to isolate testable functionality from stuff that touches specific hardware. Algorithms and communications protocols are fairly straightforward to develop/test apart from the embedded target.We don't (yet) do hardware tests or simulation.We use Google Test because it works well and all our code is C++.Embedded unit testing is trickier because of the hardware aspect, obviously. Maybe that's all?Our build uses make and gcc. Some folks use gdb for debugging. I've had good luck with Segger's Ozone. RTT is pretty handy in general.I'd be curious to hear about other people's experience with CI (Continuous Integration) for embedded builds.

author: julianhigginson
author_rank: Frequent Contributor
timestamp: March 20, 2017, 10:43:54 am
content: absolutely the best thing about unit testing, and it's main point, IMO, is it forces you to write decoupled code. as already mentioned it is very hard to do tests for hw drivers, and interrupt driven stuff, and interactions of parts in a system... but that's not the point of unit tests... unit tests just test separate testable modules (units) in the code! they are there to make sure when you edit your GPS datum conversion code (for instance) you don't wipe out positions in one quadrant of the globe.... or when you edit your temperature control state machine for you water heater, (for another instance) you don't suddenly have the control output reversed when the outside temperature goes below zero... it's about making sure small compartmentalised bits of code stay good when edited. this gives you the freedom to rewrite old legacy things and improve stuff without niggling worry "what if it was done that particular silly way for a reason I don't understand?" because once you edit it, you know it still passes the unit tests, it should still be good!For complex inter related behaviour like whole system operation, you do black/white box release testing... ideally with a proper test jig that can exercise the actual units inputs, read outputs, and time anything that needs to happen in realtime..Sent from my A0001 using Tapatalk

author: tggzzz
author_rank: Super Contributor
timestamp: March 20, 2017, 10:57:26 am
content: Always remember that "you can'ttestquality into a product; quality has to bedesignedinto a product".I've seen some truly dreadful unit tests which met magic (and I use that word advisedly) metrics, but which were actively counterproductive since they made it difficult to change anything.Work out what is important, and test that. Don't test irrelevant implementation artefacts.

author: andyturk
author_rank: Frequent Contributor
timestamp: March 20, 2017, 01:25:46 pm
content: Quote from: tggzzz on March 20, 2017, 10:57:26 amIf you've got bad code that impedes progress (and unit tests are code), why not just refactor it?

author: tggzzz
author_rank: Super Contributor
timestamp: March 20, 2017, 03:05:23 pm
content: Quote from: andyturk on March 20, 2017, 01:25:46 pmTime, money, interest, but most importantly...Almost by definition, refactoring is done is such a way that the unit tests don't break. If you want/need to do more, then you are back to trying to understand what the codeis intendedto do. Unfortunately in most commercial organisations, that understanding is defined by the unit tests themselves => "Catch 22".Now it shouldn't and needn't be like that, but too many HR droids hire people that shouldn't be let loose with a keyboard.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: March 20, 2017, 04:32:01 pm
content: So, if I understand this correctly, it basically means:1) Design your code in such a way that (at least) the code that touches the physical hardware is separated out into its own set of components / separate layer. Thin is good.2) For the rest of the code: use common/best practices for writing and testing robust code in general. This is a topic in itself and some examples have already been given.3) As an optional step you could do black/white box testing on the hardware in realtime. But this is manual integration/system testing. You need to do this at least once prior to release of course - bit late IMO.Okay, this is what I had worked out myself. I wondered if there would be any fancy test rig that could run unit tests on the device and use JTAG for instance to setup the the hardware and read back the results. (I have a faint idea of what JTAG does but never read the protocol in depth.)Is there any particular Unit Testing Framework that is recommended (Google Test was mentioned)? Or is this mainly down to personal/company choice? I am not experienced in C/C++ testing frameworks. Have read a bit about them, but there are a lot of contradicting opinions that do not really go into the why they like or dislike a certain framework.

author: tggzzz
author_rank: Super Contributor
timestamp: March 20, 2017, 05:09:01 pm
content: There is another valuable level of architecture/design in there, one which has lead to "model driven" concepts.To put it simply, there is a mismatch between theapplication domain conceptsand the hardware. You will find domain experts can express themselves in terms of application domain concepts; they will understand what the system is doing in those terms only. It is thus a very good idea to have software constructs that directly mirror the application domain concepts. That will greatly simplify requirements gathering and signoff, acceptance testing, and later system enhancements. So far that is true for all software development.For real-time embedded systems, it is highly advantageous to think in terms of FSMs, states and events. When understanding the application domain, "nouns" map onto the model's states or model's data, and "verbs" map onto the model's events. Use the standard Design Patterns to implement the FSMs, states and events.For a door controller, the states might be OpenState, ClosedState, OpeningState, ClosingState, EmergencyState, FailedState, the events might be OpenRequest, CloseRequest, Timeout, CloseCompleted, OpenCompleted, and the data might be MotorOpen, MotorClose, MotorOff, MotorDisconnected.Then the hardware adaptation layer simply maps inputs into the model's events, and model's data changes into outputs. Thus when bit 4 of PIO register changes from 0 to 1, create a new CloseRequest object and send it to the FSM; the FSM will change MotorClose to True which causes bit 13 in PIO register 22 to be set to 0.You can debug/test the application domain model without the hardware: all you do is have your test framework create the model's events instead of the hardware, etc.

author: andersm
author_rank: Super Contributor
timestamp: March 20, 2017, 09:33:18 pm
content: Quote from: obiwanjacobi on March 20, 2017, 04:32:01 pmWhat would that gain you? If you really wanted to, it wouldn't be too hard to script eg. OpenOCD to write each unit test executable to your target board, and read the results over a serial port.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: March 21, 2017, 06:14:01 am
content: I thought it would allow you to test the HAL layer (1) to make sure you tickle the hardware in just the right way. At higher integration levels it could verify components/sub-systems. See if the interrupts are handled correctly. See if timing constraints are met. Put it through extreme cases to see how robust or self-healing the design is.

author: krish2487
author_rank: Frequent Contributor
timestamp: March 21, 2017, 09:26:10 am
content: Ceedling.www.throwtheswitch.orgUnity is good. We use unity for our testing. It also comes with cmock and cexception framework to simplify testing a bit further.The original developers of the framework have a course over at udemy which walks through the usage in a easy to understand manner.Not affiliated in any manner. Just a satisfied user.

author: andersm
author_rank: Super Contributor
timestamp: March 21, 2017, 12:51:55 pm
content: Quote from: obiwanjacobi on March 21, 2017, 06:14:01 amUse mocks for that. If you have a lot of dependencies, there are frameworks that can automatically generate mock functions for you. Also as I wrote earlier, IMO testing a HAL provides dubious value. It either works or not, and once it works it's stable.QuoteAt that point you're stepping out of the realm of unit testing, and there are other tools for that.

author: tggzzz
author_rank: Super Contributor
timestamp: March 21, 2017, 01:19:49 pm
content: Quote from: andersm on March 21, 2017, 12:51:55 pmThe trouble with trying to define "unit testing" in terms of levels is that the definition of "unit" is very variable.In hardware terms, a "unit" can be a transistor, a filter, a register, an ALU, an MCU, a board, a computer, a ...In software terms, a "unit" can be a poking an IO register, transmitting a single packet, transmitting a long message, a remote function, an ACID transaction, a telephone call, a ...Typically youngsters make too much of a fetish of unit testing, and use it as part of "cargo cult" software development. The key point is tothinkabout what properties must be present, and then tothinkabout how to ensure they are present. There are many techniques that can and should be usedwhereverappropriate - and not used when inappropriate.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: March 22, 2017, 09:51:43 am
content: Quote from: andersm on March 21, 2017, 12:51:55 pmSo how do you maintain your HAL if you cannot test it (automated)? I would strive for a 100% test coverage in any normal software project. Now you may never actually reach it, but that is the goal. Anything your tests do not cover, you do not actually know if that code works.Does this not apply to embedded software in some way? Can you say: "because this interacts with the hardware we do not have to unit test it"?I have experimented with testing the hardware-related code (HAL) using some #define replacement in an AVR project. I replace the macro's that manipulate the hardware register memory locations with manipulations in a simple byte array. When initialization become more complex I thought this was a nice way to verify it did the right thing by asserting on the final values left in the byte array (at the correct offset). It will not catch everything, but at least you know you're in the ball-park...Would you say that is a waste of time?

author: tggzzz
author_rank: Super Contributor
timestamp: March 22, 2017, 10:13:28 am
content: Quote from: obiwanjacobi on March 22, 2017, 09:51:43 amQ1: if the code passes your tests, does the code work and is it "fit for purpose"?Q2: how do you know your tests work and are "fit for purpose"?You will note that both those questions mean there has to be a well-defined purpose.

author: John Coloccia
author_rank: Super Contributor
timestamp: March 22, 2017, 12:08:20 pm
content: IMHO, and I know this will be a minority opinion, traditional unit testing is not particularly valuable for most projects. I'm not talking about testing things like basic algorithms to make sure they work, edge cases work, errors are tossed back appropriately. I'm talking about traditional, function to function unit testing. In my opinion, this is only valuable if you're doing a traditional waterfall model where you truly do have a real requirements document, a real design document, and it actually makes sense to test specific functionality because you can trace it back to a requirement and the whole system is requirements driven.I can tell you that in 20 years of doing software, I have NEVER really used the waterfall model, and aside from a small subset of projects where it truly is required and that's just how you have to do it, IMHO it is the lousiest way to do software that I could possibly imagine. Even people who claim to do software like this don't really do software like this...they're just fooling themselves. At my new job, we're doing safety critical software and we do, in fact, need to have this level of documentation. There it makes sense to do real unit testing. Believe me, for 99% of the software out there you could not possibly afford to do the kind of documentation and scrutiny that would make real unit testing meaningful. Your costs would increase by an order of magnitude.So I will generally and exhaustively test some base functionality (state machine management, hardware I/O, communications and things like that) to make sure that I don't have any deep problems which will drive me nuts later. Then I will tend to concentrate on functional testing and use cases, generally creating some sort of test driver and carefully crafted input stimuli with expected outputs, and as I discover problems or potential problems I add test cases to be sure I capture them moving forward.It sounds a little fast and loose, but in practice it actually does a better job of ensuring correct behavior than traditional unit testing.

author: tggzzz
author_rank: Super Contributor
timestamp: March 22, 2017, 12:45:21 pm
content: Quote from: John Coloccia on March 22, 2017, 12:08:20 pmI have a lot of sympathy for those observations and conclusions. TDD is a bit of a cargo-cult fetish used to satisfy bosses and process charts, and to provide anillusionof quality.The one area where TDD really is useful is where the codebase is mutating in the light of experience gained - and that applies to many projects. But - and it is a very big but - you have to have the right/important/useful unit tests. If you blindly apply unit tests for every function or to achieve code coverage, there is a very strong tendency to have many tests that aren't useful. Over time, such tests often freeze completely unimportant trivial aspects of an implementation. They become part of the problem, not a part of the liberation.Good, useful tests are absolutely vital, but they require thought and understanding of what is important.