Automated hardware CI (continuous integration) from github - Page 1
Forum section: electronics / projects

author: ddavidebor
author_rank: Super Contributor
timestamp: August 23, 2017, 12:09:09 am
content: Hi guys! I wanted to share my experience.We needed to run automated tests on some firmware we wrote for scientific instruments. The firmware is hosted on githubhttps://github.com/fermiumlabs/data-chanThis includes:0. pulling from github1. compiling2. programming the mcu3. connecting though USB, sending commands, etc etc4. sending reports and notifications here and thereTo do automated testing you need a repeatable environment, generally one would use virtual machines. Unfortunately, there is no simple way to reliably share USB devices to virtual machines, especially when those USB may be unstable or change VID/PID during testing.We found a solution: using docker to share the linux usb bus, from one linux kernel to another. This creates a repeatable environment with complete USB access.We used a open source CI (continuous integration) called Dronehttps://github.com/drone/droneand multiple server solution:* 1 public facing server: UI + github webhooks* 1 behind-the-firewall worker server with the USB deviceTo replicate this setup you're going to need some basic know-how about docker and docker-composehttps://docs.docker.com/compose/Currently the server is down for maintenance, will post config in a couple of days.Here's an example of a procedure to build and test. It's basically a set of sequential bash commands:Code:pipeline:build:image: ubuntu:xenial# see https://github.com/docker/docker/issues/19763 to why we use --volumes and not --devicevolumes:- "/dev/bus/usb:/dev/bus/usb"privileged: truecommands:- export DEBIAN_FRONTEND=noninteractive- apt-get update > /dev/null- apt-get install -y gcc gcc-avr avr-libc avrdude make git usbutils clang libusb-1.0-0-dev doxygen > /dev/null- make release -j2- make install- lsusb- (cd Host/examples/Tester && ./Tester )# Build only on workers that have usb on docker enabledlabels:USB: trueI banged my head on this for a month. If you want to automatically test hardware with USB against your code, there is NO better solution.Davide

author: hans
author_rank: Super Contributor
timestamp: August 26, 2017, 08:30:31 am
content: Are these unit tests or complete integration tests of the device?I think unit tests would be useful for embedded development as well, but unfortunately doing that in a language as bare as C is quite hard.I did set up my own development pipeline using Jenkins a little while ago. It runs unit tests, code coverage and does some static analysis using SonarQube (which has the other results published in it as well). Overall it's a nice platform to track technical debt and how well the codebase is doing.My plans are to measure some performance metrics of the code on this platform, e.g. do some profiling of the system to figure out how much time (=power consumption in case of battery powered devices) a particular program takes, so those statistics can be tracked and improved as well.I imagine that if you run Jenkins non-virtualized and/or with the same interesting docker USB trick, you can probably achieve similar results. But alternatives are always nice to see

author: ddavidebor
author_rank: Super Contributor
timestamp: September 04, 2017, 03:38:52 pm
content: No we don't do unit testing on hardware, it's basically impossible on embedded microcontrollers. But we can do unit testing on the user-facing app, with the hardware connetted. that's something.Yes you can do it with Jerkins and docker. But Drone is orders of magnitude easier to set up.

author: technix
author_rank: Super Contributor
timestamp: September 05, 2017, 10:44:08 am
content: Actually for CI I would prefer a more emulated or virtualized environment. This way the Flash of the target chip wouldn’t be worn down that fast.For example, the Cortex-A7 or Cortex-A53 on Raspberry Pi can run STM32F407ZG Cortex-M4F code as is. I just set up a virtual machine on the Pi through KVM, with RAM-backed Flash code space, hypervisor emulated peripherals and NVIC, then the STM32F407ZG code can be run in this virtualized environment under hypervisor control to perform the required CI actions.

author: ddavidebor
author_rank: Super Contributor
timestamp: September 07, 2017, 01:29:06 am
content: Probably, but we use Atmega mcus....

author: jfr
author_rank: Newbie
timestamp: September 07, 2017, 02:16:35 am
content: QuoteNot at all. The current devices to do them aren't cheap (dSpace & ETAS) but it can be done. The only part we haven't added is the 'automated', we still have someone flash and kick off tests by hand.If my current project works that should be automated.We're trying to move away from testing the whole unit and go to 'fail fast'.Workflow looks something like this:Code:Hand Code -> Style Checking -> Static Analysis -> HAL tests (Some signal generators, scopes & logic analyzers to test IO) ->Simulink Models -> Static Analysis -> Model in the loop test ->Integrated Binary -> Flash Tests -> CAN Tests (make sure it says hello) -> dSpace Benches.I'm actually working on setting up some Arduino AVRs as a test bench to demonstrate how it'll work.

author: bd139
author_rank: Super Contributor
timestamp: September 07, 2017, 07:05:36 am
content: Good solution. Nice to see some automation outside of ITHow are you finding drone? I need a CI server for a few small projects which will generate docker images and don't want to take on the overhead of Jenkins or Team City.

author: ddavidebor
author_rank: Super Contributor
timestamp: September 08, 2017, 07:53:55 pm
content: Quote from: bd139 on September 07, 2017, 07:05:36 amIt's absolutely great, simple, quick to setup (using docker-compose), easy to use, well integrated with github etc etc.I really like the chaining of different images that operate on the same volume, it works like Wercker CI.This may help youhttp://plugins.drone.io/drone-plugins/drone-docker/

author: bd139
author_rank: Super Contributor
timestamp: September 08, 2017, 08:31:56 pm
content: Fantastic. Will take some time out next week to play with that. Thanks for the info.

author: hans
author_rank: Super Contributor
timestamp: September 08, 2017, 08:44:38 pm
content: Quote from: ddavidebor on September 04, 2017, 03:38:52 pmAnd you don't have to. That's the point of unit tests; mocking and stubbing other parts of software so you concentrate on the unit under test.Once you leave the boundaries of hardware, most things are quite well unit testable. Just try to avoid anything that uses timing as a (implicit) state to be tested.I'll check Drone out one day. Jenkins isn't very hard to set up, but any tool that'smoreproductive yet flexible is always welcome.Also, I don't really like configurations via UI's over plain text files. Carrying over installations is then so much easier.

author: bd139
author_rank: Super Contributor
timestamp: September 08, 2017, 10:25:58 pm
content: Spot on with the unit tests. Last time I did embedded stuff, it was on PPC. We had a HAL and the rest of the stuff was ANSI C with a ton of test cases on top of stubs that talked to the HAL. The unit tests were about a thousand or so compiled binaries (one per test) on a Linux box that puked to stderr or segfaulted and returned non zero/valgrind shit itself if it didn't pass. Simple but effective. It's also nice when you genuinely can sleep because you KNOW the test cases have been run through and that one line you changed isn't going to wake you up in the middle of the night.On the subject of Jenkins, we run a very large Jenkins installation (22 build nodes) that builds three 10-12 million line lumps of mud in Java/C++/C#. It's not hard to set up but it's VERY hard to set up securely, to keep it running, to keep it alive, to back it up etc. I think we're down about 400 million inodes on the master! Also some of the plugins are made of elephant shit. I'd take anything over it at the moment.

author: ejeffrey
author_rank: Super Contributor
timestamp: September 09, 2017, 03:35:36 am
content: Quote from: technix on September 05, 2017, 10:44:08 amFlash endurance would be about the last of my concerns. A $5 microcontroller that lasts for 100,000 builds before it needs ~30 minutes rework to replace -- assuming you can't / won't just replace the entire module? That is insignificant by pretty much any measure. If each build represents a minimum of 5 minutes of human effort, you are talking about a 8000 hours of human effort weighed against a $5 chip. Maybe if you had some really expensive dev board with flash in a tiny pain-in-the-ass BGA package it would be work conserving write cycles. But really, even then I would say that your dev board should have a flash package that can be replaced easily if at all possible. Plus, if you are working on very expensive hardware, probably all of your dev costs are higher, and replacing a part after 100,000 cycles is still probably not a big deal.Emulation is certainly a useful tool in testing, but you should base the decision on most important issues like what makes it easier to write the proper kind of test, which will provide the needed performance, whether you need to spool up multiple copies in parallel, and of course the availability of a suitably accurate emulator for your target (including peripherals if needed).

author: technix
author_rank: Super Contributor
timestamp: September 09, 2017, 04:25:41 am
content: Quote from: ddavidebor on September 07, 2017, 01:29:06 amThen just emulate it. A single Core i5 or Ryzen 5 can emulate a few thousands of them at once, with set up and tear down time of a few milliseconds.

author: technix
author_rank: Super Contributor
timestamp: September 09, 2017, 04:39:56 am
content: Quote from: ejeffrey on September 09, 2017, 03:35:36 amThen what about time? It can take minutes to load big application code into a chip using things like SWD or debugWire. Setting up and tearing down VMs or emulators takes milliseconds. You also get to run the code accelerated (1.2GHz or higher instead of 168MHz or 20MHz)

author: ddavidebor
author_rank: Super Contributor
timestamp: September 09, 2017, 11:36:36 am
content: QuoteYep. Yep.QuoteMinutes is fine.With real HW, you can test the peripherals at the rated clock.Hardware is not made of only software, it's made of hardware too.

author: technix
author_rank: Super Contributor
timestamp: September 09, 2017, 05:15:34 pm
content: Quote from: ddavidebor on September 09, 2017, 11:36:36 amTwo additional benefits of software based testing: 1) you can make use of the publicly available free CI resources like Travis CI instead of hosting the boards locally, 2) hardware testing can be accomplished using a good enough SPICE model of the board, 3) it is easier to add watchpoints to implement non-intrusive unit testing, since real hardware may not offer too much of that, if any (at least not without you ponying up a lot of $$$ for a more advanced debugger with code trace support. Consider Atmel ICE versus Atmel JTAG ICE, ST-Link versus J-Trace, or PICKit 3 versus Real ICE)One additional note on the Flash longevity: you may want to test the code before merging a branch. When your project is small this may not be a problem; but when the project start to gain traction you may have to go through though a lot of pull requests, wasting a significant humber of erase cycles on the chip.If you can find ROMless variants of your chip, try use those with SRAM going where the Flash would be; use JTAG, parallel programming or even an external monitor chip at maximum possible communication speeds over debugWIRE or SWD, and clock the target at maximum speed it can take (including trying a small to moderate overclock.) This way you won't be wearing out anything (no Flash) and can go through code revisions and pull requests as fast as possible.

author: Siwastaja
author_rank: Super Contributor
timestamp: September 09, 2017, 05:54:35 pm
content: Quote from: ddavidebor on September 09, 2017, 11:36:36 amWell said - this is something that seems impossible to understand for some people.Surprisingly often, I find I need to try to explain this over and over again, but have not found the right words to get the message through.Quite often the case is that writingusableandrealisticsimulation models for every external piece of HW would takeyearsif not more, including extensive reverse-engineering of commercial ICs - often orders of magnitude longer than "just testing the real thing".For real emulation, setting up all IOs and peripherals to work on a different HW may take man-months if not man-years of setup time, too. At least some very special and expensive emulation hardware.Minutes of download time on an MCU project sounds like there is an issue somewhere. It shouldn't take so long. Some bootloaders may be extra slow. Replace with a proper bootloader / flasher code. Also consider using code sections allocated on different flash pages for partial update. For example, the current project I'm working on with STM32, takes 2-3 minutes to fully flash with the built-in bootloader, about 30 seconds when I only flash the relevant sectors with the built-in bootloader, or about 3 seconds when I do the same with my own flasher, which doesn't handshake every freaking byte or something super dumb like that, like the built-in thing apparently does. (Own bootloader is almost always required anyways, to allow better firmware update functionality and integration)Simulation and code development on PC hardware is of course a great tool when developing software algorithms that do not do IO, and work properly with isolated or autogenerated data - especially when you want to scan input ranges, which would be difficult or impossible using the "just test the actual HW" attitude. But this is never 100% of the code; in fact, quite often it's some miniscule amount, since many MCU applicationstendto be very IO, HW and timing centric - otherwise we would just use computers for these tasks!Hasanyoneeverhad an issue with flash longevity while developing a project? Really? Maybe after working on a project foryearsfull-time, you could expect the HW has matured to rev.2 at least, with new cards available? Or if it's a devboard, maybe after $50-100k or more has been invested anyway, maybe getting a new devboard is not an issue?

author: technix
author_rank: Super Contributor
timestamp: September 09, 2017, 06:32:05 pm
content: The thing that is wearing down Flash here is Github and the collaborative coding it brings. A CI run can be triggered by a few things:* A new commit of the code,* A new pull request,* If the pull request is mergeable, the current head merged with the pull request,* An update to any of the dependencies.Now multiply those together (as a well configured CI does to check if the collaborators are helping and whether the upstream dependencies updates would impact the project) and you get a lot of Flash erases. And for CI and unit tests you need either code trace or a lot of watch points. (Inserting print statements or checks will throw off code timings resulting in hersenbugs.

author: ddavidebor
author_rank: Super Contributor
timestamp: September 10, 2017, 08:34:27 am
content: I'm gonna burn those fuckers down then I am gonna discard them like a used condom

author: technix
author_rank: Super Contributor
timestamp: September 10, 2017, 03:13:02 pm
content: Quote from: ddavidebor on September 10, 2017, 08:34:27 amThen you are creating a lot of waste. If your chip comes in a through hole variant (or you can use adapter boards) you can socket it, so only the chip is thrown away when it is worn out.