How to tell if the firmware is corrupted - Page 1
Forum section: electronics / general

author: Red_Micro
author_rank: Regular Contributor
timestamp: December 21, 2021, 03:09:41 am
content: I have some boards with STM32F072. The application features USB and some other peripherals like GPIO, SPI and ADC. Some boards are not working properly, mainly USB hanging/crashing. When I reflash the hex file with ST Link, it starts working. Is there a way to precisely tell if the firmware on those bad board is corrupted? I mean, if reflashing fixes it, it's an indicator of firmware corruption. What techniques you suggest to debug cases like this where you don't have the source code but only the hex file?

author: DavidAlfa
author_rank: Super Contributor
timestamp: December 21, 2021, 03:15:35 am
content: Read the firmware back and compare them by running any hash algorithm like md5sum.The stlink verifies the data unless you specially disable that, so flash corruption would be very rare.Have you tried increasing the heap/sizes?

author: Rick Law
author_rank: Super Contributor
timestamp: December 21, 2021, 05:07:37 am
content: If you are reading back the whole firmware to check, I don't see any advantage comparing check sum. Might as well compare byte-by-byte of expected vs the read-back. Then you know both the error(s) and the error location(s).On the other hand, if you have a check sum flashed into the firmware memory, it is useful. Say check sum is stored immediately after the used part of the firmware or at the last few bytes of the memory for firmware, you can compare the checksum of the read-back against the stored checksum. So this can be a "self contained" with everything in flash memory.But I don't know if tools exist to flash just a specific part of the flash memory...I experimented with this before: by reserving some bytes with them initialized first to 0. On the next-to-final version, read back the next-to-final burn and run a simple CRC32, presumably this is good since it has been tested to be ready as the "next-to-final" version. Then I go back to make the final version with the CRC32 value inserted into the 4 reserved bytes. The 4 reserved bytes are the 4 bytes before my version string. My version string always start with "V??? MM/DD/YY" so it is easy to find.But to check the stored firmware later, I had to read back, find that 4 reserve bytes, note the values, zero them, then do the CRC32 and compared it to the noted value. It did work for me (in simulated error), but it was such a pain to do that I gave up the idea very quickly.

author: Siwastaja
author_rank: Super Contributor
timestamp: December 21, 2021, 08:48:23 am
content: Does this reproduce the issue:1) Flash2) Not working3) Remove power, reapply power4) Still not working?5) Reflash6) Working7) Remove power, reapply power8) Still working?Only in this case I would suspect corruption. Otherwise, the flashing process itself changes some state / memory content / peripheral register / state of the external devices you communicate with.Even then, instead of random bit corruption, it's more likely that you are missing a large piece of the program, or similar.The normal way to deal with this is that the flashing process runs verify step, i.e. reads flash memory back and compares every bit. How (which software, which command line options or settings) are you flashing it? Often verification step is enabled by default. If not, do enable it.Then, just debug the issue "as usual". Different behavior between boots is fairly common and very rarely caused by corrupted flash. Most likely, it's a bug in your code.

author: 0xdeadbeef
author_rank: Super Contributor
timestamp: December 21, 2021, 09:00:28 am
content: When this (flashing the same binary fixes a stuck ECU) happens at the job, the actual reason always is that the debugger also clears the RAM, specifically the reset safe RAM. In >20 years of working with microcontrollers with integrated flash, which we reprogram frequently during development, I have seen only a very few cases of defective flash and these couldn't be saved with reprogramming. From what I can tell, even with hundreds of thousand of devices in the field, cases of corrupted flash are not very frequent.

author: Siwastaja
author_rank: Super Contributor
timestamp: December 21, 2021, 09:09:38 am
content: Yes, and now if you just debug the issue like any bug, you'll find the offending place in code. Then you can compare what happens after reflashing. Likely some peripheral configuration is different, or RAM. Seeing USB, does it have some dedicated piece of RAM you need to clear? This is the usual problem with USB, CAN, Ethernet and similar peripherals which are complex enough to come with their own RAM segments, which arenotcleared by the default startup code, but instead you need to first enable the peripheral clocks and gain access to the RAM, then clear the RAM yourself.

author: ajb
author_rank: Super Contributor
timestamp: December 21, 2021, 06:36:43 pm
content: Does your flashing process include verification after programming? If so then there should be no concern about a problem with the programming process, since it should report an error if there's any discrepancy. You can do a verification without programming to test this functionality (for example, try verifying after erasing the part and make sure the verification fails), or to confirm if something in flash has actually changed since the part was initially programmed. This would easily and definitively confirm whether or not the flash is changing somehow after programming.Is the application reprogramming the flash at all? For example with a bootloader, or for EEPROM emulation? Short of hardware failure or a significant radiation event, a problem during in-system rewriting is about the only way you'd see flash corruption after initial programming. If the system uses flash to store configuration information, then it's possible that you'll see a change to flash that corresponds with the failure mode, but is not the cause--it could just be that the system writes a bad configuration and breaks itself. Make sure that you read out and verify the entire memory space of the chip, since a configuration area would likely be at the end of memory and a verification of only the range used by the program would not catch any changes to other regions.Aside from that, these are complex parts, with complex software comprising the USB stack, interacting with a whole other pile of complex software in the USB stack of the connected host (or other device), so it can sometimes be challenging to be sure you've identified the real source of the problem.If you have the hex file but not the source code, then debugging is going to be more difficult but not impossible--the real challenge is going to be in what you do about the root cause if you manage to isolate it in the binary.

author: Bud
author_rank: Super Contributor
timestamp: December 21, 2021, 07:07:52 pm
content: I'd start from buying the processors from a reputable supplier to exclude the fake hardware variable.

author: abyrvalg
author_rank: Frequent Contributor
timestamp: December 21, 2021, 07:32:46 pm
content: Another possible reason is if reflashing performs a mass erase, as a side effect it also clears some corrupted settings area (those are more prone to corruption than the firmware).