SAMD21G18 Hardfault Debugging - Page 1
Forum section: electronics / general

author: oldfortune_
author_rank: Contributor
timestamp: June 25, 2024, 01:32:21 pm
content: Hi All,I've been working on some lighting objects recently, and I released a limited batch of 10 for testing last year. All of the units are fine, however 1 has stopped working after 8 months of continuous use. I've got the PCB back in the lab to try to find the fault, and I'm not having much luck unfortunately, so I thought I'd reach out to the forumSo far I have narrowed the issue down to a fault in the microcontroller, but here is a little info on the PCB, and what I have tested so far:- Atsamd21g18- potentiometer inputs- gpio -> mosfet to 24V LED strip- piezo buzzer- CUI buck converter module 24 -> 3.3VEvaluation:- All the hardware works fine, after swapping the microcontroller all the functionality is there.- After digging a little with the faulty micro it seems that the program gets held up because of a hardware fault.- See attached images for the register values at that point in the flash as compared to a reference micro.- There is a hardfault listed on the IPSR- I read back the flash from a freshly flashed micro, and the faulty micro, and found that there were a few differences (this feels like a red herring though)Questions:- Any idea how to identify what the IPSR = 0x003 is.- Any ideas what might have happened, I suspect some sort of voltage spike damage.- is it possible to cause this damage with faulty firmware?Hope thats enough info to have some initial insights. Happy to upload schematics also if thats helpful.Cheers, Sean

author: ataradov
author_rank: Super Contributor
timestamp: June 25, 2024, 01:57:57 pm
content: The way to identify a HF is to set a breakpoint at the beginning of the handler before any other registers are modified. And then look at the top of the stack. There will be a program counter at the time of the fault and a few other useful registers. Then you can disassemble the code around the saved PC and in combination with the saved registers, you will know what caused the fault.But this will not tell you what exactly happened on a large scale. You may not even supposed to be running that code and the actual fault is in some other place. This is not a trivial thing to investigate in general.You say you read the firmware from the faulty MCU and it has bit differences compared to the expected image? How it that a red herring? Looks like the reason for the fault, or at least something that needs investigation.Can you show example of those differences?

author: oldfortune_
author_rank: Contributor
timestamp: June 25, 2024, 02:13:30 pm
content: Thanks for the response!See attached a couple of screen grabs of a hex compare of the flash memory. Faulty MCU as compare to the fresh out of the box working MCU.I think the fact that a bunch of the registers are set to 0xFFFF FFFF seems a little strange in the previous micro status screen grabs.Cheers,Sean.

author: ataradov
author_rank: Super Contributor
timestamp: June 25, 2024, 02:26:22 pm
content: Yeah, those are bit errors. They are 100% the reason for the faults. You either have a faulty MCU or something else is wrong on the board and one MCU is a bit more sensitive.Double check VDDCORE and see that it is at 1.2 V and clean (no ripple or other noise).And you can see that bit 3 (value 4) is the reason for the difference in all cases and it is stuck at 1. This is how flash issues typically present themselves.Furthermore, you can try to do multiple reads with power cycles between them and you are likely to observe that those bit errors move around and sometimes affected bytes recover. This is also a sign of marginal flash performance.

author: oldfortune_
author_rank: Contributor
timestamp: June 25, 2024, 02:36:36 pm
content: Good spot! Is it possible that this sort of flash fragility can occur from just 8 months of use? That seems way too short. Or does this massively depend on the power supply stability etc.

author: ataradov
author_rank: Super Contributor
timestamp: June 25, 2024, 02:41:05 pm
content: It depends. If your VDDCORE is noisy or you have massive EMI issues in your design, then it can happen even faster.At the same time, you can't exclude the actual device defect. This happens too and the only way to find out is to file a support ticket and eventually submit the device for a failure analysis.

author: oldfortune_
author_rank: Contributor
timestamp: June 25, 2024, 02:53:59 pm
content: Here's a couple of scope grabs of VDDCORE: looks to be sitting around 1.26V avg, any thoughts on that? The PSU is just 24V DC in, and a CUI drop in Buck to get a 3.3V supply rail.

author: ataradov
author_rank: Super Contributor
timestamp: June 25, 2024, 02:59:05 pm
content: This is really bad. First of all, make sure this is not an observation issue. Instead of the ground alligator clip, use the ground spring contact on the probe. This will minimize the loop area and may eliminate some noise. If the noise is still present, you will need to investigate it.And then, use a multimeter to measure the voltage on the pin more accurately. It must be ~1.2 V. Multimeter will average noise and show you a real DC value.Can you show your VDDCORE part of the schematic?

author: oldfortune_
author_rank: Contributor
timestamp: June 25, 2024, 03:11:15 pm
content: Ok so multimeter measurement is at 1.23V. And see attached the samd21g18 schematic section as well as the relevant board layout.The buck module is the following.https://www.mouser.co.uk/ProductDetail/CUI-Inc/PXO7803-500-M-TR?qs=By6Nw2ByBD3KcmSyN1WOOQ%3D%3D

author: ataradov
author_rank: Super Contributor
timestamp: June 25, 2024, 03:28:13 pm
content: This is not nearly enough decoupling. Look at the schematic checklist of the datasheet. VDDCORE capacitor must be 1 uF. Ideally you want 1 uF || 100 nF.I would also add 1 uF in parallel with C9. Whether you really need it depends in the rest of the board layout, but it will not hurt in any case.And you need 100 nF for each VDDIO pin.VDDCORE is the biggest issue, and needs to be addressed immediately. The rest are just good practice.

author: oldfortune_
author_rank: Contributor
timestamp: June 25, 2024, 03:31:03 pm
content: Ok cool, and so you think that the VDDCORE insufficient decoupling could cause these permanent flash memory failures?What would an acceptable ripple on VDDCORE be, with correct probing?Thanks for all the answers

author: ataradov
author_rank: Super Contributor
timestamp: June 25, 2024, 03:37:44 pm
content: VDDCORE noise can cause permanent flash damage. Whether this noise is due to insufficient decoupling or from external factors, does not matter.Ideally you want to keep it below 10 mVpp, but I would really start looking into things at 20 mVpp and above.

author: oldfortune_
author_rank: Contributor
timestamp: June 25, 2024, 03:41:23 pm
content: Ok thanks for taking a look! I think I pulled a reference for the decoupling from an open source design, which I now see a couple of others have pointed out failure due to bad VDDCORE decoupling!Good tip to be really careful with that one!Cheers,Sean.