Need Help Troubleshooting Remote IoT SBC Hangs with Custom Cape - Page 1
Forum section: electronics / projects

author: davegravy
author_rank: Regular Contributor
timestamp: November 01, 2024, 02:04:17 am
content: I'm working on a project with an SBC that has a custom cape for:Powering the SBC via a 5V buck converter from a 12V LiFePO4 batteryCharging the 12V battery from solar powerInterfacing an LTE module over UARTCommunicating with an I2C fuel gauge and monitoring charging state + panel presence via GPIOSince this device will be deployed remotely, stability is critical. The current setup runs a test Python script that mimics the final application with a 10-minute loop:Connects via LTE to the carrier and sends a test payloadPuts the module to sleep for ~9 minutesIt’s been stable for 2+ days, but twice now I’ve found the SBC hung—SD card access LED solid ON, unresponsive over UART console—frustratingly after long periods of stable operation. I tried with a brand-new SBC, same result.Notably, I've used this same SBC and OS image in other projects without any issues, so I suspect something with the electrical interfacing on this cape.How would you troubleshoot this? Here’s my thought process so far:Key Theories:Could high current draw from a GPIO trigger an MPU reset, or even cause a system hang? I plan to monitor current on each GPIO pin over the loop duration for any unexpected spikes.Scoped the 5V line: ripple is about 40mV pk-pk, but I see brief 100mV sags during Rx/Tx. Could this be critical? This is on an old analog scope that I can't figure out how to get triggering or max/min functions working, so there could be brief voltage fluctuations outside of thisLog code location on each hang to see if there's any pattern.Measure voltage across all exposed SBC pins after a hang.Tail syslog for any kernel errors leading up to the crash.The SBC outputs VDDIO which is its 1.8V IO reference to various translator ICs - the current drawn from this output should be monitored, checking that it's not close to exceeding its ratingSystematic Testing Ideas:Disable specific code sections to isolate possible causes.Disconnect specific GPIOs/interfaces to see if that impacts stability.Try swapping the SD card (just in case).Substitute the built-in buck converter with a bench 5V supply to rule out power supply issues.Troubleshooting will be slow, given the intermittent nature of the problem and that I currently have only one prototype. I might order more units to speed up testing, though I'm hoping to resolve this issue before a new rev.I’m also considering adding a hardware watchdog to monitor the SBC with a heartbeat and cycle power if it stops. Does anyone have recommendations for simple watchdog ICs or circuits for this without using a full MCU?Would appreciate any insights or additional ideas on what to check or ways to streamline testing! Has anyone faced a similar issue with intermittent hangs like this?

author: Salitronic
author_rank: Contributor
timestamp: November 01, 2024, 11:07:54 pm
content: Which SBC are you using? Typically the PMIC chip (that would typically be on the SBC module) has a built-in watchdog that you can use.Is there any evidence of SD card corruption or filesystem errors?If you're running the OS from the SD card you should ideally disable logging to the card, mount strategic directories to tmpfs and lock the card to disable writes (not just mounting it read-only). That's the only way I've found to ensure reliability when running from SD card.

author: davegravy
author_rank: Regular Contributor
timestamp: November 01, 2024, 11:42:48 pm
content: Quote from: Salitronic on November 01, 2024, 11:07:54 pmIt's this:https://conclusive.pl/static/docs/kstr-sama5d27/DS_020100_20220401_kstr-sama5d27.pdfOooh, I'll check the PMIC datasheet in case it has a watchdog, that would be very useful. Thanks for the suggestion.Quote from: Salitronic on November 01, 2024, 11:07:54 pmNo evidence.Quote from: Salitronic on November 01, 2024, 11:07:54 pmI need nonvolatile storage in case of spotty cell reception, to cache data. Nonvolatile partly because outages can sometimes last a while and data may not fit in RAM. Not sure where else to log besides the SD card.While I've read about lots of SD card stability issues I have almost 100 of these SBCs deployed remotely with the same SD card and in 2 years never had an issue.

author: Salitronic
author_rank: Contributor
timestamp: November 02, 2024, 08:32:36 am
content: Quote from: davegravy on November 01, 2024, 11:42:48 pmThey are using an ACT8945A PMIC, it's a rather basic PMIC, no watchdog unfortunately.