intermittent Problems in Embedded Systems - Page 1
Forum section: electronics / general

author: palpurul
author_rank: Regular Contributor
timestamp: February 10, 2022, 01:31:43 pm
content: Hello,I've been working on a design for vending machines and we've been testing the device in the field for the last few months. It's been performing kind of better than I expected, but still we got a few device fail in weeks from time to time. We can obviously service the devices because we can reach them easily, simple reset would give them a week or two or maybe a month, but after that they would still fail for some reason.I can't really share the whole code because obviously it's an IP of the company I am working for, even if I do share it, it's terribly long...What I am looking for is advice on how to debug or determine intermittent problems in embedded systems. They are incredibly hard to catch and debug. I know it's not a clear question, but I can really benefit from hearing an experienced embedded designer about how to solve problems that are really hard to come by.What I was thinking about is to write aperiodic reset routinethat resets the whole systems safely after a few hours, I know it's a very primitive way to solve any intermittent problems, but that's what I can think of at the moment.I am writing bare-metal on dsPIC33CK.Thanks in advance!EDITWOW! Many thanks. A lot of sharing thier experiences about this issue, I can't tell you how happy I am. I am getting the help of my life here thanks!I am reading all of the answers as all of them are filled with experience and knowledge they are hard to digest all at once, I now have more questions

author: alexanderbrevig
author_rank: Frequent Contributor
timestamp: February 10, 2022, 01:42:50 pm
content: Are you using a watchdog timer? You probably shouldAs for answering your question, from my experience I have had luck with "fuzzing" where you basically automate a way to test your system under expected and (most importantly) unexpected inputs. F.ex if you have buttons wired in, maybe they become extremely bouncy from time to time (depending on temperature f.ex). I would then fuzz this button by attaching a signal generator and sweep it from very slow to waay too fast, in burts with delay between bursts. This is just one example, but most inputs can be fuzzed like that.I must admit though, that I too have implemented a timeout reset in one particular project that I just could not find a solution to (I saw in a later version of the softdevice  that they fixed a bug that I think may have beeen my culprit).

author: hans
author_rank: Super Contributor
timestamp: February 10, 2022, 01:53:46 pm
content: My first intuition would be:Do you have any idea which user actions have a correlation with the issue? E.g. were most reports that the machine is hanging during during vending, restocking operation? Or was it just idle?Do you use any kind of heap or other allocation structure that could overflow?Also indeed make sure the hardware is stable, and any code interfacing with hardware is rock solid. E.g. you don't want a code to hang on some kind of peripheral flag that never gets set. Or an UART that may every 1M bytes detect a frame error which is not (or improperly) handled, and therefore the code keeps waiting until the last missing byte is received..If you have time awareness (RTC) you could hard reset the MCU every night at e.g. 4AM. Probably no customer that will notice/care, and that may "solve" the issue for the customer.Still that doesn't improve the confidence you have in your firmware. I think increasing that requires a review of how firmware is developed. Do you run your debug images with assertion statements? If so, try to fix or understand any assertion failure you see occurring.What kind of QA or in-house testing is done before firmware is released? Is there any indication of a regression?Are there unit tests for high-level parts of the code? For example, I imagine that all the inventory bookkeeping can be ran on a PC and doesn't require hardware. You can stress test that code very easily on a PC. E.g. you could put together an integration test that makes the inventory code vend/restock items (pseudo-)randomly for a days, weeks, years.. You could write tests that specifically put the system in an edge-case scenario and verify it still operates correctly. etc.If there indeed is any bug found, you could write another unit test that replicates the bug first, and then write a fix for it.You could use continuous integration to check if the code develops any regression by any change thats pushed to the source repository.But even when doing all of this, it can be very hard to track these kinds of issues. I've had several occasions where rare-but-fatal bug hunts took literally weeks of man-hour time. In one case wehadto have a direct fix for it, because it was a bug in a bootloader which stopped us from booting larger firmware images.. That really prevents the capability to do anything else..

author: Siwastaja
author_rank: Super Contributor
timestamp: February 10, 2022, 02:02:41 pm
content: Get competent people to do the job.Failing to do that (I admit it's pretty idealistic!), instead of even thinking about watchdog or self-reset routines, start adding logging and asserts in the code.You need to find the issue!Working around it is a nightmare, avoid it at all costs. Instead, make finding bugs easier.Yes, you can read on the EEVBlog forum how professional firmware development is supposedly done by single-stepping code in debugger on lab table until it works and those who resort to "printf debugging" are Arduino amateurs.And this isexactlythe result.My amateur printf debugging code always include a lot of this pattern:Code:if(index > 5)error(57); // overindexing of index. This comment is helpful as you can just run "grep error(57 *.c" and directly see this text!if(thing != 0) // our assumption!error(58); // thing wasn't zero as we expected.thing = 1;And how to implement error()? Initially, make it a while(1) loop which prints the error code into UART - repeatedly, so you can catch it even if you didn't have anything hooked up - and blinks a LED showing the error code (for example, 5 long and 7 short blinks for code 57). Or make it do long and short beeps. This way, anyone can count the flashes/beeps. You can also print or save extra information, like a bunch of peripheral register values you feel like could be significant.Unlike single-stepping in debugger, these mechanisms work in production!Later, you can also come up with errorlogging, which then allows you to make error handler just reset the thing.Also, say, you have a state machine? Like this:Code:state_t cur_state;switch(cur_state){case IDLE:if(input) cur_state = STATE_ACTIVE;break;case ACTIVE:... and so on ...}How to improve it? Well, obviously first add default:, calling error(59);Then, add something like this:Code:typedef struct{uint32_t timestamp;state_t from;state_t to;... another things of interest ...} log_element_t;#define LOG_LEN 1024volatile int log_enabled; // volatile, if you log in ISR, and poll this from main threadint log_idx;log_element_t log;... and just after the state machine switch-case:Code:static state_t prev_state;if(log_enabled && prev_state != state){if(log_idx < 0 || log_idx >= LOG_LEN)error(60); // you see, let's make this a habit before indexing an arraylog.timestamp = access_some_timer();log.from = prev_state;log.to = next_state;... and so on ...log_idx++;if(log_idx >= LOG_LEN)log_enabled = 0;}prev_state = state;Turning log_enabled = 1 and either setting it back to 0 or waiting for completed trace to turn it to 0, then printing out log to UART, SD card or whatever is left as an exercise for the reader.If this feels overwhelming, let me say I didn't copypaste this from any project or even use code editor to write it, just wrote it on the forum reply box, and it didn't take many minutes to do that. For the same reason, it may have bugs.Such instrumentation is critical in complex applications anyway; for a vending machine, you might be able to simplify the problem to survive without. But seeing you have problems with it, start from adding visibility.Sometimes people are afraid of "debugging" in production, but if the alternative is a nonfunctional system... Think about it! It would bebestif no bugs were left in production, at all, but failing that, being able to solve the issues even in production is way better than total failure.

author: Berni
author_rank: Super Contributor
timestamp: February 10, 2022, 02:05:14 pm
content: Watchdogs are the best safeguard against mysterious lockups. MCU code usually runs in an endless loop so you can clear the watchdog in that loop, if the loop stops you get rebooted after a while.But this will only fix complete lockup problems. The code can still get in a weird state that keeps executing but constantly does the wrong thing. For this sanity checks are one way to do it. This means that functions always check there input parameters for all of the parameters being sensible values. So if you expect to get something in % from 0 to 100 and you get -1 that is a sign that something went wrong, this is called an assertion and typically are wired to stop the debugger there so that the programmer can find out what happened. For in production where there is no debugger connected you might either log that or just straight reboot.You also want to torture your final board in every way possible. Expose it to ridiculous heat with a heat gun, freeze it to -40°C with freeze spray, rise and lower the supply voltage to the extreemes, reduce the supply voltage to the point when it shuts off and give it back, take a cellphone and make a call, wave it around the whole board, take a piece of wire short out the I2C SPI, UART etc lines to ground to Vcc, between them, hit buttons randomly at fast rates... any form of torture you can think of that wouldn't just permanently destroy the board.

author: Siwastaja
author_rank: Super Contributor
timestamp: February 10, 2022, 02:09:50 pm
content: Quote from: Berni on February 10, 2022, 02:05:14 pmAlways do your best to solve mysterious lockups. If you just reset, the risk is high that in some particular use case, the lockups become more frequent and prevent the operation completely. Mysterious lockups are disastrous failures, IMHO, unless they are really rare and caused by things like cosmic radiation flipping a bit once in 30 years (normal for non-ECC DRAM).EDIT:And remember, this might not be obvious: Solving meansfinding and proving the root causeand rectifying that! Random changes that make the problem disappear are no good, because then the problem reappears later. Workaround can be acceptable in very rare cases (bug too difficult; no time for proper troubleshooting) but they need to be proven to work as well, and their consequences documented somehow.For a resetting watchdog to be usable workaround, you would need to be able to prove the actual bug never occurs during critical phase (for example: between the user inserting the first coin, and the machine vending the product), and you rarely are able to prove that - if you could, you would know where the bug is, and could just fix it instead!

author: Siwastaja
author_rank: Super Contributor
timestamp: February 10, 2022, 02:32:53 pm
content: Code that seems to "hang":Possible causes:* Any loop (those that do not simply count, but look at changing conditions): breaking conditions are not satisfied as you expect* Stack corruption* Wrong/corrupted function pointers* ISR that gets retriggered infinitelyBy far, #1 reason for stack corruption is over/underindexing an array. Just search for [ in your code to find every indexing, add range checking as shown above, and chances are high you catch the problem! Just call the customer, ask how many times the LED is blinking, and voila, you have a fix within a day! (Remember that if you index with signed types, negative indexes will cause a problem, too!)Of course pointer arithmetic can go wrong, too, but people usually do less of it, and are more cautious when they do, which is why I mention [] indexing as the most probable cause.If you handle strings, make sure you use the strncpy, snprintf type functions and further, read very carefully how they actually work regarding the n parameter (do they add terminating zero or not, and if they do, is it included in the n?)I mention this stack corruption first, because when you rule it out first (or, reduce the likelihood by safeguarding the most usual suspects), this either fixes the problem, or leaves:* Any loop (those that do not simply count, but look at changing conditions): breaking conditions are not satisfied as you expect* Wrong/corrupted function pointers* ISR that gets retriggered infinitelyFunction pointers are easily checked because you either have none, or have just a few, or you are an experienced C wizard who makes no mistakes with them.This leaves retriggered ISRs, and loop conditions. For ISRs, make sure you understand the clearing conditions of peripherals, and maybe you have unexpected interrupt sources turned on, like an error interrupt coming from EMI/ESD event, which you fail to check and clear, keeping it repeating?This leaves you with just loop conditions, and instead of going full-blown instrumentation first, maybe you can just go through all of your loops, looking for assumptions such as some peripheral setting a status flag like you expect, but not do that? You can probably rule out 90% of your loops as trivial enough, and instrument the rest; add a timeout and report an error.I despise of STM32 HAL code where every freaking function can return a failure or a timeout. Talk about cargo cult engineering! The problem here is, there is already enough instrumentation, error checking and recovery work to do where a failure canactuallyhappen. You don't need to check every operation which, by design, cannot fail, and STM32 designers should know the best which can fail, and which can't.

author: palpurul
author_rank: Regular Contributor
timestamp: February 10, 2022, 02:44:05 pm
content: Quote from: alexanderbrevig on February 10, 2022, 01:42:50 pmI am not actually using watch dog-timer. I don't have much experience with embedded software we have a very small team, we are kind of learning along the way really.I thought about using watchdogtimer, but I don't have much knowledge with it. I'll look into it.Thanks!

author: eutectique
author_rank: Frequent Contributor
timestamp: February 10, 2022, 03:20:43 pm
content: Quote from: palpurul on February 10, 2022, 01:31:43 pmJust make sure you do thisafteryou receive a banknote, dispense an item, and give the change. Not in between.

author: palpurul
author_rank: Regular Contributor
timestamp: February 10, 2022, 03:27:37 pm
content: Quote from: hans on February 10, 2022, 01:53:46 pmI don't have any idea which action causes MCU to fail, but I can guess that it fails during idle because most of the time we have customers calling in saying that they found our device is not working. It might be something.Quote from: hans on February 10, 2022, 01:53:46 pmThat's a good idea actually, but I don't have RTC in the board, I'll keep that in mind.Quote from: hans on February 10, 2022, 01:53:46 pmNope, that's kind of new to me actually.Quote from: hans on February 10, 2022, 01:53:46 pmWe have a very small team actually (me and some other guy who develops the app that work with the device) we are not that organized, I am trying to develop some kind of a diagnostic tracking built into the MCU, but I am also running out of space and time. I will try to make a pseudo vending machine and test the device for hours to see if it fails after thousands of vend, I think that's the only way I can get a repeatable result.

author: palpurul
author_rank: Regular Contributor
timestamp: February 10, 2022, 03:28:20 pm
content: Quote from: eutectique on February 10, 2022, 03:20:43 pmYes, I've been thinking about this actually.

author: MK14
author_rank: Super Contributor
timestamp: February 10, 2022, 03:32:18 pm
content: In some systems, you can create logs. Which might go something like this:10:12:05 Started User Input10:12:51 Normal exit of User loop14:11:02 Coin Slot Activated14:11:04 Coin Slot Approved $0.5014:11:21 Started User Input14:11:51 User Requested Refund14:12:04 User Requested Refund14:13:42 User Requested Refund14:13:49 User Requested Refund14:15:24 Anti-Tamper-Vibration Alert {Possible Cause - Machine being kicked by frustrated user}15:07:44 Unexpected recovery from Watchdog timeoutWhich can help see where any problem(s) are occurring, and determining if they are software, hardware or mechanical/other issues.

author: krish2487
author_rank: Frequent Contributor
timestamp: February 10, 2022, 03:35:21 pm
content: I would side with @Siwastaja and add that your first plan of action should be to make the issue easily reproducible.If you can write to internal flash memory / non volatile memory then keep a few hundred bytes as a log storage where you keep overwritingcontents with a log of the last operation that the MCU ran.. I would suggest you try and isolate the most suspect path which is causing this and place thelog functions there.. then when the lockup happens.. rerun the mcu under debugger and inspect the flash memory contents / variable contents in the IDE.or print it over something like uart on the power up and capture it..This ideally should be something like a ring buffer that you can simply keep clearing and overwriting..Not a long term solution since it wears out flash.. but its one option..Edit : Looks like MK14 and I were responding at the same time on the same ideas.. :-D(something about quantum entanglement comes to mind... :-P )

author: Siwastaja
author_rank: Super Contributor
timestamp: February 10, 2022, 03:43:41 pm
content: 98% likely the problem is just a bug in your code. 2% it's a HW problem, for example unstable supply voltage.Just add visibility to the code, by means I mentioned above. Don't try to do anything else but follow the leads and find the problem. It is always doable, but the work needs to start sooner than later.Even if you just spend one hour to add rudimentary logging, you can likely rule out 90% of the code as the source of the bug!The worst thing you can do are periodic resets. That would be suitable if you can first prove the problem really surfaces after some time; i.e., a probability function where probability is first zero, and increases after some time period. But 95% likely this isn't the case, the problem probably happens at semi-random* timing! Say, the bug happens randomly; on average, once every 48 hours. So resetting after every 24 hours should fix it? No, it does not do anything, it will still happen once every 48 hours, on average!*) semi-random, because the problem is still likely triggered bysomething, we just don't yet know what, so it appears random.Unless youreallyhave some running counter which wraps around causing the issue, only then are periodic resets going to help. Like the classical Windows 95 crash (if it somehow managed not to crash until the wraparound), or the Boeing rebooting requirement. Even then, it's best to just fix the bug, but failing that, periodic reset is at least an actual workaround.And about reproducibility. Best would be if you can indeed reproduce the issue in seconds or minutes, on lab table! Then you could even use a debugger! The problem is, if you don't know what causes the bug (otherwise, you would just fix it, right?), you usually don't know what affects the frequency of the problem (that would already be a huge hint). If it happens once every 48 hours on average, or if it only happens at customer's premises, or customer's specific workload, the most practical solution is to add instrumentation to your code, even if it takes a bit more work than poking at registers on a debugger.You can still try to make all kinds of modifications that try to make the problem appear more often. But donottry the opposite, trying to make it disappear with random changes and then be happy. In optimum case, you find a modification which makes it crash almost instantly, setting the stage up for systematic bug finding session.In the end, this is detective work. Try to enjoy it as such! If you specialize in coming up with workarounds, your code never gets better, and you always waste time. But if you take a systematic approach, you find the bugs, correct them, and won't make the same mistakes again (at least too many times) in the next projects. Little by little, you write less and less buggy code, and add more and more those little features that help you find any mistakes.

author: palpurul
author_rank: Regular Contributor
timestamp: February 10, 2022, 03:56:12 pm
content: Quote from: Siwastaja on February 10, 2022, 02:02:41 pmThat should be meQuote from: Siwastaja on February 10, 2022, 02:02:41 pmI like this mindset. I am not too capable as an embedded desginer, still learning and learnt much from what you wrote. Is there any open-source project that fits your requirements on the issues that we discussed here.I am still reading your replies because it's too much for me to digest all at once, thanks for all the replies, it reall helps!

author: Siwastaja
author_rank: Super Contributor
timestamp: February 10, 2022, 04:09:13 pm
content: Quote from: palpurul on February 10, 2022, 03:56:12 pmExcellent, you can always decide to become one. It's often harder to find really capable people, than to become one yourself, especially as you have already started the journey.You can boil down my long posts to: add visibility to your code, in any way you can come up with, to understand what it is doing before and when it crashes.

author: tggzzz
author_rank: Super Contributor
timestamp: February 10, 2022, 04:20:57 pm
content: You need todefine what is happening when it is locked upthe sequence of things that occurred in the runup to the lockupI presume you have explicitly designed the system as an FSM and explictly embodied the events and states in your hardware and software implementation. Keep a log of the events and transitions, on a separate device if necessary. The log can be extremely terse, provided that it can be "decoded" by a post-processing program and/or the designer.Make sure that you are using C correctly, particularly w.r.t. multiple threads and interrupts. Many people misunderstand what the C keywordsdo and don'tguarantee. if you have coded it as a superloop with lots of nested if-then-else/switch-case statements, then have fun.

author: ralphrmartin
author_rank: Frequent Contributor
timestamp: February 10, 2022, 05:23:39 pm
content: If it runs for weeks OK then fails, I'd- look at the bits of code which you don't expect to run very often - perhaps some unusual set of circumstances triggers a rarely used bit of code, which is faulty- consider the possibility of a memory leak, and over time, you slowly run out of memory until something bad happens. Of course, that may also be in infrequently exercised code- consider overflow of a 32 bit millisecond counter, which will happen after about 1200 hours (50 days, or 25 if signed)

author: NorthGuy
author_rank: Super Contributor
timestamp: February 10, 2022, 07:48:57 pm
content: You cannot fix a bug if you don't know anything about it. You need to collect some information first. Anything relevant will help.For example, create a periodic (e.g. once a millisecond) highest priority interrupt which you never disable. On the ISR entry, the stack will contain the execution address. Create a circular buffer in RAM, and store the execution addresses in that buffer. This way, when the program starts looping somewhere, your interrupt will keep recording addresses where it is looping. You will always have last 50 or 100 most recent addresses in your buffer. On startup, dump this area of RAM somewhere where you can inspect it.Next time your machine hangs, go there, reset it, and inspect the dumped addresses. This way you will know where your program is looping. This will give you something to start from.

author: peter-h
author_rank: Super Contributor
timestamp: February 10, 2022, 09:25:25 pm
content: I would start by running it under a debugger but with no breakpoints so it all runs at full speed.Then when it hangs, you can press STOP (or whatever function you have in your IDE) and it will show where it was hanging, and a stack trace. That will tell you a great deal and you can work back from there.If it doesn't hang but crashes, it probably crashes into some traps like invalid opcode, and if you trap those, again you can get a stack trace.

author: VintageTekFan
author_rank: Regular Contributor
timestamp: February 11, 2022, 12:57:54 am
content: Quote from: ralphrmartin on February 10, 2022, 05:23:39 pmI think you just solved an issue I've been having with some vendor software for the last couple of months...

author: nctnico
author_rank: Super Contributor
timestamp: February 11, 2022, 09:16:21 am
content: Quote from: Siwastaja on February 10, 2022, 03:43:41 pmI agree with this. Logging and adding checks for illegal values (which also print to the log output help) is a good way of finding bugs which happen every now and then. Even if it means putting a computer next to a machine to do the logging itself. All microcontrollers devices that I desing have logging output on a UART.

author: tggzzz
author_rank: Super Contributor
timestamp: February 11, 2022, 09:43:18 am
content: Quote from: Siwastaja on February 10, 2022, 03:43:41 pmThere are other hardware possibilities:voltage transients caused by "high" current actuators or motors, possibly fixed by changing the wiring and/or connectorsEMI, possibly mains borne, possibly due to RF from cellphones or intentional felonious discharges from gas lighters

author: Siwastaja
author_rank: Super Contributor
timestamp: February 11, 2022, 10:40:15 am
content: Quote from: tggzzz on February 11, 2022, 09:43:18 amI was thinking about this, and logging of the code gives a very good hint of this, too. Once you see that the crash happens just after initiating the actuator, and before the next big thing the code would do next, you reduce the possibilities to either a very small part of code, or hardware related to driving the actuator.Actually, it's a good sanity check to make:* Do all motors, solenoids, relay coils etc. come with a freewheeling diode (or in case of relays, maybe another way of clamping back-EMF such as zener, resistor, or diode+resistor)* is this freewheeling diode placed in parallel with the load, in series with the switching transistor,but physically right next to the transistor* is there a low-ESR, low-ESL capacitor right next to the said transistor - diode loop, over the supply rail

author: Jeroen3
author_rank: Super Contributor
timestamp: February 11, 2022, 11:53:55 am
content: Did you run a static code analyzer yet?https://cppcheck.sourceforge.io/Simple things like bad ifs, wrong arrays and such are easily found that way.Since you're using pic you're probably not benefiting from gcc's plentiful warnings.

author: snarkysparky
author_rank: Frequent Contributor
timestamp: February 11, 2022, 12:43:44 pm
content: create a unsigned int variable.sprinkle this variable throughout your code assigning it a value between 1 and 100 depending on location.create a non blocking function that will move this variables value to an LED. Blinking the led at one half second intervals to count the tens digit. Then pause two seconds and blink out the ones digit.5 second off period between output of the numbers.Hopefully on site customer can count these for you.If CPU is locked up the watchdog is the answer.Make sure your inputs to the CPU have transient protection to prevent noise spikes from getting into the CPU. Steering schottky diodes or large resistors on inputs.

author: Carel
author_rank: Regular Contributor
timestamp: February 11, 2022, 02:10:18 pm
content: Problem is you are developing in another place as your own. Now you have to travel, talk and listen to maybe wrong feedback. After that the device goes from a not working state to a finite time working state.It is not clear if you have one device or multiple devices and if so, have they all these symptoms.In any case, get one in your own working space. If only to see if that makes a difference. You can then leisurely torture it, without the stress that it must function the next moment, until the next moment.Beware of tunnel vision. If you know for sure that part of software or whatever is 100% perfect, Murphy states that the error is there, it will only take longer to find it.This week I spent a couple of hours making a program working again. After debugging I read my own warning 20 odd lines higher. Of course I read this only after I was ready.As you are learning, make in case of doubt solid tests. X + Y can be anything, 2 + 2 is still 4.

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: February 11, 2022, 11:16:19 pm
content: Quote from: ralphrmartin on February 10, 2022, 05:23:39 pmA good example of a problem like this affected the Boeing 787 a few years ago.https://arstechnica.com/information-technology/2015/05/boeing-787-dreamliners-contain-a-potentially-catastrophic-software-bug/

author: peter-h
author_rank: Super Contributor
timestamp: February 12, 2022, 06:38:58 am
content: The problem with "spikes" crashing your CPU is that there is a fine line between crashing it, and blowing something up.One must design the product so that these cannot get in to start with.Sometimes this is just not possible. Many years ago I had a customer who was installing one of my interface/protocol converters right next to a huge piezo boiler igniter. My thing was in a plastic box and passed all the standard EN... EMC tests, but this thing was so powerful. 100kV transients with microsecond risetimes will capacitively couple into anything. I told them to mount them in a metal box, shield all the cables, etc, but they would not spend the money. The product has a watchdog which was tripping, but the spikes corrupted a serial EEPROM (which takes some doing!). There is no software defence against that problem.

author: nigelwright7557
author_rank: Frequent Contributor
timestamp: February 12, 2022, 08:08:32 am
content: Embedded systems can have 2 types of problems.Software bugs or hardware problems.Hardware problems like glitches on the power supply or static discharge into inputs from people touching the device.Software bugs come in many forms but often because of silly code input mistakes.Or the designer not thinking through all possible inputs.

author: hans
author_rank: Super Contributor
timestamp: February 12, 2022, 08:48:11 am
content: Quote from: ralphrmartin on February 10, 2022, 05:23:39 pmThe challenge is that if these methods are exhausted, there can still be bugs even for the functionally "right" C code.For example race conditions can also occur when dealing with non-atomic operations. On this 16-bit platform, maybe there is a 32-bit variable shared between irq and main code. Since that variable will require 2 load or store instructions to access, if the IRQ occurs inbetween the load/store who knows what might happen (I don't know the code, so it could mean a small glitch in timing/behaviour, or a complete lock up). The chance of the IRQ occuring exactly at that access time is very low (=long time) with random intervals.

author: tggzzz
author_rank: Super Contributor
timestamp: February 12, 2022, 11:28:52 am
content: Quote from: peter-h on February 12, 2022, 06:38:58 amI first came across this in the mid 70s with a single board computer kit. The amateur radio club the other side of a brick wall would occasionally discharge a big ~2kV capacitor; the EMI locked up the computer.

author: nctnico
author_rank: Super Contributor
timestamp: February 12, 2022, 06:00:02 pm
content: Quote from: hans on February 12, 2022, 08:48:11 amOr worse... At some point I spend quite some time hunting down a buffer overflow which caused a problem every now and then depending on whether an interrupt just happened so the data on the stack was different than 'normal'.

author: ralphrmartin
author_rank: Frequent Contributor
timestamp: February 13, 2022, 04:21:57 pm
content: Quote from: hans on February 12, 2022, 08:48:11 amSure, these were just a few of the things I'd start off by looking at. This wasn't meant to be an exhaustive list. As you pointed out, race conditions are another possible cause - but rather harder to debug, so I'd check the things which are easier to check first.

author: nigelwright7557
author_rank: Frequent Contributor
timestamp: April 07, 2022, 05:44:53 pm
content: 1/ Noise on power supply.2/ Poor decoupling.3/ Static discharge into buttons. Put 47k resistor in series with button inputs.4/ Stack wind up eventually causing a crash.

author: mazurov
author_rank: Frequent Contributor
timestamp: April 07, 2022, 06:48:32 pm
content: Quote from: Siwastaja on February 10, 2022, 02:02:41 pmThis. This will be cheaper in the long run and could be faster.QuoteYou can also output the error code on the machine's display, just make it look dissimilar to a normal output. For example, print Errxx, where xx is the error code [00-ff].There are other ways to implement fault indication in the field. For examples, google "shipping with assertions", that's the proper name for the approach.