Most annoying bug.... - Page 1
Forum section: electronics / general

author: dmills
author_rank: Super Contributor
timestamp: October 20, 2015, 07:51:59 pm
content: This one that just cost me 7 working days: hard of thinking processor vendors supplying their own libc, and somehow managing to screw up the string handling functions, strtol and friends in this case, it is not like there is not a perfectly good one in the BSD libc they could have freely used (That actually works), guess what my fix was?Of course the bug they introduced was a bus error that only occurred when the string started at an address NOT divisible by four....Also of course the call was in the middle of a mess of pointer arithmetic (In the way adhock text parsers in C tend to be), so it was not at all obvious that the crash was in the library (Mips has this amusing thing it does upon function call return which made the problem appear to occur after the return).Next project is so going to use something non Microchip.So who else has had a good bughunt recently?Regards, Dan.

author: amyk
author_rank: Super Contributor
timestamp: October 21, 2015, 02:37:07 am
content: MIPS alignment requirements are certainly annoying but I can see how they never tested with anything other than aligned strings.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 21, 2015, 04:59:04 am
content: I am currently examininga situation where the Debug build puts out several bit-banged signals (LCD) too fast compared to the release version. I have tracked it down to code optimization options and have run a few isolated tests that did not reproduce the problem. It must be some structure in my code that gets 'optimized away'.I have patched it for now by making the delay times a bit bigger/longer because I fear that hunting this one down may eat up a lot of time and I want to keep moving...

author: Jope
author_rank: Regular Contributor
timestamp: October 21, 2015, 08:11:32 am
content: Quote from: amyk on October 21, 2015, 02:37:07 amUnaligned access to char arrays is actually a pretty normal use case. Let's say you have a command string with a number and you want to extract the number with strtol():Code:long val = strtol (&cmd_str, NULL, 10);Depending on index, it will be an unaligned access.

author: dmills
author_rank: Super Contributor
timestamp: October 21, 2015, 03:30:44 pm
content: Yep, a mix of strtok and strtol is fairly standard for pulling apart a string of CSV data and exactly what I was doing.....It does work when you have an strtol that was not written by the crack smoking intern.Regards, Dan.

author: Maxlor
author_rank: Frequent Contributor
timestamp: October 21, 2015, 03:50:29 pm
content: This bug cost me a couple of days a few years ago: a pointer bug caused the interrupt table to be modified, leading to a crash some time later, when the interrupt happened. (On ARMv5, the interrupt table is at address 0, so dereferencing a NULL pointer is legal, and puts you there.) That sort of problem is hard to find if you don't suspect it, because the program will crash at a random location (and usually always in the same place, if the timing remains the same), but that location has nothing to do with the actual bug. Plus, when you start debugging, the location will move. When you single step, the bug disappears altogether. It's a classical heisenbug.After having finally found and fixed the problem, I felt motivated enough to read up on the ARM MMU unit and and figure out how to write-protect most of the address space. It cost me another two days to get that working because the docs were rather sparse, but I felt that eliminating a whole class of bugs was worth it.Not strictly a bug, but painful nontheless: quite some time ago I was developing a custom printer driver for our application that would talk directly to the hardware. Apparently, the printer manufacturer wasn't a fan of error reporting. If you sent the printer invalid data, it would simply blink it's red LED 3 times and do nothing else. After pouring over a lot of hexdumps, and some guesswork, I did get it working in the end, so yay. But it's not a great feeling if you don't know whether you're 5 minutes or 5 days away from solving your problem.

author: Dielectric
author_rank: Regular Contributor
timestamp: October 21, 2015, 03:57:29 pm
content: Quote from: Maxlor on October 21, 2015, 03:50:29 pmOh man, you triggered a flashback for me. Furiously debugging a test box design, I had a couple of LEDs for user indicators. The red one was on/off/blink for various states, driven from a PWM. For some reason it was always either off or blinking, but this was a target board so the ICE wouldn't connect to it to set a breakpoint, etc. UART printf didn't show much either. After about a day, I realized I had put in just a random red LED from the junk box which turned out to be a flashing LED. I could just barely see the little ASIC in the T1 package. Hateful.

author: dmills
author_rank: Super Contributor
timestamp: October 21, 2015, 04:16:32 pm
content: Ouch!We will gloss over the week spent on a linker script (Why are they never properly documented?) due to the fact that 0x1000000 != 0x10000000Regards, Dan.

author: eck
author_rank: Contributor
timestamp: October 21, 2015, 06:00:28 pm
content: Explicitly targeting the 80517 (improved 8051), which also enables additional data pointers via a data pointer select register, without taking care to also save and restore the data pointer select register during interrupts. Random crashes every week or so, only in the fully operational system at the customer site.After month of logging, experiments and desperate hacks, the problem was finally found reading the assembly code. Fortunately caught most of the affected systems in the warehouse before shipping, and replaced the EEPROMs.

author: langwadt
author_rank: Super Contributor
timestamp: October 21, 2015, 07:15:26 pm
content: Quote from: dmills on October 21, 2015, 03:30:44 pmone thing is writing it but it is hard to imagine how little effort when into testing if something like that wasn't caught

author: rollatorwieltje
author_rank: Unknown
timestamp: October 21, 2015, 07:41:01 pm
content: Quote from: Maxlor on October 21, 2015, 03:50:29 pmAt least it just blinks a LED. A long time ago I was messing around with that HP printer configuration protocol (PJL?). Turns out when you make an error, instead of complaining it just starts printing the commands you send. Really nice when you want to configure a printer with a script and suddenly it has wasted 50 sheets of paper, just because there was something stupid like an extra space somewhere

author: AndyC_772
author_rank: Super Contributor
timestamp: October 23, 2015, 08:22:51 am
content: Quote from: dmills on October 21, 2015, 04:16:32 pmNow I don't feel so bad about the day I lost becauseload="Yes"...isn't the same thing as...Load="Yes"...in a linker placement file. One is recognised and understood, the other is silently ignored. The net effect is that, although the code is perfectly good, and appears in the IDE correctly, it's not actually present in the MCU's memory. The code that gets executed is not the same as the code that appears in the debugger.It didn't help that, in the default font used by the IDE, the two are virtually impossible to distinguish.

author: Karel
author_rank: Super Contributor
timestamp: October 23, 2015, 08:52:39 am
content: Apart from USB and TCP/IP, I don't use any libraries at all. During my career I have written all those utils/functions myself. At least I can blame myself when there is a bug.There are a couple of target/MIPS specific functions that I copied from the mchip libs to my own libs but only after a carefull examination and fully understanding what it does and doesn't.I have bumped my head too much to trust vendor specific libraries. And when they work, they are usually bloated and written in a style that makes it difficult to understand for me what is going on.But probably this is also because my coding/writing style is different than what is considered normal... For example, I hate it when the hardware is abstracted too far away. I'm a hardware guy, I like to poke registers.

author: tggzzz
author_rank: Super Contributor
timestamp: October 23, 2015, 09:05:47 am
content: Quote from: AndyC_772 on October 23, 2015, 08:22:51 amAnd that's the generic problem that you will automatically avoid in the future! Everybody knows about 0 and O, but L l 1 are just as problematic.

author: diyaudio
author_rank: Frequent Contributor
timestamp: October 23, 2015, 09:15:06 am
content: Quote from: dmills on October 20, 2015, 07:51:59 pmI spent about 2 weeks with a issue where the external flash code intermittently loads into the SHARC processor, sometimes it loaded successfully and sometimes it didn't.. during many power cycles attempts it would work ... the problem was I didn't connect the external crystal loading caps and loading resistor to the crystal, so the processors PPL didn't have a stable oscillator during start-up... I was pretty close to dumping the board in the dust bin

author: Wilksey
author_rank: Super Contributor
timestamp: October 23, 2015, 10:31:56 am
content: The question is what compiler (XC I presume, but XC8, XC16 or XC32?) and what version, I find that bugs get introduced in one version of compiler that work fine in a previous version, I believe the XC compilers are what used to be the HITEC C compiler?Cheers

author: Ian.M
author_rank: Super Contributor
timestamp: October 23, 2015, 10:37:58 am
content: No. XC8 is based on the HiTech PICC and PICC18 compilers (Microchip bought HiTech), but XC16 and XC32 are based on GCC.

author: Howardlong
author_rank: Super Contributor
timestamp: October 23, 2015, 12:47:39 pm
content: I had to deal with some code the other week where the dev guy had let a string literal go over the end of a line. In his test system where it was deployed, and worked, his editor replaced the leading spaces on the next line with tabs. In the higher environments, the spaces remained spaces, and exceeded the space allocated. The dev guy, of course, was blaming the deployment process, and not his code (dev guys are _always_ right, right?)It took far longer for me to show & explain to him what the problem was, and how to resolve it, than it did for me to notice it. This quite commonly happens, a second party sees an error like this right away, while the other party remains completely blind to it.

author: AndyC_772
author_rank: Super Contributor
timestamp: October 23, 2015, 12:59:08 pm
content: Quote from: tggzzz on October 23, 2015, 09:05:47 amN0rmaLLy I'd expect a parser to ch0ke and c0mpLain if I transp0se '0' with 'o', th0ugh. In this case, n0 error was generated; my instructi0n t0 actuaLLy d0wnL0ad this part 0f the c0de t0 the b0ard was simpLy ign0red.I usuaLLy expect capital and L0wer case Letters t0 be interchangeabLe, 0r t0 thr0w an err0r!

author: c4757p
author_rank: Super Contributor
timestamp: October 23, 2015, 01:06:10 pm
content: Quote from: AndyC_772 on October 23, 2015, 08:22:51 amThere's a font where L and l are hard to distinguish? Wow. I've never seen onethatbad.

author: AndyC_772
author_rank: Super Contributor
timestamp: October 23, 2015, 01:26:34 pm
content: Spot the difference. View at 1:1 scale from the distance you'd normally view a screen full of code!

author: rollatorwieltje
author_rank: Unknown
timestamp: October 23, 2015, 01:36:50 pm
content: Quote from: AndyC_772 on October 23, 2015, 01:26:34 pmLooks like that garbage font Microsoft introduced in Windows Vista. I think it's called Segoe or something like that. I pretty much use Dina everywhere:https://www.donationcoder.com/Software/Jibz/Dina/

author: c4757p
author_rank: Super Contributor
timestamp: October 23, 2015, 01:39:36 pm
content: Quote from: AndyC_772 on October 23, 2015, 01:26:34 pmJesus Christ, that'sbad. Whoever selected that font should beshot, as well as whoever designed it if it was intended for this sort of thing.Real fonts make those much more recognizable:

author: free_electron
author_rank: Super Contributor
timestamp: October 23, 2015, 01:50:03 pm
content: Quote from: AndyC_772 on October 23, 2015, 08:22:51 am

author: c4757p
author_rank: Super Contributor
timestamp: October 23, 2015, 01:52:59 pm
content: Quote from: rollatorwieltje on October 23, 2015, 01:36:50 pmHrm. I'd not seen Dina before, and I like one thing about it - it makes the classic 1I|l 0O distinction better even than my current preferred font (Terminus). I want that. Problem is, I find the rest of the font very tedious to look at. It's ugly.But then, I'm insanely picky about fonts. Maybe sometime I'll make myself a modified Terminus - pull in those Dina-style 1I|l 0O, and pull in the 9pt bold from Terminusmod, at least.

author: c4757p
author_rank: Super Contributor
timestamp: October 23, 2015, 02:03:52 pm
content: Quote from: free_electron on October 23, 2015, 01:50:03 pmWTF? Languages should be case insensitive? L and l are clearly different glyphs, I can't imagine why anyone would ever expect them to compare equal.

author: tggzzz
author_rank: Super Contributor
timestamp: October 23, 2015, 02:38:47 pm
content: Quote from: AndyC_772 on October 23, 2015, 12:59:08 pmWhy? Presumably the tokens were correct within the specified grammar.QuoteThat depends on the grammar in use.

author: tggzzz
author_rank: Super Contributor
timestamp: October 23, 2015, 02:39:34 pm
content: Quote from: AndyC_772 on October 23, 2015, 01:26:34 pmThat font is not fit for that purpose. End of story!

author: rollatorwieltje
author_rank: Unknown
timestamp: October 23, 2015, 03:03:26 pm
content: Quote from: c4757p on October 23, 2015, 01:52:59 pmDina 9pt feels to narrow for me, I mostly use 10pt (on a 22" full hd monitor), sometimes 8pt. And on a dark background whenever possible.

author: Wilksey
author_rank: Super Contributor
timestamp: October 23, 2015, 03:08:38 pm
content: Quote from: Ian.M on October 23, 2015, 10:37:58 amAh, OK, 1/3rd right anyway

author: krish2487
author_rank: Frequent Contributor
timestamp: October 23, 2015, 04:12:16 pm
content: Try thesource code profont family from Adobe.I believe that this particular font family was designed to clearly distinguish betweeni I l L 1 ! o O 0 and the similar.PS: It is also easy on the eyes as a terminal font.Quote from: c4757p onTodayat 12:52:59 AM

author: max_torque
author_rank: Super Contributor
timestamp: October 23, 2015, 10:12:16 pm
content: One bug that i know about, that just seems to get me time after time is the "Spi jumps into slave mode" one on AVR micros. Set it up as a spi, master but leave the SS pin as an input (which tends to be my default port setting for unused pins when just messing around on breadboards etc) and if it goes low, the micro jumps to being a slave without some much as a "by your leave"......... fricking annoying!

author: ale500
author_rank: Frequent Contributor
timestamp: October 24, 2015, 07:45:33 am
content: I remember this bug we were blamed for. Two teams, the "starts" and we. The "stars" blame us, we receive a stream of data, buffer it (double buffer) and give it to the stars' library. Boom crash. They say we overwrite the data (double buffer system with synchronized call to another task). I log every event, which buffer I use to receive and which buffer I give to the other thread... no overwrite over many many cycles... but it crashes at some point. I fire up the debugger and patiently go through the code, optimized C code is quite difficult to follow with a disassembler... after some time I realize what happens... they, the "stars" forgot to initialize a structure where the data would be copied and some fields had old data... Even showing them the problem did not convince them... but they fixed it and there were no more crashes.... I was fortunate enough to have this logging mechanism as it clearly showed that the problem was somewhere else... as someone here says "there are big lies, small lies and what is on the screen of the oscilloscope".Alignment problems in MIPS32 (PIC32), yeah I found out the hard way too... arm cortex-m3 has no problem with unaligned data...

author: amyk
author_rank: Super Contributor
timestamp: October 24, 2015, 03:29:51 pm
content: Quote from: AndyC_772 on October 23, 2015, 01:26:34 pmI found it pretty quickly because my monitor is not one of those ridiculously high-resolution ones and I can actually see the pixels. (You can thank Apple for that stupid "retina" trend...)Quote from: c4757p on October 23, 2015, 01:39:36 pmI think I use that font too! Is ithttps://en.wikipedia.org/wiki/Fixed_(typeface)?

author: c4757p
author_rank: Super Contributor
timestamp: October 24, 2015, 03:47:16 pm
content: Terminus.

author: Howardlong
author_rank: Super Contributor
timestamp: October 25, 2015, 09:13:26 am
content: Quote from: ale500 on October 24, 2015, 07:45:33 am.... Functionally maybe, but non-functionally, you can expect a hit on performance depending on the circumstances. As a real example (in code rather than data, but the same applies to data), if you use Thumb-2 extensions mixing 16 and 32 bit instructions, and you happen to have a string of adjacent 32 bit instructions straddling the word boundary, this will run significantly slower than the same code word aligned. The same applies to non-word aligned data accesses. Of course if all you're doing are a few byte by byte string manipulations there is no difference, unless it's been optimised to assume word alignment that is!That doesn't excuse the XC32 string word alignment assumption, that is bad.So there's no free lunch on ARM, and in the real case I describe of unexplained performance degradation, that took a couple of days to figure out, which is also the reason why any low level embedded programmer should not be afraid to routinely look at the disassembly in memory, even if they never intend to write any assembly language, or indeed understand what's been generated at anything more than a cursory level.

author: crispy_tofu
author_rank: Super Contributor
timestamp: October 25, 2015, 09:44:07 am
content: Quote from: AndyC_772 on October 23, 2015, 01:26:34 pmThat's horrible.Looks like the designers were having a nap while doing the l and I...