Adding capacitance (even a scope probe) 'fixes' my I2C lockup issues - Page 1
Forum section: electronics / projects

author: tom66
author_rank: Super Contributor
timestamp: May 05, 2023, 10:56:18 am
content: I have a board stack with a PIC32MM microcontroller and 8 peripherals attached to a common I2C bus. The system has worked in a stable manner before and no issues were noted on the I2C bus. The pull up resistors were set at 1k and the bus runs at 3.3V.With a recent board change (making one board smaller and removing one device from the bus), the bus became unstable. I found on the scope the microcontroller would only clock 8 bits out on the last packet, so no NACK bit, then the bus would hang indefinitely with the I2C master on the PIC indicating packet not complete until watchdog reset. I could not identify any obvious signal integrity issues on the scope. I attempted to use different pull up resistors, 3.3k and 6.8k, but the problem persisted. 10kohm proved too high to work at all, the bus did not meet timing at 300kHz, our nominal frequency. With either 3.3k or 6.8k pullup resistors, it would only take a few minutes of operation to break the system - the devices on the bus are polled around 10Hz so I'd estimate the error occurs at a rate of around 0.01% in every byte.I have found the problem got a little better with adding scope probes which suggested a possible relation to bus capacitance, and explained why the problem appeared when the board size decreased (fewer devices, shorter trace = less capacitance). So, I then experimented with adding some external capacitance. 100pF worked fine, and the device has been stable for hours now.The only thing I can think is somehow the capacitance is shifting the timing by a few tens nanoseconds or so somewhere and this is just about giving us enough setup/hold margin. But I'm wondering what others think?

author: Someone
author_rank: Super Contributor
timestamp: May 05, 2023, 11:10:13 am
content: Grab a low capacitance probe and look for signal integrity at both the master and at each slave, timing is unlikely to be the problem but non-monotonic edges on the CLK and overshoot can cause havoc.

author: David Hess
author_rank: Super Contributor
timestamp: May 05, 2023, 01:21:27 pm
content: If you do not have a low capacitance probe, then a fast low capacitance JFET or MOSFET can be used to buffer the signal and drive a normal probe.It seems incredible that I2C could drive the clock line fast enough to double clock something, but that is what I would look for. Instead of adding capacitance directly to the line, I would use a series RC with about 100 ohms at both ends to see if that works better, but that assumes finding the cause. Otherwise it is a blind fix.

author: AndersJ
author_rank: Frequent Contributor
timestamp: May 05, 2023, 02:18:31 pm
content: Could it be you are clocking/reading on the ”wrong” clock edge?So you read your bits at the ”wrong” moment.

author: tom66
author_rank: Super Contributor
timestamp: May 05, 2023, 02:39:23 pm
content: Thanks for the suggestions. We do have one high speed 1pF active probe which I can try to use (and ask the boss to get a second)Quote from: AndersJ on May 05, 2023, 02:18:31 pmThe I2C in this case is the hardware inside the PIC32MM, so I expect Microchip are doing it correctly, though I have seen some ugly I2C errata in the past, that would be an impressive blunder!

author: jc101
author_rank: Frequent Contributor
timestamp: May 05, 2023, 04:02:45 pm
content: There are all sorts of registers relating to I2C in the PIC32MM, some to do with the timing and slew rate of the signals. It may be it was on the edge (no pun intended) before, and the smaller board has tipped it over. Easy to tweak and see if things improve.

author: sicco
author_rank: Regular Contributor
timestamp: May 06, 2023, 09:14:00 am
content: Maybe the wires are long and the edges rise and fall rapidly. What you then get is transmission line bouncing effects, and those give extra edges that can be seen as extra clk pulses by the receiving I2C chip.Changing the location of pull up resistors may help.How long is the longest I2C wire?Star or daisy chain topology?What’s the I2C slave? Are there more that share the bus? What clock frequency? Is the slave specified to work for the power voltage used by the master and the pullups? Are the master output pins really open drain or could SCL be pushing in push-pull mode (and the slave aiming to hold down SCL to say wait i’m busy)?Is the PIC the master or the slave?Slew rate control on the I2C master output pins may help. But i don’t know if this PIC offers that already.Some I2C slaves can be programmed with extra filters on their SCL input lines.

author: Siwastaja
author_rank: Super Contributor
timestamp: May 06, 2023, 09:56:10 am
content: Quote from: tom66 on May 05, 2023, 02:39:23 pmI2C peripherals in microcontrollers, as well as I2C slave interfaces, are notoriously a total and utter bug nest. Don't know about Microchip but for example it took over 10 years for ST to come up with usable I2C peripheral. So the "impressive blunder" is to be expected - don't expect the problems to be described in any errata, though.I long time ago settled to an arrangement of always adding a P-channel MOSFET on the power pin of every I2C slave so they can be reset to get into correct state. I refuse to design in anything I2C when reliability or timing predictability matters at all, because I always have to count on the worst case of having to power-cycle the slaves and reset the I2C peripheral between every read.

author: IDEngineer
author_rank: Super Contributor
timestamp: May 08, 2023, 05:38:44 pm
content: Quote from: Siwastaja on May 06, 2023, 09:56:10 amTHANK YOU for being willing to state these facts out loud. When I tell people I favor SPI over I2C for these reasons they often scoff as if I'm simply too stupid to make something so easy work properly. "Utter bug nest" is my new favorite nickname for I2C!I too add PFET's to certain parts that have a propensity for lockups. The stories I could tell about ST MEMS parts, for example....

author: SiliconWizard
author_rank: Super Contributor
timestamp: May 08, 2023, 06:44:02 pm
content: Read errata just to make sure. Microchip tends to provide relatively thorough errata.I have used the PIC32MX but not the PIC32MM, so no particular experience with this one.Avoiding I2C altogether just because you had a bad experience is a bit extreme. I, and I'm sure probably millions of other people have used I2C on a variety of systems successfully.But I for one have had more issues with I2C devices themselves than with the I2C peripherals inside MCUs. Do not necessarily rule this out. You may be focusing on the MCU itself while one I2C slave may be the culprit. When you witnessed the bus "hang", were both SCL and SDA at a high level, or one of them held low?

author: Siwastaja
author_rank: Super Contributor
timestamp: May 09, 2023, 07:25:29 am
content: Quote from: SiliconWizard on May 08, 2023, 06:44:02 pmIn projects where reliability and timing is not critical, often in configuration/initialization, and users are expected to reboot the devices every now and then. There is a reason why SMBus, PMBus and similar I2C modifications that aim to improve reliability were invented.The biggest hurdle in many microcontrollers is the primitive nature of the I2C peripheral so that the code is nearly as complex, and nearly as inefficient as bitbanging; for example, having to serve interrupts for every byte and write some stupid "now generate ACK" registers. When babysitting a poorly implemented and documented peripheral, one is also more prone to making mistakes; I will never forget a project based on STM32F205 which required a massive FSM to handle dozens of ridiculously unnecessary interrupts per I2C transfer, and in production, a few% of devices failed to communicate, which was suspected to be a soldering issue, but it was some unexpectedly stuck status register bit which only came with certain ICs, all from same batch, go figure.I'm sure millions and millions have struggled with bullshit like this. But I'm also sure if the use case is simply while(!success) { do_init();} with no requirements for timing, and with the possibility of user rebooting the thing, then everybody's happy.

author: RoGeorge
author_rank: Super Contributor
timestamp: May 09, 2023, 08:07:33 am
content: I have had bad results with a I2C once. At first I thought the chips are faulty (they were engineering pre-samples). The chips were good, just that they were too fast, and thus more sensitive to the reflections in the bus wires.If proper impedance matching is not possible, at least try adding a small 20ohms or so in series with the bus wires, to mellow down the edges.

author: tom66
author_rank: Super Contributor
timestamp: May 09, 2023, 10:58:39 am
content: Quote from: SiliconWizard on May 08, 2023, 06:44:02 pmIn the failure case, the MCU appears to hold clock low indefinitely. However, the issue does only appear to occur when addressing a barometer device (TE MS5837-30BA). I don't believe this device has the capability to clock stretch, and the Vol appears to be at the same level the microcontroller uses.An "utter bugs nest" does describe my experience with I2C well too. Don't get me started on Zynq I2C either!

author: tom66
author_rank: Super Contributor
timestamp: May 09, 2023, 11:03:57 am
content: Quote from: sicco on May 06, 2023, 09:14:00 amTotal length of I2C traces ca. 150mmMostly daisy-chain topologyNumerous I2C slaves all running around 300kHz - all supply voltages OK - don't believe clock stretching is happeningPIC is the masterSlew rate control is enabled (no difference with it disabled)Have experimented with PIC's built in SDA timing adjustment (100ns vs 300ns) - no apparent differenceQuote from: jc101 on May 05, 2023, 04:02:45 pmThe only one I'm aware of is the 100ns/300ns SDA hold time adjustment (SDAHT) and the single bit slew-rate control. Are there others I'm missing?

author: tom66
author_rank: Super Contributor
timestamp: May 18, 2023, 01:24:43 pm
content: The problems continue, even one board is not fixed with 100pF added capacitance. Eventually, I tracked it down to the MS5837 barometer we are using. This pattern of I2C activity is seen when the barometer causes a bus lockup. The bus looks horrible! How do you get a 50% SDA / SCL level? The power supply to the barometer IC and microcontroller is stable throughout so it does not appear to be a power supply problem. I'm getting the barometer changed over next for a known-genuine one bought from Digi-Key but the original parts were bought from Farnell so I'm not suspecting a defective or counterfeit part just yet (especially as the sensor does occasionally work and produce useful and plausible baro- and temp-data.)The glitch seems to present at the moment register 0x48 is written to which is 'start pressure measurement'. The 3V3 supply is reasonably stable throughout. If I never address the barometer, everything is fine.Anyone ever seen this before... it has the office (and myself) baffled!Scope trace: red is SDA, yellow SCL and blue is 3V3 supply for sensor/MCUEdited for clarity.

author: IDEngineer
author_rank: Super Contributor
timestamp: May 18, 2023, 02:03:44 pm
content: QuoteGenerally when two outputs are driving the same line.

author: langwadt
author_rank: Super Contributor
timestamp: May 18, 2023, 02:15:08 pm
content: Quote from: IDEngineer on May 18, 2023, 02:03:44 pmwhich should never happen on I2C since it is open-drainmaybe the supply pin on the sensor has a bad connection and the sensor lives off the pull-up on the data clock lines via ESD diodes?

author: globoy
author_rank: Frequent Contributor
timestamp: May 18, 2023, 02:16:38 pm
content: I feel your pain as I've just gone through my own I2C nightmare ending in replacing one chip for another.I don't have any specific suggestions for your situation but a couple of thoughts to consider as you try to figure this out.1. Can you make the MS5837 fail by itself? For example on a break-out board connected to a dev board? This would help identify an issue with it as opposed to a signal integrity issue.2. What happens if you put it behind an I2C buffer/extender chip? No real hard theory on this one other than putting a different electrical interface on your bus to isolate the pressure sensor.3. Any chance at all that the pressure sensor GND isn't at the same potential as other grounds? I know, I know. Probably it's tied to a plane and perfect but if it did float up then it's possible the chip would get all kinds of confused by the I2C transaction and sometimes a hardware bug is actually somewhere we aren't even considering.4. Can your I2C Master driver on the PIC detect a timeout and then do something like clock the bus to try to free any stuck slaves and release SDA? I ask because in your first post you said the system hangs until WDT. Although this isn't a fix - and perhaps might not even reset the MS5837 - it might be good practice for the situation where an I2C peripheral has a random problem in a device in the field. Your system might be able to survive that (assuming your code can also deal with a hopefully very, very occasional I2C transaction failure).

author: wraper
author_rank: Unknown
timestamp: May 18, 2023, 02:19:55 pm
content: Quote from: tom66 on May 05, 2023, 02:39:23 pmWhich PIC32MM exactly? This has plenty of I2C issueshttps://ww1.microchip.com/downloads/aemDocuments/documents/MCU32/ProductDocuments/Errata/PIC32MM0256GPM064-Family-Silicon-Errata-and-Data-Sheet-Clarification-DS80000729J.pdf

author: Qmavam
author_rank: Regular Contributor
timestamp: May 22, 2023, 06:28:48 pm
content: I once got a job because of scope probe capacitance. I had applied, but the owner had just hired a guy. The had a tough dog projection TV that several had worked on without repair. He remembered my stack of service training certificates for Sony and called me back and ask if I'd look at it. I was working away and the owner suddenly said you have it working, I hadn't noticed it was lit up. I had hung my scope probe in an area where someone had put the wrong capacitor in, the scope capacitance was enough to fire it up. He gave me a job.Mikek

author: tom66
author_rank: Super Contributor
timestamp: May 22, 2023, 09:46:30 pm
content: Further investigation has revealed that the soldering on the bottom of the barometer was very likely defective. It was difficult to reflow the device without destroying it, so we just replaced it, and the problem went away. A second board showed the same issue and the same fix worked.I am certain it was a failure of soldering on the power input to the barometer. This is because the time period on one board for the I2C mid-level glitch was 7.4ms. The sampling period of the sensor in the OSR 2048x mode, which is what we use, is... 7.4ms.On one board, it remained at the mid level for some time until WDT. My guess there is that the sensor ended up stuck in some kind of POR or lockup condition once the bug happened and only the pulling of SCL/SDA low on boot as the first device is addressed reset it.Anyway, this is a manufacturing defect on our prototypes, so I'm gonna have to shout at the fab house ("trusted" UK supplier) again, unfortunately!

author: tom66
author_rank: Super Contributor
timestamp: May 22, 2023, 09:52:05 pm
content: Quote from: langwadt on May 18, 2023, 02:15:08 pmCredit to you langwadt for pointing this out. I had an inkling before that it could be supply related but nice to have someone else suggest the possibility. I owe you a beer.