I2C glitch challenge - Cannot figure out how to clean this up.  - Page 1
Forum section: electronics / general

author: rx8pilot
author_rank: Super Contributor
timestamp: September 08, 2016, 06:53:10 pm
content: Hi All,I have an existing project where the various systems are connected with I2C at 100Khz. The master is an 8bit AVR Mega164P and there are 5-7 slaves at any given moment from various manufacturers. I largely wrote the i2c (TWI) code starting a few years back and have not had any real trouble until now.The challenge is that there is a glitch during the ACK. From what I can tell, this was not a problem with all the slaves I have used so far so I did not try to fix it. I have added a new slave device from Maxim DS2484 which is an 12c to 1-wire driver. This device tends to lose control when the glitch happens during the ACK.What I think is happening is that there is a timing error when the MASTER releases the SDA so that the SLAVE can issue the ACK. It's hard to tell what is pulling what since any device on the line can have a problem. This is especially true for a non-expert in i2c like myself.In the scope screenshots the SDA and SCL are connected to the digital inputs and are decoded. The bottom half of the image is a ZOOMED in view. The Yellow trace is the analog channel looking at the SDA. In the first image, I believe the master is releasing the SDA about 2.3us before the the SLAVE is sending the ACK. In this case the SLAVE does not care and everything moves on.The next image is when the glitch is wide enough to be read, the SLAVE (in think) goes off the rails. This seems to happen about 50% of the time, so I would guess that it is right on the pulse width threshold of that particular slave device.More of a zoom to see that the SDA is fighting for control during the ACK.There does not seem to be any issues with rise time. Pull-Ups are 3k. Vcc is 5v. Traces are rather short - under 10in from end to end. The software uses the TWI (i2c) hardware in the AVR so I am not aware of any option to adjust the ACK timing so that I can reduce or eliminate the glitch. I also tried to adjust bus speed from below 100Khz to a little over 400Khz but that made no difference.Is anyone aware of an option to adjust the timing on the AVR side? As far as I can tell, the internal hardware timing is fixed - At least I have not found anything in the data sheet yet. I am hoping someone has solved this before and has a simple solution.

author: nctnico
author_rank: Super Contributor
timestamp: September 08, 2016, 07:35:14 pm
content: I've found some I2C devices are very picky about the right start/stop order and don't like to see a new start before a stop.

author: hans
author_rank: Super Contributor
timestamp: September 08, 2016, 07:41:16 pm
content: I believe sometimes generating a start and then a stop code can reset the state machines of some I2C devices.Maybe this is useful work-around if the glitch keeps happening at the same point in the transactions.

author: nctnico
author_rank: Super Contributor
timestamp: September 08, 2016, 07:50:22 pm
content: Also the saw tooth bothers me. Can it be a power supply problem? Are all I2C devices powered from 5V because I2C is not designed to allow mixing logic levels.

author: rx8pilot
author_rank: Super Contributor
timestamp: September 08, 2016, 07:51:04 pm
content: This one seems to be struggling with the ACK, before the STOP is even issued by the master.If all was by the book.....to write a byte....The master would:Send a STARTSend a 7-bit device addressSend a '0' bit to indicate a WRITE is about to happenRelease the SDA line so that the slave device can ACK by pulling it low during the ACK SCL pulseSend a DATA byteWait for ACK from slave by releasing the SDA line and letting the slave pull it low.Send a STOPThis one addresses the slave and sends the WRITE bit. The master immediately releases the SDA line but the slave has not yet pulled it down to ACK. There is about a 600ns delay from when the Master releases the line and the slave pulls it back down again. Sometimes the resulting glitch is long enough to send the bus into chaos and sometimes it is ignored.All the other slaves see a similar glitch and ignore it. This particular device is clearly more sensitive to an errant pulse. Looking for a way to change how fast the AVR master releases the SDA after the data is sent. If I can delay the SDA release by about 1us or so, this would be all fine. It is a very frustrating problem for sure.Quote from: hans on September 08, 2016, 07:41:16 pmThe glitch happens at the same place every time, immediately following the last bit of data but before the slave has ACK'd.

author: tatus1969
author_rank: Super Contributor
timestamp: September 08, 2016, 07:54:04 pm
content: when you look closely at all the rising edges you notice some up and downs. Looks like you have some source of interference there. Could be the supply, or some crosstalk. Does the I2C run through wires? You need to get rid of that, because this can cause false triggering on some devices. Many implement low pass / glitch filtering but not all. Apart from that I cannot see anything wrong. These short pulses before ACK are normal.

author: nctnico
author_rank: Super Contributor
timestamp: September 08, 2016, 07:55:33 pm
content: Perhaps using larger pull-ups can fix the problem. 3k Ohm is rather low; I usually use 10k.

author: rx8pilot
author_rank: Super Contributor
timestamp: September 08, 2016, 08:04:44 pm
content: Quote from: nctnico on September 08, 2016, 07:50:22 pmThe sawtooth is very weird, I really don't know how that is happening. I very closely checked power and it seems to be clean with no more than a few mV of noise. All 5V, however there are 2 different sources of the 5V.Quote from: tatus1969 on September 08, 2016, 07:54:04 pmYes, I took a close look at the rising edges and they all have small ups/downs on the rising edge. They are very consistent and I cannot correlate them with any other system. There are a lot of high-current lines close by, but they are not active. There is also a few DC-DC converters close by, but I would not expect them to be in perfect sync with the rising edge of the i2c. These glitches are very fast and I wold be surprised if they have any impact.

author: AndyC_772
author_rank: Super Contributor
timestamp: September 08, 2016, 08:37:48 pm
content: This doesn't look like any kind of interference to me. If it were, you'd see it superimposed continuously, even when there's real data being transferred.I2C is open drain and uses a passive pull-up. It shouldn't be possible for two devices to 'fight' for control of the bus; either:- no device drives low, in which case the bus is pulled up to VCC by the pull-up, or- at least one device drives low, in which case the bus is driven to GND.The intermediate voltage you're seeing, and the burst of oscillation, shouldn't be able to happen. The fact that it does implies that some device is actively driving high, which it should never do. Perhaps the AVR pin isn't set to open drain?: removed duff information about clock edges.I'm afraid I'm not familiar with the AVR, but its I2C driver is the first place I'd look. Can you single step through the code to correlate events on the bus with interrupts or changes of state in its I2C peripheral?Also, most microcontrollers' I2C peripherals that I've come across suck. They're a congenitally awful breed, and since the AVR is master, it may be a better idea to just program the pins as GPIO and write your own I2C driver.One of the better descriptions I've come across on how the I2C bus works, is in the data sheet for the PCA9501 I2C EEPROM/GPIO expander. Have a read of that and you'll be in a good position to spot other anomalies in your system.Even if you don't manage to fix the problem, at least you've had the chance to post some screen shots from your new toy. Don't tell me you didn't take a little pleasure from that opportunity

author: tatus1969
author_rank: Super Contributor
timestamp: September 08, 2016, 08:46:06 pm
content: some devices have transistors from output to vcc that get activated for a short moment after the low side is switched off, in order to help the pullup resistor rising the voltage. The idea is to have short rise times even in open drain / pullup configuratuon. Maybe that is counter productive here and can be disabled. Check for things like pullup boost or alike.

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: September 08, 2016, 08:56:01 pm
content: Since you said this started happening after you added the Maxim DS2484 to the system, can you remove all other I2C devices from the bus and see if this still occurs? Then you'll know if the issue is with the new Maxim device.It's pretty obvious that something on the bus isdrivingSDA high, which is something that should never happen on an I2C bus. If it's the Maxim part, talk to your Maxim FAE and see if they have any advice.Can you try another (non-AVR) I2C master for comparison purposes?

author: AndyC_772
author_rank: Super Contributor
timestamp: September 08, 2016, 09:13:36 pm
content: I think the glitch in your first post is a non-issue.I2C data (SDA) is allowed to change, arbitrarily, when SCL is low. The glitch is simply due to the propagation delay between the master relinquishing control of the bus on the falling edge (which it generates), and the slave device detecting the falling edge and pulling SDA low.Since I2C devices sample on the rising edge of SCL, no device should see this glitch at all.I guess it's possible that some device sees the change of SDA while SCL is (locally) still high, and misinterprets this as a start condition? That still doesn't explain some device apparently driving SDA high actively.

author: rx8pilot
author_rank: Super Contributor
timestamp: September 08, 2016, 09:17:13 pm
content: Quote from: AndyC_772 on September 08, 2016, 08:37:48 pmIt could be a single device that is pulsing the SDA very fast which would result in a sawtooth. It is hard to sort out what device is contributing though. The system design will make it difficult to completely isolate the other slaves, but it only happens when this Maxim is addressed.Quote from: AndyC_772 on September 08, 2016, 08:37:48 pmThe AVR is using the internal TWI (I2C) hardware that takes control of the pins and the programmer cannot do much (that I am aware of). If the slave device, for whatever reason, was pulsing the SDA into a low state very fast and just fighting the pull-up resistor - it may result in the sawtooth without anything actively pulling it up. Why the DS2484 would ever behave that way is beyond me, so maybe the source is something else. Just to eliminate variables, I could swap the DS2484 with a new one in case this one was damaged during soldering or some other issue.Quote from: AndyC_772 on September 08, 2016, 08:37:48 pmI can only step through so much, the I2C hardware automatically deals with the ACK after sending a byte of data. There are some outputs of the AVR connected to the scope that I am pulsing to correlate the code with the results. It is not too helpful though since I cannot control the timing of the ACK.Quote from: AndyC_772 on September 08, 2016, 08:37:48 pmAbsolutely! A proud moment indeed. It is very easy to control and grab frames from my programming workstation too. Aside from the excitement of winning the scope, it has been remarkably useful in real life challenges. It is very intuitive and easy to find difficult details.Quote from: Sal Ammoniac on September 08, 2016, 08:56:01 pmI may try to rig a simpler system to further isolate the problem. The only other I2C master I have is a Bus Pirate which I think is a PIC of some sort. That could help although I don't know how to use it very well - tried it once about 2 years ago and have not used it since.

author: rx8pilot
author_rank: Super Contributor
timestamp: September 08, 2016, 09:25:54 pm
content: Quote from: AndyC_772 on September 08, 2016, 09:13:36 pmThis is what has me confused - all the other slaves ignore the glitch and there is not an issue. The DS2484 seems to have the issue only half the time. The 1st screenshot is in fact the DS2484 being addressed with no problem. Maybe I should probe the pins directly on the DS2484 to see it there is anything happening locally - currently probing close to the master. It is behind an I2C buffer that facilitates hot-plugging to the bus. There are a few other I2C slaves operating normally on this side of the buffer, so I would be surprised to find a problem there. At this point, I am looking at anything at all. This is day two of the investigation and it is getting old. Hoping to turn a corner soon.

author: nfmax
author_rank: Super Contributor
timestamp: September 08, 2016, 09:28:19 pm
content: It looks to me as if the slave might not be pulling down on SDA as 'hard' as the master. If you look at the first screenshot, you can see the SDA line drop in voltage slightly after the falling edge of the clock following the ACK. Then in the second screenshot the pulldown is so weak the master logic is getting confused and going into some sort of metastable oscillation. This might be because of a bad connection to the SDA pin on the slave (dry joint?) or maybe ESD damage - probably not helped by the relatively low value of pull-up resistance, but one that ought to work. The steps on the rising edges of SDA seem to be at the same voltages on every pulse, regardless of which device has been driving the line low, so they probably aren't directly related. They don't correspond exactly to the sawtooth voltages.I doubt anything is actually driving the SDA line high.Try replacing the DS2484 - it may be bad

author: rstofer
author_rank: Super Contributor
timestamp: September 08, 2016, 09:59:59 pm
content: I assume all devices on the bus are 5V logic, right?Your pull-up resistors could be as low as 2.2k before they can handle an extensive bus with distributed capacitance. There's actually a formula a graph on page 42:http://www.cs.unc.edu/~stc/FAQs/Interfaces/I2C-BusSpec-V2.1.pdfThat glitch when the master releases the bus and the slave hasn't decided to handle it is expected behavior. The only thing that matter is the value of SDA when SCL transitions.The long half-voltage thing isn't normal. Maybe smaller pull-up resistors, maybe something else.So, unhook everything except the Maxim device and see how it looks.

author: AndyC_772
author_rank: Super Contributor
timestamp: September 08, 2016, 10:10:21 pm
content: Some devices do have oddly large capacitance. Might be worth changing your pull-up resistors if you haven't already? I normally use 2k on I2C as a matter of course; I've had way more issues with pull-ups being too weak than too strong.Perhaps the I2C buffer you mention is a clue. What part is it? How does it determine which direction to drive in at any given moment in time? Does it require separate pull-ups on each bus segment? How does the signal integrity compare between the two segments? Could *it* be the thing that's getting confused by the change in bus direction at the ACK time?

author: nctnico
author_rank: Super Contributor
timestamp: September 08, 2016, 10:19:08 pm
content: Quote from: rx8pilot on September 08, 2016, 09:25:54 pmAha! I'd remove that chip first and see how it works then. These chips have to guestimate which direction the data should flow and the saw tooth could indicate the buffer can't decide what to do.

author: rx8pilot
author_rank: Super Contributor
timestamp: September 08, 2016, 10:23:11 pm
content: Quote from: AndyC_772 on September 08, 2016, 10:10:21 pmI think you are right - this is the place to look. I had been probing on the master side of the bus, until now. When I looked at both sides of the buffer - problem on the master side while totally clean on the other side. Duh, the problem is between those two points which is a TI TCA4311A. I can stop looking at the code, AVR, and the DS2484 for a minute and focus on this element.http://www.ti.com/lit/ds/symlink/tca4311a.pdfThere does not seem to be any issues with other parts on the island bus, so I am looking for something that would trigger an oscillation. It does require it's own pull ups for the isolated i2c bus and I had those at 3k. I swapped for 7.5k and see no difference. Next, I will try to add some capacitance to the bus to see if that will smooth out high-frequency in case that is the trigger. Also, reading the data sheet more closely to see if I find any clues.This is progress! Thank you all for the help. I can feel a solution coming this way.

author: rx8pilot
author_rank: Super Contributor
timestamp: September 08, 2016, 10:24:22 pm
content: Quote from: nctnico on September 08, 2016, 10:19:08 pmThe microscope now knows where to focus. This has had me running is circles.

author: nctnico
author_rank: Super Contributor
timestamp: September 08, 2016, 10:29:48 pm
content: The datasheet seems to imply the chip has been designed around 10k pull-up resistors.

author: AndyC_772
author_rank: Super Contributor
timestamp: September 08, 2016, 10:33:59 pm
content: Right. It's an interaction between the buffer IC and the bus capacitance. A lower pull-up value may well sort it.An I2C buffer has to determine which device has driven each wire on the bus low at any given moment.Say device 'A' drives low. The buffer must drive device 'B' low, but also remember the direction of communication. When 'A' releases the bus and allows it to go high again, the buffer releases 'B' and vice versa. During normal data communication, this is fine.The problem comes with ACK, because the bus goes straight from being driven low by the master, to being driven low by the slave.In this case, what happens is:- master drives low- buffer drives slave low- master stops driving low- positive glitch inevitably appears on master side, because the buffer is driving master>slave and not slave>master- buffer stops driving slave low- buffer determines that slave side is still low, so starts driving master low- when slave relinquishes SDA, the buffer sees the slave side go high, and it relinquishes the master side.This is all OK, provided you don't mind the glitch (it's harmless!).But, add in some bus capacitance and it all goes horribly wrong.Suppose the buffer is passing a logic low from A to B.A starts to go high, so the buffer stops driving B low. But because bus segment B has capacitance, the buffer sees B still low, and starts driving A low again, on the basis that B must now be 'driving' the bus.B gradually floats high, and the buffer stops driving A low. But A has capacitance too, so the buffer swaps direction again, and the cycle repeats. You have an oscillator.Stronger pull-ups. You owe me one beer

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: September 08, 2016, 10:45:24 pm
content: Post your question tohttps://e2e.ti.com/support/interface/i2c/f. TI people frequently respond to posts there and may have some insight for you.

author: nuno
author_rank: Frequent Contributor
timestamp: September 08, 2016, 11:07:05 pm
content: Quote from: rx8pilot on September 08, 2016, 06:53:10 pmMan, you're BRAVE!!

author: rx8pilot
author_rank: Super Contributor
timestamp: September 08, 2016, 11:34:40 pm
content: Quote from: nuno on September 08, 2016, 11:07:05 pmIt doesn't seem so bad does it?

author: nuno
author_rank: Frequent Contributor
timestamp: September 09, 2016, 12:06:45 am
content: Quote from: rx8pilot on September 08, 2016, 11:34:40 pmI've had so many problems with I2C stuff in projects I've worked on (both professionally and hobby) or know of, even with just 1 or 2 devices, that I avoid I2C like the plague. Silicon bugs, software bugs, "routing" bugs, you name it. Last time was a bug in RaspberryPI SoC's implementation of clock stretching; fortunately I had I2C bus isolators from each slave to the bus, which really helped me debug the issue, by allowing me to see who was pulling a line low. The worst problems are those that happen sporadically, maybe once every few days; those are tough...

author: rx8pilot
author_rank: Super Contributor
timestamp: September 09, 2016, 02:39:45 am
content: Quote from: AndyC_772 on September 08, 2016, 10:33:59 pmNo dice, unfortunately. I tried a huge range of pull-ups on both sides of the buffer and there does not appear to be any change yet. I also tried a brand new set of PCB's where all the parts are from different batches made about a year apart just to eliminate any unlucky-ness.Next - add capacitance.After that, start a thread on TI board as @Sal Ammoniac suggested. I have found some help there on other projects from both TI employees and the participating public. Hope someone there can spot something they have seen and conquered before.Not sure what to do if those things don't work. I am running out of ideas quickly.

author: rx8pilot
author_rank: Super Contributor
timestamp: September 09, 2016, 03:17:01 am
content: DONE!Adding two 180pf caps on the SDA line slowed it down and eliminated the oscillation. If I had to guess, any high frequency content can confuse the buffers ability to determine direction. The caps filter the high frequency enough to eliminate ambiguity while staying within the required rise/fall time for i2c. The caps are on the master side of the buffer chip and fairly close to its pins.I did not see this problem coming and it took a ton of time since my first guess was that it was my software. It did not occur to me that a lack of capacitance would be a problem. I even put specific efforts into reducing capacitance on i2c. Nuts. I thought that taking the pull-ups all the way to 20k would have a similar effect.Thanks everyone for all the suggestions - it would have been a while longer before I looked at the buffer.

author: tatus1969
author_rank: Super Contributor
timestamp: September 09, 2016, 06:40:45 am
content: Quote from: rx8pilot on September 08, 2016, 10:23:11 pmSecond that. My next question would have had been exactly that. Too slowAka DS2484, used that on several occasions, it never made any trouble. On the other hand, I2C bus multiplexers / hubs / buffers / level shifters are very dangerous to use. They need to implement some method to sense if they themselves drive to bus LOW, or any other device. Most of them use some voltage level approach that is theoretically within the I2C spec, but not working all the time. And ground offset kills communication very quickly as well. Try replacing that buffer with shorts and I'll take bets that this inconsistency during the transition, and also the oscillation will disappear.As you appear to have more than one segment, have you checked rise times on each of them? Maybe you have a bus load problem as well? Although I don't think so, 2k2 pullups at 100kHz will allow maaaany slaves.

author: tatus1969
author_rank: Super Contributor
timestamp: September 09, 2016, 06:43:03 am
content: Quote from: rx8pilot on September 09, 2016, 02:39:45 amThe rise times in your pictures look perfect, there is much headroom (on this segment of your bus). No worries there.

author: tatus1969
author_rank: Super Contributor
timestamp: September 09, 2016, 06:47:30 am
content: Quote from: rx8pilot on September 09, 2016, 03:17:01 amNaaah, my "engineering intuition" is ringing bells here. These capacitors only mask your real problem, which will eventually start haunt you as soon as you start mass producing your design. Even a different room temperature might be enough to have it pop up again.

author: rx8pilot
author_rank: Super Contributor
timestamp: September 09, 2016, 07:08:58 am
content: I will admit that I don't really understand what the real problem is. Over the next few days, I will be beating the design as much as possible. In general, I am ok with a patch as long as the patch is tough.Sent from my horrible mobile....

author: hans
author_rank: Super Contributor
timestamp: September 09, 2016, 08:52:12 am
content: From memory, those bidirectional I/O chips that 'sense' the direction are horrid. Something like TXB0108.These chips have minimum pull-up/down resistor values, and maximum capacitance values. Don't meet them, and they go berserk. I've seen them being used on a prototype RF OEM module for interfacing to customers boards, and after a talk with the application engineers they decided on replacing them because they were a PITA.As for an I2C buffer, paragraph 10.2.1.1.2 immediately draw my attention. This may be a problem if your design includes a variety of boards with different cabling and levels of capacitance. You'd to almost cherry pick boards for this to work reliably.It also gives some hints that 10k pull-up is on the holy grail; but rather the 1.25V/us rise time should be maintained. I suppose that is the rise-time becomes slower than that, the chip becomes confused as to what direction the I/O is flowing. Contray to what being said here; 3k pull-ups doesn't sound too low for me. I've seen 2.2k being recommended in many datasheets before. At 5V this is a mere 2.27mA of sink current.At 10.2.1.2.1 (honestly who thinks of these document structures?) also talks about "boost pull-ups".So.. maybe pick a different I2C buffer is the best engineering solution?

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: September 09, 2016, 02:41:59 pm
content: When I have an I2C bus that's critical to device operation, I use active pull-ups in the form of a two transistor current mirror rather than a simple passive resistor pull up. This results in faster rise times on the I2C signals and allows for longer signal runs.As an aside, don't believe people who say that I2C is only for short, on-board connections. HDMI/DVI/VGA use a protocol called DDC2 to communicate monitor parameters, brightness/contrast, etc., between the host and the monitor. DDC2 uses I2C as its physical layer and cables can be many meters long. The HDMI cable running from my receiver to my projector in my home theater is 14 meters long and I've never seen a problem with the DDC2 communications on this cable.

author: Electro Fan
author_rank: Super Contributor
timestamp: September 09, 2016, 03:27:15 pm
content: You guys are so far ahead of me it's not a matter of how far ahead you are but how many laps you are ahead, so this could be way off.... I have found with very limited I2C experience using a scope LA that sometimes the clock edges don't consistently align with the data edges. This might be due to the DUT, or it might be due to something in how the scope and/or it's LA processes and displays signals. (My guess is that a Keysight 6000 series scope has been well designed, manufactured, and tested so there is probably no issue with the scope and it's LA - but sometimes some combinations of circuit speed, sampling rate, or other signal processing could potentially throw off a project's analysis - I'm pretty sure I have seen this (possibly due to a scope/LA firmware issue) with another brand.) Either way, whether there is a question about the test equipment or the DUT, when things don't seem to perform as expected I zoom and sometimes use cursors to examine edge alignments.Showing how many laps behind I am, I'm not sure how precise (and consistently precise) the clock edge and data edge alignments need to be (I'm not sure what an acceptable deviation range is) but it seems that if the clock and the data don't retain sufficient alignment that something foundational could be out of alignment, either to the point of throwing off measurements, or possibly inhibiting proper performance of a circuit and possibly the logic riding on the circuit.The thinking here is that beyond looking at the shape of signals I think there might be merit in first confirming sufficient timing precision - on the theory that if something gets confused by a timing transition the result could be just that - confusion in whether a signal should be rising or descending - which could lead to (or be represented by) some small amount of oscillation or ringing that could absorb enough time to throw something else out of sync. Sometimes, under most conditions most of the edges align but then after looking further and more closely through the full dataset sometimes a relatively few mis-alignments emerge into view and from there a pattern can possibly be identified and further analyzed.So one of the first things to rule out across the full dataset is any issue with respect to clock and data edge alignments. In addition to using cursors to examine clock and data alignments it's sometimes helpful to rearrange the displayed information to put the clock and data waveforms just above or below or directly on top of one another.If I'm wrong about this or it's too elementary and just plain stupid, sorry

author: AndyC_772
author_rank: Super Contributor
timestamp: September 09, 2016, 04:19:15 pm
content: If you're interested in I2C, have a read of the PCA9501 data sheet. It contains a very detailed description of the interface.Basically, SDA is allowed to change arbitrarily while SCL is low, but must be stable by the time SCL goes high, and must remain stable all the time the SCL is high.There's no requirement for accurate alignment between the two.Whichever device is transmitting (master or slave) changes the level on SDA after SCL goes low. There's a cause-and-effect relationship between the two; SDA can't change until after SCL has gone low, and should never change when SCL goes high.One problem with I2C is that, because it's a bus with multiple devices on it, it can be quite capacitive. Also, because it has passive pull-ups, the rate (Volts/sec) of rising edges can be quite slow.This in turn means each device can potentially see multiple transitions on SCL, if the signal is noisy whilst it's hovering around the transition point from 0 to 1.The fix for this is usually to use a stronger pull-up, to increase the dV/dt on rising edges. Alternatively, a small capacitor placed near the input of a susceptible device can help smooth the noise and allow a 'clean' transition, though adding capacitance does technically make the bus loading problem worse, not better.Falling edges on SCL are generally less of a problem, since they're actively driven by transistors, and therefore are faster.

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: September 09, 2016, 04:33:48 pm
content: Quote from: AndyC_772 on September 09, 2016, 04:19:15 pmActive pull-ups can also be used for faster rise times. I use a current mirror implemented with a dual transistor array for this.

author: rx8pilot
author_rank: Super Contributor
timestamp: September 09, 2016, 08:27:27 pm
content: Do you do the active pull-up to cope with long wires or a ton of slaves on the bus? My system is physically rather small and the total length of the wires is very short. The rise-times with passive pull-ups seem very safe in that setup. I am always looking for ways to improve the robustness of the i2c bus. There is a ton of error correction and fault recovery in my software but if I can eliminate circuit level faults, that is better.

author: rx8pilot
author_rank: Super Contributor
timestamp: September 09, 2016, 08:40:03 pm
content: Quote from: hans on September 09, 2016, 08:52:12 amIt is hard to tell good from evil without a pile of testing. The TI data sheet offers some guidelines and useful data but it is hard/impossible to tell where and when it's limitations will cause a problem. Testing a few would be a time consuming endeavor, wishing it was more forgiving. It also works with no issues with the other two slaves on the buffered side (hundreds of PCB's and no issues). It is still a mystery why the DS2484 triggered a problem, but only with itself and only on the un-buffered master side. The other slaves - after addition of DS2484 - still had no issues. I looked closely at the i2c from a signal integrity and timing perspective and could not see anything different relative the other slaves in the system. It would be nice to understand this enough to make an educated decision.

author: hans
author_rank: Super Contributor
timestamp: September 09, 2016, 08:47:01 pm
content: Hm, have you tried taking the I2C buffer out of circuit and bridging it directly? As I understand it's only there for hotplugging and not some kind of level translation and/or performance related spec.If the glitches and oscillations stop, you probably know the buffer plays a crucial role in this erratic behavior.Indeed, issues like these on bi-directional I/O can be hard to track down. Any of the 8 chips can cause trouble at this point, so at this point I would first start tracking down the issue along the common denominator.

author: rx8pilot
author_rank: Super Contributor
timestamp: September 09, 2016, 09:08:49 pm
content: The original reason to add the buffer was to deal with possibility that the card could be disabled (powered down) during normal operation. When that happened the uC i2c pins sink current and pull the I2C low. All communication stops at that point. The buffer is High-Z when power off which is good. The second reason was that it is good to be able to swap this card for another with the system running. The buffer coordinates this without too much turmoil on the bus. It looked like all was just fine and I stopped thinking about this part of the circuit long ago since it seemed to be doing the job - until now.If there is another SOIC-8 pin compatible option, it would be easy to try. A requirement to change the PCB would be a pain.

author: Electro Fan
author_rank: Super Contributor
timestamp: September 09, 2016, 09:09:37 pm
content: Quote from: AndyC_772 on September 09, 2016, 04:19:15 pmAndy, thanks for this - it's very enlightening (I think).Just to confirm: Any time after SCL goes low (but before it goes high again) SDA can change, yes?Therefore #1 (a statement): If the clock and data aren't aligned it's ok, but we would expect the clock to be ahead of the data, of course.and #2 (a question), Can data change more than once while the clock remains low? I'm guessing yes.Assuming I got this correct, this must imply that the transmitters listen for when the SCL is low (but not yet high) to determine the window during which they can transmit, yes?And further, any "alignments" observed between SCL and SDA are strictly the result of SDA's reacting very quickly to the "all clear to transmit" timing of the SCL going low, yes?And still further, if the SDA isn't stable when the SCL goes high, then a NAK will be issued and the data is retransmitted?Thanks, EF

author: nctnico
author_rank: Super Contributor
timestamp: September 09, 2016, 09:21:15 pm
content: Quote from: Electro Fan on September 09, 2016, 09:09:37 pmNo, I2C doesn't do retransmit. The state of SDA is sampled at the rising edge of SCL during data transfer. Start and Stop events are exceptions to the rule that SDA may not change when SDL is high to create is out-of-band signalling.

author: Electro Fan
author_rank: Super Contributor
timestamp: September 09, 2016, 09:56:42 pm
content: Quote from: nctnico on September 09, 2016, 09:21:15 pmWhat does this refer to with respect to ACK and NAK? Thxhttps://learn.sparkfun.com/tutorials/i2cThe address frame is always first in any new communication sequence. For a 7-bit address, the address is clocked out most significant bit (MSB) first, followed by a R/W bit indicating whether this is a read (1) or write (0) operation.The 9th bit of the frame is the NACK/ACK bit. This is the case for all frames (data or address). Once the first 8 bits of the frame are sent, the receiving device is given control over SDA. If the receiving device does not pull the SDA line low before the 9th clock pulse, it can be inferred that the receiving device either did not receive the data or did not know how to parse the message. In that case, the exchange halts, and it’s up to the master of the system to decide how to proceed.

author: nctnico
author_rank: Super Contributor
timestamp: September 09, 2016, 10:18:46 pm
content: You better go to nxp.com for good I2C information. After all they invented it.

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: September 09, 2016, 10:26:35 pm
content: Quote from: rx8pilot on September 09, 2016, 08:27:27 pmI use active pull-ups to deal with long wires. I use sensors with I2C interfaces that are typically at the end of a few feet of wire from the micro. This configuration has been very robust for me with very few errors on the I2C bus (typically one every few weeks on sensors that are read several times a second).

author: Electro Fan
author_rank: Super Contributor
timestamp: September 09, 2016, 11:29:55 pm
content: Quote from: nctnico on September 09, 2016, 10:18:46 pmhttp://www.nxp.com/documents/user_manual/UM10204.pdf3.1.6 Acknowledge (ACK) and Not Acknowledge (NACK)The acknowledge takes place after every byte. The acknowledge bit allows the receiver tosignal the transmitter that the byte was successfully received and another byte may besent. The master generates all clock pulses, including the acknowledge ninth clock pulse.The Acknowledge signal is defined as follows: the transmitter releases the SDA lineduring the acknowledge clock pulse so the receiver can pull the SDA line LOW and itremains stable LOW during the HIGH period of this clock pulse (see Figure 4). Set-up andhold times (specified in Section 6) must also be taken into account.When SDA remains HIGH during this ninth clock pulse, this is defined as the NotAcknowledge signal.The master can then generate either a STOP condition to abort thetransfer,or a repeated START condition to start a new transfer. There are five conditionsthat lead to the generation of a NACK:1. No receiver is present on the bus with the transmitted address so there is no device torespond with an acknowledge.2. The receiver is unable to receive or transmit because it is performing some real-timefunction and is not ready to start communication with the master.3. During the transfer, the receiver gets data or commands that it does not understand.4. During the transfer, the receiver cannot receive any more data bytes.5. A master-receiver must signal the end of the transfer to the slave transmitter.

author: jesuscf
author_rank: Frequent Contributor
timestamp: September 10, 2016, 03:22:43 am
content: Quote from: Sal Ammoniac on September 09, 2016, 02:41:59 pmThe Nunchuk and the Wii Remote communicate using I2C. That is about a meter I think. In cases like this, should the pull-up(s) resistor for SDA and SCK be placed at just one end of the cable or both? Perhaps OP can tell us how long is his bus and where are the pull-up resistors placed.Would it make sense to have more than one pull-up resistor? For example, instead of having a one pull-up of 3.3k at one end, what about having two pull-ups of 10k at both ends of the bus and one 10k in the middle of the bus.

author: rx8pilot
author_rank: Super Contributor
timestamp: September 10, 2016, 05:56:37 am
content: The total length of the bus in my system is under 12in or so. The 3k resistors are placed right at the pins of the master. There are a total of 3 PCBs with i2c that use connectors and the main purpose of the circuit is to manage power. There are 2 high-ish current SMPS's, a bunch of routing FETs, and 3 low current SMPS's. It is a rather hostile environment when it is all going full speed ahead. Over a number of iterations, I have worked out most of the noise issues and have a device that stands a chance of passing emissions tests - but it is not yet at that stage.Hoping that I can switch from 100Khz to 400Khz with no issues. I need more speed, but cannot tolerate very many errors.

author: AndyC_772
author_rank: Super Contributor
timestamp: September 10, 2016, 06:44:30 am
content: Quote from: Electro Fan on September 09, 2016, 09:09:37 pmYes. Cause and effect, time's arrow and all that.QuoteYes again. This is why the positive glitch in between the last bit of data and the ACK isn't a problem.QuoteCorrect. Different devices work in different ways here; some use SCL directly as a clock, while others treat SCL as an asynchronous signal and poll it regularly (based on some other, faster clock source).QuoteYesQuoteNo. It's the responsibility of the receiving device to make sure SDA is stable by the time SCL goes high. The receiver will sample the state of SDA at, or perhaps just after, the rising edge on SCL. If the value isn't yet stable, then there's a chance it'll read a wrong value, and then the system has an uncorrected bit error. There's no in-built way to detect that this has happened, and the result will be a malfunction of some kind. I2C isn't like, say, a CAN bus, where the hardware includes CRC checking, collision detection and automatic retransmission.

author: tatus1969
author_rank: Super Contributor
timestamp: September 10, 2016, 07:46:07 pm
content: Quote from: rx8pilot on September 10, 2016, 05:56:37 amJust browsed over that I2C buffer datasheet, looking for information about the mechanism they use to detect/control signal flow. An active I2C hub always has the following problem:1) initial condition: SDA high on both sides2) a device pulls SDA low on left side3) the buffer will activate its transistor on right side and pulls down SDA there4) SDA is pulled down by another device on right side5) both external devices release SDANow the hub would still be in a condition that it drives the right side low. How should it distinguish whether this is his own transistor, or something connected externally? Without any additional tricks, it would run into a deadlock.There are different strategies to resolve that. I had used some NXP buffers/multiplexers (e.g. P82B96) that would pull down SDA to a voltage of 0.8V, but not lower. if another device wants to pull down that SDA, ithasto drive it below 0.6V. If the buffer detects this lower voltage, it knows that something outside is pulling down the line, and it can switch off its transistor and change direction.The device that you are using does not specify clearly how it solves that conflict. But they say in the datasheet that, if it detects a LOW on the one side, it will drive the other side to that same voltage level, plus some 100mV margin. If some external device "helps" pulling the latter low, it will hopefully detect this from the now smaller voltage.All this is extremely fragile, they work with very small voltage margins here. And just imagine that you have a device using aBJT instead of a MOSFET. If that has lets say 0.5V saturation voltage, then both strategies above may not recognize it pulling the line low.All the above is also valid for SCL, and also for the other direction. These buffers do not distinguish between SCL, SDA, left, right regarding this problem.If your priority is the ability to disable sections of your I2C bus, then you may consider using ordinary analog switches. Of course you would lose hickup-free hot swapping then.

author: nuno
author_rank: Frequent Contributor
timestamp: September 10, 2016, 09:57:47 pm
content: Quote from: rx8pilot on September 08, 2016, 08:04:44 pmMaybe an issue with clock stretching? The slave trying to stretch the ack and the master trying to ignore it? Although I think it's control over SCL that allow the stretch, if I remember.

author: dmills
author_rank: Super Contributor
timestamp: September 10, 2016, 10:44:39 pm
content: I2C has given me so much random shit over the years, that these days I just don't use it (Or at least use it VERY reluctantly).SPI, sure it is a couple of extra pins, but it has NO ANNOYING STATE MACHINES WITHOUT RESET INPUTS to tie you up in knots when debugging and it runs at a reasonable speed....Those I2C bus buffers are Trouble, used them before, took them out, they have an impossible job, and do it as well as you would expect.Think I would rather deal with CAN then I2C, and CAN has pathologies all of its own.73 Dan.

author: rx8pilot
author_rank: Super Contributor
timestamp: September 10, 2016, 11:14:21 pm
content: Unfortunately - changing protocols is a very heavy handed fix for this. I would have to change all the slaves to parts that can do SPI, the software would change significantly, the PCB's would change. The new and improved configuration would no longer be firmware compatible with the older i2c based versions. I don't need the speed since 100Khz is generally enough - although I may change to 400Khz to have a ton of headroom.I2C made it into this design about 2 years ago in large part because most of the peripherals were i2c only. I feel like I will be fine if I focus the root causes of the issue and avoiding it as best I can. Periodic errors are not difficult to deal with and I have added a few extra checks in the i2c software to avoid complete communications loss. Dealing with the analog components like rise time and noise effectively reduce bus errors that I can manage in software. Some of that has been trial and error since the data sheets lack enough detail to understand what they are looking for. Since adding the capacitors to the bus, it has not gone into oscillation once - monitoring overnight. That is not an exhaustive test on its own, but clearly a giant improvement. We will see how it does when the system is hot and loaded down which creates EMI.

author: Electro Fan
author_rank: Super Contributor
timestamp: September 10, 2016, 11:48:58 pm
content: Quote from: AndyC_772 on September 10, 2016, 06:44:30 amThanksSo if I2C doesn't have much in the way of built in error detection and correction this means that a) the typical I2C environment of engineering (vs. end user) designed and tested IC to IC communications isn't expected to encounter errors from poor implementation or extraneous noise or other problems that are more likely to be encountered for example in Wide Area Network communications, and b) to the extent that highly reliable (mission critical) communications are required I2C wouldn't be the communications protocol of choice?

author: rx8pilot
author_rank: Super Contributor
timestamp: September 11, 2016, 12:57:40 am
content: Quote from: Electro Fan on September 10, 2016, 11:48:58 pmThe end user does get some feedback about the quality of the transmission, but it is not any kind of automatic or built-in correction or retry scheme. I have built me own error detection system in software that allows me to check critical data and just transmit things that don't matter as much. In general I have various levels that I can choose to help ensure it all works ok. First, I have bandwidth to check/re-try. It nearly saturates the bus with checks, but errors never get to the receiving end even though the bus does have errors here and there.The protocol has limitations, but overall it is easy and safe for low speed passing of data like I have. It does take some effort to make it work. For example, the error that started this thread had nearly no impact on the system function as a whole. The one part that had error would not work, but everything else was fine.

author: AndyC_772
author_rank: Super Contributor
timestamp: September 11, 2016, 08:01:27 am
content: Personally I'd be looking to get to the bottom of why errors exist on the bus at all, rather than worrying too much about what to do about them if they do exist.Even flat out, I2C is only running at 400 kHz. The memory bus in your PC is over 1000 times faster than this, with much smaller voltage margins, and it runs error free.I generally work on the basis that communications between parts that are within the controlled, repeatable environment of a single PCB will be reliable. If they're not, the design needs fixing in hardware, rather than having the errors masked by firmware.

author: nctnico
author_rank: Super Contributor
timestamp: September 11, 2016, 09:49:46 am
content: It also depends on the quality of the chips you use. In one of the designs I inherited an I2C bus goes off-board but there is also an Atmel controller with a bit-banged I2C interface. To make things worse the Atmel's inputs have a threshold of 0.7V so whatever glitch gets onto the wiring makes it loose the I2C bus. All the other (real) I2C devices are fine even during EMC immunity tests because a real I2C interface has a large tolerance for noise and filtering for narrow pulses. Your design has to be really bad for I2C not to work reliably because it has been designed specifically for use in TVs and other consumer electronics using single layer boards and wire jumpers.

author: rx8pilot
author_rank: Super Contributor
timestamp: September 11, 2016, 07:13:23 pm
content: Quote from: AndyC_772 on September 11, 2016, 08:01:27 amI totally agree and try my best to hunt down any circuit level errors. In reality, there are bigger fish to fry in my world of being a one-man-band. As long as those errors have no chance of taking the circuit down, I am forced to deal with them at a lower priority. A memory bus does not get integrated into a high-current power supply and has tons of layers to isolate delicate signals. One of the challenges in this project, is that the i2c lines run in close proximity to very difficult areas were big power MOSFETS are switching with fast di/dt, going through low-cost connections and I am sure some other design infractions. The PCB layout priority was first to current paths, followed by the delicate analog control signals, followed by everything else. I doubt you have to look to close at the i2c routing to find places that could be messy. The firmware error handling offers me easy routing options. Fewer layers, less layout time, etc. If I started over from scratch, it would most likely be improved, but that time is not available. For now, the software 'patch' method seems reasonable. Coupled glitches will all be very fast events and a couple of re-transmissions can easily avoid problems if they actually occur.This buffer issue is a high priority though. It has my nerves rattled for sure even though it is working right now.Quote from: nctnico on September 11, 2016, 09:49:46 amFortunately, I am using the I2C (Atmel calls it TWI - two-wire interface) hardware that has glitch filtering. One of the hard things to do is determine if an error detected on the logic analyzer is an error the DUT also sees. The Keysight LA seems to be more sensitive to noise than the DUT. I have not adjusted any of the default thresholds and I have no idea how it deals with noise, if at all. I would have to write some dedicated code to even see if the errors I see in the LA are even real errors at all inside the DUT.

author: nctnico
author_rank: Super Contributor
timestamp: September 11, 2016, 07:25:30 pm
content: I recall having some problems with decoding I2C using my Agilent DSO7k series. If an edge rises too slow the digital inputs bounce up&down an extra time and the clocking is lost. Very frustrating because changing the thresholds doesn't help!

author: rx8pilot
author_rank: Super Contributor
timestamp: September 11, 2016, 07:34:11 pm
content: I also have a Saleae to compare. Maybe on the scope, it would be quick to trigger on a bus error and look at the analog input so see if the problem is real or not. I did not have that option before getting the Keysight MSO where it was a PITA to time correlate the Saleae with the analog scope.

author: rx8pilot
author_rank: Super Contributor
timestamp: September 11, 2016, 10:00:16 pm
content: I just did a quick and dirty test to see if the LA is offering believable decoding. To do that, I manually decoded a packet marked as an error by looking at the analog SDA and SCL - it is all fine. I don't know why the LA does not like it quite yet. The bus may be cleaner than I thought.

author: exmadscientist
author_rank: Frequent Contributor
timestamp: September 11, 2016, 10:58:40 pm
content: Quote from: rx8pilot on September 11, 2016, 07:13:23 pmFor what it's worth, the default thresholds on our 4000X MSO are kind of weird. The bus decoder on the analog channels uses the same threshold as that channel's trigger... which is 0.0V by default. This is not very useful. I usually set it to something around 1.8V (for 3.3V systems). The digital channels see a lot less use (usually the 4 analog channels are enough) but I recall their thresholds, adjusted through the digital settings, are also not great. This may well be your problem.On the subject of I2C issues, I have found a lot of trouble goes away when I stick to NXP parts. Unsurprisingly, no one else seems to do I2C as well as they do. LTC also has a nice set of buffers with built-in active pullups ("rise-time accelerators", they call them) and stuck bus recovery that look very interesting but that I haven't tried yet. Given the usual high quality of LTC products (and corresponding high prices) I would expect good things. Unfortunately none of the LTC parts will drop in for your TI part unless you were lucky enough to use the VSSOP (aka MSOP) package, not the SOIC.

author: rx8pilot
author_rank: Super Contributor
timestamp: September 11, 2016, 11:26:11 pm
content: Here is a screenshot of what appears to be the decoder missing a valid STOP condition. Top two waveforms are analog and the bottom two are the digital inputs. There does not seem to be an issue with thresholds here. The left is the end of a WRITE with a good ACK. It sends a STOP and then a START where the decode is all wrong (I think).Despite the very poor probing, the analog looks to be acceptable by a large margin.Bug in the decode? User error? I don't know.EDIT: This maybe from too slow a timebase and zooming in. Going to try a shorter timebase, capture less data with a higher sample rate to see if that is the problem - ie user error.

author: nctnico
author_rank: Super Contributor
timestamp: September 11, 2016, 11:29:47 pm
content: Try to zoom in on the rising edge of the SCK and SDA lines and look for extra pulses on them in the digital trace.

author: rx8pilot
author_rank: Super Contributor
timestamp: September 11, 2016, 11:52:44 pm
content: I have not been able to detect anything. No spikes anywhere near the START/STOP transitions. I turned off the digital channels and used the analog inputs at a faster sample rate/time base. I cannot trigger directly on the problem, so I am just stopping the scope arbitrarily to catch one.So far, I cannot see an actual error - only in the decode.

author: Electro Fan
author_rank: Super Contributor
timestamp: September 12, 2016, 06:42:00 am
content: Suggestion: Make a moderately long text file - such as the alphabet, maybe with lower and upper case, plus 0-9, and maybe with special characters, all in a repeating sequence 10-100 times or more, enough so that the data cannot be viewed in the decoded mode on one screen, and enough to stress test the scope's memory and processing. Send the data between two I2C DUTs you are confident in. Probe SCL and SDA with both analog channel probes and digital channel probes. Setup an analog decoder and a digital decoder. This will give you 6 lines (2 analog waveforms, 2 digital waveforms, and the 2 decoders). Try different Time Division and sampling settings for both capture and playback. Look to see how how much consistency or inconsistency can be observed among the 6 lines. If this all looks good maybe your scope's LA is working fine; if not, maybe there is an anomaly among the analog channels or the digital channels or the decoders.