I2C bus Hangs on 70% of the assembled PCBs [Potentially Solved] - Page 1
Forum section: electronics / projects

author: AMS
author_rank: Contributor
timestamp: October 11, 2015, 09:16:56 am
content: Hi All, I am new to this forum so sorry if I am a bit rusty.I designed a circuit based on PIC18 series which reads the output of an I2C sensor (BNO055) and stores it in the EEPROM memory. There is also a RTC chip in the circuit. Everything is on an I2C bus with 10K pull-ups. I tested the circuit before sending it off to be manufacture and everything was fine. However after I received the assembled boards more than half of them don't work. After further investigation I noticed that I get NAK on the I2C bus after any command to any of the i2c chips. I should also say that i suspected that it might be the pull-ups so i changed it to 4.7K, 2.7K and 1.2K without any luck. Sometime however i can get it to work for the first measurement but it hangs after the first reading (Just read and spit out on the UART not store in memory at this stage). I would greatly appreciate if any of the members could help me (if they had similar experience) or suspect anything that I haven't checked. What is frustrating is that out of almost 50 boards 15 of them work without any issue or modification.Regards

author: Rerouter
author_rank: Super Contributor
timestamp: October 11, 2015, 09:33:36 am
content: Ok 2 thoughts,1. Please tell me you added a timeout function in your code,2. Did you implement the correct I2C start command, including waiting the maximum prescribed delays on commands from the devices datasheets? you could have been cutting it close on tolerance of how long before the device is ready and could have had a funny consequence,

author: Ian.M
author_rank: Super Contributor
timestamp: October 11, 2015, 09:50:48 am
content: There's a known bug affecting the MSSP module of a significant number of older 8 bit PICs. If you modify any bit of the TRIS register of the port that has the I2C pins during I2C bus activity it can latch one or both bus lines permanently low. They wont be released until the module is disabled and reinitialised.

author: AMS
author_rank: Contributor
timestamp: October 11, 2015, 09:54:37 am
content: Thank you for the quick replies:1.I have checked the noise on the I2C bus and can confirm that the bus is clean from noise2.Both the MCU and the sensor have external crystals (and option to work with internal) tried all possible four combinations without any luck3.I have checked the tracks and all seem fine4.MCU of the boards are programmed (and reprogrammed many times) with the same HEX file.5.I increased the wait time before start of the I2C sequence still no luck6. There is a timeout fiction in the code (so i can get out of the free) however the bus is not functional7. there is no ferrite beads on the power lineRe:Ian. M I will check your suggestionAgain thank you for any suggestions I will try some more things and post back the results. This is driving me crazy

author: German_EE
author_rank: Super Contributor
timestamp: October 11, 2015, 10:17:19 am
content: How much decoupling do you have on your power supply? Strange things can happen when decoupling goes wrong.

author: mikerj
author_rank: Super Contributor
timestamp: October 11, 2015, 10:44:34 am
content: It's probably down to some marginal timing caused by interaction between the PIC code and the bus itself. I can remember spending a long time resolving issues like this on older 18F PICs, though usually when the PIC was the slave rather than the master.Is your I2C handler polled or interrupt driven? Do you have other interrupts running during I2C transfers?Which PIC are you using?

author: nuno
author_rank: Frequent Contributor
timestamp: October 11, 2015, 12:53:19 pm
content: I2C modules are the most bug plagued module implementations I've ever seen (from AVR to Raspberry PI). I have yet to work on a project having an I2C bus that doesn't give considerable problems, so I'm at the point I avoid I2C like the plague unless is something exactly equal to something else I have done before. I vote your problem is in the hw implementation of one of the chips, or both - carefully go through erratas for both chips, google for someone having similar problems too.

author: Fungus
author_rank: Super Contributor
timestamp: October 11, 2015, 01:13:59 pm
content: Quote from: nuno on October 11, 2015, 12:53:19 pmNever had a problem myself.The only major "fail" in I2C is that there's no way to hard-reset the bus. You're limited to stuff like this:http://www.analog.com/media/en/technical-documentation/application-notes/54305147357414AN686_0.pdfQuote from: AMS on October 11, 2015, 09:16:56 amIf it consistently fails or "hangs after the first reading" on 35 out of 50 boards then something is fundamentally wrong with the implementation,probably software(plus: I would check those other 15 again to see if they're as robust as you think they are).If it turns out you have some boards thatalwayswork then start swapping components with the ones that don't. Find out which component is causing the failure.PS: It isn't causing this problem but 10K resistors seem a little high to me. Have a look at the signal, make sure it's nice and square.

author: Fungus
author_rank: Super Contributor
timestamp: October 11, 2015, 01:17:35 pm
content: Quote from: mikerj on October 11, 2015, 10:44:34 amRule number 1: Try slowing down the clock rate.If it still doesn't work: Try a software I2C. If you only have slave devices then it's a very simple protocol to implement.

author: Fungus
author_rank: Super Contributor
timestamp: October 11, 2015, 01:25:00 pm
content: Quote from: evb149 on October 11, 2015, 09:40:06 amThis is another possibility.It's easy to check if signals are arriving at chips though....

author: Kalvin
author_rank: Super Contributor
timestamp: October 11, 2015, 01:27:27 pm
content: If you have budget left, invest on a logic analyzer from the Saleae or similar. If you are low in budget, you might get away with a cheap chinese clone. Those analyzers can interpret different serial protocols, I2C included. Consider donating some funds to the original software author when you are done and you start getting money.

author: AMS
author_rank: Contributor
timestamp: October 11, 2015, 01:44:24 pm
content: Quote from: Kalvin on October 11, 2015, 01:27:27 pmI have a logic analyser and have checked the signal it seems that the PIC is sending the write command but the chip is not acknowledging it. I will post some screen shots.Quote from: Fungus on October 11, 2015, 01:25:00 pmTracks are certainly intact as the signal is fine all around the board.Quote from: Fungus on October 11, 2015, 01:17:35 pmI will try these and post the resultsI will also try to replace parts with the working ones. This might be tricky as the PIC is QFN and the sensor is a ridiculous LGA package. You guys all Rock. Thank you al for your replies. I hope I can solve this mystery soon.

author: Kalvin
author_rank: Super Contributor
timestamp: October 11, 2015, 01:59:28 pm
content: The BNO055 Pin 17 (i2c address select) is connected to VDD or GND instead of left floating?

author: AMS
author_rank: Contributor
timestamp: October 11, 2015, 02:00:25 pm
content: Quote from: Kalvin on October 11, 2015, 01:59:28 pmYes it is connected to the GND.

author: Kalvin
author_rank: Super Contributor
timestamp: October 11, 2015, 02:04:28 pm
content: Quote from: AMS on October 11, 2015, 02:00:25 pmAnd the PS1 and PS0 (pins 5 and 6) connect to GND? The oscillator works well and the waveform and the frequency are good?

author: AndyC_772
author_rank: Super Contributor
timestamp: October 11, 2015, 02:04:43 pm
content: NAK on the I2C bus doesn't necessarily mean there's anything wrong with the I2C bus, it just means that for some reason the slave device is unable to accept commands. It might be marginal timing, or it might be that the device has latched up or been soldered on upside down, and without more investigation it's impossible to know.Firstly, scope the bus and see what the I2C signals actually look like. Without this information, you're poking about in the dark and guessing. Don't just swap parts randomly; look for specific problems, then make changes that address them.With I2C there's not a lot to go wrong with the hardware. Check the logic '0' voltage and the rise time of the signal; they'll tell you whether your pull-up is a reasonable value. If rise-time is slow, you need a lower value, and if the '0' is marginal, then a higher value should help.IMHO it's unlikely to be anything to do with decoupling or power supplies, unless they're fundamentally broken in some way. I2C is slow in terms of both bits/second and edge rate, and receivers generally have Schmitt triggers in them so they have good noise immunity too. You can get away with a lot in terms of hardware.

author: Jeroen3
author_rank: Super Contributor
timestamp: October 11, 2015, 02:10:06 pm
content: Where did you buy your BNO055?

author: bobcat
author_rank: Regular Contributor
timestamp: October 11, 2015, 04:51:03 pm
content: Are you splitting the bus into more than one direction? (bad)Are the SCL and SDA lines approx the same length? (including vias)

author: free_electron
author_rank: Super Contributor
timestamp: October 11, 2015, 05:18:45 pm
content: hard resetting the bus : send 9 STOp cycles. any device( even with a badly implemented state machine ) will bomb out at that point.what is the eeprom you are using ? are you sure the eeprom and senor are on different addresses ?i have seen eeprom's that respond to more than one i2c address ... (they are the large ones like the 24c256 and 512 )

author: AndyC_772
author_rank: Super Contributor
timestamp: October 11, 2015, 05:20:40 pm
content: Quote from: bobcat on October 11, 2015, 04:51:03 pmThese things simply don't matter with I2C. Edge speeds are slow enough that transmission line effects are negligible.

author: PeteH
author_rank: Unknown
timestamp: October 11, 2015, 05:27:36 pm
content: You aren't using any isolation chips are you?I2C isolators (ADUM1250/51), bus redrivers or level translation? All devices are on 3.3v?

author: NiHaoMike
author_rank: Super Contributor
timestamp: October 11, 2015, 09:17:39 pm
content: I once had a strange I2C issue on a DAC I was working on. On a cold start, the S/PDIF receiver would sometimes act as if it never received the I2C commands but it would always work fine on a warm reset. Turns out the MSP430 can operate down to 1.8V and the big rail caps were slowing down the ramp rate such that the processor would start booting before the S/PDIF receiver got enough voltage to start working properly.

author: nuno
author_rank: Frequent Contributor
timestamp: October 11, 2015, 10:39:42 pm
content: Lucky you, Fungus!AMS, if you have more than one device on the bus, you should also test one of your bad boards leaving only one device, going through all your devices.I also think you should scope the lines with an analyzer of the "analog type" too.Your board is probably small and this doesn't apply but nevertheless, although the I2C standard limits the slew rate (actually, not all hw does it, beware, some arecompatibleand notcompliantto I2C), there can still be a relevant amount of crosstalk between SCL and SDA, if proximity and length allows.

author: photon
author_rank: Frequent Contributor
timestamp: October 12, 2015, 12:47:59 am
content: Try both the default I2C address 0x29 along with the alternative address 0x28 with both settings of pin 17 (COMM3). Make sure there are pullups on the board for I2C clk and data.

author: Fungus
author_rank: Super Contributor
timestamp: October 12, 2015, 07:38:33 am
content: Quote from: free_electron on October 11, 2015, 05:18:45 pm'any'?I assume you're referring to that doc I posted earlier. They don't propose sending 9 STOPs, they propose sendingup to9 clock pulses whenever you see the bus is being held low, followed by a STOP.

author: Fungus
author_rank: Super Contributor
timestamp: October 12, 2015, 07:40:43 am
content: Quote from: NiHaoMike on October 11, 2015, 09:17:39 pmNice one!Again we see the dangers of the "just stick a couple of big electrolytics on it, she'll be right!" approach to power supply design.

author: mikerj
author_rank: Super Contributor
timestamp: October 12, 2015, 08:59:59 am
content: Quote from: NiHaoMike on October 11, 2015, 09:17:39 pmI've seen exactly the same kind of problem in a prototype product. Fixed by polling the I2C device during power on initialisation until an ACK was received (or a timeout reached).

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 12, 2015, 01:13:33 pm
content: I am not an expert in hardware but I do know something about software.Write a new program that focuses on this problem and isolates it.When you have fixed it in this test program then go back to the original program and work it out there.

author: bson
author_rank: Unknown
timestamp: October 12, 2015, 06:52:28 pm
content: Assuming your address pins on the I2C device aren't floating - verify with a continuity/ohm test. Check the levels of the address inputs at the minimum operating supply voltage; often this is 1.5-1.8V. If the A inputs are capacitive and you pull them up with something monstrous like 1M or 100k, then the input gates may not be charged above threshold when the chip comes alive. (Or the opposite perhaps, for pull-downs.) Misconfigured/floating I2C address -> unresponsive chip. If it has a reset input, try holding it a little longer during power-on.

author: Kjelt
author_rank: Super Contributor
timestamp: October 12, 2015, 07:53:59 pm
content: Quote from: free_electron on October 11, 2015, 05:18:45 pmWith bitbanging that should work, even generating clockpulses alone till the data line is released works. But with a peripheral inside a uC you often can only generate stops if you are already started and transmitted data i thought, the peripheral of the uC itself also has a state machine built in.

author: ez24
author_rank: Super Contributor
timestamp: October 13, 2015, 11:12:04 pm
content: Maybe the OP got it working and forgot about us. Two days and no news. Seen this happen many times.

author: AMS
author_rank: Contributor
timestamp: October 13, 2015, 11:48:13 pm
content: Thank you all for your suggestions. Just wanted to let you all know that I haven't figured it out YET!!!!. I am going through the suggestions one by one(was a bit busy at work) and will post and extensive result with logic analyser data tomorrow. Again thank you all You are all Awesome

author: alank2
author_rank: Super Contributor
timestamp: October 14, 2015, 12:00:56 am
content: I've had really good luck with i2c on a couple of AVR projects and almost always use 2K pullups on it. Does the SAME response happen when you slow it down? NACK where you expect ACK? I agree with the idea of bit banging it, simplifying the code, etc. Have you posted a schematic yet? Might be nice for others to see exactly what is hooked up and how.

author: obiwanjacobi
author_rank: Super Contributor
timestamp: October 14, 2015, 10:22:59 am
content: Quote from: Fungus on October 12, 2015, 07:40:43 amI think of it as the danger of not having a proper(system-wide) reset line with correct logic (time/voltage etc)....I just thought of a piece of code that I used to find out the address of an LCD-I2C interface I was using... It scanned the I2C lines to see if there was any response at each address. Perhaps it is something you could try to see if there are one or more devices that do not respond correctly...?It was Arduino code (that was the quickest for me at the time) but basically it initiated I2C communication for each address (1-127) and see if the line was pulled down by the device..?http://playground.arduino.cc/Main/I2cScannerI am no expert on I2C but perhaps you will find it useful...

author: Kjelt
author_rank: Super Contributor
timestamp: October 14, 2015, 11:48:55 am
content: Curious what happened here but not surprising. I bailed out of a personal I2C project once when using multiple Microchip I/O expanders, Eproms and RTC.Writing now and then went perfect but when I used to read >20 times a second one of the ports the expander would freeze up within a few minutes.No reset line on that expanderand even then this should not happen.The dataline was stuck (hanging) and reclocking to free it could only be done by using another I/O port or shutting down the I2C peripheral, resetting the pins to I/O and do it like that. Garbage. Hopeless, after a week of frustration I throw the towel made a redesign using simple shift registers for the input ports and I can now do hundreds of thousands of reads a second if I wanted to, without problem.Any new design I am going for SPI instead if I2C, done with it.

author: hamdi.tn
author_rank: Frequent Contributor
timestamp: October 14, 2015, 12:03:41 pm
content: Quote from: obiwanjacobi on October 12, 2015, 01:13:33 pm+1i hate I2Ci hate more that i2c chips don't have any hard reset.i had similar problem recently ( first prototyping board work fine , second one hang ) and after hitting my head to the wall it was just a software bug ( under some condition it skip the delay time ) anyway 99.99% of the case i had to deal with it's a software issue and usually related to the delay.Your slave respond, so nothing wrong with your hardware i guess, if you not doing something wrong with slave address, it's probably due to some delay problem. it's for the best to put some time out mechanism and a way to recover the bus too.as suggested Try to isolate the problem first

author: Kilrah
author_rank: Unknown
timestamp: October 14, 2015, 12:58:42 pm
content: I'd triple check the addresses, and physically check the (real) level of each address select pins on all chips that have one.Even if you know you've set them properly on the schematic there could be a recurring manufacturing defect that happens to let one floating or shorted on some of the boards.

author: ruffy91
author_rank: Regular Contributor
timestamp: October 14, 2015, 01:33:29 pm
content: Have you already seen this thread:https://www.microchip.com/forums/m727768-p2.aspx?Someone had a similar problem with an 18F4550 and a DS1307 and solves it with moving the pull-ups closer to the PIC.You could also set/reset the SMP bit in the SSPSTAT register to change the slew rate.

author: AMS
author_rank: Contributor
timestamp: October 14, 2015, 01:50:17 pm
content: Hi everyone, I tried almost all of the suggestions (software ones first) with no Luck. However i started to suspect that the problem might be the BNO055 (possible cold joints as was pointed out in some of the suggestions) so i lifted 2 of them and voila!!! the top and bottom rows of pins (which include the i2c and address select bits) were not reflowed properly. I have attached some of the photos that i took from the chip and the board. As you can see there is solder on the board but not on the chip. Horrible LGA packages. However this doesn't explain why the other I2C chips were not working properly. For now I am going to send back the boards to the assembly house so they can rework them . You were all awesome and I learnt a lot from you all. I am looking forward to start contributing to this fantastic forum.One more thing during testing i noticed that the SCL line is putting some noise on my pic oscillator (a 100KHz signal was almost super imposed on top of 16MHz clock). I am not sure whether it has anything to do with the fault or not as I only saw this on the faulty boards.That you all again. you were all really helpful.

author: free_electron
author_rank: Super Contributor
timestamp: October 14, 2015, 01:53:03 pm
content: It is possible you have head-in-pillow syndrom on the LGA packages.just like your head makes a dimply in a pillow the lga surface can do that to the solder. there is an ohmic connection of a few kiloohm. Flex the board or do anything to it mechanically and that resistance wildly fluctuates.