One SPI slave corrupts data on MISO line from other devices - Page 1
Forum section: electronics / projects

author: Emil
author_rank: Regular Contributor
timestamp: February 06, 2018, 11:18:46 am
content: Hi,I have a project with 4 SPI devices on one bus. Each device is on a seperate board, connected with shielded cables. Let's call the devices Controller (master), X (slave), Y (slave), Z (slave). All the devices are STM32F0x microcontrollers running at 3.3V.Background:This was working most of the time, but I had some random errors where expected data would not be received. I attributed this to noise (5 steppers, a DC motor and 4 solenoids on the machine) so I started by adding ferrites on most of the cables, this seemed to reduce the problem a bit, but I'm not sure. I added ferrites on the SPI lines on Z, but SPI stopped working so I removed the ferrites. There was one unshielded ribbon cable connecting power and SPI to Z, so I replaced this with two shielded cables, one for stepper power and one for SPI and power for the MCU.Current issue:- Data on MOSI is received correctly by all slaves.- Data on MISO from Z is received correctly by the master.- Voltage on MISO line is low (~1.8V).- Data on MISO from Y or X is corrupted. For example 0x6E is received as 0x58 or sometimes 0x5C. I have confirmed with a logic analyzer and a scope that the data on the bus is the same as what is received.- The CS lines go low for one device at a time as expected. (TODO: Check actual level on scope, I have only checked with the logic analyzer.)- There are no shorts between any of the SPI lines or any of the lines and gnd/vdd.- If I disconnect Z completely: Everything else works, no corrupted data, MISO voltage ~3.2V.- With Z connected, but SPI not enabled in firmware, voltage on MISO is normal (~3.2V), but data on MISO is still corrupted.Any ideas what the problem might be or what to check next?

author: 0xdeadbeef
author_rank: Super Contributor
timestamp: February 06, 2018, 11:30:59 am
content: Obviously on a bus, the data outs of the slaves have to be passive (tristate) if they aren't active for the bus to work.But there are devices that like the 74HC595 that don't tristate if they are not activated by CS and thus disturb other devices on the bus.In case of the 74HC595, it might help to add e.g. a 10k resistor on its SPI output (SQ7), to avoid disturbance on the bus.

author: spudboy488
author_rank: Regular Contributor
timestamp: February 06, 2018, 12:37:11 pm
content: I've been burned on this in the past where the part can use a SPI like input but is designed for a daisy chain-able connection. I just cut the pin/trace of the parts MISO connection. Since then, I check if the part can be read from. If it's a write-only part, I don't even connect to the MISO pin on the part.

author: andyturk
author_rank: Frequent Contributor
timestamp: February 06, 2018, 12:50:30 pm
content: How are the MISO and CS pins configured on the slaves? Are you using hardware CS or something in firmware?If you've got signal corruption, a 'scope will be a better tool than a logic analyzer.

author: Ice-Tea
author_rank: Super Contributor
timestamp: February 06, 2018, 12:58:19 pm
content: Cables? How long? Topology? SPI is can be a high speed bus and even if it is a low speed bus it can use high speed buffers. I'm imagining your setup has considerable stubs with the reflections and SI problems to match...

author: Emil
author_rank: Regular Contributor
timestamp: February 06, 2018, 01:18:00 pm
content: Quote from: andyturk on February 06, 2018, 12:50:30 pmHardware on all devices. Actually, your question made me notice a possible problem with my SPI configuration on Z, I will post an update when I have checked it.QuoteTrue, but inconvenient on 6 signals with only a two channel scope.

author: Ice-Tea
author_rank: Super Contributor
timestamp: February 06, 2018, 01:22:57 pm
content: Quote from: Emil on February 06, 2018, 01:18:00 pmNot really. Probe MOSI and SCK at each chip and look at the edge. You don't have to capture the exact moment it goes wrong. If it's bad you'll see it regardless of the logic state.

author: Emil
author_rank: Regular Contributor
timestamp: February 06, 2018, 01:24:25 pm
content: Quote from: Ice-Tea on February 06, 2018, 12:58:19 pmThe cable for the problematic device is almost 2 meters long. The cables for the other SPI devices are ~30cm.Speed is 1.5MHz, I have also tried 3MHz and less than 200kHz with the same result.

author: Ice-Tea
author_rank: Super Contributor
timestamp: February 06, 2018, 02:19:46 pm
content: That's just... Bad.Bus speed doesn't matter. How fast your edge rises matters. If you have reflections you risk, per example, double clocking data. SPI simply is not designed to work with long cable and noisy environments.I'll repeat: set the LA aside, get a decent BW scope and start probing. But for the kind of setup you have, look at something more robust than SPI (CAN? LIN?)

author: mikerj
author_rank: Super Contributor
timestamp: February 06, 2018, 05:48:29 pm
content: Quote from: Emil on February 06, 2018, 01:24:25 pmToo long, SPI is not designed for such long interconnects, though it can be made to work. Bit rate is not your problem, it's combination of high edge speeds and a badly terminated transmission line causing reflections. A quick hack would be series resistors located near output driver for each line i.e. SCK at the master, MOSI at the master and MISO at the slaves. Series resistors in the chip select lines (at the master end) would also be wise. Use a scope to get a suitable value that slows the edge enough to prevent excessive ringing, probably in the 10's to 100's of ohms range.

author: Emil
author_rank: Regular Contributor
timestamp: February 06, 2018, 10:47:20 pm
content: Quote from: Ice-Tea on February 06, 2018, 02:19:46 pmAs mentioned in my first post, I already measued the SPI lines with the scope (except for the CS line). I have checked the CS lines with the scope now and I can't see a problem on CS.I have attached a screenshot showing CLK and MISO. Are the small dips on the rising edge and small spikes on the falling edge of the clock reflections? If so, it seems very minor(?)I do notice two problems here: The high level on MISO is low at only 1.8V and the data (0b01011000) does not match what is being sent (0b01101110).I'm planning on replacing the SPI, but I'v not decided on what to use instead.

author: jbb
author_rank: Super Contributor
timestamp: February 07, 2018, 02:04:37 am
content: Good thing you’ce got a ‘scope :-)When you get a between-the-rails output on a ‘digital’ circuit it tends to indicate two IO pubs fighting. In this case, it looks like the ‘Z’ chip on the bus is not releasing it’s MISO line.I suggest you review the firmware in Z to confirm it’s IO pin and SPI slave are configured right. It could also be hardware damage to the chip (rare but it happens).For future use, you could use SPI over LVDS for the long cable. LVDS is quite good in noisy environments. The trick is that the LVDS->CMOS converter in the MISO circuit will need an enable pin connected to the relevant CS line. I also suggest some TVS diodes and maybe small series resistors (say 5R) for ESD protection. Don’t forget the termination resistors (some receivers have them built in).LT also make SPI to something-proprietary converter chips which use CAT5 cable and isolating transformers. They could be helpful.

author: Ice-Tea
author_rank: Super Contributor
timestamp: February 07, 2018, 09:28:04 am
content: Can you zoom in on the clock edge?

author: David Hess
author_rank: Super Contributor
timestamp: February 07, 2018, 04:40:52 pm
content: Quote from: Ice-Tea on February 07, 2018, 09:28:04 amAnd tell us the bandwidth and how you are probing?What is important is the signal at the receiver which is malfunctioning.

author: bson
author_rank: Unknown
timestamp: February 07, 2018, 09:27:33 pm
content: Do you have a third channel? If so, add the Z device's SSEL/CS input to make sure it's properly deselected. Make sure to probe it on the device end so you see what the device sees.

author: Emil
author_rank: Regular Contributor
timestamp: February 07, 2018, 10:58:09 pm
content: Quote from: Ice-Tea on February 07, 2018, 09:28:04 am

author: Emil
author_rank: Regular Contributor
timestamp: February 07, 2018, 11:02:40 pm
content: Quote from: bson on February 07, 2018, 09:27:33 pmNo third channel. I'v been probing on the master end since data from (not to) the slave device is what get's corrupted. I will try probing the other end.

author: Emil
author_rank: Regular Contributor
timestamp: February 07, 2018, 11:06:29 pm
content: Quote from: David Hess on February 07, 2018, 04:40:52 pm70Mhz bandwith, probing at the board with the master which is not receiving data correctly.

author: Emil
author_rank: Regular Contributor
timestamp: February 07, 2018, 11:13:08 pm
content: Quote from: jbb on February 07, 2018, 02:04:37 amI checked the firmware and found a problem. The SPI on "Z" was not set up correctly, the initialization code for CS was missing. I added it and the levels on MISO are now normal.This has however, not solved the problem with corrupted data on MISO.

author: jbb
author_rank: Super Contributor
timestamp: February 08, 2018, 02:11:15 am
content: Quote from: Emil on February 07, 2018, 11:13:08 pmProgress!So, your 2m cable will have a round trip time of approx 20ns (assuming that the signal goes at 2/3 light speed), which explains the 20ns long flat spots in your waveforms. This is due to transmission line reflections.Given the relatively long cable, I suggest that the next step should be signal integrity testing.I suggest you check that your clock phase and polarity are the same for all devices.Then try it with Z (and the long cable) disconnected.Then I suggest you check CLK, MOSI and MISO at each node to check things haven’t been mangled (e.g. ringing above supply or below ground, not staying high, not staying low etc.). This should be done at each node for slave X talking, slave Y talking and slave Z talking. (So that’s 12 data sets I guess.) You’ll be looking for weirdness.

author: Emil
author_rank: Regular Contributor
timestamp: February 09, 2018, 09:28:46 am
content: Quote from: mikerj on February 06, 2018, 05:48:29 pmI added series resistors as suggested and the flat spot on rising edge is gone now. I ended up with 330R.

author: Emil
author_rank: Regular Contributor
timestamp: February 09, 2018, 10:49:17 am
content: A quick update:Change: Fixed firmware bug.Effect: Correct high level voltage on MISO.Status: Works most of the time, but not reliable.Change: Added 330R resistors on CLK, MOSI anc CS -lines at master and on MISO on each slave.Effect: Edge looks good.Status: Works most of the time, but not reliable.Change: Updated firmware to detect bad data on MISO and request a resend.Effect: Works as long as the errors are sporadic, not effective if resent data is repeatedly corrupted.Status: Works, but will sometimes freeze for a while as data is repeatedly resent.Change: Removed seperate 90V PSU for X/Y steppers and connected to same 36V PSU as the other steppers.Effect: No or very little corrupted data (needs more testing).Status: Works. Need more tesing, but will probably do until I respin the boards and replace the SPI bus.

author: Siwastaja
author_rank: Super Contributor
timestamp: February 09, 2018, 11:13:46 am
content: 330R sounds quite high. Probably slowing down the edge too much, and increasing noise coupling because of the high drive impedance (330R now) - now the switching takes very long, stays at mid-level for quite some time, and noise couples easily during that time, possibly double-clocking. Try 33, 56, 100R - when the drive is stronger, coupled noise amplitude is similarly lower.If very low data rates are acceptable (think about 10 kbit/s and less), adding a dumb RC filter, i.e., adding a capacitor (think about 100 - 1000 pF) right at the input pin may help.You should make it work 100% reliably without software resend hacks, and try that with different components, different batches, possibly in different temperatures if at all possible. Then test the software hack still works, keep them on and make them report an issue instead of just silently correcting. The reason for this is that if you are relying on the software tricks, the margin for it to stop working completely isextremelythin.

author: Emil
author_rank: Regular Contributor
timestamp: February 09, 2018, 01:15:22 pm
content: Quote from: Siwastaja on February 09, 2018, 11:13:46 amI tried 100R, but it was not enough to get rid of the flat part of the rising edge, but perhaps I didn't need to go as high as 330R.QuoteAgreed. But I'm following the advice I got earlier in this thread and getting rid of the SPI bus entirely and replacing it with something more robust. I have four boards that are a bit of a mess of wires and modifications so I'm redesigning everything to go on two boards with some means of communication other than SPI. So this is just a temporary fix to get it usable for a few weeks. Btw, it's not silently correcting, it also sends a message back to the PC so I will see the error.

author: jbb
author_rank: Super Contributor
timestamp: February 10, 2018, 08:57:15 pm
content: Glad you're making progress.Quote from: Emil on February 09, 2018, 10:49:17 amAny comm link more than a few cm long can experience infrequent errors. In your case, with motors and solenoids around, it's not surprising to get some errors. Did you do error checking on the MOSI path as well? A useful approach can be to add 'good packet received' and 'bad packet received' counters for analysis. If you have a spare IO pin or two, you can also get the micro to put out a pulse on error. This can be used to help track down the source (i.e. trigger a 'scope with it).Given you're talking about XYZ stages, I guess timing is really important to you (otherwise your trajectories get jaggy). So getting a link with very few errors (and thus few re-sends) is important.Quote from: Emil on February 09, 2018, 10:49:17 amI guess you might have had some kind of ground loop (note, isolated supplies always include some input->output capacitance and can form high freq ground loops) in the mix too.The SPI approach might not be a dead loss: you could try fitting digital isolators (with appropriate CS decode logic) between power domains, it might help.However, another comms scheme might work out better for you. I think CAN is used in some machine tool systems - maybe it would be a good fit (features: multi-drop bus-type, differential signalling, typical speed 1Mbit/s, hardware error check and re-send).