Sync of two FPGA at 5m  - Page 1
Forum section: electronics / projects

author: fabiodl
author_rank: Frequent Contributor
timestamp: November 13, 2020, 09:52:30 am
content: I have two fpgas, each sampling a set of signals. I want to sample all of them at the same time. Each fpga clock is 50Mhz. They are physically far, approximately 5m/ 16 feet apart. One would be master, the other slave. The sampling occurs at 1Khz.The master fpga generates the sample pulse. The slave fpga receives it, and brings it into its clock domain.What would you use to connect the two fpgas? what kind of buffering would you use? How long would you make the sync pulse? What would you do to assure the pulse is not received twice due to ringing, without relying on verilog and minimum inter-pulse interval?

author: Miyuki
author_rank: Frequent Contributor
timestamp: November 13, 2020, 05:35:28 pm
content: What about optical fiberIt can transfer a nice pulse at almost any distance

author: ejeffrey
author_rank: Super Contributor
timestamp: November 13, 2020, 06:28:03 pm
content: I'd just connect a coax line, BNC or SMA. Depending on the FPGA IO characteristics you might be able to just directly drive a short transmission line directly but it would be better to use an external driver -- just a standard CMOS inverter or buffer is probably enough. If you use something like a standard hex inverter you can connect multiple channels in parallel to increase drive current or use a dedicate line driver chip.Source terminate the line to limit reflections without reducing the DC signal amplitude.Instead of sending short pulses you can send a 500 Hz square wave and have the receiver sample on change detection. That will avoid needing clock domain crossing and help avoid reflections at the same time.

author: BrianHG
author_rank: Super Contributor
timestamp: November 13, 2020, 06:45:07 pm
content: A proper coax cable with a cable with matched impedance driver and receiver will send a perfect signal from point A to point B. This can even be your 50MHz clock.I would just transmit serial signal (something like 8N1 for a 10 bits per word) at 100 megabaud with 1 repeating character except during the 1KHz sync signal where you change the character to a new one (choose the 2 different characters so the signal looks like a 10MHz clock, with an inverted phase for the 1KHz sync character). On the FPGA, you are using serializer/de-serializer with a LVDS differential outputs to a balanced differential cable driver while on the other side you use a differential cable receiver/cable length equalizer amp feeding the FPGA differential input where that FPGA's PLL regenerates the 100MHz data clock and feeds you 10million repeating characters a second except for 1KHz sync pulse. 100 megabaud is well within the oldest, slowest, lousiest, garbage FPGAs in existence so long as the FPGA has a PLL. (You could just as easily run a 4:1 serializer with 2 bits of data, 1 for sync and another for an auxiliary channel running at 50 Megabaud,)Cheap bottom end cable transmitters and receiver exist today with gigabit performance, however, ~300 megabit would be fine for you. Simple BNC/Cable TV 75ohm coax would even be fine as this is used heavily in SDI digital studio TV.Note that cheaper direct wire solutions with proper termination would also work without the cable driver and receiver amp, the solution I mentioned is expected to operate in a hardened industrial environment where you want to protect the FPGA IOs from esd & the coax isolates you from unpredictable local EMI.If you just need a 1KHz reference from the first PCB, this is OVERKILL. Just use any RS485 differential serial transmitter and receiver with a single output to 1 single input with any twisted pair network cable transmitting a 1KHz pulse. It will work fine.If I were to use fiber-optic as a transmission medium, I would use the same serializer / de-serializer technique as it would not only communicate the sync signal, but it would regenerate the 100MHz clock on the other side.

author: Someone
author_rank: Super Contributor
timestamp: November 13, 2020, 11:22:46 pm
content: Quote from: fabiodl on November 13, 2020, 09:52:30 amQuantify "same", 1us error? 1ns? 1ps?

author: dmendesf
author_rank: Frequent Contributor
timestamp: November 14, 2020, 12:08:41 am
content: I've done this with 485 drivers and hundreds of meters of twisted pair to an error < 10ns (that was my target... Didn't bother to try further). I warn you that if you have different xtals on each FPGA the clock deviation with temperature will trump other errors after that, so better to plan for TCXOs.

author: ejeffrey
author_rank: Super Contributor
timestamp: November 14, 2020, 12:15:43 am
content: Quote from: Someone on November 13, 2020, 11:22:46 pmIt sounds like the two FPGAs have non-synchronized 50 MHz clocks. So better than 20 ns is not even possible.

author: BrianHG
author_rank: Super Contributor
timestamp: November 14, 2020, 12:40:39 am
content: Quote from: ejeffrey on November 14, 2020, 12:15:43 amUsing my serializer trick at 100megabaud allows the second FPGA's PLL to sync to that 10MHz frame clock & re-create a synced 100MHz clock. As for the 1Khz pulse, you can add any number of cycle delay so you will get a 1Khz clock with an adjustable +/-10ns step resolution (5ns if you run your internal PLL at 200MHz), no DLL re-clocking jitter.

author: Someone
author_rank: Super Contributor
timestamp: November 14, 2020, 12:51:58 am
content: Quote from: BrianHG on November 14, 2020, 12:40:39 amPlus some ns of delay in the cable, and some (variable) ns of delay in the TX and RX, and some ns of delay in the clock network inside the FPGA.....At some point of alignment accuracy it demands a switch from open loop to closed loop.

author: dmendesf
author_rank: Frequent Contributor
timestamp: November 14, 2020, 04:59:24 am
content: I used a closed loop approach that assumed each half of a round trip was the same duration. FPGA pin -> transceiver -> cable -> transceiver -> other FPGA pin and transceiver -> another cable -> another transceiver -> another pin from first FPGA. So if you have a 1kHz signal to send and receive it back with 200ns delay you need to offset it by 1ms-(200ns/2) = 900ns so both local and remote edges match.

author: dmendesf
author_rank: Frequent Contributor
timestamp: November 14, 2020, 05:01:57 am
content: Spoiler... Don't use different pairs from a utp cable to do it as each pair has a different twist rate, so a different delay per meter... Violating the rule that propagation time going is the same as propagation time returning. Don't ask how I know

author: Miti
author_rank: Super Contributor
timestamp: November 14, 2020, 01:32:32 pm
content: HDMI cable should also do it. You have 4 differential pairs to send clock on one pair, sync on another, etc. They come longer than 5m.

author: rstofer
author_rank: Super Contributor
timestamp: November 14, 2020, 06:11:58 pm
content: Quote from: ejeffrey on November 14, 2020, 12:15:43 amThe propagation delay of coax is on the order of 5 ns per meter so, right off the bat, you have 25 ns of delay. Plus the delay of running through 2 D-flops at 20 ns each to cross the clock domain at the slave. Plus there is some delay at the output drivers of the master.You could get clever on the master end and add a 3 or 4 flop delay (about 60-80 ns) and try to approximate what is happening at the slave end.You could also try to synchronize the clocks. Have the master send a clock to the slave but internally use a delayed clock (run the clock out of the FPGA, through 5m of coax and bring it back in as the master clock). You just need to play around with delays until things are nearly equal.Large mainframe computers had a similar problem. They just couldn't send signals over wire at much more than 0.9 times the speed of light. They had to add delay loops of wire here and there to equalize the delays. That's why the Cray was built as a cylinder. The speed of light was just too slow!

author: fabiodl
author_rank: Frequent Contributor
timestamp: November 16, 2020, 11:25:22 pm
content: Thank you everybody, all these comments really help!Quote from: Miyuki on November 13, 2020, 05:35:28 pmThat is a nice idea. Do you know any off the shelf solutions?Quote from: ejeffrey on November 13, 2020, 06:28:03 pmThat was indeed what I initially thought about. If I am not mistaken, however, I'd need a source and termination resistor. Imagine I use a video cable. then the fpga (I use de DE10nano) would see a 150ohm load, which is too much. Even a 74ls245 will not be able to provide 5/150=33mA.Quote from: ejeffrey on November 13, 2020, 06:28:03 pmRS485 may indeed be sufficient for my setup, and I think I'll go for it. Since the sampling is at 1khz, synchronization with a 1us jitter is totally fine. Is there any rs485 chip you'd recommend? Do you think I could use an off the shelf ethernet cable to have several rs485 lines?Quote from: Someone on November 13, 2020, 11:22:46 pmAs stated above, 1usQuote from: dmendesf on November 14, 2020, 12:08:41 amwhat drivers did you use?Quote from: dmendesf on November 14, 2020, 12:08:41 amThis is a very good point. I do timestamp everything on the master FPGA, so I can ignore this drift.Quote from: dmendesf on November 14, 2020, 04:59:24 amI need to retrieve the data anyway. I will try some "echo back" to see what the round trip time is.Quote from: BrianHG on November 14, 2020, 12:40:39 amThat's indeed really neat!Quote from: dmendesf on November 14, 2020, 05:01:57 amIs that a thing for any ethernet cable? Or were you just unlucky?Quote from: Miti on November 14, 2020, 01:32:32 pmthis is a nice idea!Quote from: rstofer on November 14, 2020, 06:11:58 pmLuckily I do not need this kind of resolution. But that's indeed something I will take care of if I will need to improve the setup. In such case, I think I will just get a clock node, which does not measure anything , and N measuring nodes, all connected using equal length cables, irrelevant of their physical position.

author: ejeffrey
author_rank: Super Contributor
timestamp: November 17, 2020, 12:38:09 am
content: Quote from: fabiodl on November 16, 2020, 11:25:22 pmFor relatively slow things like this you only need the source resistor. The signal will reflect ~100% from the load, but be absorbed after a single reflection. This drops the DC power consumption down to basically nothing. It will still draw a bit of current during the transient but it will settle down after a single round trip (10 ns). It also avoids the 2:1 voltage divider effect that drops your ouptput voltage by half, so you don't need to signal with 5V. You can probably get away with 3.3 V logic over 5 meters in coax, so you could use like an SN74LVC04 at 3.3 V.

author: langwadt
author_rank: Super Contributor
timestamp: November 17, 2020, 12:38:42 am
content: Quote from: fabiodl on November 16, 2020, 11:25:22 pmit is intentionally done in ethernet cable , different twist rate means the pairs won't align with each other socross-talk between pairs is reduced

author: BrianHG
author_rank: Super Contributor
timestamp: November 17, 2020, 01:58:34 am
content: For the100 megabaud trick, you can use a cheap fast NPN like the MMBTH10/MPSH10/PMBT6428. 2N3904 or MMBT3904 would be good for 20 megabaud, however, the receiving FPGA might not be able to regenerates a frame clock at 2MHz, but maybe it can with a 2 or 4 bit serial output. Other options include a PUSH-PULL configured NPN-PNP transistor for the driver. Offers a higher quality balanced drive output on par with any super-expensive IC which can drive >500MHz, especially if you use 2 outputs from the FPGA to drive each base, and you can use cheaper slower transistors to achieve >100MHz drive performance. (If you need schematics, just ask)For 1 transistor:Use a 100 ohm pullup on the collector to a filtered 3.3vUse a series resistor of 50 ohms from collector to feed the coax.Use a 100 ohm series resistor from FPGA IO pin to the base of the transistor.On the receiver side, place a 50 ohm resistor to GND and a second 100 ohm resistor to the base of a second MMBTH10/MPSH10/PMBT6428.Place a 100-220 ohm pullup resistor from collector to your VCCIO.Tie the collector to your FPGA clock input.This can be bread boarded for testing.For 75ohm cables, change the 50ohms resistors to 75ohm.Any 2 ICs you use will cost significantly more than 10 cents of transistors + a few load and termination resistors.Example:https://www.digikey.com/en/products/detail/renesas-electronics-america-inc/ISL3259EIBZ/1976835You need 2, 1 on each side. You also get 100 mbaud, but you know what you get has a complete spec sheet and it is easy to reverse the direction.

author: BrianHG
author_rank: Super Contributor
timestamp: November 17, 2020, 02:16:49 am
content: You can also take a look at these uni-directional guys which work on 3.3v:Transmitter:https://www.digikey.com/en/products/detail/texas-instruments/DS90LV011ATMF-NOPB/482938Receiver:https://www.digikey.com/en/products/detail/texas-instruments/DS90LT012ATMF-NOPB/1835982Though I don't think they will be good without a shielded/coax with a twisted pair inside.The shielded HDMI cable idea mentioned might work good with them.

author: fabiodl
author_rank: Frequent Contributor
timestamp: November 20, 2020, 10:28:13 am
content: Quote from: ejeffrey on November 13, 2020, 06:28:03 pmThank you, indeed allowing one bounce is ok with my timing constraints!Quote from: BrianHG on November 17, 2020, 01:58:34 amThank you for the wonderfully exhaustive reply!I attach a schematic, did I get it right?top left circuit is the transmitter, top right circuit is the receiver, bottom left is a push-pull version of the transmitter.I am concerned about the base resistor. (3.3-0.7)/100 is 26mA. The fpga is a Cyclone V.Page 19 of the datasheetreports currents in the order of 4mA (absolute maximum ratings is +40,-25 though).Quote from: BrianHG on November 17, 2020, 02:16:49 amI am building just one of this circuits, and making it work at the first attempt has the highest priority, so using an off the shelf chip would be totally ok. The same for the cable, I can buy a cable specifically for this.I browsed digikey, and cables are divided in two sections,coaxandmultiple conductors cables. Coax specify the impedance, but twisted pairs do not report it (sorry my complete ignorance in regards to transmission lines)Thedatasheet of the receivesreports a 100ohm resistor across the wires of the pair. If I buy a cable specifically for this, what should I get?