How to connect multiple FPGAs? Common bus? - Page 1
Forum section: electronics / general

author: polcad
author_rank: Contributor
timestamp: April 11, 2025, 01:53:34 pm
content: Hi, I have to build an analogue aquisition system that consists of many ADCs and an FPGA. Because there is no FPGA large enough to do the job, neither in terms of i/o pins or logic gates, I plan it as a number of channels, each composet of an ADC (100 msps, 16 bit) + a medium size FPGA (Artix 7A50T, Zynq 7020, etc) for each aquisition channel and then an FPGA (probably Zynq 7020) to collect partially processed data from each channel and send it via RJ45 to ethernet.Each channel has to be able to exchange data with other channels and of course has to be able to send tha data to the comon FPGA. Something like a PCI bus.So my question is, what kind of bus should it be? I know, one could just implement PCI like bus, but I do not want to waste the logic resources to implement a PCI bus o something like that. I am looking for efficient and economical in terms of logic resources solution.Can someone point me to a possible solution?

author: xvr
author_rank: Frequent Contributor
timestamp: April 11, 2025, 04:02:28 pm
content: Parallel bus will be slow one. If you want something fast you have to use serial point-to-point connection. Xilinx FPGA offer SERDES and Gigabit transivers for this task. SERDES is a simpler and slower (but not so critical to PCB layout and signal integrity as Gigabit transivers). Both of them supported by dedicated HW in FPGA.But for both of them you have to implement hub (in cental FPGA for example) or mesh connection between FPGAs (if you need to pass data not only to main FPGA but and between them)

author: Gleeson
author_rank: Contributor
timestamp: April 11, 2025, 07:50:13 pm
content: What is the required data rate for the communication?

author: asmi
author_rank: Super Contributor
timestamp: April 11, 2025, 08:07:05 pm
content: Even a single ADC channel 16bit@100Msps gives bandwidth requirement of 1.6 Gbps, so the only viable connectivity options are multi-lane LVDS @1.25Gbps each lane, or MGTs (Artix can do up to 6.6 Gbps, but watch out for speed grade and package - some of them have lower limit!). As far as protocol, something like Aurora 8b/10b should be good, alternatively you can roll out your own protocol - just make sure you include error correction capability, because errors over multi-gigabit links can happen quite frequently - for example, error probability of "only" 1 per billion over 6 Gbps link means that an average of 6 errors occur every second!

author: nctnico
author_rank: Super Contributor
timestamp: April 11, 2025, 10:10:33 pm
content: What is missing from the spec is whether the data rate needs to be continuous or in burst. In the latter case each FPGA can use a buffer (in the form of DDR memory if necessary) and dump the data through a network interface. I have build something like that in a distributed form where data is acquired for a specific time period and send over a network when the acquisition is done. That system uses ethernet switches (special ones which include White Rabbit time synchronisation) to transfer data from the acquisition units to a central computer.One a single board, an easy way to implement data transfer is by having ethernet MAC (one of the RMI/MII variants) interfaces sitting between the acquisition FPGA and main FPGA. In case the main FPGA doesn't have enough pins for all the MAC interfaces, an extra FPGA or switch chip with MAC interfaces can be used to manage the data streams. The UDP protocol is easy enough to implement in an FPGA. In my design I used a softcore to add / keep track of packet sequencing (to detect missing data at the receiving end) and DMA to copy the contents from memory into the ethernet frame transmit buffer. If you have full control over the MAC interfaces, you can use extra data pins to increase bandwidth.

author: Someone
author_rank: Super Contributor
timestamp: April 11, 2025, 11:48:53 pm
content: Quote from: nctnico on April 11, 2025, 10:10:33 pmTotal number of channels that need to be aggregated.If there isn't an FPGA with enough IO bandwidth to connect the ADCs, then there isn't enough bandwidth to connect the ADCs regardless of what Rube Goldberg machine is placed between them.

author: polcad
author_rank: Contributor
timestamp: April 12, 2025, 07:08:29 am
content: Quote from: Gleeson on April 11, 2025, 07:50:13 pmEach channel is 2x an ADC 16 bit , 120 Msps and all that sampling data is pre-processed by the fpga in each channel and then from each channel has to go to the common FPGA.The data from at least one channel has to go in real time to the common fpga, the data from other channels can go in bursts.Even that can be a challenge so that's why I plan that each channel will have an fpga servicing two ADCs and that fpga will do some signal pre-processing in order to reduce the data that has to go to the common fpga. The common fpga needs the data from all channels but it does not have to be in real time, but the data has to be coming from the same time frame, coherent sampling and data processing (only one channel in real time is enough in order to stream it to ethernet, after some decimation and signal processing).For real time processing of each channel is the first fpga in each channel. I need upto 9 ADCs sampling synchronously, so 4 channels , 2 ADCs each.Quote from: nctnico on April 11, 2025, 07:50:13 pm@nctnico Which WR switch exactly did you use? And why did you have to use it? Could you not do time tagging in your fpga? I am trying to understand why it had to be WR switch.>> an easy way to implement data transfer is by having ethernet MACThat is somewhat the way I plan to implement it if I cannot assemble it as a single board with a common "PCI like" bus.Channel = (2x ADC +FPGA) + FPGA with ethernet.And then assemble 5 channels to a single ethernet switch. Not sure whether WR switch is strictly required here if I can cannect a common sampling clock to all the channels. But I plan also a "remote version" of the system where the above channels can be located kilometres apart and then either GPS synchronized clock or WR switch with clock distribution will be necessary (not sure which one would do the job, if either).If a common bus was possible I would plan it this way:Channel = (2x ADC +FPGA) + FPGA with ethernet.|| ||Channel = (2x ADC +FPGA) + FPGA with ethernet.|| ||Channel = (2x ADC +FPGA) + FPGA with ethernet.|| ||Channel = (2x ADC +FPGA) + FPGA with ethernet.|| ||Channel = (2x ADC +FPGA) + FPGA with ethernet.where: || represents a serial connection between FPGAs using one of the transceivers in FPGA.And since I plan one board per channel ((2x ADC +FPGA) + FPGA with ethernet.) I would connect the boards with a standard SATA cable.

author: voltsandjolts
author_rank: Unknown
timestamp: April 12, 2025, 08:00:25 am
content: You're likely aware, but there are off-the-shelf digitizer cards that might fit your needs, at 14-bits anyways.What's the ENOB requirement?

author: nctnico
author_rank: Super Contributor
timestamp: April 12, 2025, 08:11:35 pm
content: Quote from: polcad on April 12, 2025, 07:08:29 amThis project used the 'standard' WRS3/18 WR switch for time distribution and data aggregation. The reason for this is that no other off-the-shelve technology is capable of syncing clocks (in time) to sub-nanosecond levels. Now for 120Ms/s you might not need sub-nanosecond accuracy but GPS just isn't good enough. For the project I designed (based on the WR Spec board) I used 8ns intervals to timestamp the trigger points and let the sampleclock free run at 20MHz (50ns period). For 120Ms/s you might want to lock the sampleclock to the WR generated clock so every unit samples at exact the same time so the samples are aligned perfectly. Or use the 125MHz clock generated on the spec board directly. It is quite cool to have two seperate units with a long fiber in between and have the sampled signal from a single source overlap perfectly when processing the data of the two units.

author: Gleeson
author_rank: Contributor
timestamp: April 12, 2025, 11:38:24 pm
content: 1.6Gbps would be the data rate from the ADC to the FPGA, the protocol will be defined by the choice of ADC and not something you can otherwise choose. The question in the post is "How do I connect multiple FPGAs?". It sounds like the data gets processed in the FPGA, so the data rate between FPGAs could be much lower (or not).

author: polcad
author_rank: Contributor
timestamp: June 30, 2025, 08:01:25 pm
content: >>>>> so the data rate between FPGAs could be much lower (or not).yes, each fpga will process the date from its own ADC in order to reduce the amount of the data that will need to be exchanged between FPGAs.Quote from: voltsandjolts on April 12, 2025, 08:00:25 amIt has to be a stand alone device, no PC.I know one way of solving the problem is as I draw it in a message a few posts above and use data packets with timestamps and do the data processing on the external PC.But I would prefer to do all the math on FPGA and then only exchange the results between FPGAs and a host PC.How else would you process the data from a large number of ADCs? The ADCs can be collecting the data from a phased antenna array and I need to calculate the FFT, amplitude and phase , of each ADC versus every other ADC. This does not have happen in real time, but it has to be done pretty quickly.