Distributed data acquisition system synchronisation - Page 1
Forum section: electronics / projects

author: jars121
author_rank: Regular Contributor
timestamp: October 09, 2023, 02:52:52 am
content: Hi all,I'm contemplating a distributed data acquisition system design, whereby N number of standalone data acquisition system (DAS) modules are connected via ethernet to a computer. The computer serves as the system 'master', and performs post-processing on the data collected by each of the distributed DAS modules.Each DAS module is based upon an SoC FPGA; the Programmable Logic (PL) is responsible for high-throughput, low-latency signal acquisition and measurement and maintaining the device's internal high resolution timestamp, and the Processing System (PS) is responsible for device configuration management, initial signal processing and packaging, interfacing with external control systems and communicating with the computer 'master'. I was unsure as to whether an SoC approach would be most suited for this application, but have arrived at this solution (for the time being) following some very insightful input in this threadhere.Each of the DAS modules would be the same, and would have the following high-level structure:As per the above image, the high-throughput, low-latency circuitry (ADC, DI, TC, etc.) interfaces with the FPGA, with external control module interfacing (CAN, RS-232/RS-485, etc.) and device configuration interfacing with the CPU. Ethernet 1 is the designated ethernet interface for communication with the computer master; ethernet 2 is intended for industrial ethernet communication with external control modules. Data from the FPGA will be passed to the CPU for initial processing and packaging for transmission to the master via the available internal 32/64-bit interfaces (e.g. AXI4).T1 is the internal timestamp/time domain pertaining to DAS 1, with T2 corresponding to the internal timestamp/time domain of DAS 2 and so forth up until Tn for DAS N. The timestamp functionality will be common across all DAS modules.Below is an indicative representation of the system arrangement with 4 DAS modules, each connected to the computer master via their Ethernet 1 ports:The concept I'm trying to understand at the moment is synchronisation. In order to support successful correlation of the acquired data across the distributed acquisition systems, the internal timestamps for each of the DAS modules (i.e. T1, T2, T3 and T4 in the above example) need to be synchronised with each other. From the research I've done, I believe the proposed ethernet approach could facilitate this synchronisation (with sub-microsecond accuracy) via the IEEE1588 PTP protocol. I could see this working as follows:As each DAS module comes online, it's internal 64-bit timestamp starts counting.The DAS module is configured as a PTP slave, and communicates with the computer master which is configured as the PTP grandmaster.The computer master determines the offsets, etc. via the PTP communication to establish clock synchronisation between the DAS module and the computer master.The DAS module sends its internal 64-bit timestamp value along with its data payload to the computer master, where the computer master converts the internal timestamp into the synchronised 'PTP domain' such that the data from each of the DAS modules is aligned to the same time reference/domain.Does that make sense? Is that a reasonable approach? An alternative option I've pondered is waiting for all remote devices to be connected and PTP synchronisation established with the master, and then sending a 'START_TIMESTAMP' signal from the master to each of the remote devices to start their internal timestamps at the same time. If this approach was reliable/accurate then the internal timestamps wouldn't need to be sent with each data packet from each remote device to the master, as synchronisation of each of the device's internal timestamps would be guaranteed.I had planned on using a Zynq 7000 SoC for this project, but upon further reading of the Xilinx documentation, as well as some forum questions/comments from users and Xilinx staff it appears that PTP functionality in the Zynq 7000 series SoC doesn't actually work? I've seen one or two mentions that PTP slave functionality does work, but most mentions of IEEE1588 are for ZynqMP and Versal only.With that said, it looks like the PolarFire SoC supports PTP; any comments or recommendations as to a suitable SoC platform/family would also be appreciated!Thanks!

author: jbb
author_rank: Super Contributor
timestamp: October 09, 2023, 05:04:41 am
content: I think PTP is for this kind of thing, yes. As I recall, you need an Ethernet switch with PTP support built in. And I suspect that the PC may not be an ideal master clock source - perhaps the Ethernet switch can be the master time source.I might by overkill, but I think Cern's White Rabbithttps://ohwr.org/project/white-rabbitwas built for this kind of job.

author: jars121
author_rank: Regular Contributor
timestamp: October 09, 2023, 05:45:13 am
content: Quote from: jbb on October 09, 2023, 05:04:41 amAmazing, thank you very much, I hadn't come across the White Rabbit project in my research. That looks like it achieves exactly what I'm after, but perhaps at another level of accuracy (as you've rightly pointed out).I would indeed be looking at using a PTP-capable switch as the PTP master. Given that the White Rabbit approach is built off IEEE15881 PTP, I think I'm on the right track. I just need to understand how to actually implement PTP in the CPU of the SoC, and which SoC actually supports PTP. As mentioned above I had more or less landed on the Zynq 7000 series but I really don't trust the PTP slave actually works on that SoC.

author: Berni
author_rank: Super Contributor
timestamp: October 09, 2023, 06:05:45 am
content: PTP is indeed the perfect fit.But as it was mentioned above. The entire chain must support PTP in order for this to work. You need a PTP switch and have PTP support in each node. This is not just a software feature, the network interface itself has to have hardware support for PTP.This is because the precision PTP works at is really really high. The delay of the packet being routed trough the switch could wreck things. The time the CPU takes to service an interrupt from a network interface is also too slow. So for this reason the network interface itself has to timestamp the packet.Safe to say this is a fair bit of complexity. So unless your data acquisition nodes are going to be distributed around a building or something, then it is easier to just run a common sync signal to all the nodes over coax. When it is all inside one rack, it is not hard to make all the coax cables equal lengths to not have to deal with signal time of flight delays (if you need accuracy within <10ns).

author: jars121
author_rank: Regular Contributor
timestamp: October 09, 2023, 06:33:17 am
content: Quote from: Berni on October 09, 2023, 06:05:45 amI had actually thought about some sort of 1PPS or similar signal to each node, but the nodes will indeed be distributed over a significant distance. I have end-to-end control over the architecture, so ensuring PTP hardware capability at each layer/interface is doable. I don't have any FPGA/SoC experience outside of Xilinx, so I still need to determine a suitable PolarFire, Cyclone, etc. option to achieve this.

author: Berni
author_rank: Super Contributor
timestamp: October 09, 2023, 07:37:39 am
content: Yeah then PTP has some major benefits for your application.Tho if you are going to be specific about the switch, you could build your own 'switch' and run 100Mbit(2 pairs) or 1Gbit single pair ethernet to free up some pins on a RJ45 connector for standalone sync signals.This also allows for the communication to the nodes to be simplified as you no longer have to run a full TCP/IP stack on each node. You can drop down to just layer 2 ethernet and talk exclusively using MAC addresses. This is enough to make it trough off the shelf Ethernet switch chips. Potentially removing the need to have a FPGA+SoC , moving down to just a plain FPGA directly pumping data into a Ethernet PHY. Then have the switch have a SoC that forwards the data onto a real LAN network with full TCP/IP

author: jars121
author_rank: Regular Contributor
timestamp: October 09, 2023, 08:17:26 am
content: Quote from: Berni on October 09, 2023, 07:37:39 amVery interesting idea, which I had not considered. Unfortunately however I had planned (and I should have stated this, my apologies) on using SFP optical, as the distance from a remote device to the switch could exceed 100m and will be going through some very electrically noisy environments.

author: jbb
author_rank: Super Contributor
timestamp: October 09, 2023, 08:27:03 am
content: Quote from: Berni on October 09, 2023, 06:05:45 amIt’s been a while since I looked at the docs, but I think White Rabbit went an extra step (or two, or three…) on top of PTP by:- using the high quality master clock to drive the master Ethernet PHY transmitter-frequency lockingoscillators in each mode to the incoming Ethernet bits, and thence to the master clock- doing the whole PTP thing- comparing not just the delay in bits for PTP traffic to go out and back, but also measuring fractional bit delays with a very high performance phase comparison units in FPGAs

author: jars121
author_rank: Regular Contributor
timestamp: October 09, 2023, 08:31:59 am
content: Quote from: jbb on October 09, 2023, 08:27:03 amI've just read through a lot of White Rabbit documentation and you are indeed correct; they start with PTP and then add several additional layers of phase adjustment and synchronicity on top. That level of synchronisation is definitely beyond what I'm after, but very interesting to read about and keep in mind.

author: SuntUnMorcov
author_rank: Contributor
timestamp: October 27, 2023, 12:02:35 pm
content: Hello jars121,What you're describing will take you a lot of time, effort, resources, and learning. But depending on what your intentions for this project are, it should not be impossible to put together a proof-of-concept system based on what you have here. And it will be a very interesting project too.I have implemented exactly what you're describing here for deployments with several hundred distributed DAS modules on an Ethernet backbone with device sampling + timestamping synchronised using IEEE 1588v2 and CERN's White Rabbit extensions. The deployment consists of many front-end devices, data recorders, real-time visualisation, supervisory control, etc. Several million samples per second are exchanged between devices on the deployment every second.I am also one of the original designers of the iDDS protocol (seehttps://www.ni.com/en/solutions/aerospace-defense/idds-overview.html) that allows for the loosely-coupled communication of real-time data between different nodes, as well as their configuration and operational management / control, within such a system architecture.Today I lead the implementation of time synchronisation functionality for a major automotive manufacturer, where I am responsible for the implementation of the vehicle architecture's gPTP Grand Master and associated functionality, etc.But basically, drop me a line or a DM if you'd like to discuss this project. My time is limited, but I'd be happy to give you some pointers.- Jack