highspeed data transfer newb  - Page 1
Forum section: electronics / projects

author: qwaarjet
author_rank: Regular Contributor
timestamp: March 03, 2017, 07:00:29 am
content: Currently in my work I use a large number of COTS DAQ solutions and SCADA systems. However COTS hardware is staring to become limiting in number ways mainly related to channels density, form factor and rapid reconfigurability. So Im looking into the feasibility building a custom solution. I am comfortable with building the analog front end and even digitizing a few in the 100KHz rates however with the large number of channels I'm looking at the sheer volume of raw data ~500Mbps-2Gbps I'm trying to figure out the best way to transfer that to a standard server. First though was a gigabit Ethernet but that has fair amount of overhead and is not very deterministic. What I really want is a way of doing DMA to the the server from my custom hardware but without having to design a custom physical layer also preferably it is a open or at least a industry standard. Any recommendation on what to use for physical layer.

author: Cervisia
author_rank: Regular Contributor
timestamp: March 03, 2017, 08:43:23 am
content: TI has an application reportComparing Bus Solutions:Quote

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: March 03, 2017, 08:57:02 am
content: Fibre ethernet, or Sata perhaps ?For that amount of data, maybe a local hard disk as a buffer, then gbit ethernet.Presumably you're not acquiring that amount of data continuously?

author: qwaarjet
author_rank: Regular Contributor
timestamp: March 03, 2017, 03:15:23 pm
content: Quote from: mikeselectricstuff on March 03, 2017, 08:57:02 amno but it could be several hoursQuote from: mikeselectricstuff on March 03, 2017, 08:57:02 amHave not ruled this out but doing so would require considerable change to Software infrastructure since at least some of that data is needed real timeQuote from: Cervisia on March 03, 2017, 08:43:23 amthanks this is kinda what I wanted to see i nice concise way of comparing things.

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: March 03, 2017, 04:21:27 pm
content: Quote from: qwaarjet on March 03, 2017, 03:15:23 pmIn which case storing it may be at least as big a problem as transferring it

author: CM800
author_rank: Frequent Contributor
timestamp: March 03, 2017, 04:31:28 pm
content: Sounds to me like a solution like this would be best:Honestly, not an expert here, I know more about systems level then down at the bare bones (I come up with machine level solutions and generally pass the programming / board dev to outsourcing companies in these kind of projects)EDIT:http://www.terasic.com.tw/cgi-bin/page/archive.pl?Language=English&CategoryNo=158&No=858SFP+, SATA 3.0, PCIe and I/O

author: Marco
author_rank: Super Contributor
timestamp: March 03, 2017, 06:01:36 pm
content: Quote from: qwaarjet on March 03, 2017, 07:00:29 amSo what you're saying is you want some kind of Remote DMA?Put a RoCEv2 Ethernet controller on the server, use a software implementation on the clients?

author: DaJMasta
author_rank: Super Contributor
timestamp: March 03, 2017, 08:57:12 pm
content: From what test equipment seems to have done (at least in the past), treat it sort of like a disk drive and use SATA or PCIe. Both have a lot of throughput and are designed for sustained, low overhead transfers, and both already have established standards and protocol for peripheral devices - so the way they're designed makes it easier to treat your end of the device modularly and prevents you from really special hardware or software on the interface server's side.Don't know if it's appropriate for your usage, but what about something like NI's PXI series mainframe style devices? They can integrate many modules worth of acquisition devices and have already done the hardware and software to control much of it together - probably not an unreasonable price if it seems like they fit your needs and the development of your own system seems too pricey or difficult.

author: Someone
author_rank: Super Contributor
timestamp: March 03, 2017, 09:41:21 pm
content: You're going to need to add a lot more context for anyone to make sense of this, first off the software to trawl through such large data sets will be as complex as the hardware. I've worked in teams on many systems in this scale, there are two broad approachesSync everything at the front end (typically an FPGA or distributed set of FPGAs) and send a continuous realtime stream to the computer, interleaving the data onto some high bandwidth bus. Firewire/thunderbolt are convenient with their DMA access for instance, but its equally possible with a PCIe card. You could try leveraging some sort of existing data format and hardware but this will cause huge problems in trying to unpack the data in realtime.Timestamp chunks of data and push them to a server over standard IP infrastructure, this is the sort of workload they are intended for and it works well and can be scaled by adding more servers. Now the data sources can be cheap microcontrollers and the hard work is pushed off to COTS hardware. Decide if realtime or peak bandwidth is more important (TCP vs UDP etc) and go from there, but you'll want a dedicated network setup thats been designed for these workloads. It aint gonna be cheap.

author: nctnico
author_rank: Super Contributor
timestamp: March 03, 2017, 10:26:16 pm
content: Quote from: qwaarjet on March 03, 2017, 07:00:29 amYou are not the first with this requirement. One of the projects I'm involved in is a distributed data acquisition system. All nodes are synchronised within a few ps using CERN's White Rabbit time synchronisation FPGA platform and fibre optic network switches which support White Rabbit. Data (timestamped) is send to the server using UDP (with the ability to refetch missing data) and there is also a realtime monitoring possibility in the product I'm working on. All in all it is doable but even with White Rabbit as a base it is quite a lot of work.BTW it is pretty cool to sample the same (few MHz) signal with two independant acquisition units and get an exact overlay using the timestamps.

author: qwaarjet
author_rank: Regular Contributor
timestamp: March 04, 2017, 04:18:43 am
content: Thanks for all the feedback.