# Real-Time Volumetric Ultrasound Research Platform with 1024 Parallel Transmit and Receive Channels
## 2 Materials and Methods
### Multi-Channel Electronics
#### Main Board  
The main board is the central control unit of each 256-channel sub-system. It integrates a high-performance Virtex-6 FPGA with internal MircroBlaze soft processor, which is responsible for a wide range of tasks. The FPGA handles communication with the PC; receives, interprets, acknowledges and manages incoming commands; and parameterizes the transmit and receive circuits on the front-end boards accordingly. In the receiving phase, the digitized data of all 16 front-end boards are parallelized in the FPGA of the main board, synchronized, buffered and then transferred to the PC at high transmission speeds. For this purpose, a PCIe interface of the 2 nd generation with 8 lanes per sub-system was implemented, with which it is possible to realize transfer rates of up to 25 Gbit/s per 256-channel sub-system. In total, the data of the four sub-systems are connected with an overall bandwidth of 100 Gbit/s to a single PC, which computes the actual signal processing, reconstruction and visualization. Parallel to the PCIe interface for general communication and data transfer, a bidirectional serial interface was also implemented, which can be optionally used for debugging or for service purposes for controlling a sub-system. In order to be able to ensure communication between the individual front-end FPGAs and the main board FPGA, a custom bidirectional data bus with a special transmission format was developed. Each front-end FPGA is connected to the FPGA of the main board via a total of 5 LVDS line pairs. One of these line pairs serves as a so-called CMD bus, through which commands are transferred from the MicroBlaze (running at 100 MHz) of the main board to the FPGAs of the front-end boards. At the same time the bus is used to synchronize the transmit/receive phases, controlled by the main board FPGA. In the opposite direction, four LVDS line pairs per front-end board are used as a DATA bus, in order to be able to transport not only status information but also above all the digitized data of the 16 receive channels to the main board FPGA at high transfer rates. Physically, a total of 16 slots are provided on the main board, into which the front-end boards are plugged via card-edge connection. When using 16-channel front-end boards, up to 256 parallel send and receive channels are addressed per sub-system. In addition to the parameterization of the transmit and receive characteristics and the data management on the receive side, the main board is also responsible for the synchronization of the 256 parallel channels of the sub-system. A high-precision clock network was set up for this purpose. When digitizing ultrasound signals, the choice of a suitable clock is essential in order not to affect the performance of the receiver's A/D converters. To ensure a high resolution, the applied clock must have only small phase variations. However, clocks generated by an FPGA are generally subject to comparatively high phase fluctuations. Therefore, two 14-channel clock generators of the type AD9523 from the manufacturer Analog Devices are used on the main board, which have an extremely low broadband jitter < 100 fs; i.e., they can generate very clean clocks. Each of the two clock generators integrates two PLLs and a clock distributor. To reduce phase noise and thus improve the signal-to-noise ratio at the A/D converter, an external, frequency-stable, voltage-controlled crystal oscillator (VCXO) is used to generate a clock of 80 MHz. A reference frequency of 20 MHz is applied to the input of the clock generators, which is generated by a VCO and fed via a clock distributor to the two clock generators and as a system clock to the FPGA of the main board. Both PLLs generate eight in-phase 80 MHz clocks each, synchronized to the input clock, which are routed via LVDS lines to the individual A/D converters on the front-end boards (Figure 3). In the layout, conscious care was taken to ensure that the signal line lengths to the individual front-end interfaces are practically identical.