Content from this work may be used under the terms of the CC-BY-4.0 licence (© 2023). Any distribution of this work must maintain attribution to the author(s), title of the work, publisher, and DOI

12th Int. Beam Instrum. Conf.
ISBN: 9 7 8 - 3 - 9 5 4 5 0 - 2 3 6 - 3

IBIC2023, Saskatoon, Canada
JACoW Publishing
ISSN: 2 6 7 3 - 5 3 5 0d o i : 1 0 . 1 8 4 2 9 / J A C o W - I B I C 2 0 2 3 - W E 2 C 0 3

BEAM INSTRUMENTATION HARDWARE ARCHITECTURE FORUPGRADES AT THE BNL COLLIDER-ACCELERATOR COMPLEX AND
THE FUTURE ELECTRON ION COLLIDER*R. Michnoff†, C. Degen, L. DeSanto, D. Gassner, S. Hafeez, R. Hulsart, J. Jamilkowski, J. Mead,K. Mernick, G. Narayan, P. Oddo, M. Paniccia, J. Pomaro, A. Pramberger, J. Renta, F. Severino
Brookhaven National Laboratory, Upton, NY, USA
Abstract
Many beam instrumentation systems at Brookhaven National Laboratory’s Collider-Accelerator complex are over
20 years old and in need of upgrading due to obsolete components, old technology and the desire to provide improved
performance and enhanced capabilities. In addition, many
new beam instrumentation systems will be developed forthe future Electron Ion Collider (EIC) that will be housedin the existing Relativistic Heavy Ion Collider (RHIC) tunnel. A new BNL designed custom hardware architecture isplanned for both upgrades in the existing facility and newsystems for the EIC. A general-purpose carrier board basedon the Xilinx Zynq Ultrascale+ System-on-Chip (SoC) willinterface with a family of application specific daughter
cards to satisfy the requirements for each system. This paper will present the general architecture that is planned, aswell as details for some of the application specific daughter
cards that will be developed.

EXISTING FACILITY HARDWARE
When RHIC was commissioned in 2000, the majority of
instrumentation systems were based on the VME platform.
While this has worked very well over the years, options for
system upgrades and new systems for the EIC have been
carefully considered [1].
A custom-designed Beam Position Monitor (BPM) module (named V301) was developed in 2015 that uses theVME form factor but does not include a VME bus interface [2]. Each module instead includes a direct Ethernet
communication interface for higher level control and general data transfer. The V301 is based on the Xilinx Zynq
gate array. Four 400 MSPS ADC converters are provided
for measurement of BPM pickup signals and onboard filter
options are included to process electrons or ions with various bunch repetition rates. The on-board Zynq ARM processor runs Linux with embedded software for processing
and communication to higher level systems. Beam synchronous clocks are received by a single board in the chassis and distributed via unused P2 pins to all of the V301
modules. One major benefit of this system is that up to 15
V301 boards can be housed in a single VME chassis, andmodule replacement is very simple when necessary.
Another VME form factor module (named V340) basedon the Zynq Ultrascale+ SoC was also recently developed.
This module includes a high pin count FMC connector toenable using both custom and 3rd party FMC modules. TheV340 is planned to be used for current transformer data acquisition and processing, as well as other applications.

In 2009 the RHIC RF group developed a Xilinx Virtex5 based carrier board with custom daughter cards specificfor RF applications [3]. This system is housed in a custom
developed 3U chassis and runs the VxWorks real-time operating system directly on the Virtex 5 gate array.

OVERVIEW OF THE EIC
The EIC is scheduled to be fully commissioned withbeam in 2034 and consists of the following major subsystems as shown in Fig. 1:
 Existing Hadron injector systems
 Hadron Storage Ring (HSR) Electron Pre-Injector (Polarized electron source andLinac) Rapid Cycling Synchrotron (electron accelerator)(RCS) Electron Storage Ring (ESR)
The existing Collider-Accelerator complex injector systems - Tandem, Electron Beam Ion Source (EBIS), Linacwith polarized proton source, booster and Alternating Gradient Synchrotron (AGS) - will continue to be used for injection into the new HSR. Therefore, the existing systems
need to be maintained and/or upgraded to support the EIC
operation. The HSR will reuse significant portions of theexisting Relativistic Heavy Ion Collider (RHIC) machine
which is scheduled to complete its final beam run in 2025.
The electron pre-injector, RCS and ESR will be completelynew machines.

Figure 1: Layout of EIC.
Figure 2 lists the new instrumentation systems planned
to be developed for the EIC machine. Note that this list
does not include systems that are planned to be upgraded
for the existing hadron injector systems.
___________________________________________* Work supported by Brookhaven Science Associates, LLC underContract No. DE-SC0012704 with the U.S. Department of Energy† michnoff@bnl.gov

WE2C03308

09 Data Acquisition and Processing Platforms

IBIC2023, Saskatoon, CanadaJACoW Publishing
ISSN: 2 6 7 3 - 5 3 5 0d o i : 1 0 . 1 8 4 2 9 / J A C o W - I B I C 2 0 2 3 - W E 2 C 0 3

Figure 2: EIC instrumentation list for each subsystem, excluding the existing hadron injector machines.

OVERVIEW OF COMMON PLATFORM
The knowledge gained from the RF developed Virtex-5
based system, the Instrumentation System V301 Zynqbased BPM module and the V340 Ultrascale+ based module has been instrumental in the evolution to the new Xilinx
Zynq Ultrascale+ SoC based carrier module that will accommodate 2 daughter cards via custom-defined pinouts
on high-speed connectors. A mockup of the system is provided in Fig. 3.

The major features include: Xilinx Zynq Ultrascale+ SoC
 4 GB DDR4 memory
 4 120-pin connectors to support 2 daughter sites
 8 SFP modules with crosspoint switch for configurable routing to on-board Ultrascale+ or daughter sites
 Dual RJ-45 Ethernet ports
 RJ-45 connector for RS-232 serial console port
 16 Digital I/O LEMO connectors, 4 inputs, 4 outputs,
8 bidirectional
 External clock input passed to the daughter sites
 System clock input with PLL for generating beam
synchronous clocks

Daughter Cards

Figure 3: Mockup of common platform chassis with carrier
module and one of two daughter cards installed.

Carrier Module
A block diagram of the common platform carrier moduleis provided in Fig. 4. The external dimensions of the module are 15.5" wide by 10" deep, with a 3" x 6" cut-out forthe power supply module.

A suite of daughter cards is in the process of being developed to support controls, RF, beam instrumentation andother applications.
Each daughter card will include a gate array for local
control and for communication and data transfer to/fromthe carrier module. A memory mapped interface will be
implemented between the carrier and daughter modules using AXI (Advanced eXtensible Interface) buses bridged bya Xilinx AXI Chip2Chip core over a Xilinx Aurora serial
link. For large buffers of data, dedicated DDR memory
may be provided on the daughter card and transferred tohigher level workstations either by the carrier module after
transferring the data from the daughter card to the carrier
module memory or directly through a dedicated Ethernet
connection on the daughter card.
The daughter cards slide into the front of the chassis withrails to guide the board to the carrier module connectors.
WE2C0309 Data Acquisition and Processing Platforms

309

Content from this work may be used under the terms of the CC-BY-4.0 licence (© 2023). Any distribution of this work must maintain attribution to the author(s), title of the work, publisher, and DOI

12th Int. Beam Instrum. Conf.
ISBN: 9 7 8 - 3 - 9 5 4 5 0 - 2 3 6 - 3

Content from this work may be used under the terms of the CC-BY-4.0 licence (© 2023). Any distribution of this work must maintain attribution to the author(s), title of the work, publisher, and DOI

12th Int. Beam Instrum. Conf.
ISBN: 9 7 8 - 3 - 9 5 4 5 0 - 2 3 6 - 3

IBIC2023, Saskatoon, Canada
JACoW Publishing
ISSN: 2 6 7 3 - 5 3 5 0d o i : 1 0 . 1 8 4 2 9 / J A C o W - I B I C 2 0 2 3 - W E 2 C 0 3Figure 4: Block diagram of common platform carrier board.
The carrier module will provide +3.3 V@8 A,+12 V@6 A and -12 V@2 A to each daughter card. TwoSamtec ERM8-060-01-L-D-EM2 connectors on eachdaughter card will be used for communication between thecarrier module and the daughter card.

in height. Some assemblies may require a larger height toaccommodate the necessary panel space for connectors.
For example, a 32-channel ADC daughter card with 32
BNC connectors, as planned for the beam loss monitor system, will require a height of more than 2U.

Signal Conditioner Modules

Cooling

Some daughter cards will be designed to accommodate
a signal conditioner module that can include buffers, filters, etc. for application specific needs. The signal conditioner module can also be used to provide application specific connector needs. For example, the multi-wire profile
monitor system will likely use multi-pin connectors while
the loss monitor system might use BNC connectors.
Providing this capability within a simple signal conditioner
module prevents the need to design several similar ADC
daughter cards.

As shown in Fig. 3, several small fans are planned to be
used to provide cooling across the carrier module anddaughter cards.

Power Supply
A dedicated power supply for each common platform
chassis will be provided. The exact details for this power
supply are still being developed.

ChassisSYSTEMS TO BE SUPPORTED
The beam instrumentation systems that are planned to besupported by the common platform hardware include thefollowing: Beam position monitor systems (BPM) Beam Loss monitor systems (BLM) Multiwire beam profile systems
 Beam current and charge monitoring systems
 Ionization Profile Monitor systems (IPM) Tune measurement systems
 Transverse injection damper systems
 And more

Each common platform chassis will include one power
supply, one carrier module and up to 2 daughter cards. Thechassis will be 19" rack mountable and will typically be 2U
WE2C03310

09 Data Acquisition and Processing Platforms

IBIC2023, Saskatoon, Canada
JACoW Publishing
ISSN: 2 6 7 3 - 5 3 5 0d o i : 1 0 . 1 8 4 2 9 / J A C o W - I B I C 2 0 2 3 - W E 2 C 0 3

The following sections provide preliminary system architecture details related to how the primary instrumentation systems will be supported using the common platform
hardware.

Beam Position Monitors
A block diagram of the BPM daughter card is shown inFig. 5. The EIC BPM system requirements require different filters and different data acquisition modes for each of
the major subsystems. A common filter footprint will be
designed to accommodate the different filter requirements
using one common printed circuit board (PCB).

Figure 6: Block diagram of 32-chan 5 MSPS daughter card.

Beam Current and Charge Monitor Systems

Figure 5: Block diagram of BPM daughter card.
The AD9690-500, 500 MSPS, 14-bit ADC is presently
being considered for the BPM system, but further evaluation is required before making a final decision. This ADC
uses the high speed JESD204B serial link (10 Gbit/s) fordata transfer to the Ultrascale+ SoC.
The direct Ethernet connection to the Xilinx Zynq Ultrascale+ SoC will provide options for data to be sent directly to higher level workstations, bypassing the FPGA on
the carrier module. Running Linux or other operating system directly on the on-board SoC is also an option, which
would allow an EPICS IOC or RHIC Controls ADO to reside on the daughter card.

For DC beam current measurements, DC Current Transformers (DCCT) and Faraday Cups will use a high-precision Analog Input/Output daughter card (Fig. 7). Thedaughter card will include low noise high bit count ADC
and DAC components. The DAC output will generate aremotely configurable current source for the calibration of
the DCCTs and other current transformers. This daughtercard is still in the early design phase, and the specific ADC
and DAC still need to be selected.
For bunch charge measurements (Wall Current Monitors(WCM), Integrating Beam Current Transformers (ICT),and Fast Current Transformers (FCT), etc.) another custom
daughter card similar to the BPM module will be developed, but with DC coupled inputs and 1 GSPS or fasterADCs. This module is also in the very early design phase.

Tune Measurement and Transverse
Injection Damper Systems
The tune measurement and transverse injection dampers
systems will use the BPM daughter card, but with custom
firmware to synchronize beam kicks and measurements.
The present plan is to provide beam kicks to a stripline
via high voltage Behlke switches or similar.

Multiwire Profiles, Beam Loss Monitors,and Ionization Profile Monitor Systems
A general purpose 32-channel, 16-bit 5 MSPS analog input daughter card is presently being developed to accommodate a variety of system needs, including multiwire
beam profiles, BLM and IPM systems (Fig. 6).
Each system will include custom firmware for the specific application, as well as a custom designed signal conditioner module to provide the specific front-end filter andexternal connector needs.

Figure 7: Block diagram of precision ADC/DAC daughter
card.

REFERENCES[1] R. J. Michnoff et al., “Preliminary Design of a Real-Time
Hardware Architecture for eRHIC”, in Proc. ICALEPCS'15,
Melbourne, Australia, Oct. 2015, pp. 1099-1102.
doi:10.18429/JACoW-ICALEPCS2015-THHB2O01[2] R. L. Hulsart, P. Cerniglia, N. M. Day, R. J. Michnoff, and Z.
Sorrell, “A Versatile BPM Signal Processing System Basedon the Xilinx Zynq SoC”, in Proc. IBIC'16, Barcelona, Spain,
Sep. 2016, pp. 646-649.
doi:10.18429/JACoW-IBIC2016-WEPG12[3] T. Hayes and K. S. Smith, “A Hardware Overview of theRHIC LLRF Platform”, in Proc. PAC'11, New York, NY,USA, Mar.-Apr. 2011, paper MOP283, pp. 645-647.

WE2C0309 Data Acquisition and Processing Platforms

311

Content from this work may be used under the terms of the CC-BY-4.0 licence (© 2023). Any distribution of this work must maintain attribution to the author(s), title of the work, publisher, and DOI

12th Int. Beam Instrum. Conf.
ISBN: 9 7 8 - 3 - 9 5 4 5 0 - 2 3 6 - 3