BEAM INSTRUMENTATION HARDWARE
ARCHITECTURE FOR UPGRADES AT THE BNLCOLLIDER-ACCELERATOR COMPLEX AND THE
FUTURE ELECTRON ION COLLIDER
Rob Michnoff, Brookhaven National Laboratory
on behalf of the entire development team
Chris Degen, Lenny DeSanto, Dave Gassner, Syed Hafeez,Rob Hulsart, Jim Jamilkowski, Joe Mead, Kevin Mernick,Geetha Narayan, Peter Oddo, Matt Paniccia, John Pomaro,
Adam Pramberger, Julio Renta, Freddy SeverinoInternational Beam Instrumentation Conference 2023, Saskatoon Canada
September 13, 2023

Collider-AcceleratorDept.

Outline• Brookhaven National Lab (BNL) Collider-Accelerator Complex• BNL Electron Ion Collider and required beam instrumentation• Common platform architecture overview• Custom modules to support beam instrumentation systems• Summary

IBIC 2023, Saskatoon, Canada

2

BNL’s Collider-Accelerator Complex

IBIC 2023, Saskatoon, Canada

3

Layout of the Electron Ion Collider
presently being designed

Estimated EIC fullcompletion date:
2034

IBIC 2023, Saskatoon, Canada

4

EIC Beam Instrumentation
Electron Storage Ring

Instrument
Quantity
BPMs
400
BLMs
50
DCCT/FCT2
Synch Light Mon 2
X-Ray Pin-hole1
Tune Monitor1Polarimeter
1Trans BbB FB
1Long BbB FB
1Slow Orbit FB
1BBA
1Pre-Inj Linac to RCS Transport

RCS to ESR Transport

Instrument
BPMs
Prof Mon’s
Charge Mon’s

Instrument
BPMs
Prof Mon’s
Charge Mon’s

Quantity
2272Quantity
30
72BPMsBLMs
Beam Pipe Temp Mon's
Screen Prof Mon's
Current & Charge Mon's
Faraday Cups
Wire scanners
Emittance Mon’s
Halo Mon’s
Transverse Deflecting Cavity
Synch Light Mon’s
Relative Bunch Alignment

Hadron Storage Ring

Quantity
1108870351812974121HSR Injection Transport

Rapid Cycling Synchrotron

IBIC 2023, Saskatoon, Canada

Instrument
Quantity
BPMs
13
YAG/OTR Mon’s
13
Charge Mon’s
10
BLMs
4
Faraday Cup
2Bunch Length
2
Emittance Mon
1Mott Polarimeter
2

Strong Hadron Cooling
Instrument

Instrument
Quantity
BPMs
280
BLMs
220
IPM
2
Longi Prof Mon
1
Tune Monitors
2
DCCT
1
HF Schottky
1
LF Schottky
1
HT pick-up
1
Global Orb FB
1
Polarimeter
2
BbB Damper
1
Instrument
Quantity
BPMs
554
BLMs
20
DCCT/FCT
2
Profile Mon’s
7
Synch Light Mon
2
Tune Monitor
1BBA
1Electron Pre-Injector (400 MeV)Interaction Region

5

Hadron Injection
from AGSInstrument
Quantity
BPMs
78
BLMs
50
Orbit Corr at IP
2
Crab Tilt Monitor 1

Instrument
BPMs
BLMs
Profile Mon’s
Charge Mon’s

Quantity41
20
62

slide by Dave Gassner

Architecture options• There are so many architecture platforms available. Making adecision is not a simple task.
• VME, pizza box, uTCA, TCA, VPX, and so much more

• We have opted to develop a custom 2U chassis• But first… a short history of a few other developments at RHIC

IBIC 2023, Saskatoon, Canada6

RHIC low level RF controller chassis
LLRF Controller Features• Developed in 2009
• Custom 3U chassis• Direct Ethernet for interface to controlsystem• Xilinx Virtex 5 FPGA
• vxWorks running on embedded
processor in Virtex-5• RHIC ADO software resident on board• 6 XMC sites for application specificmodules• Ability to synchronize RF clocks
between multiple chassis• Successfully used at RHIC for manyyears
IBIC 2023, Saskatoon, Canada7

V301 BPM module

V301 FEATURES• Developed in 2015
• VME form factor, but no VME interface• Direct Ethernet for interface to controlsystem• Xilinx Zynq XC7Z030-3FBG676E• Linux running on embedded processor inZynq• EPICS IOC or RHIC ADO software residenton board• Supports one 4-pickup BPM per board• Beam sync clocks and timing links
distributed to all V301s in chassis fromboard in slot 1 via unused P2 pins
• Up to 15 V301s installed in VME chassis• Successfully used for RHIC, Linac, LEReC,CBETAIBIC 2023, Saskatoon, Canada

8V340 General purpose module•••••••••••IBIC 2023, Saskatoon, Canada

9V340 FEATURESDeveloped in 2022
Modeled from the V301, but with FMC
connector instead of BPM RF front-end
circuitVME form factor, but no VME interface
Direct Ethernet for interface to control
systemXilinx Zynq Ultrascale+Linux runs on embedded processor in Zynq
EPICS IOC or RHIC ADO software can beresident on board
FMC high pin count connector
Beam sync clocks and timing links
distributed to all V340s in chassis fromboard in slot 1 via unused P2 pins
Up to 15 V340s can be installed in VME
chassisOne application for the V340 is beam
current/charge measurements

Common platform overview• For EIC and upgrades at the existing injector complex, acustom common platform system is in the process of being
developed, consisting of:• Custom 2U rack mount chassis• Carrier module, with connectors for 2 daughter cards• Custom daughter cards to support application specific requirements• First prototype of the carrier module is expected to be received
in October 2023
• This talk will focus on the beam instrumentation
applications
IBIC 2023, Saskatoon, Canada

10Layout of common platform carrier
module and daughter cards
power supply

daughter card 18”
daughter card 27.5”

IBIC 2023, Saskatoon, Canada

11Common Platform
mockup chassis withcarrier board andone daughter card

Carrier module

Cooling fans

Daughter site 1

IBIC 2023, Saskatoon, Canada

Powersupply

12Daughter site 2

Common Platform
Carrier Board

Connectors fordaughter 1

Connectors fordaughter 2

IBIC 2023, Saskatoon, Canada

13External Clock

System Clock

External Clock Site 1

Clock Buffer

CommonPlatform

External Clock Site 2

Clock1 Site 1
Clock1 Site 2
Clock2 Site 1
Clock2 Site 2
Clock3 Site 1
Clock3 Site 2
Clock4 Site 2
Clock4 Site 2
Clock OutClock to PL

Clock Control-Cleanup PLL-Clock Gen-Fanout

Dual DAC

BS CLK 1 Site 1
BS CLK 2 Site 1

Beam Sync
Clock Fanout

BS CLK 1 Site 2
BS CLK 2 Site 2

I2C

Clock to Bank #1
Clock to Bank #2

EEPROMFPGA Recovered ClockXPT Site 1 GTX1FPGA

Rear Panel
Digital I/O

Clock1 Site 1
Clock2 Site 1
Clock3 Site 1
Clock4 Site 1

TDL Site 1

MAC Address

XPT Site 1 GTX2External Clock Site 1

Daughter Site 1

Axi Chip2Chip Site 1+12V@6A

Data Link1 Site 1
Data Link2 Site 1

Digital I/OHeader

Zynq UltraScale+ CG (or EG) Family

Dual (or quad) ARM A53
Dual ARM R5 real time processor
192k to 504k Logic Cells
176k to 461k Flip Flops
4.5 Mb to 11Mb Block RAM
13.5 Mb to 27 Mb Ultra RAM
728 to 1728 DSP Slices
16 GTH Transceivers (16.3 Gbps)
4 GTR (6 Gbps) dedicated to processors
204 I/O pins
900 pin package

QSPI FlashSD Card
DDR4 4GB
512Mx64
RS-232RGMII

Ethernet 1
Ethernet 2
Voltage / Current / Temp
Monitoring

I2C

SFP 1-12V@2A

32 Dedicated IOs+3.3V@6A

TDL Site 1 Tx
TDL Site 2 Tx

162 Diff Pairs

Clock1 Site 2
Clock2 Site 2Clock3 Site 2
Clock4 Site 2External Clock Site 2

TDL Site 2

XPT Site 2 GTX1XPT Site 2 GTX2

Daughter Site 2

Axi Chip2Chip Site 2
Data Link1 Site 2
Data Link2 Site 2

BS CLK 1 Site 2
BS CLK 2 Site 2+12V@6A-12V@2A

32 Dedicated IOs

RGMII

BS CLK 1 Site 1
BS CLK 2 Site 1

CarrierBoardblock
diagram+3.3V@6A

XPT Site 1 GTX1Timing
Data LinkFanout

XPT Site 1 GTX2TDL Site 1
TDL Site 2

SFP 2
SFP 3
SFP 4
SFP 5

XPT Site 2 GTX1XPT Site 2 GTX2

Cross Point Switch12 x 12

SFP 6
SFP 7
SFP 8

IBIC 2023, Saskatoon, Canada

programmableattenuator

SMA connectors from BPM 2

SMA connectors from BPM 1

Beam position monitor daughter card block diagram

IBIC 2023, Saskatoon, Canada

AD9690-500
14-bit ADC
500 MSPS

filtergain

JESD204B serial, 10 Gb/s
ADC clk

PLL

Typical 1 of 8Notes
1. A common filter footprint will be designed to accommodate different filter
requirements in terms of low-pass, band-pass and frequency.
2. For large data buffers, data will be stored in local DDR memory, thentransferred to higher level workstations via the direct Ethernetconnection.
3. Smaller data sets and register settings will use the AXI bus from thecarrier module.
4. Machine protection digital signals will use the general-purpose digital I/O
to the carrier module in conjunction with the carrier module digital I/O
LEMO connectors.
5. Specific configuration of attenuators, gain amplifiers and filters is still
under consideration.

15Clock Mux
4Local power
supplygeneration

3.3V @ 8A
12V @ 6A
-12V @ 2A

P2
tocarrier

DDRmemory

Xilinx Zynq
Ultrascale+AXIevent/data link
Gen. purpose I/O
Ethernet MGT(to SFP on carrier)
Clocks (incl. beam sync clocks)

P1
tocarrierGeneral purpose 32-chan 5MSPS per chan analog input
daughter card
application-specific front-end circuit

gain

LTC2325-16 ADC
quad, 16-bit, 5Msps/Ch
simultaneous sampling
differential inputs

gain

Application-specific connector(s) from field devices
Multi-pin, BNC, twinax, etc.

application-specific front-end circuit
application-specific front-end circuit

gain

Serial data 4
ADC clk

gain

Local power
supplygeneration

3.3V @ 8A
12V @ 6A-12V @ 2A

P2
tocarrierapplication-specific front-end circuit

Notes
1. Signal conditioner module
will be custom for eachapplication, allowing custom
filters, external connectors,etc.

Signal conditioner module
IBIC 2023, Saskatoon, Canada

Typical 1 of 8

XilinxUltrascale+Notes

1. Configurable gain will be provided
2. Module will be configurable to accommodate
differential or single-ended inputs
3. Applications for this module include:• Multi-wire profile monitors• Ionization profile monitors• Beam loss monitors
each with a custom signal conditioner boardADC daughter card
16AXI
event/data link
Gen. purpose I/OClock Mux
4Clocks (incl. beam sync clocks)

P1
tocarrier

High precision analog input/output daughter card
application-specific front-end circuit

SerialΣ-Δ ADC
24-bit

gain

Clock

Local power
supplygeneration

3.3V @ 8A
12V @ 6A
-12V @ 2A

P2
tocarrierapplication-specific front-end circuit…
application-specific front-end circuit

BuffDAC
20-bit…

Application-specific connector(s) from field devices
Multi-pin, BNC, twinax, etc.

…

…
application-specific front-end circuit

gain

BuffNotes
1. Signal conditioner module
will be custom for eachapplication, allowing custom
filters, external connectors,etc.

SerialClockXilinx
Ultrascale+Notes

AXI

1. Configurable gain will be provided
2. Configurable precision DAC source
3. Applications for this module include:• DC Current Transformers• Faraday Cup Beam Dumps
Each with a custom signal conditioner board

event/data link
Gen. purpose I/O
Clock Mux

Clocks (incl. beam sync clocks)
Signal conditioner module
IBIC 2023, Saskatoon, CanadaHigh Precision Analog I/O Daughter card
17P1tocarrierBunch charge monitor daughter card
• For bunch charge measurements (Wall Current Monitor,Integrating Beam Current Transformers, Fast CurrentTransformers, etc.) another daughter card will be developed withthe following features:• DC coupled inputs• 1 GSPS ADCs or faster• 4 channels• This module is still in the very early design phase

IBIC 2023, Saskatoon, Canada

18

The daughter card suite• The suite of daughter cards is expected to grow for applications in:• Beam instrumentation• RF• Controls• Feedback systems• And more

• Other daughter cards presently being developed include:• General purpose module with FMC interface (1 LPC, 1 HPC) toaccommodate applications with 3rd party or custom FMC boards• General purpose module with 16 SFP ports for data link and clock fanouts,
as well as other applications

IBIC 2023, Saskatoon, Canada19

Summary• The BNL Collider-Accelerator and EIC Instrumentation, Controls, RFand other groups are combining efforts to develop a custom
hardware platform with a suite of custom daughter cards to satisfy
the hardware and software requirements for new EIC systems andupgrades at the existing collider-accelerator complex.
• The first carrier prototype module is expected to be received inOctober 2023
• Several daughter card modules are in the process of being designed• If anyone is interested in joining our endeavor, please contact us

IBIC 2023, Saskatoon, Canada

20Acknowledgements
Paul Bachek
Norm Day
Chris Degen
Lenny DeSanto
Dave Gassner
Syed Hafeez
Kayla Hernandez
Rob Hulsart
Jim Jamilkowski
IBIC 2023, Saskatoon, Canada

Joe Mead
Kevin MernickGeetha Narayan
Peter Oddo
Matt Paniccia
John Pomaro
Adam Pramberger
Julio Renta
Freddy Severino21

Thank you!

IBIC 2023, Saskatoon, Canada22