IBIC 2023
The Art of Sensing Orbits in Mile-Long
Accelerators on a Nanometer Scale

Danny Padrazo
Date 9/10/23
ID: MO3I01

Overview

Today’s RFBPM Electronics at NSLS-II• Typical Cell Configuration



RF BPM Development- Motivation• RF BPM Upgrade (Next-Generation Receiver)

RF BPM development Time-Line

RF BPM Evolution
 Generation 2 Digital Front End/BPM• zDFE- Zynq FPGA Generation 3 Digital Front End/BPM• zuDFE- Zynq Ultra-Scale FPGA Generation 2 Analog Front End/BPM• Analog Devices Dual ADC LTC2195
 Generation 3 Analog Front End/BPM• Texas Instrument Dual ADC ADS54J69
 Generation 3 Digital Front End (rfSOC)/BPM• ZCU208



Summary/Look-Ahead2

Today’s RF BPM Electronics at NSLS-II
 Successfully developed an “In-House” solution for High Performance Beam PositionMonitor

AFE Module Approximately 300 units in service since 2010
 Benefits realized by “In-House” System Design
 Organizational expertise to support a development program Flexibility to Implement State-of-The Art Technologies
 Utilize capabilities for hardware, software, and firmware development
 Innovative techniques for production, fabrication, assembly, test, and integration Independent of 3rd-party / proprietary platform Significant cost-savings compared to commercial options
 Adaptation to the growing needs of Accelerator Physics initiatives

DFE Module
1u Form Factor

 Investment for the future!

 Meets Performance Specifications including: 1 um in 378Khz (TbT)
 200 nm in 10 kHz (FA) Long Term Stability 200nm/8hrs in 10Hz (SA) Verified with beam

Data Type SA (Slow Acquisition) for orbit measurements, System Health No bunch-by-bunch capability (cannot resolve bunches within a turn)

Max Record Length

ADC Data

On-demand256Mbytes or 32M samples raw ADC per channel
simultaneously

TBTOn-demand

256Mbytes or 5M samples TbT (Frev=378KHz)

FOFB
10KHz

Streaming via SDI
Link and Ondemand

Streaming - X,Y,SUM ; For On-Demand: 256Mbytes or 5M
samples FA (10KHz)

SlowAcquisition
10Hz

Streaming

DDR3 80hr Circular Buffer SA (10Hz)

SystemHealthStreaming 10Hz

DDR3 80hr circular buffer System Health; AFE temp, DFEtemp, FPGA Die temp, PLL lock status, SDI Link status

 TbT (Turn By Turn) used for injection & kicked beam studies
 FA (Fast Acquisition) for fast orbit feedback & interlocks

Mode3

Typical BPM Cell Configuration
Temperature Controlled Racks
Cell controller

BPM’s

CML Fanout Module
VME Timing Crate(EVR and Fiber Fanout)

BPM IOC
Rack Temperature
Controller (.1degree C)

4

RF BPM Development- Why Upgrade?
 Motivation
 Growing Demands
Maintain the status of NSLS-II as a World-Class Facility
 Obsolescence
System is 10 years old and Replacement parts are no longer
available.
 Provide a Path to supporting the growing needs of NSLS-II andBeyond!

V6DFE, GEN1 AFE BPM

 Target Improvement Goals
 Long Term Stability 200nm  50nm (10Hz(SA))
 Resolution 200nm -> 50nm (10Khz(FA))
 Reliability
Mitigate Incident of failure StabilityProvision of on-board temp control for RF input chain to.001degree C (Heat Pump)

zDFE, GEN1 AFE BPM

5

RF BPM Development- The Future is Here!
 Available Features for next generation BPM:
 More advanced signal processing routines for accelerator physics applications.
 Unlimited Record length for TbT, FA, SA Data Types
 Gating Function, extract any bunch from 1320 fill (2 -> 8 gates)
V6DFE, GEN1 AFE BPM
 Photon Feed Back Integration
 Faster ethernet communication
 Embedded EPICS IOC
 LINUX Operating System
 Resources to implement active calibration
 Serve as a common processing platform for the XBPM, Cell Controller and other subsystem applications.
 Analog Front End options with higher sampling rates, implementation of an RFswitching matrix
 Bunch to Bunch resolution
 Precision on-board temperature control for each RF input chain

zDFE, GEN1 AFE BPM
6

RF BPM Development- Time LineProductionPrototype•

Development started August 2009•First Beam Test at LBNL on ALS June 2010•

All requirements demonstrated February 2011•

Booster complete March 2012•

Storage Ring complete June 2013

•SR commissioning complete May 2014•

NSLS-II Operation started 2015

PrototypePrototype

7

RF BPM Development/Evolution
We Are Here8

RFBPM Development ProjectGen-2 Digital Front End (zDFE)• 2018 Fully Developed Production Gen 2 Zynq

Digital Front End (zDFE) board for NSLSII BPM electronics• Supports Gen2 (AD) ADC AFE• In use since 2018, has found many applications
besides BPMs for NSLS2.

• Zynq FPGA XC7Z045FF900• Hard Dual-Core ARM Cortex A9 Processor• 1GB SDRAM• 14 – 10 Gbps transceivers• 6 SFP
• 8 via rear connectors• 32 Gbyte SDCARD• Gigabit Ethernet• Si5338 PLL for clocking• 4 front panel I/O• 8 front panel status LED's• 74 LVDS diff pairs to AFE• 28 single-ended 3.3v I/O• Runs Ubuntu 20.04 LinuxzDFESFPSD Card
FP I/O

E-NetConsole
9

RFBPM Development Project
RFBPM Gen-3 Digital Front End (zuDFE)
zuDFE• 2022 – 23 Fully Developed Prototype Gen3 Zynq Ultrascale

Digital Front End board (zuDFE) for NSLS-II BPM electronics• Required for the Gen3 (TI) ADC AFE• Target Application:• RF BPM• Next Gen Cell Controller• Zynq UltraScale+ FPGA XCZU9FFVB1156• 4GB SDRAM (PS) / 1GB SDRAM (PL)• 24 – 10+ Gbps transceivers• 6 SFP• 16 via rear connectors• 32 Gbyte SDCARD for booting• Gigabit Ethernet / USB 2.0• RJ-45 Serial Port• M.2 SATA SSD• Si5347 PLL for clocking• 4 front panel I/O• 8 front panel status LED's• 40 LVDS differential pairs I/O• 40 single-ended 3.3v I/O• Runs Ubuntu 20.04 LinuxSFPSD CardFP I/OE-NetConsole10

RFBPM Development ProjectGen 2 Dual ADC AFE (AD) BPM• Fully Developed Prototype AFE Board Analog Devices Dual ADC• Uses 2 – Dual channel, low power, 125Msps/16Bit LTC2195 Analog Devices (AD) ADC’s• External analog switch (Macom MASW-007587 switch at input (20Khz) for the analog inputs)• Heat pump controller board installed (not shown) and working.
External Analog Switch (20Khz)
Dual ADC• Shown connected to zDFE board.
RF Input Chain
GEN2 AFEzDFEHeat Pump Controller
Daughterboard

11

RFBPM Development ProjectGen 2 Dual AFE Temperature Stabilization• One of the main features of this AFE board was the lower power consumption and 2 dual lowpower ADC's which can support better long-term stability via temperature stabilization via resistiveheating.

• Can run in a rack without temperature control, and can stabilize temperatures of rf chains and ADC's tobetter than 0.01 degree C.

• Each RF Chain and ADC is equipped with thermistor (NTHS1206N02N1002HE) and Linear

Technology LTC2986, multi-sensor high accuracy digital temperature measurement system, which
provides up to 0.001 degree C resolution readback.

• Daughterboard connector is provided to support heat pump controller board.

Heat Pump Controller
daughterboard
12

RFBPM Development Project
Temperature Stabilization Initial Results• Developed temperature feedback control PID code using the heat pump controller board, initial results
look very promising.
• Shown below, 1 channel has feedback control running (blue). Other 3 channels has feedback control off.

13

RFBPM Development
zBPM Dual ADC (AD) Initial Testing• Gen2 AFEDual channel 125Msps / 16 Bit LTC2195 Analog Devices ADC's• Setup is at cell 28 BPM development rack. Button signal is connected to combiner/splitter• Results below show with and without temperature PID control(Heat Pump) and RF switching.
• Analog Switching is external.
zDFE
Test Conditions:
Condition 1Condition 2
Condition 3
Condition 4

GEN2 AFE

Dual ADC AFE with RF Switching OFF and PID controls OFF
Dual ADC AFE with RF Switching OFF and PID controls ON
Dual ADC AFE with RF Switching ON and PID controls OFF
Dual ADC AFE with RF Switching ON and PID controls ON

Summary Table:
Condition RFSW
1OFF2
OFF
3
ON4ONPIDOFFONOFFON

Dual AFE x
543nm
130nm
10nm
8nm

Old AFE x
1646nm1646nm1640nm1643nmDual AFE y
780nm
210nm
19nm
9nm

Old AFE y
945nm
886nm
873nm
882nm
External Analog Switch

y- std (um)

X-std (um)T(min)

Condition 4 Results
Temperature PID On switching OnT(min)Heat Pump Controller
Daughterboard
14

RFBPM Development
Gen2 Dual ADC (AD) AFE BPM w/integrated switches• Based on promising results with the G2 Dual ADC board, spin new version with switches on board.
• Uses 2 – Dual channel 125Msps / 16 Bit LTC2195 Analog Devices ADC's• Added Macom MASW-007587 switch at input (20Khz) for the analog inputs• Initial results with signal generator look good.
• Beam testing in progress.
GEN2 AFE w/SwitchingzDFE15

RFBPM Development
Gen2 Dual ADC (AD) AFE w/integrated switches• Dedicated Setup at cell 28 BPM development rack. Button signal is connected to combiner/splitter• Results below show with and without temperature PID control and RF switching.
• Analog Switching is on-board.

zDFE

GEN2 AFE w/switching• Overnight Run 3-26-23 10pm–10am, ~14hrs• Switching on, temperature PID was off• No plastic covers over rf shields.
• Rf atten = 2dB, Max ADC counts ~ 28k.

Xstd = 18nm, Ystd = 20nm
XPosControls Interface

YPos

RF Ch ATemp
BeamCurrent
16

RFBPM Development
Complete Gen-2 BPM Electronics Assembly• BPM electronics mounts into 1U chassis.
• Digital board : zDFE
• Analog board : Dual ADC (AD) (125Msps /16 Bit LTC2195 Analog Devices) ADC's• AFE with integrated switches and heat pump controller daughterboard

Heat pump controller
board
Dual ADC AFE
PowerSupplieszDFE

New Front Panel

17

RFBPM Development
Gen3 Dual ADC AFE (TI) BPM• Fully Developed Gen 3 Prototype AFE (Texas Instrument Dual ADC)
• Uses 2 – Dual channel 500Msps / 16Bit ADS54J69 TI ADC's.
• Option to go to 1Gsps using pin compatible ADS54J60• Sample Rate currently set to 250Msps. (ADC has a built in half band filter and decimate by 2.)• Requires zuDFE board, needs 8 high speed transceivers for JESD204B interface to ADC’s.
• Analog Switching is done externally.
• Initial testing w/promising results. Spin2 board assembly in progress.
• Ready for beam test in NovzuDFE
GEN3 AFEAnalog Switch

18

RFBPM DevelopmentXilinx RF SOC Evaluation Board: ZCU208
ZCU208 Eval Board• Alternate consideration for FPGA platform• Zynq UltraScale+ RFSoC XCZU48DR-2FSVG1517E siliconfeatured on the ZCU208 Evaluation board• ARM cortex-A53 processor• Integrated 8x 14-bit 5GSPS ADCs
• Integrated 8x 14-bit 10GSPS DACs

• Possible Applications• BxB position measurements including:• 5 Gsps ADC sampling each bunch (2ns bunch space)• Existing RF BPM is 117 MHz sub-sample• Extracting peak bunch from X10 sample data
• Gate function, extract any bunch from 1320 fill
• FPGA processing clock is 499.680 MHz (Sync with RFfrequency)• FCT / WCM• Fill Pattern Monitor• LLRF.

• Completed / Working• Linux running.
• Initial Results from ADC at 2.5Gsps and 5Gsps.
• LMK04828 PLL programmed• SI570 programmable oscillator programmed• Embedded Event Receiver implemented• ADC Sample clock locked to RF (via recovered clock on EVR) 19• System configured in Cell 14

RFBPM DevelopmentXilinx RF SOC Eval Bd Test: ZCU208• RFSOC sampled data (5 GSPS)• Output from Hybrid box with 560 MHz LPF• 10 Samples per RF bucket (200ps / sample)
3 RF Buckets
10 Samples/Bucket

Bucket 1

Bucket 2

Bucket 3

Standard Fill Pattern

FFT20

RFBPM Development
Xilinx RF SOC(ZCU208) For RFBPM application• Preliminary TBT Test Results (no beam)• Sampling rate = 310*40*Frf/1320 ~ 4.69GSPS• Using built-in DDC with a decimation factor of 40• Signal generator as the source, CW 499.68MHz• 5x improvement to V6 BPM• <200nm (V6 BPM = 1um)

FA Position

TbT Position

SA Position

ABABABCDC

DC

Dxyx

yxy

TbT Position (378kHz)
14000 Turns<200nmFA Position (10kHz)
14000 Samples<100nmSA Position (10Hz)
<80nm
21

RFBPM Development
Custom RFSOC DFE Board Design• Prototype RFSOC Zynq Ultrascale+ Digital Front End board design(rfsocDFE) for NSLS-II BPM electronics• High Sample Rate BPM, up to 5GSPS• Compatible with existing chassis• Target Application:• SR/Injector RF BPMs (2 BPMs in one!)• Common Platform DFE• Cell Controller• BxB RF BPM

RF AFE DaughterBoard

NSLS-II RFSOC Board• Prototype RF AFE Analog Front End board (rfAFE) for rfSOCDFE• Similar RF chain to that of the Dual AFE boards• Interfaces directly to either ZCU208 or rfsocDFE Board• Higher gain Amplifier MML09212HT1• Replaced SAW BPF with more stable Ceramic BPF• Layout Complete, PCB Assembly for test expected in Sept.

22

𝐹𝐹𝑅𝑅𝑅𝑅𝑅𝑅 Input

Today’s Xray BPM Electronics
Current Inputs from blades

XAFE
Power Supplies

ZDFE

Fiber input forevent receiver

Ethernet to Controls Network

Presently Installed at 3 Beamlines
Features:• Linux Operating System• Embedded EPICS IOC and caClient Applications• Python scripts for FFT calculations and other data processing• Knowledgeable Users have access to all internal functions• Very flexible, changes can usually be implemented within hours

23

Tomorrow’s Xray BPM Electronics
Advantages:• In-House Development• Small compact module• Designed to interface to front end equipment
such as diamond bpm's, ion-chambers, etc.
• Interchangeable Voltage/Current Monitor• Capable of providing feedback on X-ray beam
position and provides fast feedback control on
various beamline components• Multiple programmable rate data streams areavailable to the user with nominal data rates of
10Hz, 10KHz and full ADC line rate.
• Embedded Event Receiver allows the module torun synchronized to the accelerator complex
and receive global and local accelerator
commands which permit easy synchronizing ofdata and time-stamping with other beamline
components

Electrometer
Performance Parameters
Readout Channels

4 Current (independent gain control)
4 Voltage (independent gain control)

Current Measuring Range

6 programmable current ranges:
100nA,1uA,10uA,100uA,1mA,10mA

Data Rate

Programmable, up to NSLSII turn-by-turnrate , 378.545KHz

Resolution Bits/Sample Rate

20 / 400Ksps

Bias Voltage Output-10v to +10v

Current Polarity

BipolarI/O Interface
Control Interface

RJ45 – 10/100/1000 Ethernet
Embedded EPICS IOC/LINIX OSHigh Speed Interface

SFP – Gigabit Ethernet

Embedded Event Receiver

SFP

Electrometer Inputs

SMA

Bias Output

LEMO

Feedback Outputs

LEMO

Power+5v @ 1A
24

RFBPM Development Summary / Look Ahead• zDFE FPGA board in use since 2018.
• ZuDFE board prototype in testing, most features working. Spin 2 ready for Nov• Gen 2 AFE board (dual, 117Msps / 16bit (AD) ADC)
• Initial results are looking very good. More test results in the next few months• Version of the board with integrated switches complete, testing in progress.
• Gen 3 AFE board (dual, 1GSPS / 16bit (TI) ADC)
• PLL and ADC seem to be working ok. Needs more firmware development and testing• RFSoC Platform (zcu208 Evaluation Bd)• Have 1 ADC channel working at 5GSPS.
• A version with 4 channels working.
• Linux is running. IOC in progress• ADC clock is locked to the RF. Embedded event receiver is working.
• Initial data with Beam looks promising.
• Gen 4 RFSoC (Custom Board Design)• Schematics Complete• Ordered 10ea Xilinx RFSoC FPGAs (XCZU47DR-1FFVG1517I)• Board Design in progress, Target Completion Nov25

Acknowledgements
Contributions to this presentation:
Joe Mead, Kiman Ha, Tony Caracappa, Bel Bacha
Next Generation RFBPM Development/Test:
Joe Mead, Tony Caracappa, Kiman Ha, Bernie Kosciuk, Bel Bacha,
Christian Videbaek, John Bohenek, John Hammond, Mike Capotosto, Artie
Munoz, Ray Gonzales
Data Science & System Integration (DSSI):
Stuart Wilkins, Yuke Tian, Kiman Ha, Rob Reiner, Yong Hu
NSLS-II Accelerator Division
Timur Shaftan, Greg Fries, Helena Lewis

26

Thank You !

27