Towards a new Timing Systemfor Particle Accelerators
Overview of the work of theproject interest group
Patrick Loschmidt
Austrian Academy of Sciences

Research Unit for Integrated Sensor Systems

Research Unit for Integrated Sensor SystemsIntention• Introduce you to the GMT renovation
project started with the AB-CO-HT section
at CERN• Add value to the project by making it more
known and interesting for others to join
• Learn about other accelerator operators‘
requirements• Trigger a discussion– get feedback about our ideas– probably find a new „user“ or even codeveloper– identify commonalities for possible cooperations2

Research Unit for Integrated Sensor SystemsMotivation• Initiative came from CERN, AB-CO-HT
section (accelerator beam, control,hardware timing)• After (almost) completion of the new LHC,
CERN’s control group is launching arenovation project for its injector complex.
• Review the way accelerator timing is done,
since the current version has somedrawbacks• A new system will be chosen for the next
10-20 year period3Research Unit for Integrated Sensor Systems

Currently Installed GMT• Based on fibre transmission for long stretches• followed by RS-422 multi-drop over shielded twisted pair,with a bandwidth of 500 kb/s
• Time is broken up in milliseconds• 8 events max. per slot
• The master is driven by a Pulse Per Second (PPS) pulse
and a 10 MHz from a GPS Disciplined Oscillator providing
UTC synchronisation.
• One master card per accelerator due to bandwidthconstraints• Communication is exclusively downstream.

4

Research Unit for Integrated Sensor Systems

Kick-off Workshop February, 25th
• One day meeting held atCERN Meyrin• Introduction of all participants, theircompetencies and interest in theplanned co-operation• Main subjects– timing systems for large Physics
facilities– special emphasis on non-beamsynchronous timing systems– Ethernet technologies– PTP (IEEE 1588)• Long discussion in the afternoon tolaunch collaborative effort5

Project consortium

Research Unit for Integrated Sensor Systems• Participants (25 people)– „Users“• CERN (Geneva, Switzerland)• GSI (Darmstadt, Germany)• IN2P3 (Lyon, France)• ITER (Cadarache, France)• Elettra (Trieste, Italy)• planned contact to Fermilab concerning Project X– Academic• InES (Winterthur, Switzerland)• AAS, FISS (Vienna, Austria)– Companies• Cosylab (Ljubljana, Slovenia)• Oregano Systems (Vienna, Austria)• Micro Research (Helsinki, Finland)
6

Main Issues with the Current GMT at CERN

Research Unit for Integrated Sensor Systems• Lack of bi-directionality in GMT:
– Requires parallel data path(technical network) for control anddiagnostics→ only platforms with anembedded computing engine aresupported– Cabling delay compensation
cannot be done automatically→ need for costly/unreliablemeasurement campaigns• Lack of bandwidth in GMT– 500 kb/s current rate was chosen
for backwards compatibility– different networks for different
accelerators
7

Research Unit for Integrated Sensor Systems

Project Steps and Issues• Identify commonalities with other projects• Analyse promising enabling technologies– Ethernet and PTP (IEEE 1588)– Bidirectional optical links, with active or passive fan-outs• Boundary conditions– Be as standard as reasonably possible.
– End up with a completely open source product.

• Raised issues– Time scheduled vs. event based systems.
– Unclear advantages of plain Ethernet + protocol stack.
– How to mix companies and open source?

8

System Concept

Research Unit for Integrated Sensor Systems• More than a timing system– full-blown deterministic field-bus– bidirectional nature• Several approaches, what to put
on top of synchronous Ethernet– similar to WorldFIP or Powerlink• Downstream communication would
be the monopoly of the master– continuous stream so there is nodrift problem in the receiver's PLL(a must for ps timing).

• Upstream traffic would be timemultiplexed9

Workshop outcome

Research Unit for Integrated Sensor Systems• Technical issues– Synchronous Ethernet plus PTP (IEEE 1588) (pro & con)
– Form factors– Interest in higher level protocols• Project Management Issues– CERN driving project, remaining organisational structure– Open source definition– Timescale of work
– Quantities of components / machine requirements– Financing• Documentation– http://indico.cern.ch/conferenceDisplay.py?confId=28233

10

Research Unit for Integrated Sensor Systems

Selected Work Packages• Manage Project (CERN)• Gather user requirements (Cosylab)– GSI, CERN and IN2P3• Starting at the physical layer and moving up until the event
distribution processor.
• Find commonalities and identify potential incompatibilities– Study Synchronous Ethernet and IEEE 1588
– almost finished ✔

• Fibre vs. copper (FiSS)– Study the trade-offs between copper and fibre solutions
regarding delay symmetry, EMC robustness, price,performance, dependencies towards ASIC vendors– Analyse a solution involving a standard bidirectional fibretransceiver + an FPGA, no dedicated Ethernet chips
11Research Unit for Integrated Sensor Systems

Selected Work Packages• Reliability studies (CERN)• Standards compliance (InES)– Compliance of IEEE 1588 with synchronous Ethernet– Propose a delay compensation part
– Study the interplay between our home-made solution andcommercial hardware not using either synchronous
Ethernet or IEEE 1588 or none.
– Standardisation activities• Scalability system-level simulation (FiSS)– System level stability issues• Scalability studies (Oregano Systems)– Potential instabilities caused by cascading PLLs
– Study impact of the IEEE 1588 loop on these instabilities• Overall Design Specification (CERN)
12

Currently Running Efforts

Research Unit for Integrated Sensor Systems• CERN and Elettra– Transmit a 240 MHz clock encoded in a 2.4 Gbps data
stream over std. multimode fibre using commercial SFP
modules on Xilinx Virtex-5 evaluation boards• Extract the clock on the receiverside and evaluate its stability• Further manipulate this clock bymeans of DLL inside the FPGA
• Already running demo using
coax cabling• FiSS– Evaluation system based on Altera Stratix II GX dev. board– compare copper with optical link with respect to IEEE 1588
– checking climate influence on link

• Project wide
– dig out funding possibilities (national or EU)

13IEEE 1588 Standard

Research Unit for Integrated Sensor Systems• IEEE Standard (2002, 2008)– Origin in instrumentation andmeasurement• Master-Slave based– “best master” election– Stratum based (clock quality)• Defines protocol for time
information exchange– PTP (Precision Time Protocol)– Network-independent– Mostly Ethernet-based– „Delay Request“ and„Delay Response“ packets• No actual synchronization
14

Basic Clock Synchronization Core

Research Unit for Integrated Sensor Systems• High resolution adder based clock– Smooth, continuous amortisation– Different oscillator speeds• Single and periodic timers

Host– Distributed PLL– Event distribution• Programmable MII scanner• State, rate and interval
based synchronization
support– Temporary increment– Compensated timer

MIIS

AdderbasedclockEventsTimers

Ethernet15

Research Unit for Integrated Sensor Systems

IEEE 1588 Clock Synchronization• Software stack running protocol• Kernel driver accessing HW functions• HW supported timestamping of Ethernet packets– memory buffers and pre-scheduled parameter
modifications
PTP

otherUDP

JitterIP

Sync. Driver

Device Driver
MAC

CSC

Switch Fabric
MAC

MIIS
PHYCSC

MACMIISPHYPHY16

Research Unit for Integrated Sensor Systems

Achieved Goals• IP Cores (FPGA, ASIC)• PCI Ethernet NIC• 8-port Ethernet Switch• IEEE 1588 PTP Stack + Drivers– Version 2002 & 2008
– Linux & Windows

17

WP3 Evaluation Platform

Research Unit for Integrated Sensor Systems• Altera evaluation platform for gigabit communication– High performance FPGA
– Optical transceiver– Gigabit PHY• GE delay behaviour forcopper and optical• Basis for development of newtiming system– climate chamber– fibre spools (1 km, ...)• Investigation of remaining jitter sources
and control loop optimisations

18

Research Unit for Integrated Sensor Systems

Patrick.Loschmidt@OEAW.ac.at19

Research Unit for Integrated Sensor Systems

Embedded Processor HyNet32 XS/S
• Hyperstone RISC / DSP core
• UMC 0.18µm technology• Clock synchronization core
– 3rd generation– 60500 gates– 100 MHz– two Ethernet interfaces• µClinux 2.4/2.6• 400ns accuracy inrev. 0 on 10 Base-T• expected 7ns in rev. 2
using standard 100 Base-T• Current lab equipment– 700ps std. deviation for pulse generation
20