# Presentation: 656275

## LHC Control SystemTutorial for the FNAL LARP group

- Jim Patrick
- November 9, 2005

## General Overview

- Taken from a number of talks, mostly:
- ICALEPCS Conference, October 2005 (33 CERN presentations!)
- December 2004 AB/CO “Controls Day”
- Plus other miscellaneous talks
- Goals:
- Give some practical flavor of the system, but not prepare one to write actual code.
- Relate concepts to FNAL system.
- I am not at all an expert.

## Outline

- Overview
- Timing
- Device Model
- Front-End Software
- Application Software and Development
- Services
- Security
- Summary

## The LHC Control System

**The LHC Control System**

**B. Frammery**
**For the CERN - AB/CO Group **

## CERN machines

- (LEP)

**LHC**

## Strategy 

- Develop new software and hardware infrastructures
- For LHC
- To be used & tested on all the new developments
- To be spread over all the CERN accelerators **at a later stage**
- Integrate industrial solutions as much as possible

*Meaning that, meanwhile, the “legacy” controls for LINAC2,*
*the PSB, the PS and the SPS are to be maintained *

## LHC controls architecture diagram

*TCP/IP communication services*

*TCP/IP communication services*

*CERN  GIGABIT  ETHERNET  TECHNICAL  NETWORK*

*FILE SERVERS*

*Linux/HP ProLiant*
*APPLICATION SERVERS*

*PVSS /Linux PC*
*SCADA SERVERS*

*RT/LynxOS*
*VME Front Ends*

*Linux/LynxOS PC*
*Front Ends*

*PLCs*

***LHC MACHINE***

*TCP/IP communication services*

*TIMING GENERATION*

***LHC MACHINE***

*ACTUATORS AND SENSORS*
*CRYOGENICS, VACUUM, ETC...*

*QUENCH PROTECTION AGENTS,*
*POWER CONVERTERS FUNCTIONS*
*GENERATORS, ...*

*BEAM POSITION MONITORS,*
*BEAM LOSS MONITORS,*
*BEAM INTERLOCKS,*
*RF SYSTEMS, ETC...*

- WorldFIP SEGMENT
-  (1, 2.5 MBits/sec)

- OPTICAL FIBERS

*cPCI*
*Front Ends*

*ANALOGUE*
*SIGNAL*
*SYSTEM*

*LHC controls architecture diagram*

## Software

- frameworks

## The software frameworks (1)

- Front-End Software Architecture (FESA)
- 	Complete environment for Real-Time _Model-driven_ control software implemented in C++ for the LynxOS and Linux platforms
- Java framework for accelerator controls
- Uses  J2EE application servers with lightweight containers
- Plain Java objects (no EJB beans)
- Applications can run (for test) in a 2-tier setup
- Unified Java API for Parameter Control (JAPC) to access any kind of parameter.
- Runs on Linux platform

## The software frameworks (2)

-  UNified Industrial Control System (UNICOS)
- Complete environment for designing, build and programming industrial based control systems for the LHC.
- For cryogenics, vacuum, environmental controls etc.
- Supervision layer: PVSS II (SCADA from ETM)
- Cross communication with accelerator controls framework possible

- UNICOS & the Java framework for accelerator controls use
- the same graphical symbols and color codes

## Machine Timing& sequencing 

## Overview ICentral Timing. What’s the CBCM ?

- The Central Beam and Cycle Manager CBCM is a collection of hardware and software systems responsible for coordinating and piloting the timing systems of CERN’s accelerators.
- In the LHC era, the CBCM will control Linac-II, Linac-III, the PSB, CPS, ADE, LEIR, SPS and the LHC timing systems.
- The CBCM will also drive the Beam Synchronous Timing (BST) for LHC. There will be 3 distributions R1, R2, Experiments.

## Hardware IILHC MTG

**2.2 G-Bit / S optical link**
**64Mb Reflective memories**

- Main MTG

- LHC MTG

- BST

- GMT
- LHC

- CPS and SPS telegrams and timings
- and MTG synchronization when filling

- Clocks:
- Bunch Clock 40..8 MHz.
- Frev ticks at 89us.
- 40.00 MHz GPS clock
- 1PPS (1Hz) clock
- Basic period clock

- Preloaded Injector Sequences

- Preloaded LHC Sequence

- 3 BST Cards

- Safe
- Params

- Energy/Ring
- Intensity/Ring
- Safe Beam Flg
- Beam present Flg
- Extraction permit Flg
- BIC Beam permit Flg

- External Conditions
- and Events

## The LHC telegram which will contain at least the following information: 0x14xxyyyy
USER:	The cycle ID, it has values like PILOT, NOMINAL, DUMP MD...
PARTY1:	The particle type in Ring-1, Protons/Ions from LEIR
PARTY2:	The particle type in Ring-2
ENERGY1:	The Beam Energy in Ring-1
ENERGY2:	The Beam Energy in Ring 2
INTEN1:	The Beam Intensity in Ring-1
INTEN2:	The Beam Intensity in Ring-2
RING:	The next ring to be injected Ring-1, Ring-2, NONE
BUNCH:	The next target bunch position in the ring 0..35640
BATCH:	The actual batch number in the ring 1..12
BATCHES:	The number of CPS batches
MODE:	The machine mode, Pre-injection, Injection, Ramping, Physics, DUMP etc
BPNM:	The basic period number from the start of the cycle
COMLN:	Timing trigger bit patterns which are calculated by the CBCM to trigger specific actions.
STATUS:	Machine status bits like, OK, ABORT, QUENCH
BEAMID:	Identifies the next beam in all injectors
The SPS telegram	
The CPS telegram
The UTC time each second
The LHC 1KHz events 0x0100xxxx
The LHC machine events CTIM X:=: 0x13xx0000
Some CPS & SPS events such as the SPS extraction kicker warning pre-pulse.

- USER:	The cycle ID, it has values like PILOT, NOMINAL, DUMP MD...
- PARTY1:	The particle type in Ring-1, Protons/Ions from LEIR
- PARTY2:	The particle type in Ring-2
- ENERGY1:	The Beam Energy in Ring-1
- ENERGY2:	The Beam Energy in Ring 2
- INTEN1:	The Beam Intensity in Ring-1
- INTEN2:	The Beam Intensity in Ring-2
- RING:	The next ring to be injected Ring-1, Ring-2, NONE
- BUNCH:	The next target bunch position in the ring 0..35640
- BATCH:	The actual batch number in the ring 1..12
- BATCHES:	The number of CPS batches
- MODE:	The machine mode, Pre-injection, Injection, Ramping, Physics, DUMP etc
- BPNM:	The basic period number from the start of the cycle
- COMLN:	Timing trigger bit patterns which are calculated by the CBCM to trigger specific actions.
- STATUS:	Machine status bits like, OK, ABORT, QUENCH
- BEAMID:	Identifies the next beam in all injectors
- The SPS telegram
- The CPS telegram
- The UTC time each second
- The LHC 1KHz events 0x0100xxxx
- The LHC machine events CTIM X:=: 0x13xx0000
- Some CPS & SPS events such as the SPS extraction kicker warning pre-pulse.

- LHC Timing cable

- Can have NO
- “next” lines

- Payloads = 0000
- for LHC events

- CTIM X = F (code)

- LHC
- Machine = 1

- Event Type = 3

## Overview IVTiming Reception

- The CTRx V/VME I/PCI P/PMC
- Down to 1ns UTC time stamping if HPTDC installed, else 25ns
- 50MHz external clocks
- 1PPS 1KHz 10MHz and 40MHz internal clocks
- Counters are 24-bit
- 2048 actions supporting MP and PPM
- Telegram and Payload handling
- Full counter remote control
- Fully Integrated into FESA, Alarms monitor

- FESA

- Tim/Tgm Lib

- CTRx

- SPS
- Tg8

- PS
- Tg8

## Data Management

## Databases : the 4 domains of data

***Physical****** ******Equipment***

***Machine Layout***

***Controls Configuration***

***Operational Data***

*Serial Number*

*Installed *
*Equip**t*
*Type*
*Optics*
*Powering*

*Computer Address*

*Settings Measurements Alarms*
*Logging*
* **Post-Mortem*

***Equipment***
*** ******Catalogue***

***Consistent naming and identification scheme as defined in Quality Assurance Plan***

## Device/Property Model

- A *device* is a named entity within the control system, which corresponds to a physical device (Beam Position Monitor, Power Converter) or to a virtual controls entity (e.g. transfer line)
- The state of a device is accessed via *properties** *and can be read or modified by the *get* and *set* access methods. (synchronous and asynchronous)
- Uses CORBA, hidden from the user by “Controls Middleware”
- Property can be *monitored* (publish/subscribe)
- A *cycleSelector* or a polling period can be specified
- Optional on-change mode: client will be notified only when property has changed (server criteria).
- Uses Java Message Service (JMS) “publish/subscribe” technology
- Device classes can implement many properties of simple type or few properties of composite type

## Device Model

- Devices refer to higher level constructs than in ACNET
- Devices have properties; may have more than one
- All properties are like C-structures, not confined to READING, SETTING etc. as in ACNET
- You name the elements (“parameters”)
- These have associated datatype, units, dimension, minimum and maximum value etc.
- Can be atomic (single element) or composite (multiple elements)
- Scaling assumed to be done by front-end
- Device structure defined in “MetaProperty” classes for each general type
- Formal hierarchical naming scheme
- A “Working Set” device is a collection of devices

## Beam Current Transformer Acquisition as example of composite property

| tag | value type |
| --- | --- |
| cycleId | Long Long |
| timeNano | Long Long |
| numberOfBunch | Integer |
| maximumBunchIntensity | Float |
| bunchIntensity | Float[NbOfBunch] |
| minimumBunchIntensity | Float |
| bunchSpreadSigma | Float |
| statusTag | Long |

- Standard
- Entries

## Input Form for metadata

## Generic Workset Display Program

## Generic Control Knob Component

## Initialisation Parameters

- Any status or value control parameter for a device can be stored as a REFERENCE. This includes arrays for function generators.
- This can be done for any of up to 64 virtual machines which configure our accelerators for a particular operation like injecting protons in the LHC.
- Particular sets of values can be stored in named ARCHIVES for a virtual machine. Used to set up the machines for a particular operation.
- The Directory Service provides interfaces for storing and retrieving REFERENCES and ARCHIVES.

## JAPC

- “Java API for Parameter Control”
- Single API for all Java applications to access devices (physical / virtual)
- Based on the concept of parameter (device/property)
- Unified and simple access to various systems
- Hardware (via Controls Middleware – CMW; Including PVSS devices)
- Directory Service (descriptions)
- SDDS logged data, Simulation
- Virtual Parameters in the middle tier
- Provides more services to applications
- Metadata, descriptions
- Groups, Caching, Transactions

## JAPC Code example

- Counter device named Counter11 with one property with one parameter named Measurement that is an int:
- Parameter p = Factory.createParameter(“Counter11”, “Measurement”);
- CycleSelector sel = Factory.createCycleSelector(END_OF_CYCLE);
- ParameterValue value = par.get(sel);
- int counts = value.getInt();
- Code generation facility to make pseudo-”wide” API
- e.g. par.getMeasurement();
- Compile time check

## 		FESA generic services 

## 0. Outline

- Our offering to the equipment-specialist
**“****A comprehensive offering consisting of a model, method, framework, suite of tools and set of utility packages and support services”**
- Our progress at a glance
**	****“****FESA switches from being a project to being an open-ended activity”**

## 2. Service offering

- Model
- Method
- Tools
- Framework
- Utility packages
- Documentation
- Training
- Support
- Consulting
- Requirements

- ........**Formal generic-model and customization-language **
- ...........**Workflow formalized as a step-by-step method**
- ..................**One tool dedicated to each step of the above**
- ........** Reusable C++ package which can be tailored**
- ...**“A la carte” interfacing with PLC and timing**
- .........**Essentials, tools’ on-line documentation**
- .............................**mostly ad hoc, on-line tutorial**
- ..............................................**mostly ad hoc**
- ...........................................**long-term goal**
- .........................**Issue management system**

## 3. Client needs coverage as today

_**http://project-fesa.web.cern.ch/project-fesa/**_

## FESA Development

- Framework attempts to automate development
- Minimize code that must be written by the developer via automatic generation of code and configuration information
- Library support for timer cards and common devices
- Generic GUI tools guide one through 4 main phases:
- Designing the class structures (internal variables, real-time scheduling, external API etc.)
- Deploying the class on a front-end computer
- Instantiating 1 or more instances of a deployed class (defining configuration values for internal variables, real-time scheduling etc.)
- Testing over the accelerator middleware

- ICALEPCS 2005

- Arruat et. al.

## FESA Development

- ICALEPCS 2005

- Arruat et. al.

## LHC Software Architecture

- All accelerators share common characteristics
- Create a model that captures those characteristics important for control
- Have a common domain model
- Have common software components to work with this model
- Rationalize software development to reuse and extend the common parts to control all accelerators and transfer lines

## A bit of context

- CERN Controls Group
- Controls Software for the LHC
- LSA shared project between Controls and Operation
- Provide homogenous application software to operate the SPS, its transfer lines and the LHC

## Applications

**Datastore**

**Devices**

**JAPC CMW/RDA**

**JAPC**

**Hibernate / Spring JDBC**

**Data Access Object (DAO)**

**LSA Client API**

**LSA CORE  ****(TH1.4-8O)**
**(Settings, Trim, Trim History, Generation, **
**Optics, Exploitation, Reference)**

**Parameters**
**Concentration**

**JAPC**
**CMW/RDA**

**JAPC Remote**
**Server - JMS**

**JAPC-LSA**

**LSA Client implementation**

**LSA Client API**

**JAPC API ****(TH1.5-8O)**

**Spring HTTP Remoting / Proxies**

**JAPC Remote**
**Client - JMS**

**Business Tier (Web Container)**

**Client Tier**

**CORBA IIOP**

**CORBA IIOP**

**JDBC**

**HTTP**

**HTTP**

**JMS**

## Logging

- 4000 Beam Loss Monitors

**BLMs**
**Concentration**

**Publication**

**Operator**
**Console 1**

**Operator**
**Console 2**

**Logging**

**Fixed**
**Displays**

**Operator**
**Console 3**

**Operator**
**Console 4**

- ...

- Broadcasting

## Datastore

**Devices**

**Complex Business Logic**
**(Settings, Trim, Trim History, Generation, **
**Optics, Exploitation, Reference)**

**Operator Console 1**

**Operator Console 2**

**Operator Console 3**

- many applications

- many applications

- many applications

## Extrapolation to Beamline Settings

## Complexity must be handled 

- Need of Standard Services
- Service discovery (find where services are)
- Remoting (split application among tiers)
- Transaction handling (multiple device “sets”)
- Database access (object-relational mapping)
- Security (Who/what/where can access)

## One Answer : J2EE + EJB

- Infrastructure provides standard services
- Widely used in industry
- In house experience
- Change of programming model
- Intrusive
- Force the use of container
- Force the use of components
- Tie the persistency to the container
- Debug with application server on local PC
- Deployment hell

## Another Answer :J2EE – EJB + Spring Framework

- Design for 3 logical tiers
- Run 2 or 3 physical tiers
- Developers write plain Java
- No change in the programming model
- Focus on our domain
- No time for doing infrastructure

## Applications

- Trim (perform + history browser/revert)
- Orbit Steering
- Generic Equipment Control
- Fixed Displays
- SDDS Logger and Viewer
- Optics Twiss viewer
- Settings Generation
- Using the Application Frame

## Generic Equipment Control

## Generic Measurement

## Trim

## Trim history

## Visualization of the settings

## Orbit Steering

## Fixed Displays

## Optics Display

## Results and Future Targets

- Control of TI8 (October 2004)
- Steering of the SPS ring orbit
- LEIR controls
- SPS start-up
- Extraction sequencing TI2/TI8
- LHC sector test

## Development Process Issues

**Projects and code organization**
**Source versioning management **
**Build services (automation of common tasks)**
- Compilation, JAR
- Documentation generation
**Dependencies management**
**Release management**
- Releasing new versions of software in a dedicated repository
**Applications deployment**
**Issues & bugs tracking**

**(guidelines, naming conventions, directory structure)**

**(CVS)**

**(JIRA)**

## Common-Build constraints(Directory structure)

**equipstate/**
**	****build.xml**
**	****product.properties**
**	****product.xml**
**	****people**
**	****src/**
**	****    ****java/**
**	****    ****test/**

## Target examples

**Based on *****Apache Ant***
- Java based open source build tool (like *Make*)
**Compiling sources**
- ant compile
**Building distribution of the product**
- ant dist
**Releasing new version of the product**
- ant release

## What Release does

**Extracts the product from the ****CVS**** to the dedicated ****production repository **
**Builds**** the product (calling Common-Build)**
**Installs it in a ****multi-versioned**** repository**
- New version is added without modifying the old ones
- We can always use old versions** **
**Updates ****product**** aliases**** (symbolic links)**

## Eclipse IDE

## GUI Applications deployment

**We use**** Java Web Start**** deployment technology**** **
**uses a special XML descriptor (****JNLP file****) to deploy and run applications**
**ensures that all required libraries (cached locally) are up to date**
**Repository contains ****all libraries**** and ****JNLP file**
**Console Manager contains directory of applications, starts them on request**
** **

## GENERAL SERVICES

## The Alarm System

- LHC Alarm SERvice (LASER)

- « Standard » 3-tier architecture
- Java message service (JMS)
- Subscription mechanism

## Architecture

**Notes:**
Keywords are ‘alarm conditions’ and ‘LHC machine’
Let’s look into more details at what that practically means

## Logging

- Several 105 parameters will be logged
- Every data or setting is timestamped (UTC)
- Parameters are logged
- on regular intervals (down to 100 ms)
- on request
- on-change

## DEVDB9

- Oracle DB

- LHCLOGDB
- Oracle DB

***Database ***
***Server***

***Application ***
***Server***

***Client***
***Tier***

***API***

***Data Input***

***Data Output***

- TT40

- VAC

*New clients*

*Established Clients*

- Java

- XML

- ABJAS4 (PC) – Oracle AS

- ABOFS1 (HP) – Oracle AS

- R. Billen 10.08.2004

**LHCLogging**

**LHCLoggingTest**

**LHCLogging**

**LHCLoggingTest**

**Loader**

**GUI**

**GUI**

**Loader**

**Loader**
**Test**

**GUI**
**Test**

**GUI**
**Test**

**Loader**
**Test**

- Test

**production**

**test**

**backup**

- _[http://lhc-logging.web.cern.ch/lhc-logging](http://lhc-logging.web.cern.ch/lhc-logging/)__[/](http://lhc-logging.web.cern.ch/lhc-logging/)_

**Notes:**
The LHC Logging Service provided by the AB-CO-DM section and supported mainly by Chris Roderick, is composed of several components.

The clients expect that the data they send get stored. Since mid-2003 and up to 03.08.2004, the data was stored on the Oracle Database DEVDB9, the only centrally available and supported Oracle9i instance in 2003.

Several clients are already operational: TT40 extraction tests (shot-by-shot logging) in autumn 2003 and SPS Vacuum since 13 June 2004.
With the operational availability of the new dedicated database cluster(+1.5 Tbyte of disk space), the data has been migrated to the production database LHCLOGDB. This now forms part of the Production Software Environment.

The LHC Logging Service is based on a 3-tier architecture; 2 machines are used to host the Application Server (middle) tier: ABJAS4 (a PC) and ABOFS1 (a Proliant HP), both with the Oracle Application Server installed and configured. In both AS, the dedicated OC4J containers “LHCLogging” are deployed, in which the software such as Java Servlets and JSP runs.

Our clients implement an XML schema or use the Data Input API to submit their logging data to the service (as well as the meta data). This data loading chain is fully operational and in production, the Data Loading application on ABJAS4 takes care of the parsing and loading process. For Data Extraction, the web-deployed TIMBER interface allows users to get to selected logged variables, show them graphically and extract in file format.

For reasons of load balancing, the GUI application (JSP in the LHCLogging container) are deployed on the other machine ABOFS1. If for any reason, this machine is down, the TIMBER interface can be accessed via a "backup route". (i.e. another URL you’ll find on our website). Similarly, the Data Loaders are deployed on the “mirror” machine ABOFS1, but switching is less trivial (and in any case, we always emphasized that the client needs to buffer locally ;-). It is anticipated that the 2 Application Servers on the 2 hosts will work in a Real Application Cluster setup, whereby the load is taken over transparently if one machine is down. 

For our new clients, we have put in place an identical Test Software Environment in order to allow them to test out the service, and to allow us to validate the new client (checking errors, meta data, data volumes, data rates, naming, hierarchy). Similarly, they have an identical TEST TIMBER interface. Note that the data resides on DEVDB9, and will be lost by default, but can be transferred upon request when a client moves to the Production Software Environment. Once again, for reasons of redundancy and symmetry, the Test Software Environment is deployed on both machines.

## Timber

- Define standard logging parameters analogous to D43
- Java client API available so programs may log as they wish
- Access to data via GUI applications, web interface, API
- All logging to single Oracle database
- Lots of disk allocated, but scaling issues
- Log all data first into “Measurement DB” cache which has a lifetime of ~week. Periodically (~15 min) insert selected data into Oracle
- Shot by Shot Logging???

## Timber Web Interface

## Timber Web Interface

## SDDS Logging & Monitoring

## SDDS Browser & Viewer

## Post Mortem

- Automatic *(typ. when an interlock appears)** *or manual trigger
- No beam allowed if PM not ready
- Capture of
- Logged data
- Alarms (LASER)
- Transient recorder signals (OASIS)
- Fixed displays
- Analysis
- A few Gigabytes per Post Mortem capture
- Structured sorting of causes & effects
- Needed from October 2005 for Hardware commissioning
- Continuous development effort for the years to come

-  TI8 extraction test in October 2004 already proved the importance of a PM system

**To take a snapshot of the LHC vital systems.**

## Analogue signals

- Open Analogue Signals Information System (OASIS)
- To visualize and correlate in Real-Time time critical signals in the control room
- ~500 signals for LHC – 50 MHz bandwidth *(+ ~1000 in PS/SPS)*
- Distributed cPCI system using analogue MPX and oscilloscope modules *(Acqiris or other types) *close to the equipment
- Triggers through the timing network for precise time correlations
- Standard 3-tier architecture.

**The ancestor**

## Real-Time Feedback systems

- LHC orbit feedback
- 2000 Beam position parameters
- 1000 steering dipoles
- 10 Hz frequency

-   LHC tune feedback
- Modest system – 4 parameters and some 30 PCs (up to 50 Hz ?).
-    LHC Chromaticity feedback
- Considered but difficulty to have reliable measurements

## Orbit Feedback system

- Centralized architecture
- > 100 VME crates involved
- Through the Technical network
- Tests on SPS in 2004 successful
- Simulations show 25Hz capability

## Networking

- General Purpose Network (GPN)
- Desktop Computing, testing, access from outside, ...
- Technical and Experiment Network (TN and EN)
- Only operational devices
- Authorization procedure
- Inter domain communications
- Application Gateways + Trusted services
- Network monitoring and intrusion detection
- Performance and statistics
- Disconnection on “breakpoints”
- Testing
- TOCSSiC (hostile network environment)

## Security Policy

- Network Domains
- Physical network segregation + Functional Sub-Domains (FSD)
- Hardware Devices
- No USB, modems, CDs, wireless ...
- Operation System
- Central installation + Strategy for security patches
- Software
- Development guidelines, installation and test procedures
- Logins and passwords
- Traceability, no generic accounts, strong passwords
- Training
- Security Incidents and Reporting

## Cheat Sheet

| FNAL | CERN |
| --- | --- |
| TLG | CBCM/MTGs |
| MDAT/TCLK | LHC Telegram/Machine Cycle;Events |
| ACNET protocol | Controls Middleware (CMW) |
| MOOC | FESA |
| VxWorks | LynxOS/Linux |
| Data Acquisition Engine (DAE) | Application Server |
| MECCA | ant |
| VMS/Windows/Solaris/Linux | Linux |
| C/Java | Java |
| Sybase | Oracle |
| Lumberjack | Timber |
| NUMI | CNGS |
| Behind the firewall | On the Technical Network |

## Summary

- New control system being developed for the LHC that will eventually be used by all CERN accelerators. This is highly ambitious particularly on the required time scale.
- Some qualitative, philosophical similarity to FNAL system; but totally different implementation.
- More sophisticated device model. Attempts at better information management, more reuse of machine software among various accelerators and transfer lines.
- Infrastructure is new and will likely have growing pains.
- There is an enormous amount of hardware to install and software to write and make work in less than 2 years...