# Presentation: 357866

## A New Vision for the Evolution of Controls 

- J. Patrick, C. Briegel, B. Hendricks
- APT Seminar
- February 21, 2008

## Motivation

- Recent Accelerator Initiatives
- HINS
- NML
- Accelerator NUMI Upgrade (current complex without Tevatron)
- More recently Project X
- Bottom-up decisions made on control system choice:
- Needed a specific feature or support for specific hardware
- Support external collaboration
- No explicit vision for an accelerator complex-wide controls beyond 2010
- Ended up with a myriad of different controls:
- NUMI Upgrade - ACNET
- HINS; NML: EPICS; EPICS/DOOCS + ACNET
- A0: DOOCS + ACNET
- Project X: ?
- How do we integrate and support these to meet the needs of one large operating complex?

- J. Patrick

## One Control System

- Project X:
- Substantial new facility that would be built here
- Catalyst for thinking about the evolution of the accelerator complex-wide Control System
- Step back and understand WHAT a control system should do, rather than how it should be done.
- Independent of Project X
- NUMI upgrade supposed to operate beyond 2018
- Demanding operational requirements
- For maintainability, older hardware must be replaced
- System released from those constraints
- Key architects may retire, maintenance becomes difficult and upgrades become attractive

- J. Patrick

## Bottom Line

- Fermilab is the premier HEP Accelerator Laboratory in the nation, our control system needs to evolve, and we have taken the first step to understand what is needed.

- J. Patrick

## Requirements

- What is the functionality of a control system?
- Separate from how it will be done (which is the Design phase).
- We did not decide on or even discuss specific technology choices (EPICS, DOOCS, ACNET, or specific hardware, etc.)
- Requirements are:
- Agreement between users and developers on needed functionality
- If a requirement can not be met, it needs to be revisited and modified
- A controlled process
- An objective way to evaluate design and technology choices
- A chosen implementation has to meet the requirements.
- A standard in industry
- A proven way to build in quality upfront.Software contributes 30% of the failure rate. [see High Availability: A Design Primer]

- J. Patrick

## Process

- Series of discussions among Controls, Operations, HINS + Computing Division people
- Each section written by subgroup of experts
- Coordinated by and final document edited by Suzanne Gysin (CD)
- Needs broader input
- From machine experts and operations
- Hardware experts
- Not considered final, will evolve with more input

- J. Patrick

## Controls Requirements Document

- Contents
- _1_	_Introduction (J. Patrick, S. Gysin)_
- _2_	_Base Requirements (J. Patrick, S. Gysin)_
- _3_	_Low-Level Systems (R. Neswold)_
- _4_	_Central Services (A. Petrov)_
- _5_	_Application Infrastructure (A. Petrov)_
- _6_	_High Level Applications (B. Hendricks)_
- _7_	_Controls In A Box (S.Gysin)_
- _8_	_Beam-Based Feedback (J. Patrick)_
- _9_	_Machine Protection System (B. Hendricks)_
- _10 Software Development Environment (C. Schumann)_
- _11 Hardware/Operating Systems (D. Finstrom)_
- _12 Networks (D. Stenman)_
- _Document is available: __https://beamdocs.fnal.gov/AD-private/DocDB/ShowDocument?docid=2934_
- Internal Review (January), i.e. we agree that this is what is needed in an evolved control system.

- J. Patrick

## LHC controls architecture diagram

*TCP/IP communication services*

*TCP/IP communication services*

*GIGABIT  ETHERNET  TECHNICAL  NETWORK*

*FILE SERVERS*

*APPLICATION SERVERS*

*SCADA SERVERS*

*VME Front Ends*

*Front Ends*

*PLCs*

***MACHINE***

*TCP/IP communication services*

*TIMING GENERATION*

***MACHINE***

*ACTUATORS AND SENSORS*
*CRYOGENICS, VACUUM, ETC...*

*QUENCH PROTECTION AGENTS,*
*POWER SUPPLIES, ...*

*BEAM POSITION MONITORS,*
*BEAM LOSS MONITORS,*
*BEAM INTERLOCKS,*
*RF SYSTEMS, ETC...*

- WorldFIP SEGMENT
-  (1, 2.5 MBits/sec)

- OPTICAL FIBERS

*Front Ends*

*ANALOG*
*SIGNAL*
*SYSTEM*

*LHC controls architecture diagram*

- Hardware Tier

- Application Tier

- Service Tier

## Challenges

- Support large multi-accelerator complex
- Many simultaneous modes of operation
- Automation, ease of configuration
- Large number of hardware devices
- Configuration management, software support
- Easy access to information
- Many simultaneous users
- Scalable data acquisition
- Application development
- Reliability - robust against single component failure
- Better stability as beam power increases
- Diagnostics, Fast Feedback, Data Logging

- J. Patrick

## Today’s Presentation

- Present the highlights in each section.
- Invite user feedback:
- To give feedback, please send us email (gysin@fnal.gov, patrick@fnal.gov)

- J. Patrick

## Document Highlights

- Clock system with more information
- Correlated data
- Front end/service framework consolidation
- Device locking/ownership
- Self-describing mixed type data format
- Improved server-side application environment

- J. Patrick

## Document Highlights

- Information at your fingertips
- Generic device search
- Open applications with devices
- Copy/paste data between applications
- Controls in a box
- Infrastructure for beam-based feedback
- Bug tracking system
- Supported graphical development environment
- Development network

- J. Patrick

## Timing System

- XClock
- 1.2 usecs per event
- 16 bit event
- 32 bit event counter
- Self-describing payload
- 1 GHz phased to TClock

- C. Briegel

## Front-End

- Common Framework Available at all Tiers
- Common Software Development
- Memory Protection
- Maintain Data Connection
- Device/System Ownership
- Acquire Data via Multicast
- Debugging, Post-mortem

- C. Briegel

## Front-End

- Data Acquisition
- Client Specification
- One-shot
- Repetitive
- Event+Delay
- Event Sequence Number - Correlated Data
- On-change+Periodic
- Array on Event
- Array on Alarm
- Real-time (1 KHz)

- C. Briegel

## Front-End

- Data Acquisition
- 2.  Returned Data
- Raw or Scaled
- Status
- Time Stamp
- Event Sequence Number
- Self-Describing

- C. Briegel

## Front-End

- Setting
- 1.  Rapid
- Forwarded
- Archived
- Atomic Set/Read

- C. Briegel

## Front-End

- Alarms
- Events
- Exceptions
- Default Scan
- Extensible Scan
- Bypass Devices or Groups of Devices

- C. Briegel

## Network

- General
- 			DMZ/Firewall
- 2.  Operational
- 3.  Development/Test

- C. Briegel

## Network

## Server-Side Application Environment

- Server-side components run continuously, on a schedule, or on-demand
- Run on machines maintained by Controls Dept
- Server-side components in a control system:
- Central Services*(e.g., data acquisition, data logging)*
- Server-side parts of multi-tier applications*(e.g., Autotune)*
- Web applications*(e.g., Supertable, Synoptic Display) *

- B. Hendricks

## Server-Side Application Environment

- Run unattended
- Require an automated tool to start, stop, and monitor the operation
- Do not have user interface
- Require a means to report the status to users (logging)
- Multi-component
- Components (Data Acquisition Engine, OAC, servlets) are dynamically linked to each other
- Need a server-side application environment to perform these common tasks

- B. Hendricks

## Server-Side Application Environment

- Several separate “environments”:
- Data Acquisition Engine with OACs and data loggers
- Tomcat for servlets
- Custom shell scripts
- Limited number of supported component types
- It's hard to maintain custom components other than OACs, data loggers, or servlets.
- Complicated development and deployment process

- B. Hendricks

## Server-Side Application Environment

- Use a standard application server for server-side components (including shell scripts)
- Customize the server for our needs
- Provide common services, such as authentication
- Facilitate development and deployment for server-side components
- Eventually have a cluster of application servers for redundancy and load balancing

- B. Hendricks

## Information at Your Fingertips

- Information is available but scattered through applications
- Inconsistent availability of information
- Inconsistent means for accessing information

- B. Hendricks

## Information at Your Fingertips

- Need a generic solution
- Should be provided by the application framework
- Simple click and menu system
- Context sensitive menus
- Device names
- Node names
- Error strings

- B. Hendricks

## Information at Your Fingertips

- Device names
- Device database attribute information
- Device presence in standardized lists
- Datalogger, alarm, save lists
- Save files
- Saved plot setups
- Families of devices
- Send device to another application (including plots)
- Find related devices
- View alarm and setting history

- B. Hendricks

## Information at Your Fingertips

- Node names
- Keeper name(s)
- Other static attributes of the node
- Devices belonging to the node
- Current status of the node
- Error strings
- Detailed description of the error

- B. Hendricks

## Bug Tracking System

- Provides communication path
- Provides persistence of information
- Maintains history
- Provides accountability
- Requires commitment of the user community
- Provides a mechanism for getting issues resolved

- B. Hendricks

## Bug Tracking System

- Fix List
- Pros
- Web based
- Easy to make an entry
- Can add images
- Easy to link to our electronic logs
- Cons
- Passive system
- No assignment of responsibility
- Problems can remain unresolved
- No standard removal mechanism when resolved

- B. Hendricks

## Bug Tracking System

- Remedy (Help Desk)
- Pros
- Entry method is web based
- Active system with reminders
- Supports assignment of responsibility
- Cons
- More complicated issue entry
- Removal method is not straight forward
- Doesn’t support attaching images
- Limits who can be assigned to resolve an issue
- Not managed by us

- B. Hendricks

## Bug Tracking System

- What we want in a new system
- Open source or commercial product
- Web based interface
- Simple issue entry and removal
- Ability to assign responsible parties
- Notification upon entry, update, or removal
- Ability to attach images
- Easy linkage to our electronic logs
- Managed by the Controls Department
- Usable by other entities in the division/lab

- B. Hendricks

## Controls in a Box

- Control system provides many services
- Large footprint with expensive commercial software
- What about a local or remote system developer?
- What about small testing facilities or experiments?
- Need to be able to scale down

- B. Hendricks

## Controls in a Box

- Must be able to stand on its own
- Simple to download and install
- Easy configuration tool
- Shall not use expensive commercial software

- B. Hendricks

## Controls in a Box

- Shall provide a local clock (hardware or software)
- Shall provide for local data storage
- Shall include necessary central services including database
- Shall include basic applications for reading and setting data
- Self-contained

- B. Hendricks

## Next Steps

- Solicit more input
- Meet with broader user base
- Select what ideas to implement according to the following criteria:
- Benefit the current program, NUMI Upgrade, and Project X
- Incorporating user input
- Prevent a divergence in Controls

- J. Patrick

## Summary

- Realistic vision of a control system beyond 2010
- Presented specific ideas and the selection criteria to start implementing
- Independent of Project X, Fermilab is the premier Accelerator Laboratory in the nation, our control system needs to evolve
- Ready to provide an organized and systematic evolution of the control system for the FNAL accelerator complex

- J. Patrick