# Presentation: 702700

## Fermilab Control System(“ACNET”)

- J. Patrick
- February 17, 2005

## Fermilab Control System

- Overview
- Applications
- Central Services
- Front-ends/Instrumentation
- Timing
- Security and Remote Access
- Summary

- J. Patrick

- SMTF Review 2/17/05

## Fermilab Control System

- aka “ACNET”
- Unified control system for the entire complex
- 400 MeV Linac
- 8 GeV Booster Synchrotron
- 120 GeV Main Injector Synchrotron
- 1 TeV Tevatron Synchrotron
- antiproton source – target/debuncher/accumulator
- antiproton “Recycler” storage ring
- fixed target lines
- Simultaneous operation of
- Tevatron proton-antiproton collider (storage ring)
- Antiproton production and storage (~0.5 Hz)
- 120 GeV fixed target to Meson lab (~0.1 Hz)
- 120 GeV fixed target to NUMI/MINOS (~0.5 Hz)
- 8 GeV fixed target to MiniBoone (~5 Hz)

- J. Patrick

- SMTF Review 2/17/05

## Fermilab Accelerator Complex

- J. Patrick

- SMTF Review 2/17/05

## Control System Overview

- Console
- Applications

- CentralServices

- IRM
- Front-Ends

- Java
- Applications

- Web
- Applications

- MOOC
- Front-Ends

- Labview
- Front-Ends

- Open Access
- Clients

- Field Hardware

**ethernet**

- Database

**CAMAC, VME, PMC, IP, Multibus, CIA, GPIB, ...**

**field bus: VME, SLD, Arcnet, ethernet, ...**

- Servlets

- Consolidators

**Application**

**Central**

**Front-End**

- J. Patrick

- SMTF Review 2/17/05

## History

- Originally developed for Tevatron in early 80’s
- Substantial evolution over the years
- CAMAC field hardware -> very broad diversity
- PDP-11/Mac-16 front-ends-> x86 Multibus-> VME based processors + Labview
- DEC PCL network-> IEEE 802.5 token ring-> ethernet
- VAX database -> Sybase
- VAX central services -> Java
- PDP-11 applications -> VAXstations-> Java application infrastructure-> Port of VAX code to Linux in progress

- J. Patrick

- SMTF Review 2/17/05

## System Scale

- 200,000 devices/350,000 properties
- 475 front-ends
- 90 Sun Netras + 12 Linux PCs that run Java based data loggers, servlets, open access clients
- + VAXes to run VAX central services
- 70 VAXes that run high console level applications
- + 70 Linux PCs to to parallel VAXes until VAX retirement.
- X-displayed out to desktop PCs

- J. Patrick

- SMTF Review 2/17/05

## Device Model

- <= 8 character devices names
- e.g. T:BEAM
- Informal conventions provide some rationality but no enforcement
- More verbose hierarchy layer created, but not utilized/maintained
- Each device has one or more properties:
- Reading, setting, analog & digital alarms, digital status & control
- Device descriptions stored in Sybase database
- Scaling transformations defined in device database
- Add new devices with “DABBEL” command language on VAX
- Templates available for various device types
- View/modify device definitions with GUI
- Monotype data, arrays are transparently supported
- Structured data is often done, though support is not transparent
- Application (library) layer must properly interpret the data from the front-end and perform required data translations

- J. Patrick

- SMTF Review 2/17/05

## Device Database GUI

- J. Patrick

- SMTF Review 2/17/05

## Communication

- Applications/Central Services/Front-ends communicate via “ACNET” protocol
- Custom protocol now exclusively based on UDP
- Extensive network/switches etc...
- Asynchronous request/reply; request multiple reply
- Timeout/retry mitigates UDP unreliability.
- “RETDAT/SETDAT” protocol for readings/settings
- Can have multiple devices/packet
- Fast Time Plot (FTP) returns blocks of readings taken at up to 1440 Hz every 0.5 seconds
- Snapshot protocol returns blocks of 2048 points taken at an arbitrarily high rate

- J. Patrick

- SMTF Review 2/17/05

## Communication

- Application communication with front-ends passes through central layer
- Datapool Manager on VMS via shared memory
- Data Acquisition/Server Engine (DAE) in Java (via RMI)
- Consolidation of requests to avoid overloading front-ends
- Centralizes security, monitoring, error logging
- Issues include:
- Limited packet size and internal device address space.
- Timestamps not part of protocol packet, applied on data receipt
- GPS sync clock event sent to all front-ends, can be included in structured devices
- New get/set32 protocol addresses the above, only on centrals now
- Lack of **transparent** support for structured data.
- Bridging to network devices that don’t directly support ACNET is via front-ends or Open Access Clients

- J. Patrick

- SMTF Review 2/17/05

## Database

- Central Sybase database
- Three parts:
- Device and node database
- Application database
- Used by applications as they wish
- Save/Restore/Shot Data database
- MySQL used for data loggers, and for some other specialized purposes. Not for general application use

- J. Patrick

- SMTF Review 2/17/05

## Application Environment

- Two major high level application environments
- Console environment
- VAX based
- C language (+ legacy FORTRAN)
- Port to Linux in progress
- Expect some operational utilization in the fall
- Java environment
- Platform independent
- Windows/Sun/Linux/Mac all supported and utilized
- Full access to entire control system
- Both environments have a standard application framework that provides a common look and feel
- All applications launched from “Index Pages”
- Currently separate pages for Console, Java environments
- Also some browser applications of varying styles

- J. Patrick

- SMTF Review 2/17/05

## Index Page (Console)

- J. Patrick

- SMTF Review 2/17/05

## Application Index (Java)

- Web Browser

- Standalone Application

- J. Patrick

- SMTF Review 2/17/05

## Application Overview

- Utility applications
- Parameter Page
- Fast Time Plot
- Data Logger viewer
- Database Viewer
- Synoptic displays
- Diagnostics
- Save/Restore
- Alarms
- ...
- Machine Specific Applications
- ~600 Primary + 125 Secondary applications in console environment
- >100 applications in Java environment
- Mostly utility, shot data analysis

- J. Patrick

- SMTF Review 2/17/05

## Parameter Page (Console)

- J. Patrick

- SMTF Review 2/17/05

## Parameter Page (Java)

- J. Patrick

- SMTF Review 2/17/05

## Fast Time Plot (Java)

- J. Patrick

- SMTF Review 2/17/05

## Synoptic Display (Console)

- Graphical Builder“Lex Draw”

- J. Patrick

- SMTF Review 2/17/05

## Synoptic Builder (Java)

- J. Patrick

- SMTF Review 2/17/05

## Synoptic Display (Web)

- J. Patrick

- SMTF Review 2/17/05

## Sequencer (Console)

- Automatically Sequence Collider Shots, other  activities
- Relies on ACL scripting language

- J. Patrick

- SMTF Review 2/17/05

## Finite State Machine (Java)

- Graphical Builder
- Primary current usage is Pelletron control
- Keeps track of HV sparks and beam related trips

- J. Patrick

- SMTF Review 2/17/05

## Sequenced Data Acquisition

- Automated acquisition of data for collider shots
- Configure readings for various stages of collider shot
- Snapshots/FTP as well as single scalar readings
- SDAEdit tool makes it easy to add new things
- Data acquired automatically and stored in Sybase database
- Summary html reports generated when store goes in
- Step efficiencies, error reports, luminosity
- “Supertable” captures ~200 most popular items
- Html, Excel, Java Analysis Studio formats automatically generated
- Suite of tools to view detailed data
- Data can be directly read by Excel, Java Analysis Studio, potentially other external tools
- Simple Java API for user written code

- J. Patrick

- SMTF Review 2/17/05

## Alarms Display

- J. Patrick

- SMTF Review 2/17/05

## Scripting

- Accelerator Control Language (ACL)
- Console environment
- Access to much of the functionality in the control system
- VMS DCL style syntax
- Extensively used in Sequencer
- Jython
- Java API in principle accessible to Jython
- Some but limited testing/use to date

- J. Patrick

- SMTF Review 2/17/05

## External Tools

- Excel
- Excel based parameter page
- Easy cut/paste of Java tables into Excel
- Excel data can be imported into a variety of tools
- MathCAD, OptimA used locally
- MATLAB
- MATLAB supposedly has a standard Java interface, in theory it should be straightforward to integrate with the control system
- However we haven’t tried it
- An attempt to do this with Mathematica > 1 year ago was not successful, it does not use a standard Java distribution
- Java Analysis Studio
- Currently data logger and shot data plug-ins

- J. Patrick

- SMTF Review 2/17/05

## Excel Parameter Page

- J. Patrick

- SMTF Review 2/17/05

## Java Analysis Studio

- J. Patrick

- SMTF Review 2/17/05

## Save/Restore

- Devices may be assigned to save/restore lists
- Console applications to save/restore to/from database on user request
- Scheduled saves of the entire control system 4 times/day

- J. Patrick

- SMTF Review 2/17/05

## Electronic Log

- Based on Oak Ridge product
- Web browser client
- Perl/cgi back-end
- Shift log + machine / subsystem logs
- Used by A0
- API to add entries programmatically, including images
- Both Console and Java environments
- Java application framework includes standard menu item to make posts.

- J. Patrick

- SMTF Review 2/17/05

## Central Layer

- Open Access Clients (“virtual front-ends”)
- Java Servlets
- Bridge requests from outside firewall
- Request Consolidation
- Alarms server
- Front-end download
- Other things ...

- J. Patrick

- SMTF Review 2/17/05

## Open Access Clients

- Processes with no user interface, always running
- Obey same communication protocol as front-ends
- Compare to EPICS “Soft IOC”
- Several classes:
- Utility – Data loggers, scheduled data acquisition, virtual devices
- Calculations – Both database driven and custom
- Process control – Autotune of fixed target lines
- Bridges –to ethernet connected scopes, instrumentation, control systems...
- Easy access to devices on multiple front-ends
- Friendlier programming environment than VxWorks front-ends
- Framework transparently handles ACNET communication
- Access to database, other high level operating system features
- Do not provide hard real-time response
- Clock events via ethernet multicast
- > 100; Almost all run in Java framework; VAX framework deprecated

- J. Patrick

- SMTF Review 2/17/05

## Data Loggers

- ~70 instances running in parallel on central nodes
- ~80 GB compressed storage per node in MySQL database
- Circular buffer; wraparound time depends on device count/rates
- Data at maximum 1 Hz rate extracted to “Backup” logger every 24 hours
- Backup logger is available online => data accessible forever
- Generally allocated to various departments to configure as they wish
- No central management of archiving lists, so inevitable duplication
- Tasks follow the Open Access Client architecture
- Simple GUI to specify logging lists
- Single readings or snapshots; on clock events or at fixed rates to 15 Hz
- Programmatic logging interface available for custom loggers
- Variety of retrieval tools available
- Standard console applications, Console and Java environments
- Web applications
- Java Analysis Studio
- programmatic APIs

- J. Patrick

- SMTF Review 2/17/05

## Front-Ends

- Interface field hardware to the high level control system
- Several architectures
- MOOC – Minimally Object-Oriented Communication
- Very wide diversity of functionality
- Processors all VME (VXI) based
- Very wide diversity of field hardware supported
- Links to many other form factors such as CAMAC, etc.
- MVME 162 (68k) and 2400 (Power PC)
- VxWorks Operating System
- ~275 in the system

- J. Patrick

- SMTF Review 2/17/05

## Front-Ends - IRM

- IRM – Internet Rack Monitor
- MVME 162; PSOS operating system
- Chassis includes 64 channel x1 KHz 16 bit continuous digitizer
- 8 x 10-20 MHz option available
- 8 bytes external digital I/O
- 8 x 12 bit DAC outputs
- Expandable via IndustryPak I/O
- ~125 in the system
- Eight used at A0 photo-injectorVariety of functions
- A few at TTF for modulators

- J. Patrick

- SMTF Review 2/17/05

## Front-Ends

- Linac front-ends
- VME based Power PC processors
- Non-MOOC infrastructure; tied to 15 Hz Linac cycle rate
- ~50 in the system
- HRM – Hotlink Rack Monitor
- Evolution of IRM architecture
- I/O chassis separate from processor chassis
- 64 x 10 KHz ADC; up to 2 per chassis
- 10-20 MHz snapshot board
- Timestamps included with ADC data
- 8 byte digital I/O per chassis
- Multiple chassis’ per PPC processor
- VxWorks (not MOOC)
- Initial build of 15 units in progress

- J. Patrick

- SMTF Review 2/17/05

## Labview

- Front-Ends
- General control/acquistion from high level controlsystem
- Interact via ACNETprotocol written inLabview VI language
- No FTP/Snap
- Non-local consoleaccess viaPCAnywhere,Timbuktu etc.
- ~25 in system
- Instrumentation:
- Flying Wires
- Sample Bunch Display
- ...
- Both PCs & Macs

- J. Patrick

- SMTF Review 2/17/05

## Field Hardware

- Wide variety of hardware supported
- CAMAC, VME, VXI, Multibus, CIA, ...
- Too extensive to describe here...
- Support for wide variety of commercial instrumentation
- Scopes, spectrum analyzers, signal generators, DVMs, ...
- Ethernet->GPIB; direct ethernet
- Through front-ends; through OACs; direct in some cases
- Mostly in console environment, ports to Java as needed
- Siemens-Moore APACS PLC supervisor
- Used for cryogenic control
- Interface through Open Access Client using library from company
- Other PLCs also supported through front-ends

- J. Patrick

- SMTF Review 2/17/05

## Bridge to External Systems

- Accelerator NETwork control system (AccelNET) is a software package designed for control of electrostatic particle accelerator systems.
- Runs on PC hardware under the Linux operating system
- Modular Linux programs that communicate to each other through a central database server using TCP/IP.
- CAMAC- GROUP3 I/O
- ACNETAccelNET Bridge

- J. Patrick

- SMTF Review 2/17/05

## Timing and Control Networks

- TCLK (Clock Events)
- 10 MHz Manchester encoded signal with 8 + 2 bits/event
- 256 possible events defined for the complex
- Decoding hardware/signal generators for variety of buses
- Standalone version of this system used at A0 photo-injector
- MDAT (Machine Data)
- 10 Mbit/sec serial link; 8 bit type + 16 bits data per frame
- Originally developed to rapidly distribute ramp currents around the main ring and Tevatron; usage has broadened
- Beam Sync – revolution markers derived from RF
- States
- State devices defined
- Any node may issue transition request, gets multicast everywhere
- Can be distributed via MDAT
- Beam permit/abort
- CAMAC interfaces....

- J. Patrick

- SMTF Review 2/17/05

## Security

- Control system network inside a firewall
- Restricted access from/to outside the accelerator division network
- Privilege classes assigned based on account, service, (node)
- Control from where applications may be run
- Control which programs/users/nodes may do settings
- Emphasis on accountability as well as security
- Setting log
- Application log

- J. Patrick

- SMTF Review 2/17/05

## Remote Access

- Console environment via remote X display
- Administrative controls on usage, particularly settings etc.
- Java applications can be launched from anywhere from a browser via Web Start
- Access to inner parts of control system only via  VPN
- Applications to be run remotely w/o VPN must be designed for that
- Applications that work via servlets;
- “Secure Controls Framework” – Subset of data access API implemented in this way, better performance than servlets/xml
- Access from outside the standard application environments
- xml-rpc is preferred method
- Readings of any device; setting of “virtual” devices
- No setting of hardware associated devices
- Used by all current experiments

- J. Patrick

- SMTF Review 2/17/05

## ACNET Summary

- Should meet requirements for SMTF
- Has been/is being continually modernized
- Extensive experience at FNAL.
- Less time to get a system running
- A broad range of people could contribute quickly
- Limited experience outside of Fermilab
- Remotely accessible but full system not portable
- Documentation...

- J. Patrick

- SMTF Review 2/17/05