# Presentation: 235600

## Diagnostics Software Team

- W. Blokland
- Liyu
- C. Long
- M. Sundaram

## FY05 Tasks

## Short Term Tasks

## Slide 4

## Server:Network Attached Device
PC-based with XP embedded
LabVIEW with Shared Memory to IOC and Channel Access
Client: Any Channel Access Client, e.g.:
EDM
LabVIEW
Archiver

- PC-based with XP embedded
- LabVIEW with Shared Memory to IOC and Channel Access
- Client: Any Channel Access Client, e.g.:
- EDM
- LabVIEW
- Archiver

- Current Diagnostics Architecture

## NAD Software Architecture

- PC with XPe

- EPICS IOC+CA

- Shared Memory

- Template tools

- Channel Access

- Application

-  Standard Examples
-  NADs implemented: BCM, BPM, ES, FC, TTS
- (M. Sundaram,  C. Long, W. Blokland, LANL, BNL)

-  db & cmd file generator at startup
-  Development tool
- -Cloning/documentation

-  Shared Memory Records
-  LabVIEW Library
- (D. Thompson)

- Accelerator Controls

- LabVIEW

- Graphical programming
-  Drivers to DAQ
-  Processing routines
-  Graphs/Plots

## Client Software Architecture

- Mac/Linux/PC

- CA/LV Interface

- LabVIEW

- Display

- Channel Access

- App

- Examples:
-  Channel13: dynamically loaded display modules
-  NAD Control screens (W. Blokland, D. Purcell, M. Sundaram)

-  Drag and Drop Display Client for implementing screens that don’t require programming (W. Blokland)
-  Tools to select data to log or set (M. Sundaram)

- Accelerator Controls

- Graphical programming
- Processing routines
- Graphs/Plots

- EPICS CA

- Our first implementation of supervisory level uses the client software architecture

-  Multi-platform Interface  to buffer CA messages and handle interrupts (A. Liyu)

## Diagnostics Architecture

- Supervisor

- Console

- Altiris

- Database

- Holds all individual instrument configuration and program data and describes instrument to instrument relations (all diagnostics)

- Manages NAD:
-  Backup, reboot, upload
-  Retrieves and stores configuration and software on database (D. Purcell, D. Murphy)

- Monitors NADs:
-  analyzes results
-  monitors settings
-  provides alerts

- Tools to:
-  handle alerts from super
-  supervise NADs: e.g look at all BPM settings
- (M. Sundaram, C. Long, w. Blokland)

- NAD

- results

- settings

- Program files

- Program files

- Program files

- Development
- NAD

- results

- Program files

- results

- Instrument layout

- Identity

## Hierarchical System

- Planning

- Execution

- Supervisory

- Coordination

- Control

- Group Control. For Example: Laserwire Transfer line

- Implementation

- NAD

- NAD

- Server

- Plant-wide data collection and optimization

- Data-acquisition and processing:
- PLCs, NADs

- Operations

- Management

- Run Coordinator

- Applications

- Network

- (Private) Network

- Wires

- Sensor/Actuator

- Operation of the plant, executing
- the plan, e.g run beam to 50mA

- Long term planning of the accelerator,  e.g.CD4 goal

- Levels

- Examples

- Accelerator

- Toroids, BMP plates, Harp, etc

**Diagnostic Data Servers collect data from instruments to perform on-line statistical analysis and derive system performance**

## Planning

- Execution

- Supervisory

- Group Control

- Control

- Time Scale

- Data

- Accelerator

- Small amount of
- high level data

- Large amount of
- Low level data (60Hz)

- > Days

- < Seconds

- Minutes

- Seconds

- Hour

- Levels

## Hierarchical System Lower Levels

- Supervisory

- Coordination

- Control

- The Supervisory level optimizes the operation of the accelerator. This is where:
- Data is collected from the whole accelerator
- Data is analyzed to evaluate performance
- Determination is made how to set devices

- The Coordination level combines a group of systems in the control level to act as one to the supervisory level
- Data is collected from its similar subsystems
- Data is compressed and sent upward
- Commands are send downward to coordinate operation

- The Control level is our NAD. It stands by itself to acquire, process , and make the data available. It should not know about other nodes.
- 1) Data is compressed (=analyzed) within the context of a single node

## Supervisory& Coordination Levels Features

- Supervisory

- Coordination

- Desired features:
-  On-line Accelerator Performance Analysis:
-  Use data from multiple NADs (results to EPICS/database)
- Use of smart triggers to acquire/analyze and log/mark data
- Qualification&correction of measurements
-  Create reports to EPICS/HTML/Oracle
-  Tools to acquire and view data (in and outside ICS)

- Combine multiple NADs into one virtual NAD to:
- Reduce data going up the hierarchy
- Hide complexity: present multiple NADs as one NAD

## Supervisory Level: Current Status

- One CPU
- No concentrator
- CA only
- LabVIEW
- Summary data on webpage
- >1 sec timescale, 180 PVs

- First “Logger/Setter” implementation: Console Tools
- Tools to view and set  diagnostics PVs from console
- Mass select of PVs
- Use file to load PV names (should become database)
- Log data for later analysis

- Concentrator (Middleware) for BLM by Andrei, Sasha
- One CPU per server
- Calculations
- Works with packaged data

## Supervisory Level: Current Status

- EDM CA User

- BPM NAD

- BCMNAD

- Concentrator

- Channel Access

- Ch13 CA User

- Presents instrument/detectors to control system

- Current configuration
-  Typically dedicates one CPU per task
-  No global database

- Users:
-  Console
-  Database
-  Monitor
-  Archiver
-  PV gateway

- Offloads computations from multiple IOCs

- BLM

## Supervisory Level Wish List Details

- Can request data from any IOC/NAD
- Gets correlated data
- Can get data from database
- Has generic interface to request and get data
- Can generate results that are available to EPICS and a database
- Can get data using different protocols
- Can be written in different languages
- Runs on-line or off-line
- Can handle all data rates

- Wish list for off-line and long term data
- Handle all requested data streams
- Easy access
- Fast access
- Access from within and outside ICS
- Tools for remote users (display, retrieval, programs, etc)
- ...

## Supervisory Level Implementation

- Do we use a cluster?
- Do we need a real-time data pool/middleware server?
- What database?
- Do we need data-concentrators to reduce bottlenecks?
- Do we use PV gateway?
- Do we use EPICS archiver?
- ...

- Guidance
- Least amount of work for most amount functionality -> use other peoples work & commercial software
- Start simple (Ch13, Console tools, middleware)
- Provide teaser quickly
- Information gathering this month

- Test Beds
- NADs/PCs
- Xserve
- PC server

- Development Environments
- LabVIEW
- Visual C/Basic (C++, ...)
- Xcode (C++,Cobj, Java)

## Supervisory Level: Future

- Database
- (long term storage)

- NAD

- NAD

- NAD

- NAD

- Analyzer Z

- Supervisory Cluster

- Real-time
- data pool

- Data

- Data

- Request

- Data

- Data

- Analyzer Beam Accounting

- Data

- RAID

- Gateway

- SMB

- CA

- Processed Data

- Analyzer Trend A

- Gateway

- Processed Data

- Could look like this:

**Notes:**
The data gateway acquires the data over the net using, e.g. Channel Acess. The data is stored in a real-time pool for the on-line monitors and also send to a database for long term storage. The real-time data pool can be a RAM disk or anything that is fast enough. 
Trend analysis and off-line monitors retrieve their data from the database which provides corellated data if so requested. Data is stored toa RAID system where the database and others can access the data.
A monitor process tells the acquisition process what PVs it wants and for how long it needs it. The acquisition process makes sure that the data is acquired and available. Additional tools are available to make data-acquisition request to the concentrators without having an on-line or off-line monitor available
Most of the software must be commercial. If not commercial, then in corporation with others (think of how the instrument group is data archiving). The monitors must be independent of the acquisition protocol (Channel Access or ...) to be able to use as many commercial tools as possible., Matlab, mathematica, IDL,etc. Multiple data-concentrators are sued to write to the same data pool if different acquisition protocols are  used. The caveat is speed, above all it must be fast enough to be of use for on-line processing. Initially, the on-line monitors will operate in human time (seconds). 

## Control Level: Waveform Data

| Instrument | Waveform data | Rates @ 6 Hz | Rates @ 60 Hz |
| --- | --- | --- | --- |
| BLM (295) | 400 samples * 2 channels * 8 bytes | 11.2 Mbytes/sec | 112 Mbytes/sec |
| BCM(21) | 1500 samples * 8 bytes‡  | 1.5 Mbytes/sec | 15 Mbytes/sec |
| FC(7) | 1500 samples * 8 bytes‡ | 70 Kbytes/sec† | 700 Kbytes/sec |
| BPM(157) | 1500 samples *4 channels* 8 bytes | 45 Mbytes/sec | 450 Mbytes/sec |
| Harp (1) | 1500 samples * 16 channels * 8 bytes‡ | 200 Kbytes/sec§ | 200 Kbytes/sec§ |
| IPM (2) | 64 samples * 10 profiles *1024 turns *2 bytes | 2.6 Mbytes/sec§ | 2.6 Mbytes/sec§ |
| Tune(1) | - | - | - |
| Video(2) | 512 lines * 480 pixels *1 byte | 3 Mbytes/sec | 15 Mbytes/sec ¬ |
| Instrument |  | Bytes / Scan |  |
| WS(38) | 200 samples * 4 channels * 8 bytes | 6 Kbytes |  |
| BSM (4) | 5000 samples * 8 bytes | 40 Kbytes |  |
| ES (2) | 5000 samples * 8 bytes | 40 Kbytes |  |
| LS (8) | 500 samples * 2 channels * 8 bytes | 6  Kbytes |  |
|  |  |  | ~500 Mbytes/sec |

- † Only one Faraday Cup to be active at the same time
- § Max Repetition rate is 1 Hz
- ‡ Double precision data could be downgraded to float (4 bytes)
- ¬ 30 Hz frame rate maximum (still unlikely to be attained)

**Notes:**
So that’

## Supervisory Level: Result Data Needs

- Each Scalar has a 8 byte timestamp -> double precision scalar requires 16 bytes.
- Continuous logging needs at 1Hz are very moderate: < 4kbytes/s (60Hz -> 240kbytes/s
- If we want to log waveforms per pulse then for all BPMs this becomes about 40Mbits per pulse. At 1Hz that would be ~40% of the 100Mbit/s ethernet bandwidth.

|  | Normal | High | Data Rates (double prec) |
| --- | --- | --- | --- |
| BPM (157) | Position X and Y Scalar @1Hz  | Position X and Y, Phase waveforms, @pulse | 6kbytes/s
7.5 Mbytes/pulse |
| BCM (21) | Current Max and Min, Scalar, @1Hz | Current waveform
@pulse | 1kbytes/s
50kbytes/pulse |
| BSM (4) | 2D intensity plot,
Results, @scan | 2D intensity plot,
Results, @scan | 10kbytes/scan
Scan < 0.01Hz |
| ES (2) | 2D XX’I array,
Results, @scan | 2D XX’I array,
Results, @scan | 25kbytes/scan
Scan < 0.01Hz |
| WS (36) | Results, @scan | Profiles, Result, @scan | 10kbytes/scan – 100kbytes/multi-scan, Scan < 0.01Hz |
| LS (8) | Results, @scan | Profiles, Result, @scan | 10kbytes/scan – 40kbytes/multi-scan
Scan < 0.01Hz |
| BLM (295) | Report @ 1 hour |  | ~50kbytes/hour (if all value available in coordination level) |

## Coordination Level: Data Needs

| 
 | Normal | High | Rates (double prec) |
| --- | --- | --- | --- |
| BLM (500)
 | Integral Slow Data plus Timestamp @60Hz |  | 500kbytes/s
 |
| LS (8) | Mirror Control @ scan |  | < 1kbyte/s |

- BLM data rates can be reduced by combining single PVs in
- One PV to save on bytes needed for timestamps

**Notes:**
BLM integral of Slow signal. Frequency is 60 Hz Volume is double (8 bytes) +timestamp (8 bytes).During one minute (500 detectors) 57,6 KB * 500 = 28,8 MB
BLM full data. Frequency is one time per minute. Volume consists from fastsignal (220 double), slow signal (400 double), timestamp, gains, HVpotential, HV current, trigger event, trigger delay. It should be lessthan 700 double. During one minute (500 detectors) 5,6 KB * 500 = 2,8 MB
(A. Liyu)

## BLM Configuration

- BLM IOC

- BLM Coordinator

- Concentrator

- Channel Access

-  Scale losses
-  Packages to coordinator

- IOC

- Concentrator

-  Collect all ~1 Hz data (packages and results)
-  Store data per week  (database)
-  Prepare ~1 Minute Summary

- Supervisory Level

-  Collect ~1 Minute Summary
-  Prepare report each hour/day
-  Collect any data for Analyzers

-  Digitizers
-  MPS hardware
-  Packaging to 1 Hz

- 4Mbits/s

- IOC

**Notes:**
Data rate must be  reduced going up the hierarchy, 
The concentrator off-loads the VME IOC by performing 60Hz calculations but produces lower rate, e.g. 1 Hz
The BLM coordinator collects data from multiple concentrators and stores the data for a period of time, e.g. 1 week. It provides a summary to the supervisor who will then use this summary to generate the report.
The supervisory level can also collect data from coordinator or even concentrator as defined by Analyzer modules.
Stored Data on coordinator should be made accessible to Supervisor or the ~4Mbits/sec BLM data stream must be stored by the database on the supervisor.
Questions: How many IOCs per concentrator? How many Coordinators are needed (can we get away with just one?)

## Analyzer setup

-  1Hz
-  Synchronized clustered data

- BCM200
-  6Hz
-  scalar

- BCM11
-  6Hz
-  scalar

- BCM02
-  6Hz
-  Scalar

- BLM1-20
-  1Hz
-  Array

- To define a Measurement Points:
- Select from Measurement Points
- Combine the measurements
- Select trigger
- Monitor
- Advanced trigger
- Select packaging mode
- All
- Slowest common rate
- Select output rate
- All data is packaged and correlated at specified output rate and available to analysis program

- Measurement Points:
- PV(s)
- Measurement Point(s)

**Notes:**
Data rate must be  reduced going up the hierarchy, 
The concentrator off-loads the VME IOC by performing 60Hz calculations but produces lower rate, e.g. 1 Hz
The BLM coordinator collects data from multiple concentrators and stores the data for a period of time, e.g. 1 week. It provides a summary to the supervisor who will then use this summary to generate the report.
The supervisory level can also collect data from coordinator or even concentrator as defined by Analyzer modules.
Stored Data on coordinator should be made accessible to Supervisor or the ~4Mbits/sec BLM data stream must be stored by the database on the supervisor.
Questions: How many IOCs per concentrator? How many Coordinators are needed (can we get away with just one?)