Control System Studio- CSS Alarm Handling
Kay Kasemir, Ph.D.
ORNL/SNSkasemirk@ornl.gov

June 2011 at KEK
Managed by UT-Battelle
for the Department of Energy

Previous Attempts at SNS
ALH; manual “summary” displays; generated soft-IOCs +displays
Issues– GUI• Static Layouts• N clicks to see active alarms– Configuration• .. was bad  Always too many alarms• Changes required contacting one of the 2 experts, wait~days, restart CA gateway, hope that nothing else broke– Information• Operator guidance?
• Related displays?
• Most frequent alarm?
• Timeline of alarm?
2 Managed by UT-Battelle
for the Department of Energy

Now: Best Ever Alarm System Tool

Yes, alarms are always a little scary…
3 Managed by UT-Battelle
for the Department of Energy

Alarm System Components
Cool UI

Alarm Server

Configuration

Control System

1. What you see
2. Technical details
3. How to use it
B. Hollifield, E. Habibi, "Alarm Management: Seven Effective
Methods for Optimum Performance", ISA, 2007

4 Managed by UT-Battelle
for the Department of Energy

1. What you see
Alarm GUI used by Operators

5 Managed by UT-Battelle
for the Department of Energy

What you see: Alarm Table• All currentalarms– new, ack’ed• Sort by PV,Descr., Time,Severity, …
• Optional:Annunciate• Acknowledge one or multiple alarms– Select by PV or description– BNL/RHIC type un-ack’
6 Managed by UT-Battelle
for the Department of Energy

Another View: Alarm Tree
• All alarms– Disabled, inactive, new, ack’ed• Hierarchical– Optionally only show
active alarms– Ack’/Un-ack’ PVs or sub-tree

7 Managed by UT-Battelle
for the Department of Energy

Guidance, Related Displays, Commands

 Basic Text
 Open EDM/OPI screen Open web page Run ext. command
Hierarchical:
Including info of parententries
Merges Guidance etc. fromall selected alarms
8 Managed by UT-Battelle
for the Department of Energy

Integrated with other CSS Tools
 Alarms
 History of PV
 EPICS Config.

9 Managed by UT-Battelle
for the Department of Energy

CSS Context Menus Connect the Tools
Send alarm
PV to anyother CSS
PV tool

10 Managed by UT-Battelle
for the Department of Energy

E-Log Entries• “Logbook”
from context menu
creates text w/basic info about
selected alarms.
Edit, submit.
• Pluggable implementation, not limited toOracle-based SNS ELog
11 Managed by UT-Battelle
for the Department of Energy

Online Configuration Changes.. may require Authentication/Authorization
 Log in/out while CSS is running

12 Managed by UT-Battelle
for the Department of Energy

Add PV or Subsystem
1. Right-click on ‘parent’
2. “Add …”
3. Enter name
Online. No search for config files, no restarts.

13 Managed by UT-Battelle
for the Department of Energy

Configure PV• Again online• Especially usefulfor operators to
update guidance
and relatedscreens.

14 Managed by UT-Battelle
for the Department of Energy

2. Technical details
Behind the GUI;
Tools to monitor performance

15 Managed by UT-Battelle
for the Department of Energy

Technical ViewIOCs
PV Updates (Channel Access, …)

TomcatAlarm Server
Current Alarms: Acknowledged? Transient? Annunciated?
Log Messages

Annunciations

Alarm Updates

Ack’; Config Updates

JMS
LOG

TALK

JMS
2RDB

JMS
2SpeechMessage
RDB

16 Managed by UT-Battelle
for the Department of Energy

ALARM_SERVER-Reports

ALARM_CLIENT

Alarm Cfg & StateRDB

Alarm Client GUI

CSS Applications

General Alarm Server Behavior• Latch highest severity, or non-latching– like ALH “ack. transient”• Annunciate• Chatter filter ala ALH• Alarm only if severity persists some minimum time
• .. or alarm happens >=N times within period• Optional formula-based alarm enablement:– Enable if “(pv_x > 5 && pv_y < 7) || pv_z==1”– … but we prefer to move that logic into IOC• When acknowledging MAJOR alarm, subsequent MINOR
alarms not annunciated– ALH would again blink/require ack’
17 Managed by UT-Battellefor the Department of Energy

Logging• ..into generic CSS log also used for error/warn/
info/debug messages• Alarm Server: State transitions, Annunciations• Alarm GUI: Ack/Un-Ack requests, Config changes• Generic Message History Viewer– Example w/ Filter on TEXT=CONFIG

18 Managed by UT-Battelle
for the Department of Energy

Logging: Get timeline• Example: Filter on TYPE, PV

6. All OK

4. Problem fixed
5. Ack’ed by operator

3. Alarm Server
annunciates

19 Managed by UT-Battelle
for the Department of Energy

2. Alarm Server
latches alarm

1. PV triggers,clears,triggers again

All Sorts of Web Reports

20 Managed by UT-Battellefor the Department of Energy

3. How to use it
This may be more important
than the tools!

21 Managed by UT-Battelle
for the Department of Energy

Best Ever Alarm System Tools, Indeed.. but Tools are only half the issue
Good configuration requires plan & follow-up.

B. Hollifield, E. Habibi,"Alarm Management:
Seven (??) Effective Methods
for Optimum Performance", ISA, 2007

22 Managed by UT-Battelle
for the Department of Energy

Alarm Philosophy
Goal:
Help operators take correct actions– Alarms with guidance, related displays– Manageable alarm rate (<150/day)– Operators will respond to every alarm(corollary to manageable rate)

23 Managed by UT-Battelle
for the Department of Energy

What’s a valid alarm?
• DOES IT REQUIRE IMMEDIATE OPERATORACTION?
– What action? Alarm guidance!
• Not “make elog entry”, “tell next shift”, …
• Consider consequence of no action• Is it the best alarm?
– Would other subsystems, with better PVs, alarm at thesame time?

24 Managed by UT-Battelle
for the Department of Energy

How are alarms added?
• Alarm triggers: PVs on IOCs
– But more than just setting HIGH, HIHI, HSV, HHSV– HYST is good idea
– Dynamic limits, enable based on machine state,...
Requires thought, communication, documentation• Added to alarm server with– Guidance: How to respond– Related screen: Reason for alarm (limits, …), linkto screens mentioned in guidance– Link to rationalization info (wiki)
25 Managed by UT-Battelle
for the Department of Energy

Impact/Consequence Grid
Category

So WhatMinor Consequence

Major Consequence

Personnel Safety

PPS independentfrom EPICS?

Environment,Public

Can EPICS cause
contained spill ofmercury?

Uncontained spill??

No effect

Beam off <10 min

Beam off >10min

Beam off< 1 sec?

<$10000>$10000

Cost:
BeamProduction,Downtime,
Beam Quality• Mostly: How long will beam be off?
26 Managed by UT-Battelle
for the Department of Energy

.. combined with Response TimeTime to Respond

Minor Consequence

Major Consequence>30 Minutes

NO_ALARM

MINOR

10..30 minutes

MINOR

MAJOR

<10 minutes

MAJOR

MAJOR + Annunciate– This part is still evolving…

27 Managed by UT-Battelle
for the Department of Energy

Example: Elevated Temp/Press/Res.Err./…
• Immediate action required?
– Do something to prevent interlock trip

• Impact, Consequence?
– Beam off: Reset & OK, 5 minutes?
– Cryo cold box trip: Off for a day?

• Time to respond?
– 10 minutes to prevent interlock?
• MINOR? MAJOR?
• Guidance: “Open Valve 47 a bit, …”• Related Displays: Screen that shows Temp, Valve, …
28 Managed by UT-Battellefor the Department of Energy

“Safety System” Alarms• Protection Systems not per se high priority– Action is required, but we’re safe for now, it won’t
get worse if we wait

• Pick One
 “Mommy, I need to gooo!” “Mommy, I went”(Does it require operator action?
How much time is there?)

29 Managed by UT-Battelle
for the Department of Energy

Avoid Multiple Alarm Levels• Analog PVs for Temp/Press/Res.Err./…:– Easy to set LOLO, LOW, HIGH, HIHI

• Consider:• In most cases, HIGH & HIHI only double thealarm traffic– Set only HSV to generate single, early alarm– Adding HHSV alarm assuming that the first one isignored only worsens the problem
30 Managed by UT-Battelle
for the Department of Energy

Bad Example: Old SNS ‘MEBT’ Alarms• Each amplifier trip:≥ 3 ~identicalalarms, no guidance• Rethought w/ subsystemengineer, IOC programmer
and operators: 1 better alarm

31 Managed by UT-Battelle
for the Department of Energy

Alarms for Redundant Pumps

32 Managed by UT-Battelle
for the Department of Energy

Alarm Generation: Redundant Pumps
the wrong way• Control System– Pump1 on/off status– Pump2 on/off status• Simple Config setting: Pump Off => Alarm:– It’s normal for the ‘backup’ to be off– Both running is usually bad as well
• Except during tests or switchover– During maintenance, both can be off

33 Managed by UT-Battelle
for the Department of EnergyRedundant Pumps

Required Pumps:

1• Control System– Pump1 on/off status– Pump2 on/off status– Number of running pumps– Configurable number of desired pumps• Alarm System: Running == Desired?
– … with delay to handle tests, switchover• Same applies to devices that are only needed
on-demand
34 Managed by UT-Battelle
for the Department of Energy

Weekly Review: How Many? Top 10?

35 Managed by UT-Battelle
for the Department of Energy

A lot of information available• How often did PV trigger?
• For how long?
• When?
• Temporary issue?
Or need HYST,alarm delay,
fix to hardware?

36 Managed by UT-Battelle
for the Department of Energy

Weekly Check: Stale, Forgotten?

37 Managed by UT-Battelle
for the Department of Energy

Summary• BEAST operational since Feb’09– Needs a logo
– For now without BEAUtY– DESY AMS is similar and has been
operational for longer• Pick either, but good configuration requires work inany case– Started with previous “annunciated” alarms• ~300, no guidance, no related displays• Now ~330, all with guidance, rel. displays– “Philosophy” helps decide what gets added and how• Immediate Operator Action? Consequence?
Response Time?

– Weekly review spots troubles and tries to improve
configuration
38 Managed by UT-Battelle
for the Department of Energy