SNS OperationsTools andAutomation
The 7th International Workshop on Accelerator
Operations – Automation and Tools

April 13, 2010
KAERI, Daejeon, Korea
C.C. Peters, Accelerator Machine Specialist
On behalf of:N.P. Luciano, T.B. Southern, and many others
Presented by: George Dodson

Outline• SNS Overview• SNS Operations Overview• Operator Downtime– Training– Repetitive tasks– Communication and Alarms• Status and Future Plans

2

Managed by UT-Battellefor the U.S. Department of Energy

WAO 2010

SNS Overview
NS is a ~1GeV 1.4 MW 6% DF Pulsed Superconducting
H- LINAC with a full energy Accumulator Ring

Central Control Room
3

Managed by UT-Battelle
for the U.S. Department of Energy

WAO 2010

Central Control RoomCRSS“Chief”
CRAS

TSTCRAS

4

Managed by UT-Battelle
for the U.S. Department of Energy

WAO 2010

SNS Operations• Control Room Shift– 2 Control Room Shift
Supervisors (but 1 “Chief”)– 1 Control Room Accelerator
Specialist1– 1 Target Systems Shift
Technician• On call– Machine Specialist

5Managed by UT-Battelle
for the U.S. Department of Energy

WAO 2010

Operator Downtime• Operators do not “own” systems which causedowntime• Unfortunately operators do cause downtime– Training• Lack of training sets up personnel for failure– Repetitive Tasks (Human Performance Issues)• Slow and/or incorrect machine turn on• Tasks with many steps that are not automated with will produce errors.

– Communication and Alarms• Even with training and automation, communication is needed to assure
operators receive accurate, up-to-date information.
• Alarms must be configured correctly (limits) and communicate theproblem, severity and point to actions which should be taken or areas
6
Managed by UT-Battellefor the U.S. Departmentof Energy
WAO 2010
to investigate.

SNS Operations Training Program• Basic steps in the Systematic Approach to Training– Develop a Training Manual (defines goals, Roles
Responsibilities, Authorizations and Accountabilities)– Determine and Develop Objectives of the Training including
Evaluation Standards– Develop Assessment Tools (Tests)– Develop Training Materials– Maintain the Training Program

7

Managed by UT-Battellefor the U.S. Department of Energy

WAO 2010

Training• Control Room Accelerator Specialist Qualification
Levels– New Hire
– Control Room Accelerator Specialist I– Control Room Accelerator Specialist II– Control Room Shift Supervisor– Mentor (mentors are assigned by Operations Management)

8

Managed by UT-Battelle
for the U.S. Department of Energy

WAO 2010

Training• Control Room Accelerator Specialist I– Secure area Sweep Team Qualification– Provides training on basic accelerator concepts and systems technology– Intended to especially assist training of personnel with little or no previous
accelerator experience• Control Room Accelerator Specialist II– Provide training that is more specific to SNS systems, their operation, andmachine tuning– Includes both knowledge and performance requirements (written tests andhands-on training)– Training areas divided into major global systems (e.g. PPS, TimingSystem) and by areas of the machine (Front End, SCL)
– Upon completion, considered fully qualified to operate facility• Training is done, but there people still make mistakes!
9Managed by UT-Battellefor the U.S. Department of Energy

WAO 2010

Repetitive Tasks - Breakdown• Breakdown time is divided into 4areas.
• Diagnosis and Recovery times
are operator downtimes• Diagnosis – time it takes for operators
to find the cause of a problem.
• Recovery – time from equipmentturnover to when beam is returned totarget.

10

Managed by UT-Battelle
for the U.S. Department of Energy

WAO 2010

Repetitive Tasks• Automate most tasks– Magnet power supply turn on (operator written php script)– RF power supply turn on (EDM script)– Tunnel entry preparation “LAZEE” (operator written pythonscript)– Beam repetition rate ramp up (operator written python scripts
and then EDM sequencer)– In process Machine Protection System mode change (EDMsequencer)

11

Managed by UT-Battelle
for the U.S. Department of Energy

WAO 2010

Repetitive Tasks• Example– Beam repetition ramp up• Python gui crashed, ranslow, too many Channel
Access “Gets”and ended
up being very complicated• An Operator came up withthe idea of using an EDM
program based on theLLRF State Sequencer.

Supposed to be 5 seconds!
And missing beam repetition rates!

12

Managed by UT-Battellefor the U.S. Department of Energy

WAO 2010

LLRF EDM Sequencer• LLRF detail EDM– Idle– Ramp– Tune– Closed loop

• Operator suggested using the same state sequencer forramping the beam power on target• Plus we can use the same code
13

Managed by UT-Battelle
for the U.S. Department of Energy

WAO 2010

Repetitive Tasks• Beam Power Ramp upSequencer– Ready = Idle
– Ramp = Ramp
– Goal = Closed Loop

• Great addition– Recovery• Added initially because ofring injection foil failures• When beam trips for any
reason, power will recoverquickly and automaticallyusing beam repetition rateand LEBT chopper Pulse
Width modulation.
14

Managed by UT-Battellefor the U.S. Department of Energy

WAO 2010

The SNS Downtime – Operator Error• MPS PLC trips in “Changing Machine Mode” are the largest component ofoperator caused downtime• Simple in principal;– Turn on power
supplies, put inbeamstops, close
vacuum valves in thecorrect sequence.
• But it is difficult to becorrect every time withmanual operations (HPI).
– The “Hurry Up!” factor• MPS must be in“Standby” mode or
the Front End is
SCRAMed
15

Managed by UT-Battelle
for the U.S. Department of Energy

EDM SEQUENCER TO THE RESCUE!
WAO 2010

The SNS Downtime – Operator Error• MPS mode changesequencer– Code is written– Gui is built– Testing is needed

16Managed by UT-Battelle
for the U.S. Department of Energy

WAO 2010

Communication• How to communicate the ever changing machine,procedures, and daily activities.
– Rotating crews make communication difficult• Elog– A lot of entries and not all important for operations– Crews can’t read every elog entry• Email– A lot of emails and not all important for operations• Want easier and faster way to convey information

17

Managed by UT-Battelle
for the U.S. Department of Energy

WAO 2010

Communication• Have a web-based tuning guide and wiki
– Tuning guide lists “recipes” for tuning• Great for procedures that do not change often, but not fast because
operators have to search– Wiki lists changing conditions• Great for daily updates, but again not fast because operators have tosearch• Want an even easier and faster way to notify operatorsof changing conditions– Direct Wiki pushbutton link directly from EDM screens– EDM blogger attached to screens
18

Managed by UT-Battelle
for the U.S. Department of Energy

WAO 2010

Communication• Wiki links from EDM– Fast links to page specific wiki
– Anyone can update including
the system experts

19

Managed by UT-Battelle
for the U.S. Department of Energy

WAO 2010

Communication• EDM blogger– Goal is to be able tocommunicate about a specificsystem on the system’s EDM
page– Should lead to faster operator
response, and reduce error tochanging conditions– BUT it is not easy to program
into EDM, and is currently quite
limited in EDM

20Managed by UT-Battelle
for the U.S. Department of Energy

WAO 2010

Alarm Transitions• MPS chirper– Initially had no audible alert of beam trips.
– Using first BASH(Bourne Shell) and then python. Monitored
MPS, and chirped.

• Stoplights (example of the WRONG way!)– EDM, lots of drilldowns, and not all annunciated21Managed by UT-Battelle
for the U.S. Department of Energy

WAO 2010

Best Ever Alarm System (BEAST)
Toolkit• Running now• BEAST– Running in CSS (DESY)– Alarms in list form
(Current, Acknowledged,and Annunciated Alarms)– Alarms link to response
instructions, wiki
rationalization, and EDM
pages– Using a lot of operations
input

22

Managed by UT-Battelle
for the U.S. Department of Energy

WAO 2010

Conclusions and Future• Goal is to reduce operator downtime– Thorough training program• Input from new, experienced operations staff– Measuring operator downtimes helps locate issues– Creative tools to automate repetitive tasks– Improved communication and decreased response time
– Operator involvement has made huge improvements• Into the future…

23

Managed by UT-Battelle
for the U.S. Department of Energy

WAO 2010

Future Plans 1 & 2• Real time downtime calculator– Using the beam power ramp sequencer– Time outside of the “Goal” is downtime– Downtime timer will start automatically when outside of“Goal” state (e.g 80% nominal beam power for 0.1 Hr.)

• Operator troubleshooting database using XAL– Easily searchable database containing common problems– Faster than e-log, wiki, or tuning guide– Updated by operations

24

Managed by UT-Battelle
for the U.S. Department of Energy

WAO 2010

Future Plans 3• XAL Sequencer (java)– Want software so easythat anyone can easily
create a sequence• Scripts and EDM
sequences have learningcurve.

– Libraries of operations– Operations placed intosequences

25

Managed by UT-Battelle
for the U.S. Department of Energy

WAO 2010