Changing Gears:
Transition from Shutdown to Operational Modes atRHIC

Background: RHIC (Relativistic Heavy Ion Collider) is part of the CAD(Collider Accelerator Department) at Brookhaven
National Laboratory, Upton New York, USA.
 The facility is comprised of two pre-injectors (Proton
LINAC and Tandem for ions), a Booster, a post booster(the AGS) and RHIC itself.
 In addition to the primary purpose of providing beam for
injection into RHIC, injectors provide beam for various
secondary users throughout the complex• Brookhaven Lab Ion Production (BLIP): Protons from 200

MeV LINAC• NASL Space Radiation Lab (NSRL): Ions from 2 GeV Booster• Proton Radiography (PTR): protons from 30GeV AGS

WAO 2010 April 15th 2010

Paul W. Sampson

2

NSRL and BLIP

WAO 2010 April 15th 2010

Paul W. Sampson

3

RHIC
 RHIC is an accelerator and collider• Circumference = ~3.8km• Main ring consists of ~2200 main SC magnets• Cooled by 4k Helium gas• Max Dipole field = ~3.6T• Injection Momentum = ~30GeV (protons)• Max Collision = 500GeV CM (protons)• Cycle time varies with species and energy (nominally 10minutes to ramp and 4-12 hours at store)• Capable of colliding Polarized Protons and ions throughAu.

WAO 2010 April 15th 2010

Paul W. Sampson

4Experiments at RHIC
 Six collision points
 RF, Jet polarimeter + 4 experiments possible
 PHENIX and STAR are the two major experimentsthat presently run at RHIC

WAO 2010 April 15th 2010

Paul W. Sampson

5STAR and PHENIXWAO 2010 April 15th 2010

Paul W. Sampson

6

CAD complex at Brookhaven National Lab

WAO 2010 April 15th 2010

Paul W. Sampson

7

The running season:
 Typically, RHIC has a single multi-month run eachfiscal year.
 The run will normally start late in the fall andcontinues until the end of June.

WAO 2010 April 15th 2010

Paul W. Sampson

8Insert Schedule

WAO 2010 April 15th 2010

Paul W. Sampson

9Transition to shutdown Configuration of the accelerators is quite different
during shutdown periods.
 Machines are secured:• Higher level of LOTO.
• RS LOTO is applied.
• Water and AC systems are secured or modified.
• Controls hardware is secured or modified.

 Areas secured during the run are unlocked for freeaccess.
• Berms, rooftops and other areas.
• Radiation and other signage removed, covered or modified.
• Road gates are opened.
WAO 2010 April 15th 2010

Paul W. Sampson

10

WAO 2010 April 15th 2010

Paul W. Sampson

11

Scope of work defined During each shutdown major projects for upgrade
and improvement projects are executed
 Prior to shutdown, group project priorities are set
and individual work schedules developed Group schedules and workloads are then submitted,
reviewed
 A departmental meeting is held where overall
project priorities are discussed and set with upper
management accord
 Individual groups then re-adjust their schedules forprojects and PM accordingly.
WAO 2010 April 15th 2010

Paul W. Sampson

12Tracking and planning tools
 The status of shutdown work is:• On a Web based display• Viewable by all in the department, general pages viewable

the world• Displayed on CCTV throughout the lab

 Job and project progress data stored in a database• Display tools used to view individual job data

 During Startup• Global access schedule• Startup progress schedule• Weekly detailed schedule• Startup and maintenance elogs

 Annual Shutdown Post Mortem/ Shutdown preview
department wide assembly
WAO 2010 April 15th 2010

Paul W. Sampson

13General Access page
 All  Web pages
 JRS

WAO 2010 April 15th 2010

Paul W. Sampson

14

Overview with links to specifics

WAO 2010 April 15th 2010

Paul W. Sampson

15Group progress by job…

WAO 2010 April 15th 2010

Paul W. Sampson

16

Overall Startup Progress by Machine

WAO 2010 April 15th 2010

Paul W. Sampson

17

By individual job or task

WAO 2010 April 15th 2010

Paul W. Sampson

18

During the shutdown: Systems undergo PM, upgrades, modification andtesting.
 New components and systems are installed, someold systems may be removed or upgraded.
 Most of the commissioning and testing is performed
during a well defined startup period.
 In order to complete these tasks effectively:• Sub-systems restored may need to be restored (water,cooling power etc.)• Specific controls must be made operational• Other items such as interlocks, vacuum valves, heaters,building AC…

WAO 2010 April 15th 2010

Paul W. Sampson

19Transition to Operations: There are many challenges posed by the transitionof the accelerator complex from shutdown mode tooperational mode.
 After several months of free access and majorworks, it is necessary to change not only equipment
and systems but culture.
• Workers tend to get into “Shutdown Mode” and can get

frustrated when reverting to “Operations Mode”.

 In addition, many systems have been put into a
power savings mode and need to be tested at fullpower before routine operation.
• Scheduling power usage is a critical part of shutdown/startup and can save (or cost) big $$WAO 2010 April 15th 2010

Paul W. Sampson

20

Access Changes
 Many of the larger shutdown projects need the entire
shutdown period (+) for completion.
 At times, materials or components arrive late causing alast minute scramble for installation and testing.
 Critical steps in the end stages of a shutdown project
may include:
••
•
•
•

Final component installation
Controls and instrumentation installation and connection
Bake out/leak check for High vacuum
Final pre-beam testing
Final safety inspection and certification

 Most of these activities require free access to the ring
and/or other areas normally secured during operations.

WAO 2010 April 15th 2010

Paul W. Sampson21

Site wide changes
 Postings for radiation and ODH restrict access tosome areas• Film badges/SRDs• Changes to access roads – gates closed and locked Limits on ring access when ring >50K- ODH1
• ODH training must be current• Escape packs and ODH monitors

 Current limits in the main ring
• No Access in the tunnel for currents in the Main PS above

500A.

 Cool down activities in the tunnel• Thermister calibration• Power supply work
WAO 2010 April 15th 2010

Paul W. Sampson

22

Live testing periods
 Access limited
 No Access to areas during Access Controls recertification tests (~50hrs/year for RHIC)
 Safety concerns• Testing permits, supplemental LOTO Power consumption- peak power usage
 Assessment and Reaction• Adjustment• Repair• Reconfiguration• “Back out” plans

WAO 2010 April 15th 2010

Paul W. Sampson

23

Dry Runs:
 As they become ready for a scheduled running
period, each accelerator undergoes a dry run which
is utilized to complete a final checkout before beam
activities commence.
 During this period:• Access not normally permitted• All equipment is energized• All subsystems are on and in normal running configuration• All controls are in operational configuration• Expected initial super-cycles device settings, timing andsystem functions are loaded

WAO 2010 April 15th 2010

Paul W. Sampson

24Early beam studies: If it is desirable to complete beam studies before thescheduled run one or more of the injectors.
• Before scheduled running• Special setup?
• Special permissions?

 Have proven to be helpful for the run• Early detection of potential major problems• Decrease likelihood of delays in scheduled start up

 Impact on other work:• No access• Some equipment testing is not compatible (cause beamcrash)• Shift in workforce focus (reduced efficiency)
WAO 2010 April 15th 2010

Paul W. Sampson

25

Run delays avoided when starting early Booster cooling water flow restrictor replacement:• Measure flows dislodged after water pressure low (coolingwater off)
• 2 weeks to repair/restore cooling (>350 devices replaced) Booster MMPS transformer oil leak• 2 days for replacement

 AGS Main Magnet Water system• Leaking diaphragm valves:• 1 week for removal and restoration of MMPS cooling Main Magnet in AGS Ring short• 5 days for repair (in place rebuild)

WAO 2010 April 15th 2010

Paul W. Sampson

26

WAO 2010 April 15th 2010

Paul W. Sampson

27

Other complications
 Late definition of running parameters• Species, energy, optics etc.

 New systems:• Inspection• Testing/Commissioning• Integration

 Commissioning:• EBIS (Electron Beam Ion Source)- Require parts of LINAC and Booster ready for beam
• ERL (energy Recovery LINAC) power consumption andsafety- Monitoring necessary.

WAO 2010 April 15th 2010

Paul W. Sampson

28

Comments on other users:
 Normally run on alternate schedules
 Must resolve clashes with main (RHIC) agenda
 May use resources and/or personnel otherwise
available for RHIC
 Limits shutdown and duration in some cases(NSRL)
 May force early startup• Some priorities many need re-assessment

 Allows for early diagnosis and rectification ofproblems that possibly avoiding a delay in RHIC
startup.
WAO 2010 April 15th 2010

Paul W. Sampson

29

Conclusions: Smooth transition from shutdown is a key element of
any successful run and can be achieved by:• Planning and flexibility• Communication between groups, divisions, departments••••

and the rest of the world
Early definition of running parameters
Scheduling
Taking advantage of early turn onContingency plans

WAO 2010 April 15th 2010

Paul W. Sampson

30

Thank you

WAO 2010 April 15th 2010

Paul W. Sampson

31