# Presentation: 450125

**LHC@FNAL**

**Patricia McBride**
**Fermilab/CD**

** **

## Key components for remote operations

**For successful participation in operations at a distance, collaborations must first address issues of *****communications***** and *****trust*****.**
**These issues can be partially addressed through choices of appropriate *****technologies***** and the establishment of clear collaboration *****policies*****.**
**A successful long-term remote operations center takes the strong commitment of the local *****community *****and the right *****environment*****.**
**Exchange of *****personnel***** between sites is key.   Nothing can replace time spent at the experiment.**
**The principle goal is to enable people to participate in operations when they are unable to travel to the experiment.**

## LHC remote operations center at Fermilab

**Fermilab  **
**has contributed to CMS detector construction,**
**hosts the LHC physics center for US-CMS,**
**is a Tier-1 computing center for CMS,**
**has built/delivered LHC machine components, and **
**is part of the LARP collaboration.**
**The LHC physics center (LPC) had plans for remote data quality monitoring of CMS during operations. Could we expand this role to include remote shifts?  What are the limits?**
**We saw an opportunity for US accelerator scientists and detector experts to contribute their expertise during LHC commissioning.  Could they help with studies without moving to CERN for a year? **
**The idea of a joint remote operations center at Fermilab was formed and people from each area joined together to plan a single center.**

## What is LHC@FNAL?

**A Place**
**That provides access to information in a manner that is similar to what is available in control rooms at CERN**
**Where members of the LHC community can participate remotely in CMS and LHC activities **
**A Communications Conduit**
**Between CERN and members of the LHC community located in North America**** **
**An Outreach tool**
**Visitors will be able to see current LHC activities**
**Visitors will be able to see how future international projects in particle physics can benefit from active participation in projects at remote locations.**

## Planning for LHC@FNAL

- We formed a task force with members from all FNAL divisions, university groups, CMS, LARP, and LHC.  The advisory board had an even broader base.
- The LHC@FNAL task force developed a plan with input from many sources including CMS, LHC, CDF, D0, MINOS, MiniBoone and Fusion Energy Sciences.
- We worked with CMS and US-CMS management, as well as members of LARP (LHC Accelerator Research Program) and the LHC machine group at all steps in the process.
- A detailed requirements document for LHC@FNAL was prepared and reviewed in 2005.
- We visited 9 sites (e.g. Hubble, NIF, ESOC) to find out how other projects do remote operations.
- We are now engaged in construction, integration, software development and outreach activities.   CMS Remote Operations Center is already in operation.
- The goal is to have LHC@FNAL ready for detector commissioning and startup of beam in 2007.
- We plan to work with the ILC controls group to develop plans for ILC remote operations.
- We hope to collaborate with the fusion community to develop tools.   Together we are exploring connections to OSG and other GRID projects.

## Remote operations for LHC and LARP

**CCC**

- LHC remote operations:
-  training prior to stays at CERN
-  remote participation in studies
-  ‘service after the sale’: to support components we built.
-  access to monitoring information

- CCC at CERN

**LARP:**   The US LHC Accelerator Research Program (LARP) consists of four US laboratories, BNL, FNAL, LBNL and SLAC, who collaborate with CERN on the Large Hadron Collider (LHC).
- The LARP program enables U.S. accelerator specialists to take an active and important role in the LHC accelerator during its commissioning and operations, and to be a major collaborator in LHC performance upgrades.

## Connecting to CERN

**Reliable communications tools and robust secure software are critical for operations.**

**Some general requirements:**
**Remote users should see applications as they are in the main control room(s) when possible.  However, they might not have the same privileges.**
**Communication channels should be kept open.  **
**Establish clear policies and protocols for shifts.****   **
***The goal is to assist in operations, and not to place additional requirements on CERN personnel.***** **

**Issues:  access to information on technical networks (TN,EN), latency, authorization, authentication, 24x7 communications****  **

## LHC Accelerator Fermilab Software (LAFS)

**It will be difficult for outside visitors to make significant contributions to the LHC once beam commissioning has started.**
**Unfamiliarity with the control system**
**Critical problems will most likely be assigned to in-house staff****. **
**Fermilab will be more welcomed at CERN if the lab can bring **_**real**_** resources to the table and has the ability to solve operational problems.**
**Fermilab has experience in the software issues of running a collider complex**
**The Fermilab control system based on Java is similar to the LHC Java based control system and has a large pool of Java software expertise to draw on.**
**Fermilab is already collaborating with CERN on a number of software projects **
**Goal of LAFS: Develop a suite of software products to enable Fermilab accelerator physicists to make key contributions to the beam commissioning of the LHC****.**
**A small team of computer professionals, operational experts, and accelerator physicists has been assembled to contribute to select LHC software tasks. **
**Software projects underway - in collaboration with CERN:**
- Role Based Access
- Sequenced Data Acquisition (SDA)
- Sequencer
- Tune Tracker

## An approach to restrict system access to authorized users. 
What is a ROLE?
A role is a job function within an organization.
Examples: LHC Operator, SPS Operator, RF Expert, PC Expert, Developer, ...
A role is a set of access permissions for a device class/property group
Roles are defined by the security policy
A user may assume several roles
What is being ACCESSED?
Physical devices (power converters, collimators, quadrupoles, etc.)
Logical devices (emittance, state variable)
What type of ACCESS?
Read: the value of a device once
Monitor: the device continuously
Write/set: the value of a device
Requirements written and in EDMS
Authentication
Authorization

Status:  Design document in progress 
Fermilab/CERN collaboration

**What is a **ROLE**?**
**A role is a job function within an organization.**
**Examples: LHC Operator, SPS Operator, RF Expert, PC Expert, Developer, ...**
**A role is a set of access permissions for a device class/property group**
**Roles are defined by the security policy**
**A user may assume several roles**
**What is being**** **ACCESSED**?**
**Physical devices (power converters, collimators, quadrupoles, etc.)**
**Logical devices (emittance, state variable)**
**What type of**** **ACCESS**?**
**Read: the value of a device once**
**Monitor: the device continuously**
**Write/set: the value of a device**
**Requirements written and in EDMS**
**Authentication**
**Authorization**
**Status:  Design document in progress **
**Fermilab/CERN collaboration**

**Role Based Access (RBA)**

**The software infrastructure for RBA is needed for remote operations.   Permissions can be setup to allow experts outside the control room to read or monitor a device safely. **

## US-CMS and remote operations 

**US-CMS and remote operations **

**The remote operations center will serve the US-CMS community.  **

**There are nearly 50 US institutions in CMS and about 600 signing physicists/engineers/guest scientists in the US.**
**The LPC will provide a place in the US for physics/analysis discussions and meetings.  The CMS remote operations center at Fermilab will provide a US hub for operations activities.**
**The goal is to provide a facility that will enable CMS collaborators to take shifts (for example, DQM shifts)**

**Fermilab**

## CMS Remote Operations Center

**The LHC Physics Center (LPC) first developed the idea of a remote Operations Center (ROC) for CMS at Fermilab.  The center is already in operation for the cosmic tests of the full detector in the surface building at LHC Point 5 (MTCC).  **
**CMS ROC activities will move to the LHC@FNAL center next year.**

## Remote operations for CMS MTCC

**CMS temporary Control Room at CERN P5**

- Data quality monitoring
- Software tests
- Tier0-Tier1 transfers
**ROC at Fermilab**

- MTCC@CERN
- Integration test
- Detector tests with cosmics
- Field mapping
- Global DAQ commissioning
- Data Transfer to Tier0

- Summer-Fall 2006

## An approach to provide a snapshot of a graphical interface to remote users. 
What is a snapshot?
An image copy of a graphical user interface at a particular instance in time.
Examples: DAQ system buffer display, operator control program, ...
A view-only image, so there is no danger of accidental user input.
Initially envisioned for application GUIs but could be expanded to desktops.

What is the role of the service?
Receives and tracks the snapshots from the monitored applications.
Caches the snapshots for short periods of time.
Serves the snapshots to requesting applications/users.
Prevents access from unauthorized applications/users.
Acts as a gateway to private network applications for public network users.

How will this work?
Applications capture and send snapshots to the service provider in the background.
Users would access snapshots using a web browser.

Status:  
The capturing of snapshots from Java applications has been demonstrated.
The transfer of snapshots is being investigated.

SSS is being developed at Fermilab for CMS, and may be applicable to the LHC.

**What is a snapshot?**
**An image copy of a graphical user interface at a particular instance in time.**
**Examples: DAQ system buffer display, operator control program, ...**
**A view-only image, so there is no danger of accidental user input.**
**Initially envisioned for application GUIs but could be expanded to desktops.**
**What is the role of the service****?**
**Receives and tracks the snapshots from the monitored applications.**
**Caches the snapshots for short periods of time.**
**Serves the snapshots to requesting applications/users.**
**Prevents access from unauthorized applications/users.**
**Acts as a gateway to private network applications for public network users.**
**How will this work?**
**Applications capture and send snapshots to the service provider in the background.**
**Users would access snapshots using a web browser.**
**Status:  **
**The capturing of snapshots from Java applications has been demonstrated.**
**The transfer of snapshots is being investigated.**
**SSS is being developed at Fermilab for CMS, and may be applicable to the LHC.**

**Screen Snapshot Service (SSS)**

**Monitored**
**Application(s)**

**Snapshot**
**Service**

**Web**
**Browser(s)**

**snapshots**

**snapshots**

**requests**

## Collaboration on Tools for Remote operations

- Although we are making good progress on the development of remote operations capabilities for the LHC, there is _room for improvement_. Better collaborative tools will contribute significantly to our ability to participate in LHC, and plan for the ILC.
- We could benefit from improved _communications tools_ by
-  exploiting convergence of telecom and internet technologies (e.g. SIP),
-  deploying integrated communications (voice, video, messaging, email, data)
-  improving electronic logbook capabilities,
-  and advanced directory services for identification, location and scheduling.
- We could benefit from a true _collaborative control room_ by
-  deploying distributed, shared display walls for remote collaborative visualization.
- We could benefit from security enhancements (role-aware & easier-to-use security).
- Some of items are already being addressed by the ILC, Fusion and/or GRID communities.  We are exploring possible collaboration for development of these tools.

## Dedicated facility to support both CMS and LHC commissioning and operations.
Remote shifts for CMS   
Enable experts to access information in real time
Participate in studies and commissioning
Training

 Facilitate communication with CMS and LHC control rooms.
Call center for US  - to contact experts, answer questions.
Test collaboration tools to improve communication

 Take advantage of a unique opportunity to have detector and accelerator experts working together to solve problems.

**Remote shifts for CMS   **
**Enable experts to access information in real time**
**Participate in studies and commissioning**
**Training**
** ****Facilitate communication with CMS and LHC control rooms.**
**Call center for US  - to contact experts, answer questions.**
**Test collaboration tools to improve communication**
** ****Take advantage of a unique opportunity to have detector and accelerator experts working together to solve problems.**

**Goals for joint facility: LHC@FNAL**

## LHC@FNAL Location & Layout

**LHC@FNAL Location & Layout**

## LHC@FNAL – renderings

## Construction of LHC@FNAL

**LHC@FNAL **
**October 27, 2006**

**LHC@FNAL**
**November 10, 2006**

**Construction of the center is on schedule.**
**Consoles to be installed before**
**the end of the year.**
**Equipment: **
**Many items were purchased in FY06**
**Still need to buy computers and projectors.**

## Equiping LHC@FNAL

- Equipment purchased in FY06
- Consoles
- Consoles will be the same as in the CCC.
- Installation scheduled for Dec 19-20.
- Chairs
- Video conferencing equipment
- Networking
- Prototype console PCs
- LCD monitors (19 in)
- VGA2WEB & video matrix switch hardware
- Public display (demo held last Friday)
- Linux Server
- Remaining purchases to complete the center
- console PCs
- Projectors (for shared wall displays and public displays)
- Misc. equip/licenses for communications (web cams, headsets...)

## LHC@FNAL Integration Task Force

## LHC@FNAL Outreach

**Outreach Planning and Activities:**
- Developed scenarios for various escorted and non-escorted outreach.  Examples:
-  tour for senior citizens,
-  VIP tour,
-  tour as part of a physics conference,
-  non-technical Fermilab employees on their lunch hours,
-  general public attending a lecture or concert ,
-  a bike rider who stopped at the hi-rise
- In the process of writing recommendations based on these scenarios
- Using the plywood construction wall for displays charting the progress of construction and also general info on LHC and CMS
- The large 4' by 7' display has been received and the first demo was very successful.
-  Have begun work on specifications for an interactive public outreach display.

**The location of LHC@FNAL in the main building makes the facility an ideal place for creative outreach activities. **

## Summary

- The CMS ROC is already in operation for CMS detector commissioning.  (See Kaori’s talk.)
- The LAFS development program has been successfully launched.
- Plans for a joint CMS and LHC remote operations center are well advanced.  Construction of LHC@FNAL began in Sept 2006, and will be completed by the end of the year.
- LHC@FNAL will be a place to support commissioning and operations for both the LHC accelerator and CMS.   The center aims to enable the US HEP community to remain actively engaged in activities at CERN.
- We are exploring possible synergies with fusion and grid communities for the development of tools.
- We look forward to developing plans for ILC remote operations along with the global ILC community.

## Additional Slides

## LHC@FNAL Task Force

- Erik Gottschalk – Chair (FNAL-PPD)
- Kurt Biery (FNAL-CD)
- Suzanne Gysin* (FNAL-CD)
- Elvin Harms* (FNAL-AD)
- Shuichi Kunori (U. of Maryland)
- Mike Lamm* (FNAL-TD)
- Mike Lamont* (CERN-AB)
- Kaori Maeshima (FNAL-PPD)
- Patty McBride (FNAL-CD)
- Elliott McCrory* (FNAL-AD)
- Andris Skuja (U. of Maryland)
- Jean Slaughter* (FNAL-AD)
- Al Thomas (FNAL-CD)
- The formal LHC@FNAL task force had its last meeting on March 29, 2006.
- The group has evolved into an integration task force with a new charge and a few new members.

-  Task force was charged by the    Fermilab Director in April, 2005.
-  Task force wrote a requirements    document and WBS.
-  Work completed in March, 2006.

## Some assumptions (CMS operations)

**For CMS**
- CMS will have a shift schedule, a run plan, and a protocol that defines responsibilities and roles of shift personnel. We assume that a shift leader is responsible for CMS shift activities.** **
- LHC@FNAL will have shift operators who will be able to assist US-CMS collaborators with CMS activities during commissioning and operations.** **
- LHC@FNAL will participate in CMS shifts. Neither the duration nor the frequency of the LHC@FNAL shifts has been determined.
- The CMS Collaboration will have a protocol for access to the CMS control system (PVSS), and a policy for how access to the control system will vary depending on the physical location of an individual user.
- The CMS Collaboration will have a policy that defines how DAQ resources are allocated. This includes allocation of DAQ resources to various detector groups for calibration and testing.** **
- The CMS Collaboration will have a protocol that defines how on-demand video conferencing will be used in CMS control rooms and LHC@FNAL.
- The CMS Collaboration will provide web access to electronic logbook and monitoring information to collaborators worldwide
- The CMS Collaboration will maintain a *call tree* that lists on-call experts worldwide for each CMS subsystem during commissioning and operations** **
**For both CMS & LHC**
- LHC@FNAL will comply with all CERN and Fermilab safety and security standards.