# Executive Summary {#executive-summary .unnumbered}

This status report...

Executive Summary [i](#executive-summary)

1\. Introduction [1](#introduction)

2\. Engineering Working Group [2](#engineering-working-group)

2.1. Location and Layout [2](#location-and-layout)

2.2. Consoles [5](#consoles)

2.3. Network [6](#network)

2.4. Computing [7](#computing)

2.5. Communications [8](#communications)

2.5.1. Video Conferencing Systems [8](#video-conferencing-systems)

2.5.2. Telephone System [9](#telephone-system)

2.5.3. Webcams [9](#webcams)

2.5.4. WebEx [9](#webex)

2.6. Environment Control [9](#environment-control)

2.7. Operations Support Team [10](#operations-support-team)

2.8. Summary of Requirements [11](#summary-of-requirements)

2.8.1. CMS/LHC Combined Requirements [11](#cmslhc-combined-requirements)

2.8.2. Constraints [12](#constraints)

2.9. LHC@FNAL Construction [13](#lhcfnal-construction)

3\. CMS Working Group [15](#cms-working-group)

3.1. ROC Infrastructure [15](#roc-infrastructure)

3.2. Plan for CMS Commissioning Activities
[16](#plan-for-cms-commissioning-activities)

3.3. Model for CMS Remote Operations
[17](#model-for-cms-remote-operations)

3.3.1. Support [17](#support)

3.4. Requirements [18](#requirements)

3.5. Conclusion [18](#conclusion)

4\. LHC Working Group [19](#lhc-working-group)

5\. Outreach Working Group [20](#outreach-working-group)

# Introduction

LHC@FNAL is a remote operations center at Fermilab. It will be
implemented in two phases. Phase 1 encompasses the construction of the
operations center and upgrades to the adjacent conference room. The
primary focus is to establish a remote operations center that supports
**commissioning** of CMS and LHC and is operational by the spring of
2007. Phase 2 includes expansion of LHC@FNAL to provide capabilities
that are needed to support **operations** of CMS and LHC.

As Phase 1 is nearly completed, it is important to establish a plan that
integrates the three functions of the LHC@FNAL Remote Operations Center
(ROC):

-   support for CMS commissioning and operations,

-   support for LHC commissioning and operations,

-   and outreach.

It is also important to engage the US HEP community to take full
advantage of the ROC.

The requirements for remote operations and a Work Breakdown Structure
(WBS)\
were developed by a previous task force that completed its work on March
29, 2006. A new task force (LHC@FNAL Integration Task Force) has been
established with four working groups: Engineering Working Group (EWG),
CMS Working Group (CWG), LHC Working Group (LWG), and Outreach Working
Group (OWG). The tasks for the working groups are listed in the October
18, 2006 Charge to the LHC@FNAL Integration Task Force \[[^1]\]. There
is some degree of overlap in tasks associated with each of the four
working groups. For example, each working group is responsible for
developing requirements and cost estimates for Phase 2. After each
working group completes tasks assigned to the group, the task force will
integrate everything into a unified plan for LHC@FNAL.

This document includes status reports from each of the four working
groups. The report from the EWG is presented in Section 2, with an
overview of the location and layout of LHC@FNAL and information on
consoles, network, computing, communications, and a cost and schedule
summary of the construction for Phase 1. Sections 3 and 4 include
reports from the CWG and LWG, respectively, and Section 5 presents a
report from the OWG on our outreach efforts.

# Engineering Working Group

The Engineering Working Group (EWG) is responsible for Phase 1
construction of the LHC@FNAL ROC and conference room, for Phase 2
requirements and cost estimates for any additional construction
activities, and for developing a plan for ongoing support of LHC@FNAL.
In particular the EWG is responsible for the following tasks:

-   coordinate activities to complete construction of Phase 1;

-   develop requirements and cost estimates for Phase 2;

-   develop a support model for LHC@FNAL infrastructure;

-   and develop an organizational structure that identifies roles and
    responsibilities.

These tasks were identified as belonging to the scope of the EWG after
considering the tasks outlined in the October 18, 2006 Charge to the
LHC@FNAL Integration Task Force \[Error: Reference source not found\].

Members of the Engineering Working Group are:

-   Erik Gottschalk (Group Leader)

-   Elvin Harms (Group Leader)

-   Mark Kaletka

-   Patty McBride

-   Elliott McCrory

-   Alan Stone

In this section we present an overview of the location and layout of the
ROC and adjoining conference room. We provide a brief description of the
consoles as well as the network, computing, communications
infrastructure, and climate control for the ROC. We describe the
requirements that have been satisfied by Phase 1, and discuss
requirements that still need to be addressed. We describe the mission of
the Operations Support Team, which consists of people from different
divisions at Fermilab, to support LHC@FNAL operations. We conclude with
a cost and schedule summary of the Phase 1 construction, which was on
time, under budget, and had no injuries.

## Location and Layout

The LHC@FNAL Remote Operations Center (ROC) is located on the first
floor of Fermilab's Wilson Hall. Figure 1 shows a floor plan of the
first floor with the location of the ROC and adjoining conference room.
The front entrance to Wilson Hall is at the bottom of the figure.

![](media/image1.pct){width="5.139583333333333in"
height="8.439583333333333in"}

Figure 1: Floor plan for Wilson Hall 1^st^ Floor. The LHC@FNAL remote
operations center and conference room are located in two rooms shown in
the middle, on the left side of the floor plan.

The layout of LHC@FNAL is shown in Figure 2. The ROC is shown on the
left. The layout of four consoles is shown, along with screen and
projector locations. Four of the projectors are assigned to consoles
(one projector per console), and are labeled "PROJ1" through "PROJ4."
The fifth projector (labeled "PROJ5") points towards a rear projection
screen shown at the top of Figure 2. This screen, which is referred to
as the "Public Display," is used for outreach and has a wide viewing
angle so that the display is easily visible from the atrium. A round
table at the center of the room provides additional seating as well as
power and network connections for laptop computers.

![](media/image2.pct){width="5.997916666666667in"
height="4.360416666666667in"}

Figure 2: Layout of LHC@FNAL. The ROC is shown on the left with the
console layout as well as screen and projector locations. The
confererence room includes two LCD displays, a screen and a projector
used for video conferencing.

A large window with switchable privacy glass separates the ROC from the
adjoining conference room*,* which is shown on the right side of Figure
2. Part of the LHC@FNAL construction project included the installation
of a videoconference system in the conference room (see Section 2.5.1),
and installation of electrical outlets for people using laptop computers
at the conference table. The figure shows the new screen that was
installed in the conference room, and the location of a short-throw
projector. Two LCD displays in the bottom, left corner of the room are
used as video displays for the videoconference system. Audio for this
system is provided by four speakers and numerous microphones (not shown
in the figure) in the room. All of the electronics of the system are
located in an equipment rack that is only accessible from the ROC.

## Consoles

One of the primary functions of LHC@FNAL is to provide the means to
access and display information as though one were located in an LHC or
CMS control room at CERN. One of the ways of satisfying this requirement
was accomplished by designing the operations center to accommodate four
consoles (see Figure 3) that match the specifications of consoles
installed at CERN. This is important, since it allows us to install as
many PCs and LCD screens per console as are installed at CERN, and it
gives us the ability to implement any changes to the configuration of
the consoles that might be made during the course of LHC and CMS
operations. We purchased the same consoles for LHC@FNAL that were custom
designed and built for CERN.

![](media/image3.png){width="4.104166666666667in" height="4.84375in"}

Figure 3: Two views of the consoles designed and built for CERN.

Each console has two workstations. Each workstation is equipped with one
*console PC* with a "desktop" that spans three LCD screens. These
screens are located at eye level on the lower tier of a two-tier
monitor-mounting system. Located on the upper tier, and centered between
each pair of workstations, are three LCD screens attached to a single
*fixed-display PC* that is used to display monitoring information. Each
fixed-display PC includes a projector assigned to that console. With
this arrangement an operator can easily move a window on the PC's
desktop from one of the LCD screens to the projector so that the window
is visible from anywhere in the operations center, from the conference
room, and from the atrium. In addition to console and fixed-display PCs,
some of the consoles are equipped with a videoconference system (see
Section 2.5.1) that can be used to participate in meetings and for
point-to-point video connections to CERN.

## Network

The LHC@FNAL network provides connectivity to the local Fermilab network
and wide area network (notably the CERN network!). Figure 4 shows a
diagram of the network. The network has two parts: one with *open
access* that has systems connected to the network through a gigabit
switch, and one with *protected access* with systems connected through a
dedicated router. Systems connected to the switch have *default permit*
access from off-site and include some of the console PCs and
fixed-display PCs used by CMS, hardware used for videoconferencing,
wireless access points in the ROC and the conference room, the building
environment monitoring system, and the keycard access system. Systems
connected to the router are protected from off-site by having a very
restrictive *default deny* access policy that is implemented by Access
Control Lists (ACLs) on the router. These systems are on a restricted
subnet (subnet 252) that is only available in the ROC, and include
console PCs and fixed-display PCs used for LHC software development and
access to the LHC controls systems. Systems must be registered to use
subnet 252. This prevents someone in the ROC from connecting a laptop
computer, for example, and thereby gaining access to the protected
network.

![](media/image4.pct)

Figure 4: The LHC@FNAL network. On the left are systems with open access
from off-site. On the right are systems connected to a dedicated router
that uses ACLs to restrict access.

Figure 4 shows that devices that are used for outreach are on the
protected network since most of these systems do not require access to
the wide area network. In particular, the projectors and Apple TV are on
the protected network, and all inbound and outbound traffic for these
devices is restricted to subnet 252. Figure 4 also shows a *gateway PC*
that allows very restricted Kerberos access to systems on the protected
network for system management purposes.

From the beginning the LHC@FNAL network was designed to have a dedicated
network for the consoles, with tightly restricted access into and out of
that network. We have proposed \[[^2]\] a scheme that implements a
secure tunnel between subnet 252 and the CERN General Purpose Network.
This would establish a tunnel that is implemented in routers and is more
secure than standard methods that are commonly used to connect to
systems at CERN. The proposal, which has not been accepted by the CERN
networking group, is particularly important for work on the development
of software for the LHC controls system, since this work benefits from
access to databases and web pages that are only visible inside the CERN
General Purpose Network. For CMS it is not yet known whether this type
of access to the CERN network is needed.

In its current configuration the LHC@FNAL network is implemented with
gigabit uplinks for both the gigabit switch and the router. Each device
supports "port channeling," whereby two (or more) gigabit Ethernet ports
can be combined to share traffic between individual channels. This way
the bandwidth can be easily increased if that should be necessary in the
future.

## Computing

LHC@FNAL entails several different types of computing systems. Table 1
lists the different systems that are used. The table includes systems
that are installed in the ROC and conference room, and a server
installed at the Feynman Computing Center (FCC).

  ------------------------------ ------------------------ ----------------
  Type                           Operating System         Quantity

  Console PC                     Scientific Linux         8

  Fixed-display PC               Scientific Linux         4

  Gateway PC                     Scientific Linux         1

  FCC Server                     Scientific Linux         1

  Outreach Mac                   Mac OS X                 2

  Presentation PC                Windows                  1

  VRVS PC                        Windows                  1
  ------------------------------ ------------------------ ----------------

Table 1: Computing systems used by LHC@FNAL.

The console PCs and fixed-display PCs are for LHC@FNAL users and are
operating with Scientific Linux version 4.4 (SLF 4.4)[^3]. The gateway
PC is used by system managers for remote access to PCs in the protected
network, and the server provides disk storage space and is used as a web
server.

Two Apple Mac computers and an Apple TV system (not included in the
table) are used for outreach. All of these systems are connected to the
Public Display through a switch that can be controlled via remote
control from outside the ROC.

One Windows PC and a laptop computer are installed in the conference
room. The PC is used to display content on the projector, while the
laptop computer is used to connect to VRVS videoconference meetings.

## Communications

Communications is a vital component of any remote operations center. For
LHC@FNAL the ability to serve as a communications conduit between people
working at CERN and members of the LHC community in North America is one
of three primary functions[^4]. LHC@FNAL is expected to function as a
communications hub by serving as an extension of the CERN Control Centre
(CCC), where all CERN accelerator activities are coordinated; as an
extension of the main CMS Control Room, the primary site for operating
the CMS experiment; and as an extension of the CMS Centre at Meyrin,
where most of the online and offline monitoring of CMS will be done. As
a communications hub, LHC@FNAL could serve as a point-of-contact for
individuals involved in LHC accelerator or CMS detector activities. For
example, university researchers could get firsthand information about
the operational status of CMS and the LHC, including detailed status of
accelerator and detector subsystems. To satisfy communications
requirements, LHC@FNAL is equipped with a variety of communications
systems.

### Video Conferencing Systems

LHC@FNAL has two types of systems for videoconferencing. One of these
systems is the high-definition (HD) videoconferencing system installed
in the conference room[^5]. The goal for HD videoconferencing is to
establish a high-quality video and audio link to control rooms and
operations centers at CERN and other remote locations. With high-quality
video and audio our ability to be part of activities at CERN is expected
to significantly better compared to standard videoconferencing that is
used today. The success of this venture into HD videoconferencing
depends to some degree on CMS and LHC installing HD equipment at CERN.
While HD systems are still relatively new (this is the first HD system
at Fermilab), these systems are beginning to be evaluated for use in
high-energy physics.

The second type of videoconferencing system that we have installed is a
small-scale system that is best used for point-to-point
videoconferencing between two sites, or for one or two people located at
one site to connect to ESnet or VRVS meetings in the usual way. We have
installed two of these systems in the ROC. They are standalone
videoconferencing systems with a built-in camera and microphone and are
attached to an LCD display with speakers. We imagine that these systems
will be used for point-to-point communications between the ROC and a
control room or test facility at CERN during the course of a shift, or
to allow people working in the ROC to connect to meetings while working
at a console. We have recommended this type of system to CMS physicists
who are looking for an inexpensive, easy to operate, and easy to
maintain videoconferencing system.

### Telephone System

The telephone system that is installed in the ROC consists of eight
standard Fermilab telephones and a speakerphone at the round table in
the center of the room. Each of the eight workstations has a two-line
telephone with a phone number that is unique to the phone and a common
number that is shared by the eight phones. International calling
capabilities have been enabled for all phones.

We are in the process of evaluating a new telephone system with
additional capabilities. One of the capabilities offers the option to
assign a specific telephone number to each group of users that is
expected to use the ROC. This way one can arrange to have the number
assigned to a particular console when the group is using that console.
When the group is not using the ROC, then phone calls to that number can
be directed to a different location.

### Webcams

Two cameras with built-in web servers have been installed in the ROC.
The two webcams have different capabilities. One webcam provides very
basic pan and tilt capabilities. The second webcam provides pan, tilt
and zoom capabilities with its 10X optical zoom. Both webcams can show
live video images of the ROC, in addition to emailing photographs
according to a predetermined schedule to specified people. The second
webcam has the ability to store specific view settings (pan, tilt and
zoom settings), and to cycle through a sequence of these settings
automatically and continuously. Both webcams require a username and
password for access.

### WebEx

LHC@FNAL pioneered the use of WebEx \[[^6]\] at Fermilab. This
commercial web-based tool can be used as an alternative to, or in
addition to videoconferencing for large and small-scale meetings. WebEx
offers screen sharing and application sharing capabilities that are
potentially very useful for operations by providing communications links
between control rooms, remote operations centers, and collaborators at
other institutions to exchange monitoring information. For now the tool
is being used by the CMS HCAL group and the LHC@FNAL Software group for
regularly scheduled meetings. It has also been adopted for ILC meetings,
and the ILC community is currently the heaviest user of WebEx at
Fermilab.

## Environment Control

The environmental controls for the ROC include an HVAC system, a Lutron
\[[^7]\] lighting system, and two types of roller shades to control the
amount of light that enters the room from outdoors. The HVAC system was
installed so that the ROC can be used for 24/7 shift operations, since
heating and cooling for Wilson Hall is not operational at all times of
the day. The Lutron lighting system in the ROC is similar to the system
that is installed in the conference room. Two types of roller shades are
installed at the windows to the outside. A "solar shade" reduces the
amount of light that enters the ROC from outdoors while still permitting
a view outside. A second shade that is not transparent can be used to
block direct sunlight during certain times of day, and certain times of
the year.

The HVAC system consists of a 2-ton ceiling unit in the center of the
ROC to provide cooling for the room. A dedicated wall-mounted ductless
split-cooling unit provides cooling for the equipment rack. Both units
have their own individual thermostat. Both units share a common 4-ton
condensing unit that is located on the ground floor of Wilson Hall near
the loading dock. The HVAC system is a Variable Refrigerant Volume (VRV)
system that provides individual zoning, since the equipment rack is
expected to require cooling at all times. The setpoint temperature and
actual room temperature can be monitored using the building HVAC
monitoring system.

The ROC is also served by the Wilson Hall HVAC system, which has
Variable Air Volume (VAV) boxes that are located in the ceiling. This
system provides fresh air to the room from outdoors. Any heating that
may be needed is provided by the building hot water baseboard heating
system that existed prior to construction of the ROC.

## Operations Support Team

LHC@FNAL needs a group of people who will be able to help with the
diverse requests that we expect to get for resources. Since the opening
of the ROC in February we have had numerous requests to provide tours of
the ROC, schedule meetings for the conference room, assist in setting up
videoconferencing and projection equipment, make the ROC available for
television interviews, prepare specific content for projectors in the
ROC (e.g. CMS event displays), and display specific content on the
Public Display. Based on these requests and considering the activities
that LHC@FNAL was designed for, we can summarize the types of requests
that we expect to receive for LHC@FNAL resources. We expect requests
for:

-   consoles for CMS and LHC commissioning and operations,

-   training on hardware and software used in the ROC and at CERN,

-   developing software that is needed to operate equipment in the ROC,

-   maintenance of computing, networking, and audio/visual equipment in
    the ROC,

-   scheduling the conference room for meetings, conferences and tour
    groups,

-   presentations on CMS, LHC and remote operations for outreach,

-   developing educational material for outreach purposes,

-   special tours for visiting committees and VIPs,

-   and planning for special events.

To address the diverse requests that we expect to receive, we have
assembled a team of liaisons from different departments at Fermilab.
This team is called the Operations Support Team for LHC@FNAL. Members of
the team are:

-   Sheila Cisko (videoconferencing)

-   Steve Fry (networking)

-   Elvin Harms (LARP)

-   Erik Gottschalk (chair)

-   Elliott McCrory (LAFS)

-   Ben Segbawu (PC and server support)

-   Jean Slaughter (outreach)

-   Alan Stone (CMS)

-   Fred Ullrich (media)

Many of the liaisons belong to organizations that are able to provide
support in the event that the liaison is unable to respond to a
particular request. Moreover, we are instructing users of LHC@FNAL that
all requests for computing and network support should be sent to the
Fermilab Helpdesk. All other types of requests can be directed to the
Particle Physics Division Experimental Physics Project (PPD/EPP) support
staff (Cathryn Laue and Cindy Kennedy), who can forward requests to the
correct person or department.

To handle requests from users we have established a mailing list
(<lhc@fnal.gov>) and a phone number (630-840-4152).

## Summary of Requirements

The requirements document \[[^8]\] that was developed by the previous
LHC@FNAL Task Force continues to provide guidance on what is needed to
establish a joint remote operations center for CMS and LHC. In this
section we refer to specific requirements in the requirements document.
We refer to requirements that have been satisfied by Phase 1, and
discuss requirements that still need to be addressed. In some cases we
find that the implementation of LHC@FNAL is still somewhat premature
with respect to certain requirements, and that in time we will receive
the necessary guidance from CMS, LHC, LARP and CERN to implement
required functionality. For example, while LHC@FNAL is already being
used for some shift activities, it seems premature to maintain and post
a detailed shift schedule (showing shift personnel, time on shift, and
contact information) so that the schedule can be accessed by CMS and LHC
personnel. The requirement to post a detailed shift schedule will be
addressed at a later date.

This section addresses *combined requirements* for CMS and LHC presented
in Section 3.3 in the requirements document, and *constraints* that are
presented in Section 3.4.

### CMS/LHC Combined Requirements

Combined requirements for CMS and LHC address both the general
capabilities of LHC@FNAL and the operational environment of the ROC.
Section 3.3 of the requirements document has nine requirements for
general capabilities, and seven for the operational environment. These
requirements are listed in Table 2.

  ---------------------------- ------------------------------------------
  Requirement Number           Requirement Name

  3-1                          Safeguards

  3-2                          Hardware and Software Consistency

  3-3                          Consoles

  3-4                          Communications

  3-5                          Shifts

  3-6                          Record of Shift Schedule

  3-7                          Directory

  3-8                          Web Page

  3-9                          Lifespan and Effectiveness Reviews

  3-10                         Shift Area

  3-11                         Common Area

  3-12                         Display Sharing

  3-13                         Working Area

  3-14                         Social Area

  3-15                         Outreach

  3-16                         Clocks
  ---------------------------- ------------------------------------------

Table 2: LHC@FNAL requirements for general capabilities and the
operational environment.

Requirements 3-1 through 3-9 in Table 2 address general capabilities.
Requirement 3-1 is a very important requirement on establishing
safeguards that prevent actions from jeopardizing or interfering with
CMS and LHC operations. For CMS this requirement is still somewhat
premature to be considered for remote operations, while for LHC we are
involved in the development of Role Based Access Control for the LHC
controls system. Requirements 3-2 through 3-4 have been mostly satisfied
by Phase 1. Those that have not been satisfied are under active
development. Requirements 3-5 through 3-9 address remote shifts and
reviews of LHC@FNAL. These requirements will be addressed as we become
more involved in shift activities.

Phase 1 construction has satisfied all of the requirements pertaining to
the operational environment of LHC@FNAL, except requirements 3-13 and
3-14 that call for separate working and social areas, respectively. At
this time we are planning to provide this need for additional space in
the Wilson Hall Mezzanine. We are in the process of setting up several
office cubicles, two meeting areas, and are looking into a location for
a small kitchen facility.

### Constraints

Constraints are presented in Section 3.4 of the requirements document.
There are requirements for communications, computing and networking,
software development that is done in the context of LHC@FNAL, security
and safety. These requirements are listed in Table 3.

  ---------------------------- ------------------------------------------
  Requirement Number           Requirement Name

  4-1                          Communications

  4-2                          Computing

  4-3                          Computing for LHC@FNAL Users

  4-4                          Reliable Networking

  4-5                          Software Compliance

  4-6                          Software Repository

  4-7                          Version Control

  4-8                          Parameters Database

  4-9                          Software Testing

  4-10                         Computing and Networking Security

  4-11                         LHC@FNAL Space Security

  4-12                         General Safety

  4-13                         LHC@FNAL Safety
  ---------------------------- ------------------------------------------

Table 3: Requirements for communications, computing, networking,
software, security and safety.

Requirements 4-1 through 4-4 in Table 3 have been mostly satisfied. The
only area in which we need some improvement is in reliability and
redundancy. For example, a power outage in the ROC will interrupt
operations by disabling console PCs and networking hardware. As soon as
the hardware infrastructure in the ROC is established we will install
uninterruptible power supplies that will let us keep a few console PCs
and the network hardware operational during a power outage. We also
intend to have a few "hot spares" available to replace PCs that have
failed.

Requirements 4-4 through 4-9 address software development that is done
explicitly for LHC@FNAL. This is distinct from CMS experiment software
or LHC accelerator software that we are developing according to coding
standards established by the CMS and LHC projects. At this time we have
not developed any software explicitly for LHC@FNAL.

Requirements 4-10 through 4-13 address security and safety. These
requirements have been satisfied during Phase 1 construction, and we
will continue to satisfy these requirements as LHC@FNAL evolves.

## LHC@FNAL Construction

Design and construction of the LHC@FNAL Remote Operations Center proved
to be a model in inter-divisional collaboration, and the end result
surpassed the expectations of nearly everyone involved. This General
Plant Project (GPP) was jointly managed by the Accelerator Division
(AD), Facilities Engineering Services Section (FESS), and the Particle
Physics Division (PPD). The general contractor, Wegman Construction,
proved to be an excellent partner.

Aside from the discovery of undocumented electrical feeds, which brought
construction to a halt for a few days until the situation was understood
and resolved, work proceeded smoothly. The project was completed on
time, under budget, and with no injuries.

A timeline of the construction is shown in Table 4.

  ---------------------------------------------------- ------------------
  Milestone                                            Date

  Construction Directive Authorization received        May 2, 2006

  Engineering started                                  May 2006

  Construction started                                 Sept. 2006

  Beneficial Occupancy granted                         Dec. 18, 2006

  Construction completed                               Feb. 8, 2007
  ---------------------------------------------------- ------------------

Table 4: LHC@FNAL Phase 1 construction timeline.

The initial construction budget for LHC@FNAL was estimated at \$600K,
including Engineering, Design, Inspection and Administration (EDIA) as
well as contingency. As very favorable construction bids were received,
it became apparent that the project budget could be reduced by \$100K.
Expenditures through the end of March 2007 are summarized in Table 5.

+----------+---------+---------+----+---------+-----+-------+-------+
|          | AUT     | OBLI    | CO | OPEN    | RIP | B     | \%    |
|          | HORIZED | GATIONS | ST | COMM    | \'s | UDGET | OBLI  |
|          | BUDGET  |         |    |         |     | BA    | GATED |
|          |         |         |    |         |     | LANCE |       |
+----------+---------+---------+----+---------+-----+-------+-------+
|          |         |         |    |         |     |       |       |
+----------+---------+---------+----+---------+-----+-------+-------+
| > Direct | 410.0   | 405.2   | 4  | 0.0     | 0.0 | 4.8   | 98.8% |
| > Costs  |         |         | 05 |         |     |       |       |
|          |         |         | .2 |         |     |       |       |
+----------+---------+---------+----+---------+-----+-------+-------+
| >        | 90.0    | 78.3    | 78 | 0.0     | 0.0 | 11.7  | 87.0% |
| Overhead |         |         | .3 |         |     |       |       |
+----------+---------+---------+----+---------+-----+-------+-------+
|          | 500.0   | 483.5   | 4  | 0.0     | 0.0 | 16.5  | 96.7% |
|          |         |         | 83 |         |     |       |       |
|          |         |         | .5 |         |     |       |       |
+----------+---------+---------+----+---------+-----+-------+-------+

Table 5: Budget table for General Plant Project G06184, "LHC@FNAL
Upgrades."

# CMS Working Group

The CMS Working Group of the LHC@FNAL Integration Task Force is
responsible for providing remote operations and monitoring capabilities
for US CMS at the Remote Operations Center (ROC) located on the first
floor of Wilson Hall at Fermilab. This ROC provides an operational
component to the LHC Physics Center (LPC) \[[^9]\] at Fermilab whose
goal is to help CMS scientists at U.S. universities and labs to
contribute to CMS and to participate efficiently in CMS physics
analyses. The CMS Working Group coordinates with the Data Quality
Monitoring groups and the detector groups to help provide them with
needed infrastructure. The actual data quality monitoring is, of course,
the responsibility of the detector groups themselves. A major goal of
the CMS Working Group is to enable physicists to take remote shifts at
Fermilab. The CMS Working Group will provide both the necessary hardware
to make this possible as well as help coordinate with the overall CMS
remote operations efforts, which include the CMS Centre at the Meyrin
site \[[^10]\].

In particular the CWG is responsible for the following tasks:

-   develop CMS requirements and cost estimates for Phase 2;

-   develop a plan to support CMS commissioning activities;

-   describe remote operations capabilities for CMS;

-   develop an operations model for remote operations for CMS.

These tasks were identified as belonging to the scope of the CWG based
on the October 18, 2006 Charge to the LHC@FNAL Integration Task Force
\[Error: Reference source not found\].

Members of the CMS Working Group are:

-   Bill Badgett

-   Kurt Biery

-   Nick Hadley (Group Leader)

-   Shuichi Kunori

-   Kaori Maeshima (Group Leader)

-   Patty McBride

-   Alan Stone

## ROC Infrastructure

Construction of the LHC@FNAL ROC on the first floor of Wilson Hall at
Fermilab is described in detail in Section 2. It became operational in
February 2007. The ROC will be home to commissioning and operations
activities for the LHC accelerator as well as the CMS experiment.

A ROC was originally built and commissioned on the 11^th^ floor of
Wilson Hall (WH11) in 2005. The WH11 ROC was first used during the CMS
HCAL test beam data taking at CERN. The WH11 ROC was also successfully
used for quasi-online data monitoring during the Magnet Test Cosmic
Challenge (MTCC) in August 2006 (Phase I) and in October 2006 (Phase
II). Data from the MTCC were transferred from the Point 5 facility in
Cessy, France to the computing facility at CERN in Meyrin, Switzerland,
and then to Fermilab. The MTCC experience at the Fermilab ROC is
described in detail in a recent CMS Note \[[^11]\].

It is obvious that a successful CMS Remote Operations Center is more
than just physical infrastructure. Effective operation of a ROC depends
on how CMS operates as a whole. To be effective a ROC must facilitate
communication and interactions between local and remote experimenters.
In addition, a ROC is comprised of a suite of portable,
location-independent software tools. Ongoing efforts to realize these
goals have been in progress by the CMS ROC group, which is part of the
LPC, since 2005 \[[^12]\]. The CMS remote monitoring operation at the
LHC@FNAL ROC can benefit from what has been learned from these efforts.

The CMS ROC activities at the WH11 ROC have migrated to the LHC@FNAL.
This increases the visibility of the LHC effort at Fermilab, and
provides a place for accelerator scientists and engineers working on LHC
commissioning and operations.

## Plan for CMS Commissioning Activities

The first user group for the ROC is the CMS tracking group, which is
taking remote shifts for the commissioning of the tracker at the Tracker
Integration Facility (TIF) in Building 186 at CERN. The tracker group
began taking remote shifts at the ROC in March 2007. This activity will
continue through the end of May 2007. Two consoles have been allocated
to the tracking group as needed for the periods of data taking when
experts and shifters gather at the ROC for remote monitoring.

The Tier-1 computing facility at Fermilab operates 24x7 support of
critical services and facilities. The Fermilab team of administrators
and operators are responsible for the local Tier-1 resources, which
represent roughly half of the CMS computing in the U.S., and provide
substantial central support for the seven university-based Tier-2
centers that operate with small local operations teams. The primary
responder from the administration group rotates on a weekly basis and
resides in the ROC during weekday, business hour shifts. The Tier-1 team
is responsible for ensuring that mass storage and processing facilities
at Fermilab are operational. The primary administrator monitors the
functionality of the storage interface, which receives data from the
Tier-1 center at CERN as well as exports data to requesting Tier-2 sites
in the US and abroad. The administrator also monitors the grid
interfaces to the processing systems, which accept remote
reconstruction, user analysis and simulated event production jobs.

The facility teams have developing detailed monitoring displays and
alarms to improve the efficiency for operating these complex distributed
computing facilities and ensuring high availability of the facility
services. The control room layout of the ROC maps very well to the needs
of the Tier-1 administration operations team and the feedback from the
operators has been positive.

During the late spring and summer of 2007, the ROC will be used to take
shifts during the global running weeks that are planned during the
installation and commissioning of CMS at Point 5. These will be weeks of
more or less stable data taking for detector debugging and
commissioning, interspersed with weeks of more intensive installation
activities when services such as power and cooling may not be available.

The members of the CMS Working Group have contacted the leaders of all
of the CMS detector groups and made them aware of LHC@FNAL capabilities,
and requested that they include the LHC@FNAL in their commissioning and
DQM plans. An advisory panel with members \[[^13]\] from the various
detector groups has been formed, and they will review and provide advice
on CMS Working Group plans.

## Model for CMS Remote Operations

Members of the CMS Working Group are actively participating in the
development of the operations model for remote operations of CMS. We
have participated in meetings of the CMS Centre Requirements and
Technical Assessments Group (RTAG) \[[^14]\], which is designing a ROC
at the CERN Meyrin site.

The US CMS Research Program Manager is assuming that remote shifts will
be counted as "official" CMS shifts in his planning and in discussions
with CMS management. It is possible that there will be other ROCs
besides the ones at Meyrin and Fermilab. ROCs are expected to begin as
remote monitoring centers; the actual control operation of the detector
(e.g. changing voltage settings) will likely be restricted to Point 5
for safety reasons for the foreseeable future.

The development of software tools for common monitoring will be
continued by the CMS ROC group at the LPC, including Web-Based
Monitoring (WBM) \[[^15], [^16]\], Screen Snapshot Service \[[^17]\],
the Data Quality Monitoring (DQM) data server functionality in the
Storage Manager \[[^18]\], and the integrated Trigger Monitoring system.
One major goal is to provide monitoring tools which can be used by
experts and shift people from any location for the efficient operation
of CMS.

### Support

A Plone site \[[^19], [^20]\] has been created for LHC@FNAL ROC shift
people as a repository for general documentation, contact information
and shift operations.

Users of a ROC are expected to have a constant stream of requests, and
the ROC infrastructure requires continual maintenance. The CMS Working
Group recommends that this role be filled by a full-time employee whose
job would include:

-   Maintaining and updating documentation

-   Handling and prioritizing requests from ROC users

-   Scheduling of ROC rooms and shifts

-   Repair and maintenance of ROC equipment

Since PCs and monitors have the same configuration for all consoles in
the ROC, if there is a hardware failure on one console PC, a shifter may
simply move to an open PC or take over a PC running non-critical
applications. If all PCs are being used by other shifters to run
critical applications, then the faulty hardware needs to be replaced. A
system administrator should be called, and a spare PC (monitor, network
card, etc.) can be swapped. For such a model to work, users of the ROC
would need to be able to reach a system administrator 24/7, or
indirectly through the Fermilab Helpdesk. Furthermore, LHC@FNAL should
maintain an inventory with service dates of all available hardware,
including spares.

Much of the software for CMS monitoring tools resides on a
doubly-redundant Blu-arc disc, and is available to any of the PCs and
users. The Blu-arc discs are protected by an uninterruptible power
supply (UPS). We expect failures to be very rare and that the response
time from the Fermilab Computing Division will be timely.

CMS management foresees a period after which CMS has reached steady
state running when the shift people at the Fermilab ROC will be
responsible for detector monitoring, especially during the evening shift
at Fermilab, which corresponds to the midnight to 8AM shift at CERN. It
should be noted that a ROC that is used for shifts must be operational
with the same reliability as an on-site control room. This will require
24/7 support.

## Requirements

Essentially all of the requirements that were specified in the
requirements document \[[^21]\] that was developed by the previous
LHC@FNAL Task Force have been at least minimally met. There are a few
requirements (e.g. simultaneous communication with multiple control
rooms) for which it is premature to ask whether the requirement has been
met, since multiple control rooms do not yet exist.

## Conclusion

Construction of the Remote Operations Center on the first floor of
Wilson Hall was completed on schedule and the ROC is already in use and
being commissioned as the CMS experiment is being commissioned at P5.
Usage is expected to increase as CMS will be commissioned during the
spring and summer of 2007. However we must remain flexible, especially
at this stage of the experiment. This will be an extremely busy period
as we prepare for data taking, which is scheduled to begin in November
2007.

# LHC Working Group

The LHC Working Group (LWG) is responsible for...

In particular the LWG is responsible for the following tasks:

-   develop LHC requirements and cost estimates for Phase 2;

-   develop a plan to support LHC and LARP commissioning activities;

-   describe capabilities that LHC@FNAL will provide for LHC and LARP;

-   and develop an operations model for remote operations for LHC and
    LARP.

These tasks were identified as belonging to the scope of the LWG based
on the October 18, 2006 Charge to the LHC@FNAL Integration Task Force
\[Error: Reference source not found\].

Members of the LHC Working Group are:

-   Wolfram Fischer

-   Suzanne Gysin

-   Elvin Harms (Group Leader)

-   Michael Lamm

-   Mike Lamont (CERN)

-   Elliott McCrory

-   Dave McGinnis

-   Jim Patrick

-   Jean Slaughter

-   Alvin Tollestrup

In this section we...

# Outreach Working Group

The Outreach Working Group (OWG) is responsible for...

In particular the OWG is responsible for the following tasks:

-   Develop requirements for outreach;

-   and develop an operations model for outreach activities.

These tasks were identified as belonging to the scope of the OWG based
on the October 18, 2006 Charge to the LHC@FNAL Integration Task Force
\[Error: Reference source not found\].

Members of the Outreach Working Group are:

-   Marge Bardeen (Education)

-   Erik Gottschalk (LHC@FNAL Integration Task Force member)

-   Judy Jackson (Public Affairs)

-   Liz Quigg (Education)

-   Kurt Riesselmann (Public Affairs)

-   Sue Sheehan (Education)

-   Jean Slaughter (Group Leader, and LHC@FNAL Integration Task Force
    member)

-   Alvin Tollestrup (LHC@FNAL Integration Task Force member)

-   Fred Ullrich (Visual Media)

In this section we...

![](media/image5.jpeg){width="6.0in" height="2.025in"}

Figure N: LHC@FNAL logo.

**References**

[^1]: \[?\]<http://docdb.fnal.gov/REMOP-public/DocDB/ShowDocument?docid=981>.
    Charge for the LHC@FNAL Integration Task Force.

[^2]: \[?\]
    <http://docdb.fnal.gov/REMOP-public/DocDB/ShowDocument?docid=964>.
    Network proposal for LHC@FNAL.

[^3]: Most of the systems are running with the 32-bit version of
    Scientific Linux, and the rest are running are running with the
    64-bit version. The gateway PC and the server installed at FCC are
    both running the 64-bit version of Scientific Linux 4.4.

[^4]: The three functions are to provide a place that provides access to
    information in a manner similar to what is available in control
    rooms at CERN, to serve as a communications conduit, and outreach.

[^5]: Installation of the HD videoconferencing system is schedule for
    the beginning of May 2007.

[^6]: \[?\] <http://www.webex.com/>

[^7]: \[?\] <http://www.lutron.com/>

[^8]: \[?\]
    <http://docdb.fnal.gov/REMOP-public/DocDB/ShowDocument?docid=165>.
    LHC@FNAL Requirements Document.

[^9]: \[?\] <http://uscms.org/LPC/LPC.htm>.

[^10]: \[?\] CMS Note 2007/010, Dave Barney et al., "Functions and
    Requirements of the CMS Centre at CERN."

[^11]: \[?\] CMS Note IN 2007/008, William Badgett et al., "CMS Magnet
    Test Cosmic Challenge Phase I -- Data Monitoring at the Fermilab
    Remote Operations Center."

[^12]: \[?\] <http://uscms.org/roc/>. The CMS ROC group at the LPC is
    coordinated by Kaori Maeshima.

[^13]: \[?\] The members of the Advisory Panel are: Ken Bloom, John
    Conway, Colin Jessup, Andrey Korytov, Lisa Shabalina, Mayda Velasco,
    Rick Wilkinson, and Peter Wittich.

[^14]: \[?\] <http://lucas-nice.web.cern.ch/lucas-nice/cms-centre/>. CMS
    Centre Requirements and Technical Assessment Group.

[^15]: \[?\] <http://cmsmon.cern.ch/>. CMS Web-Based Monitoring.

[^16]: \[?\] CMS Note IN 2006/044, William Badgett et al., "Proposal for
    Web Based Monitoring and Database Browsing."

[^17]: \[?\] <http://home.fnal.gov/~biery/snapshot/>. Screen Snapshot
    Service by Kurt Biery.

[^18]: \[?\]
    <https://twiki.cern.ch/twiki/bin/view/CMS/FilterUnitOutputHandling>.
    CMS Storage Manager.

[^19]: \[?\] [http://plone.fnal.gov](http://plone.fnal.gov/).

[^20]: \[?\] [http://lhcatfnal.fnal.gov](http://lhcatfnal.fnal.gov/).
    LHC@FNAL Remote Operations Center Shift Page.

[^21]: \[?\]
    <http://docdb.fnal.gov/REMOP-public/DocDB/ShowDocument?docid=165>.
    LHC@FNAL Requirements Document.