# The U.S. ATLAS Research Program: 

# Empowering U.S. Universities for Discoveries at the Energy Frontier

1.  **Introduction**

The scientific investigations of the 20^th^ century have revealed a
universe which is fourteen billion years old and of enormous extent.
Very sparsely filled with matter, the universe is composed of types of
particles revealed through many years of study. New ideas have explained
how these particles interact and many aspects of how the universe
evolved. An important consequence of these ideas is that the material
particles do not have intrinsic masses. The masses arise through
interaction with a field. Since particle masses are everywhere the same,
we have to conclude that this field exists uniformly everywhere in the
universe. Thus what we think of as empty vacuum in fact is filled with a
field that determines crucial parameters of our material existence. The
mysterious mass-generating field is called the Higgs field. There is a
physical "Higgs particle" associated with this field, and we know that
its mass must be greater than 110 GeV, since it has not been seen, but
less than a few hundred GeV, or it will not do its job of generating
mass. More generally, other possible mechanisms to generate mass should
be revealed by detectable effects in the same mass range.

Although the Higgs field is everywhere, the material particles are
mostly gathered into galaxies under a different influence, that of
gravity. The dominant gravitational force comes from the second
mysterious cosmic substance, dark matter, which may consist of new
particles with masses in the same range of a few hundred GeV. The new
particles that may well be found at the Large Hadron Collider (LHC) are
important not only for extending unknown dimensions of the physics of
elementary particles, but also for understanding key issues in
cosmology. This is a time when we have unusually strong indications that
the step taken by the LHC at CERN, leading to collision energies far
beyond those available at existing facilities, will lead to especially
important discoveries with implications over a broad field of
fundamental science.

2.  **The Instruments of Discovery: the Large Hadron Collider and the
    ATLAS detector**

Progress in understanding our universe has been made possible by the
construction and use of novel scientific instruments. The most powerful
discovery instruments have attracted some of the best scientists in the
country, driven by the potential to gain a deeper understanding of
nature. In addition they have provided outstanding training on science
and technology for students at numerous universities.

The LHC at CERN, in Geneva, Switzerland, instrumented with powerful
particle detectors, is such a discovery instrument. It is now under
construction with major U.S. participation. The goal of the U.S. ATLAS
Research Program at the LHC, the subject of this proposal, is to provide
a broad based understanding of the physical origin of mass, how it
relates to our other concepts about the material universe, and what
other new forms of matter may exist related to the origin of masses and
cosmological puzzles. Answers to these questions would provide a major
advance toward completing a unified view of the particles in nature, the
forces with which particles interact and their role in the past and
future of our universe.

The parameters of the LHC have been specially chosen to take advantage
of the opportunity to make important discoveries in the energy range of
a few hundred GeV. Building on over forty years of CERN member state
investments in infrastructure and specialized personnel, the LHC offers
a major opportunity for U.S. scientific participation in this particle
physics project. ATLAS is a detector carefully designed to reliably
extract the scientific information from the particle collisions provided
by the LHC and achieve the discoveries we expect^1^.

This proposal rests on the especially strong scientific case for
investigating this energy region and benefits from a program in which
the international community is providing a major part of the investment.
This creates a situation where the return on an investment is unusually
favorable, but requires that we overcome the problems of working at a
distance. We describe in this proposal how we will do this by advancing
the tools of Information Technology. The bold initiative of the NSF in
joining the DOE to carry out a very significant portion of the
construction of the ATLAS detector has opened up this opportunity for
the U.S. scientists. The present proposal will follow up that vision by
a program to exploit the operation of the facility and perform the
search for new discoveries.

3.  **U.S. Leadership**

The importance of the role of U.S. scientists and engineers in the ATLAS
Construction Project is more significant than might be expected from the
fraction (about one-sixth) of the people and resources that we provide
to the Collaboration. The U.S. made a very large effort to develop
detectors for the SSC, and much of that work has been adapted
effectively for the LHC. Similarly there has been significant synergy
with the development of the detectors at BaBar, the Tevatron and RHIC.
It would not be unfair to describe the effect of these factors as a
leadership role, certainly relative to the proportionate size of the
U.S. effort. There are similar factors at work in the ramp-up to the
research phase of ATLAS. Operation of this extremely complex facility
will be at least as challenging as constructing it, and those who best
understand the working of the detectors will be most successful in the
operation phase. We expect that the members of the U.S. team will be
important to the success of the operations. We have already seen that
the experience gained in developing the software and computing for the
big U.S. experiments has been of great benefit to ATLAS, and has again
led to a leadership role. It is in the interests of the U.S. to maintain
this trend, which will depend in large part on the adequacy of the
funding during the transition to the research phase.

4.  **The U.S. ATLAS Research Program**

The U.S. ATLAS Research Program aims at providing U.S. university
scientists, teams of faculty, postdoctoral, graduate and undergraduate
students, with the opportunity to maximize the science from the LHC
accelerator. This science includes careful tests of presently available
theoretical ideas and the ability to follow individual inspirations. The
latter often come from carefully trying to deduce the "message" being
carried by the data.

The ATLAS research program will include the technical effort associated
with maintenance and operation of the detector. A properly functioning
detector and the knowledge of how it responds to the products of the
beam collisions is the first requirement for using the data to
understand the physics. A second element of the research program is in
the area of computing, which is a most crucial resource for allowing the
university scientist access to the data. What is needed is rapid access
to as complete a set of data as possible, and the processing power to
analyze it. This is required both to collaborate efficiently in an
international group effort as well as to follow individual directions.
Open access to the data is a critical principle of scientific
collaboration, but requires adequate computing to make the principle
meaningful. In the very large collaborations that have come into
existence, the analysis of the "hot topics" that are thought to provide
the best chance of a new discovery often take on a competitive aspect
between different scientists or teams. It is important to make sure that
the U.S. scientists are provided with tools second to none, on a global
scale.

The last element of the ATLAS research program involves R&D for
upgrades, which will be needed to maximize the physics output during the
long period of running expected for the LHC. The luminosity and
radiation exposure will be increasing throughout this time, allowing
ever more rare events to be studied. These upgrades are absolutely
necessary for the components of the experiment that are known to have a
limited lifetime in the radiation environment of the LHC. These will
have to be replaced, and should be replaced with new technology with
improved performance.

The ATLAS detector will be completed over the next few years. As parts
are assembled they will enter the pre-operations phase, which will need
to start promptly after detector element completion. The ATLAS detector
is significantly more complex than any previous particle physics
detector and the pre-operations phase will be more complex as well.
Similarly, the computing effort is very large and will last many years.

5.  **Information Technology in ATLAS: Computing, Software and
    Networking Challenges**

The computing demands of the LHC scientific program far exceed anything
seen before in high energy and nuclear physics, and indeed are
unprecedented in scientific research in general. Vital aspects of LHC
computing, in particular the global management of huge LHC data volumes,
also go well beyond the practice and capabilities of the commercial
world. LHC computing is breaking new ground in information technology
(IT), and requires the development and application of new technologies
and tools at the leading edge of IT in order to be successful. A robust
and successful computing infrastructure for the LHC experiments is
particularly crucial for the U.S. and other remote participants because
many of the most challenging (and therefore vulnerable) aspects of LHC
computing are associated with providing an effective *distributed*
computing environment that fully enfranchises remote participants in
physics analysis. U.S. leadership in building LHC computing is an
essential prerequisite to U.S. leadership in physics analysis.

An indication of the LHC computing scale relative to current and
previous generation HENP and astronomy experiments is given by the table
below, showing the host lab computing requirements of a single LHC
experiment in comparison with other experiments. Note that for the LHC,
the host lab (CERN) provides only one third of the total computing
resources.

![](media/image1.wmf)

The greatest IT challenges involved in ATLAS computing involve the
scale, complexity and global management of the data. When the LHC
reaches its nominal luminosity the ATLAS experiment alone will have a
storage accumulation rate of about two Petabytes per year. The LHC data
volume is expected to reach Exabyte scale by approximately 2012-2015.
This tremendous data volume arises from the complexity of the events and
the very high rate in the LHC physics environment and the enormous scale
of its detectors. A typical bunch crossing in the ATLAS detector will
involve about 20 individual proton-proton collision interactions. When
one of these has physics characteristics of interest, the detector is
triggered and the event -- together with the superposed "background"
interactions -- is read out. Crossings occur every 25ns, a time shorter
than the sensitive time window of the detector for a triggered event, so
background interactions from several adjacent beam crossings are also
superposed on the "signal" event that triggers the readout. Hundreds of
background interactions must be filtered out from the physics signal of
interest in an event. The high occupancy and measurement precision
requirements on the detector impose highly granular detector
segmentation and readout, which contributes to the data volume -- ATLAS
has about 100 million readout channels and an event size of about 1
Megabyte -- and the complexity of the event reconstruction task.

Once the data is acquired and reconstructed, the nature of the LHC
physics environment makes its analysis a complex challenge as well. Much
of the most interesting physics expected at the LHC, such as the Higgs
boson and supersymmetry, is predicted to occur at very low rates. A
typical light Higgs signal, for example, may involve of order a thousand
signal events distilled from 10^14^ events occurring in the detector in
a year of data taking. The ATLAS trigger provides a rejection factor of
10^5^, but the further selection of one event in a million must be
performed in offline processing. This presents one of the central IT
challenges in LHC computing: the rapid and efficient extraction of
sparse physics samples from extremely large datasets.

Data volume and complexity and the scale of the physics analysis task
lead directly to very large processing requirements. Processing a
nominal year\'s ATLAS data will require computing power in excess of one
million SpecInt95. Because events in a high energy physics experiment
are independent and uncorrelated (apart from time dependent detector
state such as calibrations) with modest I/O rate requirements,
economical computing farms built from standard commodity components
interlinked by conventional Local Area Networks (LANs) are a good match
to our needs. The IT challenge in this area comes from the scale
involved: processor counts are in the thousands, and processors and
other resources at different sites must be linked in a seamless, secure
and fault tolerant collaborative computing environment that provides for
managed resource sharing and monitoring. Commercial tools for managing
this degree of complexity do not exist even for a single site, let alone
for a distributed ensemble of centers. The IT and computer science (CS)
worlds recognize, however, that such capabilities will be required in
the future, so we are addressing these areas through collaboration with
CS on the Data Grid discussed below.

The data management, processing and analysis tasks required by ATLAS
must be conducted in the context of a very large collaboration
distributed worldwide. ATLAS involves about 2000 physicists from 150
institutions in more than 30 countries worldwide with a U.S. contingent
of about 300 collaborators in 33 institutions. The computing system must
foresee \~1000 simultaneous users distributed globally and requiring
full enfranchisement in physics analysis when working from their home
institutes. The ATLAS computing task is thus inherently distributed, and
must provide a global analysis environment in which the distributed
nature is transparent to the physicist working at their home institute.
Herein lies the newest aspect of LHC computing -- current and previous
experiments have had much more centralized computing operations -- and
the aspect that most captures the interest of the computer science
community as an exciting and leading edge IT challenge.

The IT challenge presented by the distributed aspects of LHC computing
has led to a close collaboration between High Energy and Nuclear Physics
(HENP) and computer scientists on the development of the Data Grid as
the distributed "middleware" software infrastructure upon which
distributed HENP applications can be built. ATLAS has developed a
worldwide computing model in which CERN resources provide only one third
of the required computing capacity, focusing on raw data archiving and
reconstruction; about five regional centers distributed worldwide
provide another third (so-called "Tier 1" centers), focusing on
simulation, analysis and limited reconstruction; and the remainder is
provided by smaller regional centers ("Tier 2" centers) and institutes,
focusing on analysis and simulation. Most importantly, this hierarchy of
centers must serve the physicist at their home institute as
transparently as possible. Most often the physicist will not be at one
of the "Tier" centers but at a typical university. The physicist at
their home institute must be well integrated into a worldwide
collaborative environment and be fully able to participate effectively
in physics analysis through the distributed services of ATLAS software
and collaborative tools. Realizing this model is dependent upon the
successful development of the Data Grid as its foundation. The Data
Grid's roots are in the U.S. where the concept and founding software was
developed (largely by the Globus project). As recognition of its
importance has grown, activity has expanded greatly both in the U.S. and
elsewhere. The U.S. retains a leading role and the U.S. ATLAS computing
program contains a strong Data Grid element both to leverage the U.S.
strengths in this area and out of recognition that the distributed
capabilities the Data Grid is addressing are crucial for the U.S. ATLAS
Research Program.

ATLAS computing also presents challenges that are more mundane from an
IT perspective but no less important to ultimate success in the physics.
The extraordinary complexities of LHC physics, the ATLAS detector and
the ATLAS physics analysis environment outlined here are manifested in
an extremely complex software base and distributed computing
environment. ATLAS and U.S. ATLAS are employing IT professionals to
develop the software and computing infrastructure required. It is vital
to the success of ATLAS physics that there be a solid linkage between
the sophisticated computing infrastructure and the physicists who must
make use of it. Specialists -- typically computing experts with a
physics background -- must be employed to work at the interface between
the core computing infrastructure and its physicist users; for example,
to support physicists developing and deploying simulation,
reconstruction, calibration and analysis software in the context of the
software infrastructure.

6.  **Software and Computing for the Research Program**

Computing, software and physics in U.S. ATLAS are tightly coupled in
order to maximize the impact of U.S. physicists in both the building
stage and in providing the opportunity for leadership in physics
analysis in the data taking phase of the experiment. In the past, in
order to be effective, investigators in high energy physics experiments
had to be located at the host laboratory. Individuals who could not
travel to the host laboratory would only have marginal participation in
major analyses. This was, in part, a result of the dominance of meetings
at the host laboratory, but mainly a result of the availability of
computing and support resources at the central location. If U.S.
physicists are to achieve a leadership position in new discoveries
associated with the LHC, a new computing model must be implemented which
allows a far more global access to data, support and computing
resources. A substantial part of this proposal is focused on creating
the infrastructure to allow U.S. physicists to take a leadership
position in LHC physics. This includes the ability to accurately
simulate physics processes, visualize data, analyze data, and
participate in the creation of software of common use within ATLAS.

The goals for the physics, software and computing part of the Research
Program are to have the following in place in 2007, the start of data
taking:

-   A fully functional, grid-enabled, computing environment consisting
    of a Tier 1 center at Brookhaven National Laboratory, a set of Tier
    2 centers at U.S. ATLAS universities (included as part of this
    proposal), and the associated grid middleware to knit the centers
    together into a true grid environment capable of supporting both
    production and "chaotic" user analysis.

-   A well trained, highly integrated group of physicists in the U.S.
    who are in the position to do forefront analysis, and supervise
    research by students in the ATLAS Collaboration from their home
    institutions. This is closely coupled with "deliverables" of core
    and detector-specific software to the ATLAS collaboration.

-   Support for U.S. ATLAS physicists in analysis in the form of
    help-desks, librarian and infrastructure support at CERN to
    incorporate software into releases.

-   Deliverables to international ATLAS in the form of supported pieces
    of software that have been written and are supported by the U.S.
    ATLAS Research Program.

-   Support in doing physics analysis with fast turn-around in
    evaluating the signatures of physics processes.

Below we describe the physics, software and computing aspects of the
U.S. ATLAS Research Program in more detail, and identify the components
that this proposal addresses.

**6.1. Physics**

Physics analysis per se is an activity carried out by physicists, and is
not considered part of the research infrastructure supported by this
proposal. There is no plan to centrally manage physics analysis.
However, support of physics tools for U.S. investigators doing physics
analysis will be essential if our scientists are to be at the forefront
of the discoveries we expect at the LHC. In particular, when new physics
processes are predicted, or an unexpected class of events appears, a
substantial effort must be invested in testing these new processes
against theory. This involves a simulation of the physics processes in
the detector. Simulation of physics events is a vital part of both
software and physics activities on an experiment. During the time prior
to data taking this simulation is used for testing of the software,
understanding the capabilities of the detector and developing strategies
that will be used to exploit the data. ATLAS has embarked on a series of
Data Challenges (described in more detail below) of ever-increasing
complexity, leading up to the preparation of a *Physics Readiness*
report in 2006. Once data taking has commenced, simulation plays an even
more important role in modeling and extracting the physics signals. The
ability to rapidly respond to potential new physics and evaluate it is
critical to the success of an analysis.

We propose to support a person full-time who will provide interfaces
from event generators into the ATLAS software environment. The task will
evolve from preparing the code for integrating the event generators into
the ATLAS software, to a maintenance and support role, as the number of
users increases during the Data Challenges and into the data taking
phase. Such support is best carried out by persons who take it as their
primary responsibility and who can respond to the needs of users. This
proposal requests support for one FTE dedicated to this task, working
under the U.S. ATLAS physics manager. Given the size of the
collaboration and the need to be as competitive as possible, this
support person is vital to our ability to carry out physics analysis
rapidly and efficiently.

![](media/image2.wmf)

Table 1. *Estimated Physics support costs in FY02k\$.*

**6.2. Tier 2 University Computing Centers**

Global access to computing resources and support is a critical aspect to
achieving leadership in physics analysis for the U.S. investigators. The
overall computing model for the LHC experiments is to have a worldwide
distribution of computing centers, with CERN at the hub of the hierarchy
of centers. CERN functions as the main data reconstruction center.
Summary data from the reconstruction is distributed to a set of Tier 1
regional centers, associated with individual countries or clusters of
countries. Below this are Tier 2 centers, which are large aggregations
of computing resources located at university centers. The "fabric" of
computing resources consisting of regional centers, Tier 2 centers, and
individual users is linked together via grid middleware, which is being
developed via funded collaborations between physicists and computer
scientists. In the U.S., these consortia include the NSF-funded
GriPhyN^2^and iVDGL^3^ collaborations, and the DOE-funded PPDG^4^
collaboration. Many of the grid middleware products being developed by
these groups are used conjointly with the U.S. CMS Collaboration, and
consist of tools such as Globus, RLS, Condor, Condor-G, to name a few.
Much of the effort in testing out these tools is fed back to the
computer scientists in the collaborations to help them improve the grid
middleware.

One of the primary mechanisms for the development and validation of
ATLAS software and facilities is a series of Data Challenges. These are
exercises whereby data are produced via an event generator (which
simulates the raw physics processes) and information output in the form
of a list of stable particles that would appear in the detector. The
event generator output is fed into a detector simulation, with the
resulting output mimicking the format of real data. These simulated data
are then used as input to reconstruction algorithms. Data Challenges
have a twofold purpose: to produce data that evaluates the detector
performance and hence can be used to help configure the trigger, examine
staging options, etc. and to stress the computing system and evaluate
its performance in scalability, functionality, etc. Data Challenges also
provide milestones for software releases and often drive the
implementation of increasing functionality of software deliverables.

Already, for the so-called "Data Challenge 1" (DC1), U.S. ATLAS has
integrated many grid tools. Here, the most notable are:
GriPhyN-sponsored GLOBUS, iVDGL-sponsored PACMAN and GRAPPA, and
PPDG-sponsored MAGDA. These tools were meshed into a coherent package
for the ATLAS DC1 production, which was highly successful. We intend to
use future Data Challenges to add increasing functionality as grid tools
become available from these collaborations.

The U.S. regional (Tier 1) center, located at Brookhaven National
Laboratory, is funded by the Department of Energy. We are proposing to
fund the Tier 2 sites via this proposal.

At present, U.S. ATLAS has a testbed that is used for exercising grid
middleware, and is being employed for production for Data Challenge
exercises for international ATLAS. The iVDGL collaboration has funded
two prototype Tier 2 centers in the U.S., one at Indiana University and
one at Boston University. The iVDGL, working closely with the GriPhyN
collaboration, has been deploying GriPhyN-developed grid middleware in
the form of the Virtual Data Toolkit (VDT). Our prototype Tier 2 centers
are working with these collaborations to ensure that the grid middleware
is useful to ATLAS and that ATLAS software is co-functional with VDT. We
continue to ramp up the scale of tests conducted on the testbed and are
participating in ATLAS Data Challenges to stress-test ATLAS software and
grid middleware. The iVDGL funding has already provided extensive
leveraging of existing and additional university resources. We expect
this trend to continue into the selection of the final Tier 2 sites.

![](media/image3.wmf){width="0.24803149606299213in"
height="0.4074803149606299in"}![](media/image4.wmf){width="0.28346456692913385in"
height="0.4956692913385827in"}![](media/image5.wmf){width="0.2251968503937008in"
height="0.4437007874015748in"}![](media/image6.wmf){width="0.16181102362204725in"
height="0.2937007874015748in"}![](media/image7.wmf){width="0.16181102362204725in"
height="0.2917322834645669in"}![](media/image8.wmf){width="0.16181102362204725in"
height="0.2937007874015748in"}![](media/image9.wmf){width="0.21811023622047243in"
height="0.38110236220472443in"}![](media/image10.wmf){width="0.21929133858267716in"
height="0.3799212598425197in"}![](media/image11.wmf){width="0.21811023622047243in"
height="0.38110236220472443in"}![](media/image12.wmf){width="0.12165354330708661in"
height="0.25826771653543307in"}![](media/image13.wmf){width="0.12165354330708661in"
height="0.26023622047244094in"}![](media/image14.wmf){width="0.12165354330708661in"
height="0.25826771653543307in"}![](media/image15.wmf){width="0.12165354330708661in"
height="0.25826771653543307in"}![](media/image16.wmf){width="0.12165354330708661in"
height="0.26023622047244094in"}![](media/image17.wmf){width="0.12165354330708661in"
height="0.25826771653543307in"}![](media/image18.wmf){width="0.12283464566929134in"
height="0.25826771653543307in"}![](media/image19.wmf){width="0.12283464566929134in"
height="0.26023622047244094in"}![](media/image20.wmf){width="0.12283464566929134in"
height="0.25826771653543307in"}![](media/image15.wmf){width="0.12283464566929134in"
height="0.25826771653543307in"}![](media/image16.wmf){width="0.12283464566929134in"
height="0.26023622047244094in"}![](media/image17.wmf){width="0.12283464566929134in"
height="0.25826771653543307in"}![](media/image18.wmf){width="0.12283464566929134in"
height="0.25826771653543307in"}![](media/image19.wmf){width="0.12283464566929134in"
height="0.26023622047244094in"}![](media/image20.wmf){width="0.12283464566929134in"
height="0.25826771653543307in"}

Figure 1. *The present U.S. ATLAS Grid Testbed*.

A substantial part of this proposal has to do with the funding of Tier 2
centers. With the scheduled start of the LHC in 2007, the designation of
Tier 2 sites should occur in 2004, based on local expertise, ability to
support the collaboration and maximum geographical impact. We anticipate
that there will be five Tier 2 centers in the U.S., sited at
universities across the country. The first site would have significant
hardware funding in 2004, with the selection process for the remaining
Tier 2's occurring also in 2004. The Tier 2 selection process has been
described in the U.S. ATLAS Computing Project Management Plan^5^, and
includes criteria such as the leveraging of local resources, the
expertise of the personnel at the University, background in the ATLAS
program, and the possibilities of collaboration with local Computer
Scientists. Costing for the Tier 2's is based on a ramp-up of
capabilities to 100k SpecInt95 of CPU power and 1.2 TB of disk storage
at all sites at the start of data taking in 2007. Costs are estimated
using currently available Moore's law data. Hardware costs also include
licensing, backup, firewall and LAN capabilities. We anticipate that two
of the sites (A and B) will have tertiary storage capability, likely in
the form of robotic tape storage. Two systems FTE's are included at each
site, estimated at 150k\$/year/FTE, based on collaboration-wide averages
for personnel in this category. The ramp-up of Tier 2 centers over time
is shown in the table below.

![](media/image21.wmf)

Table 2. *Estimated Tier 2 costs in FY02k\$*.

We have found that the efficiency of networking in a grid environment is
strongly dependent on the so-called "last mile" of connectivity between
Internet backbones and local computing centers. We propose to augment
the Tier 2 teams with a central group that is used to monitor network
performance and debug last mile problems, particularly with the addition
of routers. Also, the monitoring and tuning of TCP protocols in
optimizing bandwidth issues is required to derive and maintain maximum
efficiency.

**6.3. Software**

There are several important roles that the U.S. should play to establish
leadership in computing and physics: core software, detector-specific
software, infrastructure and grid integration. We are proposing to fund
portions of the first three in this proposal. Grid integration is
carried out in conjunction with several grid computing consortia, such
as iVDGL, PPDG, GriPhyN and with the U.S. CMS Collaboration. Core
software is funded jointly with the Department of Energy, which has a
large component at the U.S. National Laboratories. A major portion of
detector-specific software comes from base program contributions from
physicists working at U.S. universities and National Laboratories, but
to be effective, and certainly to be in a leadership position, an
adequate complement of computing professionals who have a strong physics
background is necessary. National Laboratories and universities are
suffering from great stress in their base program funding, jeopardizing
the level of participation that has been planned. Without adequate
support of groups who are directly engaged in detector subsystem
activities, the U.S. cannot establish leadership in physics analysis.
Infrastructure support is less visible but is the line that divides
success from failure. We propose to have a component of infrastructure
support for U.S. physicists via dedicated software professionals, funded
by the U.S. program but residing at CERN to increase the efficacy of
U.S. physicists participating in the international program.

The software components for which we request funding in this proposal
are described in more detail below.

***Core software*:** Generalized software that is used to access data,
provide I/O, configure the order in which algorithms run, data
management in a local environment, etc. Core software is independent of
detector-specific algorithms which are unique to detectors and the
detector subsystems. This effort is coordinated within the international
ATLAS collaboration and is the subject of Software agreements between
ATLAS and the U.S. ATLAS Research Program^5^.

The U.S. has taken on the ATLAS control/framework and data management as
two of its major deliverables. The control/framework is the main
steering code that configures the order in which algorithms run, and is
tightly coupled with the actual technique of analysis as it is carried
out by physicists. The ATLAS control/framework is being jointly
developed with the LHCb experiment, and is being further explored by the
LHC Computing Grid Project and by the CMS Collaboration as part of the
overall suite of tools to be used for LHC computing. We expect this
commonality among the LHC experiments to extend to the development and
use of common grid tools, driven by the needs of the experiments. We
propose to fund a total of three FTE's for this share of the
control/framework and analysis effort in the U.S. Costing for software
professionals in the core categories is estimated at 170k\$/year/FTE,
based on collaboration averages.

Data management is also an area where the U.S. ATLAS program has a
significant leadership role. The current baseline for all LHC
experiments is to employ a hybrid ROOT-SQL data management solution
originally proposed by the U.S. ATLAS participants. A significant
fraction of expertise in this common effort comes from U.S. ATLAS
software experts and represents a leadership role which can be
capitalized on given adequate funding. We propose to fund two FTE's for
the data management effort.

***Detector software*:** Software specific to a given detector
subsystem, including simulation and reconstruction. The U.S. is playing
a major role in the construction of all detector subsystems for ATLAS.
We are in a unique position to capitalize on this investment by funding
software support for physicists in specific areas. A faithful detector
simulation is critical to comparing physics results against event
processes. The complexity of the ATLAS detector in particular is a key
aspect to the simulation. Coupled with this is a faithful, yet efficient
geometric description. Both of these areas are places where the U.S.
ATLAS collaboration has a substantial amount of expertise. Successful
exploitation of these requires adequate software support. We are
proposing a total of six FTE's for detector software - three for
simulation, two to support the geometrical description of the detector
and one in event visualization.

In the domain of detector simulation, the U.S. is deeply involved in
many efforts. One major focus here is in the modeling and reconstruction
of electrons and muons, which are key signatures for several important
physics processes, such as Higgs production, supersymmetric particles,
and top quark production, among others. Principally, this simulation
effort is directed at incorporating and tuning the GEANT-4 simulation
packages to accurately replicate the response of the detector to real
particles. In part this requires a comparison of test beam data with
GEANT predictions and subsequent tuning. Also, one major activity
involves an evaluation of the response of the detector via GEANT to form
fast simulations (e.g. via G-flash). Here we are proposing three
computing support people to work with U.S. ATLAS physicists, one
associated with the liquid argon calorimeter, one with the muon
subsystem and one with the transition radiation tracker. These detector
subsystems have very complex geometries, with many elements, and the
management of the associated software is a considerable challenge. FTE's
in the category of detector subsystems are costed at 170k\$/year/FTE,
based on averages throughout the collaboration.

The description of the detector geometry is also an area of U.S.
expertise. We have taken the CDF geometrical description format and
applied it to ATLAS. It has now become the baseline description of
ATLAS, and, again, the ability of U.S. physicists to track changes to
geometry and feed it back into physics analyses is important to our
ability to play a leadership role. Geometric descriptions tie into event
visualization, and this is also an area that aids tremendously in the
analysis of data. We are proposing to fund two FTE's in the area of
detector geometrical description and one in event visualization

***Infrastructure support:*** Work packages developed by U.S.
researchers have to be supported, documented and made available to the
international collaboration. U.S. physicists working at CERN need a
substantial amount of support in order to be competitive with other
collaborators. We are proposing two infrastructure support personnel
dedicated to U.S. physicists who would be resident at CERN. These are
estimated at 180k\$/year/FTE, based on the typical cost of supporting
computing professionals at CERN. These infrastructure support personnel
guarantee the effectiveness of U.S. efforts, including the integration
of U.S. software into general ATLAS software. Additional support roles
include help-desks, librarian and documentation functions, among others.

***Grid integrated software*:** Typically the computer scientists in
collaboration with the LHC experiments provide extensive middleware
tools for grid applications. However, the integration of grid middleware
with both core and detector specific software is the responsibility of
the experiments. The funding for this is the subject of other proposals,
notably ITR opportunities, in addition to the existing iVDGL, PPDG and
GriPhyN consortia.

![](media/image22.wmf)

Table 3. *Estimated Software costs in FY02k\$*.

In all, this proposal represents a significant step in the direction of
empowering U.S. physicists to position themselves in a leadership role,
assuring the U.S. collaboration of the ability to engage in successful
forefront data analyses. The funding requested will enable us to obtain
the leadership position we are seeking.

7.  **Maintenance and Operations**

The NSF-supported participation of U.S. scientists in the construction
of ATLAS was structured with two goals in mind: to make the
contributions to the construction of the experiment as effective as
possible within the resources available, and to use these resources to
strengthen the scientific capabilities in the U.S. universities. These
goals are compatible because there is in the universities a large
intellectual capital developed in past projects to construct major
physics experiments, which can be tapped by adding resources to build up
teams to produce new detectors for ATLAS. This enables us to make
contributions that are extremely valuable to the international ATLAS
Collaboration, more valuable than might be expected by looking at the
level of funding alone. This has been a great opportunity for the
universities, since the scientific infrastructure in the universities
has not kept pace with the technological opportunities over the last
decades. The ATLAS Construction project is far enough along that we can
now report that the universities have risen successfully to the
challenge of some really substantial construction projects that many
observers might have believed needed the scale of a national laboratory
to complete.

Examples, among the thirty university construction efforts, are the
collaboration of seven universities to build hundreds of square meters
of Monitored Drift Tube tracking detectors for muons with a construction
and alignment accuracy of about twenty-five micrometers. This effort
included the electronic readout and laser alignment systems, and
included no national laboratories. Another group consisted of four
universities with no national lab participation, building a system
combining tracking and transition radiation detection, with hundreds of
thousands of small "straws" for tracking, together with very advanced
radiation-hard electronics. The net effect has been to make an
investment on university campuses that has raised the level of the whole
physics enterprise in a qualitative manner. Quantitatively, we note that
twenty U.S. ATLAS universities are involved in very advanced electronic
projects, including the most advanced sub-micron technologies and the
most advanced radiation-hard processes. Twenty-one of the universities
are carrying out mechanical construction using advanced techniques,
often involving large scale construction typically using undergraduate
students as part of the construction and testing teams. These efforts
have been used very successfully as an outreach tool with high school
students and teachers. One particularly notable example of the effect of
investing in university infrastructure is reflected in the recent naming
of Hampton University as a Physics Frontier Center (PFC). The Hampton
group, funded by the NSF, is playing a key role in producing components
for the ATLAS Transition Radiation Tracker, operating a dedicated clean
room facility on the campus, a new facility provided by Hampton
University. This work, on the energy frontier of fundamental particle
physics, is a centerpiece of the Hampton PFC program and will attract
new students, postdocs and visiting professors to the university. The
expertise and infrastructure that have been accumulated in the U.S.
ATLAS institutions during detector construction will be essential as the
experiment is installed and commissioned, and begins operation.

The U.S. commitments to the ATLAS Construction project are specified in
terms of well-defined deliverables to the experiment^6^. The
construction of U.S. ATLAS Deliverables is proceeding well, and is
expected to meet the LHC and ATLAS schedules and need-by dates. The U.S.
groups will have continuing responsibilities for the detector components
they have built as the detector is installed in the experimental cavern,
and during the commissioning and operation phases. Three categories of
Maintenance and Operation (M&O) activities are included in this
proposal - pre-operations, operations and maintenance:

-   Pre-operations: Includes final testing (and associated material,
    electrical, gas, water supplies etc.), calibration, and integration
    with other detector components or subsystems.

-   Operations: Includes activities during experimental run periods, for
    example calibration and monitoring, or implementing improvements to
    software code, etc.

-   Maintenance: Includes activities both during run periods and
    shutdowns, for example costs of spare parts, routine replacement of
    parts, additional tooling/infrastructure required for accessing and
    repairing detector components, etc. In some instances, particularly
    for the various radiation-hard ICs needed in ATLAS, spares must be
    bought early as a "lifetime buy", since it is known that the
    relevant technologies will become obsolete.

In addition to the M&O costs for specific U.S. deliverables, there are
Common Costs for each detector subsystem, shared in proportion to the
respective contributions to the Construction project. Such Common Costs
might include the costs of contract labor supplied by CERN (e.g. for
cabling, plumbing, on-site machining, etc.), consumables (gases, coolant
fluids, etc.), and general support of surface assembly equipment and
infrastructure. We propose to fund one third of the total U.S.
commitment to the Common Costs via this proposal.

The M&O tasks for each subsystem are outlined below.

1.  **Semiconductor Tracker**

The semiconductor tracker (SCT) construction activities in the U.S.
should end in the early part of the 2004 fiscal year. Continued system
engineering will be needed during pre-operations after modules are
mounted on support structures, during surface testing at CERN, and
during first data collection. The SCT system has approximately 60 square
meters of detectors, very long cable runs between several patch panels,
and a complex cooling and shielding arrangement. These make for
difficult system issues requiring great attention to detail and much
testing. Of paramount importance will be keeping the noise of each
channel of the strip detector in the full system to a value close to
that for an isolated detector. Our technical responsibilities will
involve diagnosing system problems and working on system optimization.
In addition we will have responsibility for repairs of modules made in
the U.S., should any develop problems during surface testing.

The SCT will first be assembled on the surface in a building being
specially equipped for pre-operations and long-term maintenance work.
This will require ventilated clean space and assembly of readout to
perform all checks. The work on this building and its infrastructure has
already started and will proceed in parallel with detector construction.
The full set of detectors will then be assembled to make sure that they
fit and function together mechanically and electrically. This will be
the major activity for FY05 and FY06 as the SCT, assembled from fully
integrated and tested individual barrel and disk sections, is brought
into full operation in the surface building. Along with the other
tracking detectors, the SCT will subsequently be installed in the
experiment cavern, along with the beam-pipe. Many new connections will
have to be made and proper functioning re-established.

UC Santa Cruz will play a key role in the maintenance and operation of
the SCT. Their responsibilities during the research phase of the
experiment flow out of the responsibilities carried by the group during
construction, namely electronics coordination and systems engineering
for the SCT, and subsystem management for the silicon subsystem. The
group anticipates providing a scientific team of about six physicists
(including graduate students) with major involvement in pre-operations
and operations. They will work with typically two technical staff whose
funding is requested in this proposal. The scientific team will
participate in general SCT pre-operations activities, while the
technical staff will have more focused responsibilities.

At each step of pre-operations and operations (in the upstairs service
building, in the cavern, and during initial data taking) the UC Santa
Cruz technical staff will take responsibility for electronic systems
engineering and the functioning of the electronics as a whole as well as
repair or replacement as needed for individual detector parts. We
anticipate that the initial work, (FY05 and FY06) will require
significant participation of electrical engineers with replacement by
technicians as tasks become more routine and problems are solved. We
anticipate that the initial year of data taking will require the largest
manpower commitment (shown in the accompanying table).

Over the longer term running of the detector, that is after data
collection begins, two other U.S. universities will have maintenance
responsibilities within the silicon systems of ATLAS, in this case the
pixel system. The Ohio State University group has developed the optical
readout for pixels and will be responsible for monitoring, annealing,
and calibration of the optical links as they degrade. The Oklahoma group
has developed the flex hybrids used to collect information from the
pixel chips by the controller chip. They will be responsible for
maintenance and monitoring of this part of the pixel readout chain, in
collaboration with the groups that developed the individual chips.

![](media/image23.wmf)

Table 4. *Estimated Silicon M&O costs in FY02k\$*. *Total Costs include
labor and materials.*

2.  **Transition Radiation Tracker**

The activities covered by M&O for the transition radiation tracker, TRT,
span the period 2002 to 2008. These involve TRT barrel module checkout
after shipping to CERN, electronics PC board checkout prior to
installation, and system checks on the TRT modules and electronics after
assembly into the space frame. It also covers the period when the SCT
and TRT will be tested together prior to insertion into the detector in
the cavern, and checkout and maintenance between that time and beam turn
on. The printed circuit board checkout, connection and large-scale
validation for the endcap TRT modules will also be a responsibility of
the U.S. TRT group.

# *Checkout of modules:* The TRT modules will be completed at Duke University, Hampton University and Indiana University. The modules are gain scanned and then sent to CERN. It is expected that the first completed modules will arrive at CERN in late 2002. After arrival they must be checked for damage in shipping, and maintained and prepared for installation into the inner detector (ID) space-frame structure. This work includes measuring the wire tension, measuring the gas tightness, and re-measuring the gain. Staff physicists from Indiana University and Duke University will supervise this work. Two technicians from the U.S. will be on site to assist with checking and maintenance operations. One electronic technician from University of Pennsylvania will work on the electronics checkout during this period.

# *Six module checkout:* The mechanical structure that supports the TRT and SCT barrel is called the space-frame. When it is completed in December 2003, the installation of TRT modules will begin. The pre-operations activities include monitoring and testing of the service connections to a group of six modules. This includes the cooling liquids, the ionization gas, and the CO~2~ flushing gas. The HV operation, the low voltage connections, and the data links will be tested and maintained at this point. This is the most time critical, and labor-intensive aspect of the TRT assembly. It will require two mechanical technicians, an electronics technician, and a computer person. This work will be supervised by the project engineer and an electrical engineer.

The checkout procedure occurs after the installation, when groups of six
modules are in place space-frame. At that point the entire set of six
modules is checked for cooling operation, active gas function, and the
integrity of the flushing CO~2~ lines. While all of the electronics
elements will have been fully tested prior to installation, this is the
first time that a significant number of modules will be assembled, and
the first time they will be mounted in the support structure, and we
expect that considerable effort will be required to determine the
optimal connection for HV operation, electronics readout, and noise and
grounding. During this same period, electronics will be being installed
on the endcap wheels. We expect that after all the electronics is
installed on a substantial portion of a wheel (typically 1/2 - 1 wheel,
or 5000 - 10,000 channels), we will cable and operate the electronics.
Here also, especially during the early phases, there will no doubt be
problems encountered related to the occasional poor connections, broken
pins, noise, etc.

*TRT ID checkout and maintenance:* Once the barrel TRT is installed and
completely hooked up to services, the ID is turned over to the SCT for
installation of the barrel silicon detector. During this time the barrel
modules will be monitored and continually checked. Once the SCT is
installed there will be alignment checks with an X-ray source to
establish the orientation of both systems. This will serve as a shakeout
of the combined system and allow noise sources and DAQ issues to be
jointly confronted by the ID community. As an additional test we plan to
move at least one endcap wheel in close proximity to the barrel detector
(likely both TRT and silicon) and operate both systems. Again, while
careful attention is being paid to the isolation of these different
systems, it is likely that some problems will be encountered and will
have to be solved. At this point the principal work will be done by
physicists with assistance from one electronic technician and one
professional programmer. Mechanical issues can be handled by short-term
technicians.

# *ID barrel installation and services checkout:* The initial move of the ID barrel to the detector in the interaction area will occur in August 2005. Service connections to cooling, active gas, flushing gas, will require careful coordination and work by both CERN professionals as well as physicists and project technicians. After installation of the detectors underground, testing and operation will occur for the first time in situ. This will bring in a whole new layer of electronics, in particular the LV and HV power distribution, and the patch panels which include receiving and regenerating the Clock, Command Data for initializing the system, and data transmitted from the detector to the ROD\'s. Furthermore, it will of course be the first time the detector is operated in the presence of all of the other ATLAS systems. There will no doubt be additional sources of noise that will have to be understood and eliminated. We anticipate that this will require two mechanical technicians for about two months, and an electronics technician for several months. DAQ checkout will require a full time programmer for this period. After the barrel is checked out, the endcaps are installed and completely cut off access. During this time, however, the maintenance of the TRT barrel must continue, until the detector is ready for close up in November 2006.

# *Preparation for data taking and running:* This period will be the final checkout of the barrel operations with cosmic rays, and running the system with all of the ATLAS components. During this period the data acquisition for the TRT will be evolving from that required for large stand-alone system tests to a system which is fully integrated with the ATLAS DAQ software and which is capable of handling the initialization, testing, and readout of the full detector. We have estimated that 0.5 FTE of a computer professional will be required for this DAQ evolution. There will also be maintenance and final checkout of all services, which will require a part time technician in addition to the growing group of physicists that will be running the operations during data taking. By December 2006, we expect that the TRT will be fully operational. However, there will of course be a significant learning curve in terms of how to operate it most efficiently: optimal settings of thresholds, how to initialize the detector quickly and efficiently, need for periodic resets to maintain synchronization, etc. Once data taking begins, most of this work will be done by postdoctoral physicists and graduate students, but we will require some continual services support from mechanical and electrical technicians which we estimate at 1.0 FTE each and 0.2 FTE of a computer professional.

![](media/image24.wmf)

Table 5. *Estimated TRT M&O costs in FY02k\$*. *Total costs include
labor and materials.*

3.  **Liquid Argon Calorimeter**

The U.S. involvement in the Liquid Argon (LAr) Calorimeter represents
the largest fraction of the total U.S. collaboration financial
contribution to ATLAS. It is also a very large intellectual and
technical effort. Because of its crucial role in the experiment, the
calorimeter must perform well and must reach the designed energy
resolution. To ensure this, the U.S. collaborating institutions are
involved in all aspects of the calorimeter design and construction,
ranging from the construction of the cryostat housing for the detector,
through all aspects of cryogenics, cables and module construction, to
the detailed work on readout electronics and interface to Level 1
trigger. The intellectual involvement is maintained even in the areas
with little or no U.S. financial responsibilities. After completion of
the construction of subsystem components, U.S. physicists and engineers
plan to maintain their leading roles in all aspects of the Research
Program including subsystem operations, data collection, and data
analysis.

The logistics of the ATLAS assembly requires the Liquid Argon
Calorimeter to be completed early. The central barrel cryostat is
completed and operational and the installation of the 32 modules of the
barrel detector will start in FY03. The installation of the two endcap
detectors into the corresponding cryostats is planned for FY04. All
three sections of the calorimeter -- barrel and two endcaps - will be
completely installed in the cryostats at CERN in the surface building
180. Their operations will be tested with temporary cryogenic, full HV
and partial electronic readout for several months before the transport
to the experimental hall at the LHC. Since the electronics and the data
acquisition system will not be finalized at that time, special
development of cabling and of the readout system will be needed to check
the detector operations. Transport and lowering of each cryostat will
require removal of the external cryogenic lines and partial disassembly
of pedestals and baseplanes. After each cryostat is lowered and moved
into the experimental hall, the final cryogenics system must be
installed. Then crates with Front End Boards and all associated
electronics, cooling systems, fibers, cables and power supplies will be
installed on each cryostat. After the crate installation, the main work
of the pre-operation of the electronics will start in 2005 and continue
until the start of the LHC operations in 2007. Procedures are being
planned which will permit testing and pre-operation of all of the boards
in the front-end crate using the data acquisition facilities internal to
the liquid argon system. All tests of functionality, including tests of
the Level 1 trigger interface electronics will be carried out as part of
the commissioning of the front-end crates. This procedure will be
followed for each of the calorimeters, either in parallel or in
sequence, depending on the state of readiness of the hardware and the
available manpower. The main data acquisition system will be tested in
parallel to these activities, and once both systems are sufficiently
mature, readout through the main DAQ will begin, permitting tests of the
entire liquid argon system to commence.

This proposal describes the work of four U.S. institutions: Columbia,
Stony Brook, Pittsburgh and SMU. The dominant effort in these
institutions is related to the problems of readout electronics. In
addition, Stony Brook is also responsible for the HV feedthroughs and
cables. The schedule described in the previous paragraph indicates a
substantial overlap in time of installation and pre-operation of
electronics components and services. This is largely due to the fact
that the assembly and transport to the cavern of the three cryostats
will span about a year. The manpower needed for the completion of the
production tests, installation and for pre-operations largely consists
of the same people. Since a large fraction of the production and testing
occurs in U.S. and the installation and pre-operation occurs at CERN,
frequent travel and a substantial total travel cost is expected. After
completion of tasks related to the production and installation of
components, the technical staff will become available for commissioning
and pre-operation. The expected schedule for such transitions is
reflected in the profile of the proposed Research Program.

In addition to the engineering and technical staff, software specialists
will be needed during the pre-operation stage and during operations of
the experiment. Software professionals will be responsible for the
creation and maintenance of the monitoring programs for the status and
performance of the hardware and for creation and maintenance of the
databases and the corresponding graphic displays.

![](media/image25.wmf)

Table 6. *Estimated Liquid Argon M&O costs in FY02k\$*. *Total costs
include labor and materials.*

4.  **Scintillating Tile Hadronic Calorimeter**

The Tile Calorimeter, along with the Liquid Argon Calorimeter and
cryostats, is the first detector component to be installed in ATLAS,
following the initial installation of lower parts of the toroid magnet
system. By the time of installation, the calorimeter must have been
fully pre-assembled above ground, calibrated with radioactive source
(all modules) and test beam (1/8 of modules), and fitted with fully
tested and validated readout electronics. The pre-assembly process has
begun already in FY 2002 and will continue into 2004. Final assembly in
the ATLAS underground cavern will take place during the next two years,
finishing early in calendar 2005. There will then be extensive
calibration and checkout with cosmic rays and the three calibration
systems with maintenance of electronic systems and completion of
software required for monitoring the operation during data taking, which
begins in 2007.

Responsibilities of U.S. groups during this period will be connected
with U.S. responsibilities during construction of the calorimeter
components. Work on calibration and pre-assembly tasks will be conducted
by joint Tile Calorimeter working groups, mirroring the way that work on
submodules, modules, and instrumentation was carried out at several
sites worldwide. Where the U.S. had an exclusive role during
construction, as in the front-end electronics design and fabrication, we
will continue to take primary responsibility for validation and
maintenance. Responsibility for photomultiplier tubes (PMTs) is somewhat
intermediate, with the U.S. having responsibility for roughly one-third
of the total number.

The work to be done is described in more detail below with four time
periods identified: Pre-assembly and Calibration; Installation;
Pre-Operations; and Maintenance and Operations.

#### Pre-assembly and Calibration: During this period each of the three Tile Calorimeter barrel sections (Barrel and two Extended Barrel), consisting of 64 modules each, will be fully assembled above ground in Building 185 at CERN. Prior to the pre-assembly, extensive structural calculations are being carried out to determine stresses, deflections, and stability at key points in the assembly process when different numbers of modules are installed. During the pre-assembly process, deflections will be measured and compared in detail with calculations. In addition, the shims that determine relative positions of modules will be sized and preserved for the final installation. Following the pre-assembly of each barrel section, the section will be partially or fully disassembled in preparation for final installation underground. Physicists, engineers, and technicians from the U.S. will be needed for this process.

An extensive program of module calibration will be carried out during
this period in parallel with pre-assembly. Each module will be fully
calibrated with the Cs^137^ source system which illuminates and tests
each scintillator tile and its two readout fibers. In addition, eight
modules from each barrel section will be calibrated with electrons,
pions, and muons in a test beam. Readout electronics and PMTs will be
given final pre-installation tests also. Each of these checkout and
repair tasks requires a modest test bench setup at CERN. Physicists and
technicians are needed for these calibration tasks.

#### Installation: Although the actual process of installation is included in the ATLAS construction project, the name defines the period of work. Once modules are installed and connected to services, there will need to be an additional set of module calibrations and tests of electronics and PMTs. The test beam calibration program will continue in this period. Physicists and technicians will be needed for these tasks.

##### Pre-Operations: Once the Tile Calorimeter is fully installed, while later components of ATLAS are still being installed, there will be periods when it is possible to operate the calorimeter in a way that tests and increasingly approaches the final operational configuration. Data from cosmic rays will be recorded from the calorimeters and used to exercise reconstruction and monitoring programs. Both monitoring software and reconstruction software will be developed and refined during this testing and pre-operating period. Spare modules will continue to be tested in the test beam if CERN runs the test beams in that period. Personnel needed will include physicists, technicians (especially electronic), and software developers.

#### Maintenance and Operations: Once data taking begins in 2007, there will be a continuing process of calibration and monitoring of the calorimeter. Electronics and PMTs will be accessible roughly once each year and some effort is anticipated for repairs. Gaining access to the detector during the shutdowns requires support by engineers and technicians for operation of the large moving systems. Work will vary through the year according to the data taking and shutdown schedule. At different times, support by physicists, engineers, technicians, and software professionals will be required. 

![](media/image26.wmf)

Table 7. *Estimated Tile Cal M&O costs in FY02k\$*. *Total costs include
labor and materials.*

5.  **Muon Spectrometer**

Transforming ATLAS endcap muon chamber and alignment components built in
the U.S. into a fully integrated operational muon system at CERN,
capable of data collection leading to forefront physics analyses, will
be a large task. All components must be shipped to CERN, pre-operated,
certified and integrated -- an effort strongly affected by the large
physical size of some of the components and by the large geographical
region over which this work must be conducted.

It is envisioned that the M&O effort will begin in FY03 and continue
through the analysis of the commissioning beam results at the end of
FY08. Covered in the plan will be the setup and operation of component
test stations that will be employed for final testing just prior to the
installation on the support structures through the final integration of
chambers and alignment with the trigger and off-chamber readout
electronics. The U.S. ATLAS Muon Team will work closely with their
non-U.S. counterparts by sharing personnel, equipment and space when and
wherever possible.

The M&O Plan is formulated to address the labor and materials needed to
ensure the full operation of the deliverables of the U.S. ATLAS muon
endcap task. The U.S. ATLAS Muon Group is responsible for 240 Monitored
Drift Tube (MDT) chambers and readout, deployed in the inner and middle
layers (Small and Big Wheels, respectively), 32 Cathode Strip Chambers
(CSCs) and readout deployed at high rapidity in the innermost muon
plane. Included in the U.S. Deliverables are components of the endcap
alignment system consisting of some 800 BCAMs (measures angles),
components for 80 alignment bars and proximity sensors that relate
chambers to endcap alignment grid established by the angle monitors and
alignment bars.

Each major equipment category requires a subtask in the M&O plan. During
the FY03 to FY08 time frame the plan concentrates on fully certifying
components to be ready for installation, maintaining and operating these
components on the support structure elements prior to lowering these
into the cavern, and final integration of the components into a fully
operational muon system.

Since there are many similar tasks in the overall M&O task the plan is
broken down into distinct phases -- each component following each phase.
These are described below:

*Phase I:* Work performed on chambers and alignment components at CERN
prior to installation on the support structure. Included in this phase
is the setting up of chamber and alignment test stations where finished
chambers and alignment components are certified to be operational before
installation on support structure. The main thrust of Phase I is the
complete certification of chambers and alignment components. During this
phase complete remediation of problems will be mandatory since this
period is the last efficient time where such action is possible.

*Phase II:* Pre-operation of chambers and alignment components on
support structures following their installation. Work covered is the
certification of operation of chamber HV, readout as well as
temperature, B-field and alignment sensors in their installed location.
The EI chambers (MDT and CSC) and global alignment components are
installed on fully assembled Small Wheel (SW) sectors, whereas the
middle layer MDT chambers and alignment components are installed on Big
Wheel (BW) sectors -- one for each set of five EMS and one for five EML
chambers. Both the installation of chambers and alignment components
will be performed on the SW and BW sectors above the ATLAS UX1 cavern.

*Phase III:* Testing and debugging of chambers and alignment in ATLAS
cavern. Phase III will be the most critical one for the global alignment
in that it is when the system becomes fully integrated and operational.
It is in Phase III when the polar alignment lines are established and
all the azimuthal lines for the Big Wheel become operative.

*Phase IV:* Pre-operation and operation of endcap system in LHC beam.
Effort will be concentrated on integrating the precision tracking and
global alignment with the muon trigger. Backgrounds will be studied,
resolutions measured and DAQ software certified under data collection
conditions.

Of the four phases, Phases I and II are the most critical for chamber
pre-operations since they will be conducted under conditions where
access and remediation of problems are possible. Phase III will be the
most critical one for the global alignment system since it is only then
that the fully integrated system (polar and azimuthal lines and chambers
connected via proximity monitors to alignment bars) is realized.
Remediation of problems encountered during this phase will be difficult
since work must be conducted in the cavern and under very limited
access.

Installation of chambers and alignment components is considered part of
ATLAS Construction project and is not covered under this plan.
Installation occurs between Phases I and II.

![](media/image27.wmf)

Table 8. *Estimated Muon M&O costs in FY02k\$*. *Total costs include
labor and materials.*

6.  **Trigger and Data Acquisition System**

The Trigger/DAQ (TDAQ) philosophy of ATLAS is characterized by the
construction of Regions-of-Interest (RoI), selective data collection,
and sequential selection of event candidates. A Region-of-Interest
Builder (RoIB) will supply RoI information from Level 1 to a Level 2
Supervisor. This approach has been developed from a series of test-bed
measurements that have proved invaluable in evaluating candidate trigger
architectures. Major emphasis has been put on the development of
integrated prototypes that demonstrate functionality and performance
scaling. The scalability of the TDAQ system is particularly important
for staging considerations during the initial running of ATLAS. The
combination of RoIB and Supervisor is the principal hardware
responsibility of the U.S. TDAQ groups. These groups have thus become
lead institutions in the TDAQ test beam activity. Readout electronics
for interfacing S-link to gigabit Ethernet, and software to allow S-link
readout via a commodity network interface have been developed and used
in the test beam. A new trigger framework is being developed that will
be able to run in both online and offline environments.

*Pre-operations:* The pre-operations category encompasses test beam
activity that is essential for the successful integration of the entire
system from detector to data acquisition prior to TDAQ commissioning.
Since CERN does not plan to have test beams during 2005, no funds are
requested in this category for 2005.

*Operations:* The ATLAS High-Level Trigger (HLT) and Data Acquisition
(DAQ) System is a complex system composed of commodity computer and
networking equipment complemented by a small number of custom
electronics developments that include the U.S.-provided RoIB. The
scientific capabilities of ATLAS are directly dependent upon the
performance of this system and upon its reliable operation. The
commercial computing equipment employed by this system will be operated
at its performance limits. The performance of the system as a whole is
also dependent upon the performance of custom software developed by the
Collaboration.

Maintenance and operation of the system will require rolling replacement
of processors and network components. These costs are part of the M&O
Common Cost expenses, and thus will be covered by common fund
contributions. Labor costs for computer system management are also
covered in the Common Costs. Maintenance and operation will also require
ongoing software development in order to maintain the high performance
of the system in the face of required software changes. Changes to
software will be regularly demanded for a number of reasons. During the
early years of operation, the increase of luminosity with time and the
experience gained regarding experimental conditions will demand nearly
continual refinement of the HLT selection software. Increasing
luminosity will also increase the performance demands on the DAQ
software. HLT software must be changed to implement changes in the
physics priorities of ATLAS as time progresses. Changes in trigger
software will often demand complementary changes in DAQ software in
order to achieve optimal performance. In addition, version changes in
commercial software employed by the system, especially changes to
operating systems, will demand software changes. Because of the rolling
processor replacement and because of CERN computing support policies, it
will be imperative to keep pace with the evolution of operating systems.

Although most of the effort to adapt and change HLT and DAQ software
will be provided by physicists, the essential demands of extremely high
performance and reliability will require the expertise of software
professionals as well. Software professionals will generally write
highly specialized code, such as device drivers, that require
specialized software training and experience, as well as writing code
with the most stringent performance requirements. In addition, software
professionals will assist physicists writing code, by providing guidance
based upon their training and experience. Maintenance and operation will
also require system troubleshooting and repair during operations. This
task will also demand the specialized training and experience of
software professionals to complement the activities of physicists
operating and maintaining the system. Accordingly, the maintenance and
operation budget for the Trigger/DAQ Subsystem includes salaries for
software professionals plus associated costs for minor engineering
materials and for travel to CERN. In addition, equipment funds are
included for computer workstations and for small test systems for
testing newly developed code. In order to support hardware that is
solely a U.S. responsibility, the Region of Interest Builder (RoIB), the
costs of travel, test facilities and labor are included in the
operations costs.

![](media/image28.wmf)

Table 9. *Estimated Trigger/DAQ M&O costs in FY02k\$*. *Total costs
include labor and materials.*

8.  **Upgrade R&D**

The initial ATLAS detector for the start of LHC operation will be a
staged version of the planned full detector. While the staging plan has
been carefully optimized to minimize the impact on the initial physics
program, there will inevitably be some reduced capabilities. For
example, the discovery potential for a Higgs signal in several final
states will be degraded such that 20% more LHC running time will be
needed to compensate for this loss. This is acceptable for this high
priority initial goal, but the degradation for higher mass states like
those at the TeV scale suggested by the new theories of extra dimensions
will be more serious. This means that upgrades to the detector will be
highly productive investments. The upgrades will replace the
capabilities lost by staging, but the advance of technology will allow
more powerful and more cost effective detectors instead, in many cases,
of just replacing originally planned devices. This is much more
important taking into account that a vigorous program of machine
development is planned for the LHC, leading to a continuous increase of
luminosity over time. We note that the U.S. accelerator scientists who
have contributed to the construction plan to play an important part in
the Upgrade of the LHC to higher luminosity. This is good for the
scientific results, but places increasing stress on the detector
performance. The detector upgrades will allow ATLAS to exploit fully the
high luminosity running of the LHC, and will be chosen to profit from
the best new ideas and techniques, with examples given below. This will
require substantial R&D, which should start in FY06 in order to be able
to carry out the Upgrades in later years.

The full potential of the LHC will only be achieved in the high
luminosity running of the machine, which is expected to be reached
gradually starting in mid-2008. In order to cope with the expected
background conditions, the full robustness and redundancy of the ATLAS
detector will be required, according to its design criteria. This means
restoring both the pattern recognition capabilities and the resolutions
for the large variety of expected (and unexpected) signals.

The major upgrade R&D activities foreseen for which we are requesting
support in this proposal include the semiconductor tracker and the
liquid argon calorimeter subsystems, described below.

1.  **Semiconductor Tracker**

The tracking systems for ATLAS are expected to need upgrading after
initial running. This will be a result of radiation damage to detector
components as well as more stringent requirements arising from very high
luminosity as the luminosity is increased. An estimated date for an
upgrade might be approximately six years after start-up, allowing a few
years to reach design luminosity and a few years to collect data at this
luminosity. This should allow an evaluation of what new physics is
discovered and what the implications of these discoveries are for
upgraded tracking. Assuming three years to construct a new tracking
detector, preceded by three years of R&D on design questions, we arrive
at a start date of FY07 for intensive R&D.

The R&D area targeted in this proposal is the front-end for silicon
strips. The Santa Cruz group originated the present SCT design concept
(called binary readout) as one allowing adequate resolution with
significant simplicity in the readout chain following the front-end. The
implementation of this front-end is in a bi-CMOS technology which will
not be available a few years from now. We will need to move to a design
which is in whatever "deep-sub-micron" technology is used by industry
toward the end of this decade. A number of design issues will need to be
resolved, in particular the most appropriate low noise analog amplifier
design and how to deal with the low operating voltage requirements of
the new technologies, where a significant voltage drop occurs on the
services and the drop varies with chip activity. These issues will
require an integrated design approach that takes into consideration all
aspects of the readout system: ASIC technology, services and cooling. We
anticipate that the future technology will be sufficiently radiation
hard. This will, however, have to be tested, including issues of single
event upset. We anticipate a year of design work, which would be
followed by fabrication and very significant testing, with the test
results used to further optimize the design.

We anticipate that running at the LHC will teach us much about the
vulnerabilities of the front-end that cannot be learned in short beam
tests involving only small numbers of detectors. We expect to be able to
bring this experience to bear on the design of the next generation
front-end chip.

> ![](media/image29.wmf)

Table 10. *Estimated Silicon Upgrade R&D costs in FY02k\$*. *Total costs
include labor and materials.*

2.  **Liquid Argon Calorimeter**

The increase of the LHC luminosity will inevitably generate an increase
of the instantaneous and of the integrated radiation doses for various
components of the detector. This will not have an adverse effect on the
detector modules. However, the radiation doses received by the front-end
electronics will exceed the specifications required for the current LAr
readout. A completely new front-end electronics will have to be
designed. It must use different technologies than those used in the
present system and may include a radically different readout scheme.
Past experience indicates that an extended period the R&D program is
necessary for such a major enterprise. We propose a four-year R&D
program to develop and test new LAr readout schemes capable of surviving
high radiation doses. This R&D will require several steps starting with
a high-level conceptual and engineering design, followed by an extensive
prototyping, testing and verification program for the individual
components and finally the construction of a system prototype and the
corresponding system test in radiation. The beginning of this R&D
program will coincide with the start of the LHC operations in FY06. It
matches both manpower availability and the expected time scale of the
project.

![](media/image30.wmf)

Table 11. *Estimated Liquid Argon Upgrade R&D costs in FY02k\$*. *Total
costs include labor and materials.*

9.  **Education and Outreach**

Many of the NSF-supported institutions in ATLAS have been active for a
number of years in a wide range of Education and Outreach activities.
Since its launch in 1999, the NSF- and DOE-funded QuarkNet program has
been a natural focus for these diverse Outreach efforts.

QuarkNet provides community high school science teachers with the
opportunity to experience particle physics research at first hand.
QuarkNet teachers spend their summer assignments working alongside
physicists, technicians and students on tasks as varied as constructing
and testing components for future detectors (such as ATLAS) or analyzing
data from running experiments. At many institutions, the teachers also
profit from "crash courses" in a wide range of topics in modern physics.
One of the goals of the QuarkNet program is that the teachers return to
their classrooms and pass on some of the excitement of forefront
scientific research to their students, including aspects and examples of
modern physics in their curriculum where possible. Feedback from many of
the teachers in the program indicates that this has succeeded beyond
their most optimistic hopes, and has led to an appreciable number of
high school students getting interested, and participating, in the
research work at ATLAS QuarkNet sites around the country.

The first four years of the NSF-ATLAS groups' involvement in QuarkNet
have been a clear success, and we expect to build upon and extend our
activities as we enter the research phase of the ATLAS Experiment. One
example of a recent initiative to extend the reach of our Education and
Outreach activities was the inclusion of seven high school teachers from
sub-Saharan Africa in our 2001/2002 QuarkNet programs. Groups at
Columbia, Hampton University and the University of Pittsburgh invited
the teachers to join their summer workshops, adding a very stimulating
international dimension to the QuarkNet experience. This year, in
partnership with these teachers, we inaugurated a new follow-up program
in Africa, aimed initially at disadvantaged high school students in the
Gauteng and Western Cape provinces of South Africa. The project is based
on a mobile physics laboratory, accompanied by scientists, teachers and
university students, equipped to run a physics program and to provide
teacher support. The physics program includes visually exciting and
theatrical demonstrations, an "Exploratorium", and a variety of hands-on
laboratory experiments targeted at high school students. July and August
2002 saw the first visits of the mobile lab, to more than thirty schools
and communities in Gauteng and the Western Cape. Two teams from the U.S.
(one from Columbia and one from Pittsburgh), comprising particle
physicists, teachers and students, were joined by students and staff
from local universities and labs, and by local educators, in visits to
township and rural schools and community centers. The pilot project
appears to have been extremely successful and we are now developing
plans to strengthen and expand the program for the future, with the
primary goal of making it locally sustainable. It is our hope that this
program, and others like it, can play a part in bringing some of the
excitement of science and scientific research to all parts of the globe.

We place a strong emphasis on targeting diverse populations in our
Outreach efforts. We hope that this emphasis will be strengthened by
Hampton University's recent designation as a Physics Frontier Center
(PFC). The Hampton University PFC is named the Center for the study of
the Origin and Structure of Matter (COSM), and has as its physics focus
the energy frontier, with ATLAS, and the precision frontier, with
experiments at Jefferson Lab. COSM will host workshops in physics,
visiting professorships, summer programs for students and postdocs, high
school physics activities, and much more. A key component of COSM is the
linkage of Hampton to other Historically Black Colleges and Universities
(HBCUs) in the U.S. So far, three HBCUs are a part of COSM: Hampton,
Norfolk State, and North Carolina A&T State, Universities. The plan is
to increase this network to include several other HBCUs over the next
several years. The HBCU network will provide a pipeline for students to
come to the Hampton University Ph.D. program in particle physics from
the other, primarily undergraduate, institutions, and the new Ph.D.'s
from Hampton will then be attracted to these HBCUs as faculty members.
It is also expected that these new African-American Ph.D. recipients
will compete successfully for postdoctoral and faculty positions at
majority institutions. We anticipate that there will be a very fruitful
collaboration between the Hampton PFC, the associated expanding network
of HBCUs, and our new program of physics outreach in sub-Saharan Africa.

Other activities that have been supported by NSF funding of ATLAS to
date include the production of educational videos and CD-ROMs about the
experiment, assistance in setting up and constructing demonstration
experiments for high school classrooms, and lectures on topical
scientific subjects for the general public. We plan to continue and
expand such programs in the future.

Aside from our activities aimed at high school populations and the
public at large, essentially all of the NSF-ATLAS groups have
significant undergraduate and graduate student involvement in their
research and detector projects. Experience has shown that this is an
invaluable way of giving students some of the research experience and
technical training they will need in order to become our future
scientists.

10. **Schedule and Budget**

The LHC is scheduled to begin operation in the spring of 2007, with the
first physics run foreseen for the second half of 2007. The initial
ATLAS detector must be installed by late 2006 and U.S. scientists fully
prepared to reap the rewards of the many years of construction and
commissioning effort once the LHC starts colliding beams.

Construction of the U.S. Deliverables to ATLAS is well past the halfway
point, with many of the subsystems now delivering completed detectors
and detector components to CERN. Although the LHC schedule has slipped
somewhat, we intend to finish the U.S. ATLAS Construction Project as
planned by the end of 2005, thereby maximizing the construction of
deliverables and avoiding "standing army" costs. The good progress on
U.S. Deliverables is very gratifying, however once the detector modules
arrive at CERN, we must begin operating and maintaining them as soon as
possible after delivery. For those detectors which must be installed
first into the experimental cavern, notably the calorimeters, this leads
to significant M&O responsibilities already in the early years of this
proposal. Similarly for those subsystems, such as the muons, in which
module construction spans many years, the modules that are completed and
delivered early in the production schedule must be maintained until the
start of LHC operations.

For the software and computing, the schedule is largely determined by
the high-level ATLAS computing milestones^5^, in particular the Data
Challenges, and by the important goal of having the network of U.S. Tier
2 centers fully operational by the start of LHC data taking. We are
behind the schedule we evaluated as necessary several years ago, due to
a much slower beginning of the work due to funding stringency. In order
to be fully prepared for the start of data taking, and for the U.S. to
be positioned to take a leadership role in the physics analyses, we now
need to recover this lost ground. The funding profile proposed here,
with a significant ramp-up of effort in FY03, is intended to allow that.

The total cost objective of this proposal is 60.0M\$ (AY\$), for the
six-year period FY03-FY08. A summary of costs at WBS Level 2 is given in
Table 12, with a breakdown at WBS Level 1 shown in Figure 2. Cost data
are derived from fully resource-loaded schedules for both Maintenance
and Operations and Software and Computing. These have been the subject
of NSF and DOE joint agency Reviews, most recently in April 2002.

Further details on cost estimates are given in the Budget Justification.

![](media/image31.wmf)

Figure 2. *Estimated total Research Program costs (FY02k\$) for Software
and Computing, Maintenance and Operations, Upgrade R&D, Outreach, and
Project Management.*

![](media/image32.wmf)

Table 12. *Estimated total Research Program costs in FY02k\$, except for
the Grand Total which is escalated to AYk\$.*

11. **Project Management**

For the Construction phase of ATLAS, the NSF Division of Physics
delegated financial accountability to Columbia University, inclusive of
line management authority, responsibility and accountability for overall
operations implementation, and contract administration. We are proposing
that Columbia University continue to assume the financial responsibility
for NSF funding contributions to the Research Program of ATLAS, and that
Columbia continue to serve as the NSF-ATLAS Project Office, working
closely with the U.S. ATLAS Project Office at BNL.

Eighteen U.S. ATLAS institutions, including Columbia University, have
been supported to date by the NSF under Cooperative Agreement No. PHY
97-22537 for Construction of the ATLAS Detector. The eighteen groups
have coordinated their activities during the construction phase of the
experiment, with Columbia University assuming responsibility for the
work, subcontracting to the other universities.

Project Management for the U.S. ATLAS Construction program has been
provided by the U.S. ATLAS Project Manager (Willis), and Project Offices
at Columbia University (for the NSF) and Brookhaven National Laboratory
(for the DOE). Project oversight has been provided by BNL and by the
NSF/DOE Joint Oversight Group. Columbia has also administered and
assumed financial responsibility for some of the initial NSF funding for
the U.S. ATLAS Physics and Computing project.

The NSF-ATLAS Project Management role of Columbia University during the
Construction phase of the experiment has been functioning efficiently
and smoothly. Columbia is currently managing more than 25 subcontracts
for the Construction, Computing and Education and Outreach components of
the NSF-ATLAS program. The implementation of the U.S. ATLAS Construction
Project Management Plan and the U.S. ATLAS Computing Project Management
Plan included a number of features that were not very familiar to some
of our collaborating institutions. For example the operation was under
defined Memoranda of Understanding, rather than a Grant, with a
necessity of supplying detailed invoices for the subcontracts on a
timely basis. On the technical side, we require regular progress reports
for each activity in our Work Breakdown Structures with a firm deadline
each month. We plan on implementing a similar framework, including
well-defined Memoranda of Understanding, for the Research Program.

The management system now in place at Columbia for NSF-ATLAS maintains
close control over expenses and progress tracking, with a minimum burden
on the Subcontractors. We have found our organization to be flexible
while maintaining a tight managerial control. We expect that our model
and our experience will prove valuable for the research phase of the
ATLAS experiment. Further details of the proposed Columbia role, and the
overall management of the U.S. ATLAS Research Program, may be found in
the (Draft) U.S. ATLAS Research Operations Management Plan^7^.