Department of Energy/National Science Foundation

U.S. LHC Program Internal Reviews, August 15, 2006

URA Headquarters (1111 19^th^ St. N.W.)

(August 7, 2006)

**[2.0 Software & Computing (S&C)]{.underline}**

#### Management -- J. Shank

1.  **The implementation of the plan for support of user analysis must
    be adapted to the present U.S. ATLAS S&C budget guidelines.**

> The support for physics analysis in U.S. has three distinct
> components: a) the Analysis Forums, b) the Analysis Support Groups and
> c) the Analysis Support Centers. The Analysis Forums is a place to
> facilitate communication between U.S. physicists working on similar
> themes. It is convened by a U.S. physicist on base program funding.
> The Analysis Forums utilize the Analysis Support Group (ASG) to help
> with their analysis. The ASG consists of expertise in a broad range of
> software issues from core software to reconstruction to analysis
> tools. Some of the ASG members, who are well versed with the software
> issues and can provide technical assistance to physicists are funded
> through the Research Program funds. Finally, there are three Analysis
> Support Centers (ASC): BNL, ANL, LBNL that form focal points for U.S.
> physicists to meet and also get necessary training. We are planning
> support for one software FTE at each of the ASCs and one at CERN for
> assisting U.S. physicists based there.
>
> The funding for the ASC personnel must be accommodated in the present
> S&C guidelines. As the LHC turn on approaches and the core software
> effort matures, we must redirect effort in support of the Analysis
> Support Centers. We have developed a prioritized list and identified
> currently supported software effort that must be terminated in order
> to support the ASCs.

2.  **As experience accrues, the size and composition of the Resource
    Allocation Committee should be optimized.**

> We have not found that the current size (9 people) of the RAC is a
> problem. We have had too few meetings and the committee is still
> ramping up in activity, but it has helped shape the way we assign
> priorities. It has also made it clear that the grid middleware needed
> to implement the RAC policy needs more work. The current members of
> the RAC are:

-   K. De

-   R. Gardner

-   B. Gibbard

-   I. Hinchliffe

-   B. Mellado

-   S. Rajagopalan

-   J. Shank

-   S. Willocq

-   B. Zhou

> If further experience warrants a change in size or composition of the
> RAC, we will implement changes as appropriate.

3.  **The U.S. ATLAS collaboration should remain open to new
    collaborators; however, ATLAS should develop a strategy that does
    not compromise the success of the project due to insufficient
    computing resources.**

> We continue to have a policy of being open to new Institutes, but
> making sure that each new collaborator delivers real value to the
> ATLAS experiment overall and to US ATLAS. An estimate of the
> additional computing costs assuming a growth of 40 people is \$450k --
> this need has been presented to P5 and is incorporated in their
> planning.

4.  **Since the collaboration is likely to grow, U.S. ATLAS should
    continuously assess its physics priorities and the corresponding
    computing needs, to pursue other funding opportunities and to work
    with the funding agencies to obtain adequate resources.**

> We are always open to new funding sources and are exploring this, for
> example with the OSG. We also revise our total facilities needs and
> projections every 6 months (corresponding to the DOE/NSF reviews) to
> ensure that US physicists play a lead role in ATLAS analysis.

5.  **To minimize impact on overall U.S. LHC goals, and to address the
    questions posed by the LHCC, U.S. CMS should work with U.S. ATLAS to
    clarify the reasons for differences in their respective computing
    requirements. With U.S. CMS**

> We believe we understand the differences in the computing requirements
> as they appeared in the respective Computing TDR's for ATLAS and CMS.
> The new revision of the computing model requested by the LHCC after
> the new LHC schedule was released in July has common assumptions about
> the LHC machine running with all 4 LHC experiments. The process of
> revising the common assumptions will not be completed before Sept.
> 2006, but meetings with all LCG groups have already started to make
> sure all experiments and the LHC machine people are all on the same
> page. On the US specific side, US ATLAS is planning to put more than
> the LCG MOU pledged amount of Tier 1 CPU/disk capacity to enhance the
> ability of US physicists to be leaders in LHC analysis.

6.  **What are the U.S. ATLAS plans and expectations for Tier-3 centers?
    What are their roles? How will they be funded?**

> A task force was formed early this summer to write a white paper on
> the role of Tier3 centers in the US ATLAS Computing Mode. A draft of
> the white paper is attached. The T3 centers are funded from funds
> outside of the US ATLAS Research Program and can vary in size
> accordingly. It is clear that some local CPU/disk is needed for
> physics analysis and how these local resources (T3) interact and fit
> together with the T2/T1 infrastructure is outlined in the white paper.

7.  **Efforts should be undertaken to define the relationship and
    responsibilities of physicists at Tier-2 centers and those at the
    Analysis Support Centers.**

> The complete U.S. ATLAS Analysis Support Model was spelled out it a
> white paper released in 2005 and further explained in the DOE/NSF
> review of Feb. 2006. The Analysis Support infrastructure has three
> main components: Analysis Forums (AF), the Analysis Support Group(ASG)
> and the Analysis Support Centers(ASC). Some physicists at the Tier 2
> centers are part of the AF and ASG. The compute resources at the Tier
> 1 and Tier 2 centers are made available for physics analysis through
> the Resource Allocation Committee, which includes members from all
> T1/2 centers, the AF, ASG and the ASC. This makes a tight connection
> among physicists at the Tier 2 (and other sites) and the 3 bodies of
> the Analysis Support Model. The U.S. ATLAS Tier 2 selection process
> had as one of the selection criteria that the sites should have a
> tight connection to the ATLAS physics program.

8.  **ATLAS should continue to refine its computing model, and optimize
    its overall performance and resources.**

> The ATLAS computing model as specified in the Computing Technical
> Design Report was always recognized to be a moving target. The active
> ATLAS Computing Model working group
> (<http://indico.cern.ch/categoryDisplay.py?categId=156>) continually
> works to re-assess the operating parameters and required resources.
> This is now (in light of the new LHC schedule) being coordinated among
> the 4 LHC experiments to ensure they all make the same input
> assumptions.

**2.2 -- Software -- S. Rajagopalan**

1.  **Comment on the impact of the delay (perhaps it is only from June
    30, 2007 to August 31, 2007) expected in the start-up of 24x7
    operations**

> There is no significant impact in the software development milestones
> due to the announced delay in the LHC startup. While most of the
> significant U.S. software milestones have been met, lower prioritized
> milestones have not been met. The software is not yet in a robust
> state and rigorous testing of the software in recent commissioning and
> data challenge exercises have revealed weaknesses in the system. We
> expect some data from the LHC in 2007 (the same schedule as last year)
> and we need to have the software ready for this (low-E/low-L) data to
> shake out the problems and make a robust system capable of analysis of
> the full energy, higher luminosity data in 2008 and beyond. Our
> software milestones are essentially independent of machine luminosity
> as we assume we will always record data at the maximum trigger rate.
> We are not changing our software development schedule due to the
> announced LHC delay.

2.  **U.S. ATLAS should develop a means of tracking the progress and
    success of the model for support of analysis of data. How do
    preparations for analysis compare to those of other countries in the
    collaboration?**

> Activities at the Analysis Support Centers are organized to provide
> effective analysis support for US ATLAS physicists. This is achieved
> via physics workshops, software tutorials, analysis weeks
> ("jamborees") and one-on-one help. US ATLAS physicists who participate
> in these activities have been asked to fill out feedback
> questionnaires to track whether the support is effective and properly
> focused (partial results are in the table below). Activities of the
> Analysis Support Group have so far been at the level of one-on-one
> help but we are in the process of setting up two dedicated US ATLAS
> HyperNews forums, one for physics analysis support and the other for
> software/release support to better track questions from users and
> their resolution.
>
> Several other countries (e.g. Canada, France, Germany, Italy) within
> ATLAS hold major yearly national meetings to provide an overview of
> the different activities in each country. This is similar to the North
> American Physics Workshops that have been held for the past 3 years,
> the most recent held in Boston, July 26-28, each having about 100
> participants. In addition, these countries hold several software and
> analysis tutorials directed at physicists from their respective
> institutions, which are similar to the tutorials we hold in U.S. There
> generally are no formally organized physics groups within each
> country, as this is not encouraged by ATLAS Management. Overall, US
> ATLAS has been a lot more proactive in providing analysis support for
> its physicists. We have formed two physics forums -- one on Standard
> Model physics and the other on Beyond the Standard Model physics as a
> way of encouraging communication between physicists with common
> interests and as a way to provide across the board help. These forums
> also have held meetings discussing their interests and work. While
> these forums foster communication of people with common interests, it
> does not provide "coordination" as this aspect is managed centrally by
> the ATLAS physics groups. Hence the U.S. has been leading the
> collaboration in several of these support areas with the development
> of analysis software tutorials, analysis jamborees and physics
> workshops.

+--------------------------+-----+--------+--------+---------+-------+
| **BNL June 2006 Analysis |     |        |        |         |       |
| Jamboree Feedback**      |     |        |        |         |       |
+--------------------------+-----+--------+--------+---------+-------+
| Total number of feedback |     |        |        |         |       |
| forms received: 12       |     |        |        |         |       |
|                          |     |        |        |         |       |
| Total number of          |     |        |        |         |       |
| participants: 33         |     |        |        |         |       |
+--------------------------+-----+--------+--------+---------+-------+
| Rating                   | 1   | 2      | 3      | 4       | 5     |
| (1=bad,\...,5=excellent) |     |        |        |         |       |
+--------------------------+-----+--------+--------+---------+-------+
| Analysis software        |     |        |        | 3       | 6     |
| tutorials:               |     |        |        |         |       |
+--------------------------+-----+--------+--------+---------+-------+
| ASC infrastructure:      |     | 1      | 4      | 3       | 1     |
+--------------------------+-----+--------+--------+---------+-------+
| ASC help:                |     |        |        | 4       | 6     |
+--------------------------+-----+--------+--------+---------+-------+
| Analysis Jamboree        |     |        |        | 3       | 7     |
| format:                  |     |        |        |         |       |
+--------------------------+-----+--------+--------+---------+-------+
|                          |     |        |        |         |       |
+--------------------------+-----+--------+--------+---------+-------+
| Attend another jamboree  |     |        |        |         |       |
| at BNL?    Yes: 11   No: |     |        |        |         |       |
| 0                        |     |        |        |         |       |
+--------------------------+-----+--------+--------+---------+-------+

3.  **U.S. ATLAS should encourage international ATLAS to define a plan
    for the complete cosmic-ray run that would exercise the trigger,
    data acquisition, and software reconstruction chain.**

> There are several planned tests that address this, in addition to
> activities associated with the commissioning of the ATLAS detector
> itself. These tests are part of the so-called Computing System
> Commissioning (CSC) activity that is planned to bring the ATLAS
> computing system to full production capability in time for ATLAS
> turn-on. CSC tests cover:

a.  Full software chain and data streaming test

b.  Calibration and alignment

c.  Trigger chain

d.  Tier-0 scaling

e.  Distributed data management

f.  Distributed processing and re-processing

g.  Analysis Tools and Distributed analysis

h.  Full Dress Rehearsal

> The culmination of these tests, which are already underway, is the
> "Full Dress Rehearsal", scheduled for June-July 2007, where the goal
> is to inject simulated events at high bandwidth into the actual TDAQ
> hardware, pass them through the HLT, the transfer to Tier-0, Tier-0
> processing itself and thence distribute them to Tier-1 and Tier-2.
> Although it is likely that full bandwidth will not be achieved because
> of limitations in the rate with which events can be injected into the
> TDAQ, the goal is to achieve as close to full design bandwidth for
> approximately 7 days (\~10^7^ events), and to test not only the
> primary physics chain, but also the calibration/express processing
> chain. The detailed planning for this is still underway, but it is
> consistent with the ability to perform full system tests using cosmic
> rays and is expected to be coupled with these.

4.  **U.S. ATLAS should periodically revisit the matter of staffing of
    the analysis support group to determine whether it is working
    effectively, especially with regard to the use of software
    professionals.**

> The current Analysis Support Group relies on a number of software
> professionals who are engaged in software development. One aspect of
> this task is to provide documentation and user support for that
> software. This task therefore fits naturally as part of the activities
> of the analysis support group. It is our intention to revisit the
> effectiveness of this group at the end of the year 2006, based on the
> experience gained at that time.
>
> The current membership and other information on the Analysis Support
> Group can be found here:
>
> <http://www.usatlas.bnl.gov/twiki/bin/view/AtlasSoftware/AnalysisSupport>

5.  **Are U.S. Grid efforts progressing at a rate that can ensure
    adequate software infrastructure at LHC turn-on? Are there any
    common ATLAS/CMS plans to mitigate the risk of dependence on
    external resources (e.g., the OSG)?**

> U.S. Grid efforts on software that is critically important to U.S.
> ATLAS seem to be progressing at an adequate rate, assuming continuity
> in support and science-directed development, which we expect to come
> principally via the OSG (therefore we rely on adequate OSG funding).
> Distributed computing in U.S. ATLAS has by design as few middleware
> dependencies as possible, in order to minimize the risk to computing
> capability at turn-on. Our dependencies are on essential and (wherever
> possible) relatively mature components at the foundation of the grid
> infrastructure: GSI (grid security infrastructure for
> certificate-based authentication); SRM compliant storage services and
> associated transfer tools (srmcp); dCache; gLite's file transfer
> system (FTS); basic file transfer tools (which serve as a backup to
> FTS when and where it is unavailable); and Condor. On this foundation
> we have successfully built in-house systems for distributed data
> management (DQ2) and processing (PanDA) designed to meet the scaling
> requirements of ATLAS data taking. These in-house systems can
> selectively integrate and leverage higher-level middleware components
> as they are proven, but only the foundation is on our critical path.
> Examples of higher level components which we have already integrated
> or will explore -- but which are not on our critical path -- are
> Condor-G and other Condor components, SRM capabilities beyond those in
> our current SRM1 baseline, xrootd, VOMS based role management, and LFC
> (LHC file catalog).
>
> U.S. ATLAS and U.S. CMS are collaborating on meeting our essential
> grid infrastructure needs in the context of OSG. The new OSG
> Applications Area, jointly led by Wuertheim (U.S. CMS) and Wenaus
> (U.S. ATLAS), will focus on ensuring that the critical foundation
> middleware is validated as meeting experiment requirements, and that
> higher level components from which we can both benefit will be
> developed in a science-driven way, often with direct input from
> experiment in-house efforts (e.g. Panda). The OSG Facilities Area has
> responsibility for validating, releasing and deploying the middleware
> stack defined by the requirements of LHC and other stakeholders. We do
> not view the OSG itself as an external dependency and risk factor, we
> view it as an essential mechanism, integral to the U.S. ATLAS program,
> for mitigating grid middleware risk today and ensuring middleware
> development is on a science driven trajectory for the future.

6.  **U.S. ATLAS should articulate a plan for verifying that the
    proposed solution for schema evolution works sufficiently well to
    meet the needs of the collaboration.**

> The intermediate state representation model ("transient/persistent
> separation") proposed as a means to support schema evolution has been
> extensively tested, and international ATLAS has adopted a plan wherein
> the simulation data model will have migrated entirely to such a model
> by Summer 2006, and the reconstruction data model will have migrated
> by Release 13 (Fall 2006). Many event data objects have already been
> migrated, and, in every case, this migration has led both to speedups
> in I/O time and to storage reduction. (The median reported I/O speedup
> has been 20-30%, with factors of 4 reported in some cases; average
> storage savings is nearly 50%.) Work to streamline the introduction of
> state representations for complex components of the ATLAS event data
> model is underway.

**2.3 -- Facilities -- B. Gibbard**

1.  **Comment on the impact of the delay (perhaps it is only from June
    30, 2007 to August 31, 2007) expected in the start-up of 24x7
    operations**

> Since the evolution of the BNL Tier 1 facility, in terms of capacity
> and operational status, is defined by a set of LCG and ATLAS agreed
> milestones such a delay will have little effect on the Tier 1. In
> terms of operations, the RHIC/US ATLAS facility at BNL already
> operates on a 24x7 basis in support of RHIC when it is running, making
> use of 16x7 operator coverage augmented by automated monitoring
> systems and an on-call person whom principal users can contact to
> initiate remedial action on a problem on a 24x7 basis. The intent is
> to strengthen the monitoring, both automated and human, of Grid
> related additional services which are critical to ATLAS so that this
> system of round the clock operations support will be fully effective
> for ATLAS by the beginning of 2007 independent of the detailed
> schedule for LHC turn-on. In terms of the procurement and installation
> of capacities, at both the T1 and T2 centers, such a delay might allow
> the selective adjustment of the timing of some procurements to exploit
> release schedules for new products to the extent that it does not
> compromise ATLAS computer system commissioning activities.

2.  **U.S. ATLAS should ensure that personnel at grid middleware
    projects become engaged in this year's service challenges, and
    especially in issues pertaining to scaling.**

> The primary middleware scaling issues which need to be addressed in
> the service challenges are those having to do with data storage and
> movement and workload management, especially the databases and
> communication protocols supporting these activities. To this end we
> are interacting with the primary developers of FTS, dCache, the dCache
> SRM, Condor, the 3D project, DQ2, and PanDA to assure that these
> products and services along with the capabilities of the Tier 1 and
> Tier 2 facilities are scaling appropriately with increased load.
> Problems identified to date have included ones for which primary
> developers are responsible and ones for which hardware capacity and
> details of deployment require improvement.

3.  **U.S. ATLAS should pursue acquisition of redundant network
    connectivity to the BNL Tier-1 center. With Scott Bradley and BNL
    upper management.**

> ESnet, as part of building its next generation network, will upgrade
> the Long Island MAN ring, by which BNL is WAN connected, to be fully
> redundant and diverse. Additional circuits will be added to give
> circuit redundancy as part of a first phase in 2007. An additional
> site on the MAN in NYC, additional equipment in NYC and additional
> equipment at BNL will be added as part of either this first phase or
> as part of the second phase in 2008, to produce full redundancy and
> diversity in connections to both the ESnet production IP network and
> the Science Data Network / USLHCnet for the BNL Tier 1. Thanks to
> Scott Bradley, Bill Johnston and Dan Hitchcock this is now resolved.

4.  **U.S. ATLAS should test the scaling properties of dCache-managed
    distributed disk systems.**

> Study of the scaling properties of the BNL Tier 1 dCache distributed
> disk installation is being done in the context of Service Challenges
> and production operations. These studies have included the development
> of simple tests stressing individual components of dCache and have
> resulted in significant modifications of the BNL dCache configuration.
> The primary source of scaling problems to date has been PNFS which
> supplies the name space service mapping between the physical space
> internal to dCache and the logical space exposed to users. Current
> activities and directions under consideration include upgrading the
> PNFS hardware, further distributing across separate servers individual
> PNFS functions, changing out the database underlying PNFS with a more
> reliable and performant Oracle or MySQL cluster based database, and
> investigating the new Chimera PNFS technology expected to be release
> by developers in the next six months. The current dCache instance at
> 150 TB's is now being used for SC4 exercises, production processing
> and some user based production and analysis. It seems to still have
> significant headroom in PNFS service. It will soon be expanded to
> \~500 TB's and usage is expected to increase. Study of the performance
> of this expanded system with improvements underway will indicate the
> extent to which partitioning of Tier 1 dCache will be necessary to
> achieve acceptable performance.

**[3.0 Maintenance & Operations (M&O)]{.underline}**

**3.1 -- Silicon -- A. Grillo**

5.  **Comment on the impact of the delay expected in the start-up of 7X7
    operations**

> The pixel project has suffered severe delays during integration, first
> with the corrosion of stave cooling pipes (FY05) and recently with
> defects in the Type 0 cables for the barrel, which turned out to be
> present from manufacturing. The recent Type 0 cable crisis cost the
> project roughly 4 months delay on an already critical schedule. The
> latest ATLAS schedule (produced after the delayed LHC schedule)
> affords the pixels 2 months extra time before installation. Therefore,
> one could say that the delay in the LHC schedule makes the
> installation of pixels in 2007 possible, although even with this the
> installation of the pixels in 2007 is very challenging. Nevertheless,
> installation in 2007 is still the target. The delayed LHC start also
> provides some additional needed testing time for pixels. The delay
> will require US personnel to be resident at CERN for more of FY07 than
> originally planned resulting in the forecasted FY07 budget request to
> increase from \$582k to \$1,064k.
>
> In the case of the SCT, the extra time is being put to good use in a
> more thorough check-out of cooling, grounding and shielding and in
> running of cosmic ray tests. The cosmic ray testing in the surface
> building has already allowed initial check out of track finding and
> alignment tools. We expect the continued testing once the SCT is
> installed in the ATLAS detector but before beam to give us a first
> test of the complete detector and readout system, including the full
> ROD system, and a good first pass at alignment parameters. The extra
> time before beam will be put to good use and hopefully will shorten
> the time for complete "shake-down" once the beam is available. We do
> not foresee the need for extra SCT funds at this time due to the delay
> in LHC startup.

6.  **U.S. ATLAS management is asked to re-assess the timescale and plan
    for the detectors to reach design-performance goals, and present
    this information at the next MEG meeting.**

> For Pixels every effort is being made to install the detector in 2007.
> This will allow initial operation with the rest of ATLAS and
> commissioning with initial LHC beam, which is expected to permit
> having a fully operational detector ready for the first physics
> collisions. A schedule review of the pixel integration was conducted
> at CERN on 18/July/2006 (the written result of the review is not yet
> available at the time of this writing). A significant preliminary
> result of this review was the identification of CERN technical staff
> to be made available to join the pixel project in order to help meet
> the 2007 installation date. Prior to installation inside ATLAS, the
> only operation of a sizable fraction of the pixel system with the full
> service chain will be the endcap C cosmic ray run in the surface
> assembly building, on track to begin taking cosmics in Sept. 2006.
> This activity will enable the installation of the full pixel detector
> with significant confidence in the operating parameters. The US pixel
> institutes have taken the leading role in this activity (LBNL, OU,
> OSU, and SLAC) including the entire mechanical assembly, cosmic
> trigger system, overall organization, simulation, and a large fraction
> of the software. With the cosmic ray endcap test and the installation
> of the full detector in 2007 the pixel system is expected to reach
> performance goals in a timely manner for the start of physics data
> taking at full beam energy. On the other hand, should installation be
> delayed until 2008 due to further schedule slippage in the integration
> of the detector, this would make it impossible to commission the pixel
> detector on day 1 of physics data taking (since this would be
> essentially the first time the pixel detector would be integrated with
> the rest of ATLAS). In this case (delay of installation of pixels
> until 2008), a significant new effort would be needed to operate the
> full pixel detector on the surface, in order to try to accelerate the
> commissioning.
>
> As noted above for SCT, the pre-commissioning phase is now in progress
> and will continue until beam arrives. The cosmic ray testing will
> allow many of the performance related parameters to be established,
> e.g. alignment, tracking efficiency, and noise. This will allow the
> SCT to reach design-performance goals quickly once colliding beams are
> achieved.

-   (also see Upgrade R&D below)

**3.2 -- Transition Radiation Tracker -- H. Ogren**

1.  **Comment on the impact of the delay expected in the start-up of 7X7
    operations**

Short term installation delays: ( May 19- August 29, 2006)

1.  We are trying to minimize the impact of the delay by shifting the
    work of Chuck Long at Hampton to the September period. However, the
    delay has had an effect on extending the stay of Jack Fowler through
    the end of the year in order to complete the installation and
    commissioning.

2.  The extended time has had an effect on the electronics support from
    the University of Pennsylvania. We expect that the full support crew
    will be needed through October for the barrel installation, and for
    help on the endcaps installation support will be needed through the
    spring, 2007. Work on commissioning the RODs and TRT detector for
    data taking will take all of 2007.

3.  Kirill Egorov will be a leader for the installation and available
    for full time operations when they begin, but there is no change in
    the budget line, since it was assumed that he would be full time
    during data taking, also.

4.  The TRT community is working on a plan for resources for
    commissioning and data taking. The majority of these people will be
    postdocs, and graduate students, and will be supplied by the
    institutes base funding.

Longer term Operations delays: June, 2007 to November, 2008 for beam
operations.

Our principal M&O resources cover Kirill Egorov for technical services
(1 FTE) , and an electronics technician and computer professional ( each
a 0.5 FTE) . These were scheduled to be in place for pre-operations and
data taking, so a short delay will not have much effect. However, the
day-to-day operation when shift work starts will be covered mainly by
graduate students and postdocs from the TRT university base budgets.
These people need to be in place early and trained, so the main impact
will be a delay in their data taking activities.

2.  **U.S. ATLAS management is asked to re-assess the timescale and plan
    for the detectors to reach design-performance goals, and present
    this information at the next MEG meeting.**

> See above.

**3.3 Liquid Argon Calorimeter -- R. Stroynowski**

1.  **Comment on the impact of the delay expected in the start-up of 7X7
    operations**

The delay appears to be substantially longer than two months since it
includes several months of "controlled access" preceding and during the
LHC machine commissioning. The effect of extended period of access to
the detector is an extension of the commissioning activities. In
particular, work has been scheduled to minimize the number of shorts,
i.e., the number of problem channels that are presumably due to
conductive impurities inside the cryostats. These repairs include, among
others, attempts at burning the impurities by high current discharges,
rapid emptying and slow refill of liquid Argon in the cryostat and
others. These activities are likely to lead to a better performance of
the detector. They are, however, manpower intensive and will require an
extended presence at CERN of a number of US based experts and will
result in higher M&O costs.

2.  **U.S. ATLAS management is asked to re-assess the timescale and plan
    for the detectors to reach design-performance goals, and present
    this information at the next MEG meeting.**

The Liquid Argon Calorimeter appears to be in good shape. The
commissioning of the electronics will be completed before the start of
the colliding beam period. The detector calibration uses either the
calibration system (for the electromagnetic subsystem) or test beam data
(for the Fcal). The verification of the calibration data will be done
with the data -- e.g., using Z→ e+e- decays. This will require about one
year's worth of the data sample and is unlikely to be finalized before
the end of 2008.

3.  **The U.S. LAr team should work with the Int'l ATLAS LAr Calorimeter
    group and ATLAS management to obtain a more detailed understanding
    of plans for the entire LAr subsystem (overall needs, available
    resources and a schedule for its deployment) for the next few years.
    This will make it possible to evaluate how the U.S. contribution
    fits into the Int\'l effort, and whether the additional U.S. (MR)
    funds for technical support are both necessary and sufficient to
    assure success of the LAr subsystem in ATLAS.**

The International ATLAS LAr Calorimeter Collaboration formed the
"Operations Model" group charged with an estimate of tasks and manpower
needed for the operation of the calorimeter and for its maintenance
activities. The initial estimate in form of an Excel spreadsheet and
accompanying note explaining the methodology has been distributed. This
estimate is now correlated by the ATLAS wide OTSMOU group charged with
optimizing the manpower and overall cost. The process is likely to
continue until the end of the year. Initial estimate for the LAr
manpower needs is for \~75 FTE/year for the first 2 years of operations
divided among various levels of expertise. This number may be reduced
when the optimization process will be complete. It will certainly be
reduced to \~60% for 2009-2010 run period. US is responsible for \~20%
of the overall operations and maintenance cost of the LAr Calorimeter.
For the present, not yet optimized estimate, the U.S. manpower
requirements would be \~15 FTE/year. The next step in the planning will
determine how many of these FTEs are Experts and how many are
Non-experts. Then the support of these people either from the Research
Program or the Core Research Program can be planned.

4.  **The U.S.  LAr team should expedite the rework of at least a few
    low-voltage power supplies, and begin acceptance testing to see
    whether their corrective measures were adequate.**

All recommendations from the December 2005 review of the low-voltage
power supplies have been implemented and the MDI Company has started to
implement changes. The initial tests were positive and about 50% of the
units have undergone repairs. With the exception of workmanship
failures, all units passed initial testing. One unit has survived 3
months of the operations. We still observe occasional failures for which
causes have not yet been understood. We plan to convene another review
in early fall to establish further courses of action.

5.  **MEG is interested in learning the results of the FY'05 LAr and
    Tile calorimeter beam tests. A summary statement and conclusions
    from these FY'05 activities was asked to be sent to the U.S. LHC
    Program Manager and the MEG prior to the April 11 U.S. LHC JOG
    meeting! Coordinate with L. Price**

The combined LAr+tile+ID+muon test beam collected data using final
electronics components for the Front- and Back-end and a close to final
DAQ system. All hardware and software components performed well. The
collected data are still being analyzed and the final results are
expected for end of 2006. Data analysis of the electromagnetic
calorimeter leads to improvements of the clustering algorithms and was
used to verify the geometrical description of the detector elements. All
results completed so far indicate electromagnetic calorimeter's
performance close to the design values.

Test beam data are also used to produce the calibration of the FCal.
Data analysis has not yet been finalized. The preliminary results
indicate excellent resolution for electrons:

$\text{σ/E} = a \oplus \frac{b}{\sqrt{E}} \oplus \frac{c}{E}$

with a=(3.76±0.06)%, b=(24.5±0.84) $\sqrt{\text{GeV}}$%, c=(145.5±1.6)
GeV (see next page)

The corresponding preliminary resolution for pions indicate a\<5% and a
slope of about 90%/$\sqrt{E}$.

![](media/image1.png){width="7.243055555555555in"
height="3.3069444444444445in"}

**3.4 -- Tile Calorimeter -- L. Price**

1.  **Comment on the impact of the delay (perhaps it is only from June
    30, 2007 to August 31, 2007) expected in the start-up of 24x7
    operations**

Detailed commissioning of the modules and sectors (groups of eight
modules) of the Tile Calorimeter is proceeding more slowly than planned
for a number of reasons. The additional time before intensive data
taking operations begin will be fully utilized in completing the
commissioning program. In addition, additional personnel will be
required to finish in the available time.

2.  **U.S. ATLAS management is asked to re-assess the timescale and plan
    for the detectors to reach design-performance goals, and present
    this information at the next MEG meeting**

Extensive non-collider calibrations will be done with a Cs source and
laser and charge injection systems before any collisions are observed.
These will be correlated with test beam data to arrive at a preliminary
energy calibration. The initial collisions in 2007 will largely be a
test of the data taking chain from the point of view of the detectors.
However, the low energy jets obtained will be used to study correlations
between energy deposited in Tile and LAr calorimeters. The first high
energy (and hopefully higher luminosity) data in 2008 will permit the
first serious iteration on the hadronic energy scale. At the same time,
there will be iteration and optimization of optimal filtering
coefficients used to transform the measured samples of calorimeter
pulses into energies. All of these calibrations and correlations between
the calorimeters will be studied and improved in subsequent runs. Basic
calorimeter resolutions are likely to be well understood for the 2009
run. Reliable energy scales combining dissimilar electromagnetic and
hadronic calorimeters have been notoriously difficult and may take until
2010 or beyond, depending on the performance of the LHC.

3.  **MEG is interested in learning the results of the FY'05 LAr and
    Tile calorimeter beam tests. A summary statement and conclusions
    from these FY'05 activities was asked to be sent to the U.S. LHC
    Program Manager and the MEG prior to the April 11 U.S. LHC JOG
    meeting! Coordinate with R. Stroynowski.**

A realistic slice of the ATLAS detector was operated in a combined test
beam in the latter part of FY 2004 and the beginning of FY 2005. Pixels,
SCT, LAr, TileCal, and several varieties of muon chamber were operated
with magnets to test and assess the combined performance of ATLAS. Work
with the calorimeters emphasized joint operation of the Liquid Argon
(electromagnetic) and Tile (hadronic) calorimeters, especially for pion
beams, as a function of η. Runs were taken in two energy series. The
first was from 50 to 350 GeV and from η of 0.2 to 1.2 and was aimed at
assessing the high energy performance. Events were separated into those
that deposited their energy fully in the Tile Calorimeter and those that
started showering in the LAr calorimeter. The former showed energy
resolutions of approximately 50%/sqrt(E) in accordance with prediction,
with only weak dependence (slightly improving) on increasing η. Combined
energy resolution was slightly worse, but not all effects of cryostat
energy loss, leakage, and non-compensation were accounted for. Since a
surprising fraction of jet energy (25% at 150 GeV) is contained in pions
below 10 GeV, special runs were taken with beams between 1 and 9 GeV.\
The fraction of beam energy visible in the calorimeter is observed to
rise rapidly with energy in this range in agreement with simulations.
The results of these combined runs are now being used to establish a
strategy for energy calibration of the calorimeters and for establishing
an absolute energy scale.

**3.5 -- Endcap Muon -- F. Taylor**

1.  **Comment on the impact of the delay expected in the start-up of 7X7
    operations**

> The V8.0 schedule has the beam pipe closing on 31-Aug-06. The effort
> will concentrate on installing both BW systems (4 wheels/side) and the
> two SWs until then. The '24x7' operations will start thereafter during
> the cosmic commissioning phase in order to be fully prepared for the
> first collisions by the end of CY07.

2.  **U.S. ATLAS management is asked to re-assess the timescale and plan
    for the detectors to reach design-performance goals, and present
    this information at the next MEG meeting.**

> The effort on reaching the design performance is proceeding on several
> fronts. First will be the commissioning of the detector and
> certification that all chamber and alignment elements are in their
> nominal locations and that all are working according to
> specifications. Data from cosmics and first collisions will be
> employed. The second phase will be to use calibration centers to
> determine and apply the various calibration constants -- such as the
> distance-time relation, temperature corrections, alignment
> corrections, etc. This phase will probably take 6 months after the
> first collisions are recorded. The third phase will be establishing
> steady state corrections with high statistics from data taken at 14
> TeV.
>
> This effort will probably last through CY08. Much of this schedule may
> be dependent on parameters unknown at this time -- such as what is the
> actual level of background, how close will the final chamber
> placements be to specifications, will the environmental conditions of
> the muon system be stable, etc.

3.  **MEG is concerned about the continued slippage of commissioning and
    installation milestones of the Endcap Muon (EM). For some tasks the
    slippage has been as much as twelve months. Any progress?**

> Much of the schedule slippage has been for causes outside US control
> -- such as beneficial occupancy of surface buildings 180 and 191,
> where the BWs and SWs are and will be assembled, respectively. At this
> writing (24-Jul-06) we have enough space in B180 to complete and store
> BW C (finished) and ½ of BW A. Negotiations are underway with ATLAS TC
> to gain more space on the Jura side where the second ½ of BW A will be
> stored. But in order to gain this space the ECT coils will have to be
> moved form their present location. If this space issue is resolved the
> BW A will be complete by the end of September pending the timely
> delivery of Al parts for the sectors from Russia.
>
> The second major space issue is for the US muon EC group to gain
> occupancy of part of B191, where the ECTs are presently being
> assembled, for starting the SW-JD installation and integration work.
> Plans have been made with TC to obtain some space in B191 this fall,
> after the first ECT is moved out the building. While, less efficient
> for the JD work, this plan will enable the jigging and first SW
> assembly to start this fall, while the assembly and integration work
> of the second ECT takes place. Assuming no more major delays in the
> integration of the ECTs this plan will enable both SWs to be finished
> in time to install them in the pit.
>
> Overall, there is a greater awareness in TC that more resources will
> have to be allocated to the completion of the muon endcaps. As the
> installation of other parts of the detector near completion more
> resources should be available to be applied to EC muons.

4.  **Unfortunately new delays in the ECT threaten this plan. There is a
    risk that not all of the Endcap Muon system will be installed and
    commissioned in time for the 1st LHC run in 2007. U.S. ATLAS should
    evaluate what steps can be taken to improve the likelihood that the
    EM will be installed and commissioned for day-one. These steps could
    include:**

    a.  **Adding more labor to speed up Phase II and Phase III
        commissioning.**

    b.  **Expediting beneficial occupancy of needed CERN support
        buildings.**

    c.  **Providing additional funds from the Management Reserve to
        close the Endcap Muon budget shortfall in FY'06 and FY'07.**

    d.  **Investigating whether the EM schedule can be advanced by
        adding more core-supported scientific personnel to the effort.**

> An analysis has been done of the pacing items for the BW assembly.
> Experience over the last 6 months indicates that the deployment of US
> personnel for the BW sector assemblies is fairly well optimized.
> Pending the resolution of storage space for the second ½ of the BW A
> and the timely delivery of Al parts from Russia for the sector
> construction, the progress and plans are congruent and extrapolate for
> a BW completion by the end of September.
>
> However there are essential items beyond US control that delay the
> project. As an example at this time (24-July-06) the storage fixtures
> needed for the remaining BW A sectors and supplied by Pakistan have
> not been delivered therefore effectively stopping the production of
> sectors in B180. In addition, the aluminum needed for the next sectors
> has not been delivered from Russia. These factors will delay the
> assembly of BW sectors by about two weeks.
>
> The schedule for the SW assembly and Phase II commissioning is more
> problematical because it must follow the completion of at least one
> endcap toroid being done in B191. The plan is to deploy our 'Surface
> Assembly Crew', who are now working on the BW assembly in B180, to
> B191, where they will concentrate on the SW. ATLAS Technical
> Coordination has proposed a plan where some of the space in B191 will
> be devoted to the assembly of the first SW after the first ECT moves
> out in the fall of '06.
>
> The plan is consistent with completion of the installations of the BWs
> and SWs by the beam pipe closure 31-Aug-07. However, there are a
> number of exposures to significant delay related to the technical
> progress on the ECT integrations.
>
> The installation of the first BW (Side C) is expected to commence in
> the early fall of '06 following the completion of the TGC1 C. Some of
> the experts now working in B180 will work in UX15 as the sectors are
> installed to check that no damage occurred to them during the
> B180-UX15 transport. When the wheel becomes integral some of the
> services will be mounted and connected to the patch panels on the
> periphery of the BW sectors. After the BW Service Bridge is installed
> the 'pit crew' will undertake the final connection of service cables,
> gas, etc. around the wheel to the service integration points.
>
> Phase III commissioning will start when all the services have been
> connected. More base-supported personnel would be helpful during this
> critical period. At this time support for US personnel to have a
> greater presence at CERN would have the most immediate benefit and for
> the longer term it would be helpful to have more base support for
> postdocs resident at CERN.

5.  **A contingency plan should be developed for completing the EM
    installation and commissioning in the eventuality that the EM is not
    finished in time for operation in the 2007 run. Coordinate with D.
    Lissauer**

> The pacing item for the completion of the EM installation is the
> production and installation of the 72 sectors of the TGC system. In
> aggregate there are 104 TGC and MDT sectors (72+32) to install in
> roughly one year (200 working days). This will require an installation
> rate of 1 sector every two days. This rate is probably possible
> (assuming that assembly of the sectors in B180 is ahead of the
> critical path) and has been roughly demonstrated by the installation
> of the second TGC1 sector.

If the installation of sectors is delayed until after the beam pipe is
installed then the present installation technique will have to be
modified to manipulate a sector under the beam pipe for the bottom
sectors. It is believed that this maneuver is possible but detailed
engineering has not been done. This option is not the present baseline
plan but will be continually evaluated.

**3.6 -- Trigger/DAQ -- A. Lankford**

1.  **Comment on the impact of the delay (perhaps it is only from June
    30, 2007 to August 31, 2007) expected in the start-up of 24x7
    operations.**

> Delay in the start-up of 24x7 operations has no significant impact on
> Trigger/DAQ. The system will be ready for 24x7 operations on the
> original schedule, and actual 24x7 operations will start when needed
> for ATLAS commissioning. In fact, the Pre-series system has already
> been operated for short 24x7 periods in order to gain operational
> experience.

2.  **U.S. ATLAS management is asked to re-assess the timescale and plan
    for the detectors to reach design-performance goals, and present
    this information at the next MEG meeting.**

##### TDAQ hardware

> Procurement and deployment of the full data bandwidth and processing
> power capability of Data Acquisition and High Level Trigger portions
> of the ATLAS Trigger/DAQ system was "deferred" in 2002 in order to
> redirect financial resources to completion of more time-critical
> aspects of ATLAS, such as superconducting toroids. This plan was not
> inconsistent with the desire to purchase commodity hardware in a
> "just-in-time" fashion (i.e. at as late a date as is consistent with
> providing performance when required), in order to take advantage of
> rapidly increasing performance-to-cost ratio (e.g. Moore's Law). The
> deferral plan is also not inconsistent with the fact that HLT/DAQ
> design-performance goals are not required until the end of a two to
> three year accelerator commissioning period.
>
> At present, enough DAQ hardware is installed to support commissioning
> without beam of all installed detectors. Approximately 10% of the
> number of HLT processors is installed, in order to enable TDAQ
> commissioning and to study performance issues. By the end of 2006,
> nearly the full DAQ bandwidth capability into the central network
> switches will be installed. HLT processing power and associated
> network bandwidth will be progressively installed in 2007 and 2008 to
> prepare for triggering on first collisions in 2007 and during the
> first physics run in 2008. We anticipate that the full "post-deferral"
> HLT/DAQ scope, which provides roughly half of the design rate handling
> capability, will be installed during 2008. The full scope of U.S.
> Construction Project procurements for HLT/DAQ will be complete in
> FY08. In particular, we expect that DAQ capabilities will be complete
> in FY07 and that procurement of HLT processing power will proceed into
> FY08. Procurement and deployment of the deferred scope for runs in
> 2009 and after depends upon the delivery during this period of
> financial contributions promised by collaborating ATLAS funding
> agencies in order to restore the full scope. The U.S. Research Program
> is committed to providing additional HLT processing power (2 MCHF)
> towards restoral of deferred capability. During ATLAS commissioning in
> 2007 and 2008, the system will be configured such that output rate
> capability to mass storage will exceed the design-performance goal, in
> order to facilitate detector and trigger commissioning studies.

##### TDAQ software

> Data acquisition software to achieve the design-performance goal is
> complete. HLT event selection software to achieve the
> design-performance goal is expected to be complete before the first
> colliding beams in 2007. Nonetheless, demonstration that this
> software, particularly selection algorithms achieve the full
> design-performance goal in the face of actual detector response and
> backgrounds must await colliding beams. At present, considerable
> attention is paid to developing additional event selection algorithms
> to facilitate ATLAS commissioning. Furthermore, considerable
> investment in changes and refinements to software in response to
> actual operating conditions and increasing luminosity can be expected
> for a number of years after first collisions.

3.  **U.S. ATLAS should encourage international ATLAS to define a plan
    for the complete cosmic-ray run that would exercise the trigger,
    data acquisition, and software reconstruction chain. With S.
    Rajagopalan**

> Test of a complete trigger, data acquisition, and software chain
> during the ATLAS cosmic-ray run has always been a central element of
> the plan for that run. This test is now planned for the period
> September to October 2007, after the LHC beam pipe is closed.
> Meanwhile, tests of the chain are already starting using installed
> detectors, e.g. TileCal and MDTs.

**3.9 -- Project Management -- M. Tuts and H. Gordon**

1.  **Comment on the impact of the delay (perhaps it is only from June
    30, 2007 to August 31, 2007) expected in the start-up of 24x7
    operations**

> We believe the delay of two months will hardly be noticeable in terms
> of planning for U.S. support at CERN. The peak of the Pre-operations
> and Commissioning will be in the Summer of 2007 with the transition to
> 24x7 operations for Cosmic Rays and initial 0.9 TeV running after
> that.

2.  **As background material, please provide in tabular form your
    expected profiles for your major research commitments in M&O and S&C
    for FY '06 -- '09. Please include management reserve, project
    management, upgrade R&D, Tier-1 and Tier-2 facilities (both
    equipment and personnel), core software (personnel: direct to
    experiment and to OSG/LCG/etc), large detector subsystems, Category
    A, Category B, cost of operating LPC or support centers for
    analysis. Split the table into M&O and S&C components, where the
    totals correspond to the current financial guidance from the
    agencies.**

![](media/image2.png){width="6.823611111111111in" height="6.4875in"}

3.  **Last year MEG, requested that the tracking and reporting of cost
    and schedule performance be more timely and should be developed into
    a better tool for planning and making decisions. While management
    has implemented a clear change-control process to manage changes
    both in Management Reserve and scope, the tracking and reporting of
    costs still does not appear to have evolved into an optimal planning
    tool.**

> We track and report milestones and costs in our quarterly report. We
> believe this is effective.

4.  **In light of the many subsystem requests, MEG is concerned about
    the low level of Management Reserve (MR) in FY'06-'08. As a result,
    U.S. ATLAS must optimize the allocation of MR to M&O to ensure
    successful commissioning and pre-operations of the detector. Because
    lower-priority items cannot be supported within the current plan,
    how does U.S. ATLAS management expect to ensure that this problem
    does not compromise detector readiness?**

> We agree with the implied priority that detector readiness must be our
> first priority. Therefore, we have been steadfast in allocating funds
> to understand what is necessary for the start of data taking. We are
> concerned about FY08 when there is the need for a large increase in
> Computing. We have been managing the Computing personnel so that we do
> not build up too many people which would then need to be supported in
> FY08 and beyond. Lastly we can throttle down the Upgrade R&D if
> needed.

5.  **The contention over the U.S. ATLAS Management Reserve is an
    internal matter that should be resolved accordingly.**

> We agree.

6.  **U.S. ATLAS management is asked to re-assess the timescale and plan
    for the detectors to reach design-performance goals, and present
    this information at the next MEG meeting.**

> We can give an interim appraisal now. The two systems which risk now
> being completely installed for the 2007 run are the End Cap Muon
> system and the pixels. We are working on plans to bring those systems
> to full design performance for the 2008 run (see more detailed
> discussion above).

7.  **U.S. ATLAS plans for detector-upgrade R&D appear comprehensive,
    aggressive and appropriate, especially given the long lead-times
    required for such upgrades. In light of other M&O and S&C budget
    pressures, it is not clear whether the needed resources will indeed
    be available for this purpose.**

> As we said above, we realize that Upgrade R&D has the lowest priority
> in the Research Program. However, the work which is being done there
> needs to proceed for several reasons. First, the development of
> completely new detector technologies as is required for the most inner
> tracking regions needs the longest lead time. We are still benefiting
> from R&D done during the SSC era. Second, Upgrade R&D work done now in
> the U.S. preserves the intellectual leadership of the U.S. in the
> Upgrade process. Third, Upgrade R&D work captures the best engineers
> who otherwise might be diverted to non-ATLAS tasks.

**3.10 -- Technical Coordination -- D. Lissauer**

1.  **Comment on the impact of the delay expected in the start-up of 7X7
    operations**

> The ATLAS installation and commissioning plans have taken in to
> account that the beam start up is delayed to the end of '07. The plan
> is to close the experiment as soon as possible -- and use the delay to
> increase the time we use for commissioning multiple systems. There are
> small delays in the ATLAS installation schedule and the time will be
> used to make sure that we have a full detector at beam start up.

2.  **Despite the significant U.S. effort in Technical Coordination and
    integration, there is concern that the ATLAS integration team will
    be weakened by the departure of several key engineers.**

> This is a continuing worry and we are working with other collaborators
> to maximize the retention of expertise. This is not possible in all
> cases. There is still no solution for some critical items like the
> lead designers at CERN, resident magnet expert, etc.

###### Also note 4. in 3.5 above

**[4.0 Upgrade R&D -- A. Seiden]{.underline}**

1.  **Around 2012, the inner Pixel layer ("B-Layer") will lose
    functionality from radiation damage. This is somewhat earlier than
    the target date of 2015 for installation of the upgraded detector.
    With conclusions and a plan of action needed prior to the end of
    2006, has a task force as yet been appointed to resolve the
    technical issues, responsibilities, and funding questions?**

#######  Also note and contribute to 7. above under 3.9 

> Planning for the B-Layer replacement has been slower than in earlier
> plans given the priority to complete, test, and install the pixel
> system. There will be no explicit activity in FY07, although work on
> developing a next generation pixel chip within the Upgrade R&D program
> may provide the front-end electronics used in the B-layer replacement.
> The intent of the pixel collaboration still is to produce a proposal
> by the end of the calendar year. There will likely be a one-day
> workshop during the September ID week. The results of this workshop
> would be presented at the Oct 1-2 ATLAS-wide upgrade meeting at CERN.
> The actual proposal would be presented at the ID-wide upgrade meeting
> in Liverpool Dec 6-8.
>
> It is crucial that adequate resources are provided for the Upgrade R&D
> program. With both the development of the appropriate detector items
> and first-article larger assemblies needed prior to real construction
> the schedule is extremely tight. Europe has reaffirmed the centrality
> of the Upgrade in the recent document The European Strategy for
> Particle Physics with the statement: "R&D for machine and detectors
> has to be vigorously pursued now and centrally organized towards a
> luminosity upgrade by around 2015". We have created the central
> organization and now need to vigorously pursue the R&D, which we feel
> has been carefully targeted to areas where we can have major impact.
> The FY07 budget is the most difficult with about a third of the needed
> funds in management reserve (that is the guidance is presently about
> \$2M, with the required funding closer to \$3M). We can compare our
> numbers with the ILC Detector R&D budget, which is generally viewed as
> inadequate, and provides a point of comparison for what is needed for
> an R&D program. The number for the ILC in FY06 is nearly \$7M, as
> presented at the recent HEPAP meeting (for universities \$1,048k from
> DOE and \$300k from the NSF and \$5,607k for the national labs). As in
> our own program the primary costs are for various parts of the
> tracking system and precision calorimetry.