![](media/image1.wmf){width="3.3895833333333334in" height="4.0in"}

**Computing and Physics PROJECT STATUS REPORT No. 6**

**REPORTING PERIOD**

##### Q4 2002

**July-September 2002**

####  

**Table of Contents**

# 

# 

# 1. Project Objective

# 2. Project Manager's Summary (John Huth, Harvard)

3\. WBS 2.1 Physics Manager's Report (Ian Hinchliffe, LBL)

4\. WBS 2.2 U.S. ATLAS Software Manager's Report (Torre Wenaus, BNL)

5\. WBS 2.3 U.S. ATLAS Facility Manager's Report (B. Gibbard, R. Baker,
BNL)

6\. WBS 2.3.2 Distributed IT Infrastructure (Rob Gardner, Indiana
University)

## 

## 7. WBS 2.4 Financial Report (Chuck Butehorn, BNL)

# 

# **1. Project Objective**

The U.S. ATLAS Project consists of the activities to design, supply,
install and commission the U.S. portion of the ATLAS detector. The
detector will become part of the Large Hadron Collider (LHC) at CERN,
the European Laboratory for Particle Physics. The ATLAS detector is
being designed to understand the dynamics of electroweak symmetry
breaking. The U.S. ATLAS collaboration is funded jointly by the U.S.
Department of Energy and the National Science Foundation.

The fundamental unanswered problem of elementary particle physics
relates to the understanding of the mechanism that generates the masses
of the W and Z gauge bosons and of quarks and leptons. To attack this
problem, one requires an experiment that can produce a large rate of
particle collisions of very high energy. The LHC will collide protons
against protons every 25 ns with a center-of-mass energy of 14 TeV and a
design luminosity of 10^34^ cm^-2^ s^-1^. It will probably require a few
years after turn-on to reach the full design luminosity.

The detector will have to be capable of reconstructing the interesting
final states. It must be designed to fully utilize the high luminosity
so that detailed studies of rare phenomena can be carried out. While the
primary goal of the experiment is to determine the mechanism of
electroweak symmetry breaking via the detection of Higgs bosons,
supersymmetric particles or structure in the WW scattering amplitude,
the new energy regime will also offer the opportunity to probe for quark
substructure or discover new exotic particles. The detector must be
sufficiently versatile to detect and identify the final state products
of these processes. In particular, it must be capable of reconstructing
the momenta and directions of quarks (hadronic jets, tagged by their
flavors where possible), electrons, muons, taus, and photons, and be
sensitive to energy carried off by weakly interacting particles such as
neutrinos that cannot be directly detected. The ATLAS detector will have
all of these capabilities.

The ATLAS detector is expected to operate for twenty or more years at
the CERN LHC, observing collisions of protons, and recording more than
10^7^ events per year. The critical objectives to achieve these goals
are:

-   Excellent photon and electron identification capability, as well as
    energy and directional resolution.

-   Efficient charged particle track reconstruction and good momentum
    resolution.

-   Excellent muon identification capability and momentum resolution.

-   Well-understood trigger system to go from 1 GHz raw interaction rate
    to \~100 Hz readout rate without loss of interesting signals.

-   Hermetic calorimetry coverage to allow accurate measurement of
    direction and magnitude of energy flow, and excellent reconstruction
    of missing transverse momentum.

-   Efficient tagging of b-decays and b-jets.

# **2. Project Manager's Summary**

The major driving forces for our computing effort in this quarter were,
once again, the ATLAS Data Challenges (DC) and demonstrations for the
SuperComputing 2002 meeting in Baltimore November 2002. During this
period the US effort added a new element to the ATLAS Data Challenges by
using much of the existing grid middleware tools such as the iVDGL
Virtual Data Toolkit(VDT) for real production. This used facilities at
the U.S. Tier 1 center at BNL as well as those of the U.S. ATLAS
testbed.

Work continues on the LCG common project. POOL with substantial US ATLAS
participation. We are also active in a number of RTAG's (Requirements
Technical Assessment Group).

Because of our budget constraints we continue to hold off on the
expansion of the Tier 1 facility at BNL which needs ramping up to
participate fully in future ATLAS Data Challenges. The ramp up of
manpower for critical core software areas like database management has
also been delayed

**[3. WBS 2.1 Physics Manager's Report (Ian Hinchliffe,
LBNL)]{.underline}**

This quarter saw the completion of the production for DC1 phase 1 and
the preparations for Phase 2. More parameter sets were provided for the
productions related to Higgs bosons to be used primarily for the study
of b tagging algorithms. These can be found at

<http://www-theory.lbl.gov/~ianh/dc/>

STDHEP has been removed from the atlas releases, providing a
simplification of the ATLAS generator codebase. Support for the minimum
bias generator Phojet was established. This program is thought to have a
better model for these events that that contained in the multipurpose
generators such as Herwig. Good modeling of these events is important
for understanding the effect of pile up at high luminosity when as many
as 400 event can be overlapped in the LiAr calorimeters.

The Pythia interface was extended to allow users more control over
internal parameters and the type of output provided.

A prototype GUI

interface for the setting of Monte-Carlo parameters has been

written. It could be used as the basis for a more general system for

setting user options in Athena. This will be demonstrated at the ATLAS

week (October 7) and user feedback will be solicited. It is expected

that this tool will become part of the release during the next quarter.

Integration of the ALPGEN special purpose package was completed. This

package calculates rates for W+2 bquark jets and a number of other

jets in kinematic regimes where the multipurpose generators are either

slow or unreliable.

An RTAG mandated by the CERN-LCG was set up to consider the support

that CERN-IT should provide to the experiments in the area of

Monte-Carlo event Generators. Hinchliffe was the ATLAS representative

to this which was chaired by A. De Roeck (CMS/CERN). Its report will be

presented on October 8th. The report calls for the establishment of a

repository at CERN where the Monte Carlo codes required by the

experiments will be held and the manpower to maintain it.

If established this repository will replace the \"external\"

area in ATLAS which is currently maintained on an ad-hoc basis by

volunteers. The experiments will be asked to help in the establishment

of at test suite that can be used for validating new releases. This

would guarantee a level of quality control that is currently

lacking. The RTAG also addressed the level of support needed from the

CLHEP team at FNAL for the HepMC interface package that was originally

written by Atlas members but has now become a common product.

**[4. WBS 2.2 Software Manager's Report (Torre Wenaus,
BNL)]{.underline}**

Software project efforts in the quarter have largely focused on
infrastructure and operations development and support for the first
phase of Data Challenge 1, which was underway during the period, and
preparatory work for DC 1 phase 2 (reconstruction production and physics
dataset processing) which begins in the next quarter. Other activities
were associated with contributions the LCG project, in which US ATLAS is
an important contributor to software design and development in the LCG
Applications Area. A long awaited creation of a software support and
quality assurance team took place in ATLAS, in which the U.S. is
strongly involved, providing a better context for continuing efforts in
the U.S. to improve QA particularly through automated testing.

### 

### WBS 2.2.1.1, 2.2.1.2, 2.2.1.4 Framework, Architecture and Event Model 

Work during this reporting period has been focused primarily towards the
Second Phase of Data Challenge 1 and the Trigger TDR. Significant
developments during this reporting period are:

o After a common integration effort with the LHCb core software team
Gaudi release 11 incorporates all new features introduced by Atlas in
the common code repository.

o To allow modules to declare proxies for the data objects they can
provide, a new Gaudi Incident is now triggered by the event loop
manager. A better engineered implementation of the same ProxyProvider
concept is underway and should be completed by November.

o A new python package AthenaAsk has been introduced to address the
needs of new athena users. The Athena Startup Kit provides a GUI and a
command line interface to run common Atlas programs (e.g. the fast
simulation AtlFast or the \"Common Reconstruction Example\"), and even
to create new Algorithms in a new package, compile, link and run them.

o The pile-up framework has reached pre-production quality. New features
like in-memory caching and re-use of events have dramatically improved
the performance. There are pile-up algorithms for InnerDetector hits and
for the LAr calorimeter. Performance studies using both sub-detectors
and data from Zebra and/or Root have started with results expected by
Jan 2003.

o A major reengineering of StoreGate internals has been performed to
address the code review findings and the expected access patterns from
High Level Trigger algorithms. A new reference-counted pointer and
containers of \"data pointers\" have been introduced to provide better
memory management. The persistable data object references API (DataLink)
has also been simplified to ease the job of the developers and of the
automatic code generator.

o Data Dictionary and ADL data definition language have been extended to
support the DataVector\<T\> collection and ElementLink\<T\> object
association class. The data model for the Atlfast fast simulation and
event generators are in the process of being converted to use ADL.

o The emphasis on the event data model and detector description for the
Trigger TDR continues. The ByteStream infrastructure is in place and
preliminary converters exist for the inner detector, liquid argon and
tile calorimeters, with that for the muon detector under active
development.

o The handling of non-event data (e.g. alignments, calibrations)
requires the framework to update asynchronously data structures seen by
the algorithms which will see e.g. the position of a detector element
change transparently when the interval of validity of the relevant
alignment data is crossed. As this position will be in general
determined by the combination of several alignment constants the
updating procedure is highly non-trivial. An Interval Of Validity
service that steers the updating as well as the interaction with the
geometry and calibration databases has been designed and is being
prototyped. A fully functional service is expected to be delivered in
January 2003.

o A prototype monitoring service has been developed that allows an
Athena application to be instrumented with a binding to the Grid
Monitoring Architecture (GMA) and in particular the NetLogger component
of this. This will form part of the SC2000 demonstration in November.

### WBS 2.2.1.3 Databases and Data Management 

Emerging themes in this reporting quarter include an increasing emphasis
on database support for detector description, the beginning of component
development for the LCG common persistence project (POOL), a growing
role for MySQL, and a transition from Data Challenge 0 to Data Challenge
1.

At the request of the ATLAS detector description group, the U.S.
database group delivered a relational database infrastructure for
storage and retrieval of \"primary numbers\" parameterizing the ATLAS
detector description, and the means to populate this database from data
used in ATLAS Geant3 simulations. In this reporting quarter, the U.S.
also delivered an Athena conversion service for access to these primary
numbers, so that reconstruction and simulation are indeed relying upon
the same sources, and Athena-based applications are reading by means of
services that are completely consistent with those used for access to
event data. The U.S. group continues to maintain this database and the
Athena services to read it, and to ensure that it is appropriately
updated with each software release.

Access to conditions data is becoming an increasingly high priority in
ATLAS. The U.S. group has initiated a collective effort involving the
offline, TDAQ, and Technical Coordination database communities to define
the scope, requirements, use cases, and priorities for ATLAS-wide
storage of and access to conditions data. A synopsis of the current
offline database strategy, a bibliography, and a survey to users have
been produced. The first face-to-face meeting of this group will take
place in the coming quarter.

Much of the U.S. development effort is now directed toward advancing the
LHC-wide common persistence project called POOL. ATLAS is committed to
using POOL as its principal technology for building and managing its
event store. The U.S. has principal responsibility for the collections
and metadata components, which are due for delivery in the coming
quarter. The U.S. group is contributing substantially to the
infrastructure for ROOT persistence of non-ROOT objects, to the
definition of reflection and dictionary services, and to the overall LCG
persistence architecture.

Data Challenge 0 was declared officially over in this reporting quarter.
Its conclusion has allowed a reduction of database effort in support of
Objectivity/DB infrastructure and services to a subsistence level, and
redirection of that effort to LCG software development. Objectivity/DB
will be retired from ATLAS software releases after Release 5.0.0, so
that Release 6.0.0 (January 2003) will be Objectivity-free. Old
Objectivity-based releases for the Lund physics workshop and for Data
Challenge 0 will continue to be available, but no new development with
Objectivity will be supported. Support for existing Objectivity
databases and servers will likely be phased out in 2003.

The U.S. group has substantial production and support responsibilities
for Data Challenge 1, which is now underway, and is using this effort as
a testbed for virtual data concepts developed under the aegis of the
U.S. GriPhyN (Grid Physics Networks) project. Details of the substantial
role of Magda in this data challenge appear in the following section.

An increasing number of components in ATLAS offline software releases
rely on MySQL. MySQL deployments to date (including a principal
deployment at BNL providing services throughout ATLAS) have been managed
by local site database administrators, with no overall ATLAS strategy
and no support for a CERN-hosted database. A U.S.-led working group was
formed in this reporting quarter to propose a near-term server
deployment strategy, and long-term requirements that will serve as input
to an LHC-wide group on fabric management. Partly as an LCG POOL
contribution and partly to ensure coherence between ATLAS and LHC-wide
strategies, the U.S. ATLAS database group has volunteered to take
responsibility for MySQL-related external packages in the LCG POOL
project. The ATLAS working group will report during the November ATLAS
Software Week, and will host an LHC-wide discussion at a November LCG
applications meeting. In related PPDG work, the U.S. group has agreed to
a limited evaluation of grid certificate authorization technologies for
use with MySQL servers.

The first implementation of the dataset model was completed. It includes
an abstract base class Dataset which defines the dataset interface,
supporting classes and a few dataset implementations. One
implementation, TDatasetFileDataset, allows one of the new ATLAS ROOT
event data files to be used to define a dataset. Another,
EventMergeDataset, merges events in multiple datasets to form a single
dataset.

Discussions were continued on integrating the dataset model with event
collections envisioned in the ATLAS database architecture document. Once
these are resolved, then the common product will be included in the CERN
LCG. This integration is targeted for the next quarter.

The Argonne database group lost Chris Lain, who had been working
half-time on ATLAS, most recently on LCG POOL event collections, due in
part to shortfalls and delays in funding. The expected FY \'03 budget is
intended to suffice to support his replacement with another software
engineer.

### WBS 2.2.1.8 Distributed Analysis 

Most of the pieces needed for the first implementation of DIAL
(Distributed Interactive Analysis of Large datasets) were put in place
during the quarter. In addition to all the supporting components, these
include a local scheduler which can be used to run jobs on the local
machine.

DIAL provides a connection between an analysis framework and a
data-processing application. Our first choice for the former is ROOT and
we plan to integrate DIAL with ROOT in the next quarter.

The obvious data-processing application in ATLAS is Athena. ATLAS plans
to move much of its data into ROOT files and so ROOT itself might be a
candidate for this application. Most of the terabytes of DC1 data
generated during the quarter are in zebra format so atlsim might also be
a candidate. During the next quarter, we plan to choose one of these and
integrate it to provide users with the DIAL-based capability to directly
analyze ATLAS data from ROOT.

The next step will be to provide a distributed scheduler which makes it
possible to carry out an analysis task on multiple nodes at a site.
After this, the plan is to GRID-enable DIAL (or more precisely the DIAL
scheduler) so that it can locate and process data over the GRID.

**WBS 2.2.1.10 Distributed Data Management and Processing Software**

During this period Magda was used in the testbed production for the
Atlas Data Challenge 1 (DC1), and was shown to be a useful and practical
tool. \'globus-url-copy\' and \'globus-job-run\' were integrated into
magda_putfile. magda_putfile can be used to do third-party transfers,
put files to BNL HPSS directly, and register files in the Magda
database. \'globus-url-copy\' was also used in the bulk data replication
between BNL and CERN. Another trans-Atlantic transfer engine used is
\'bbftp\', which worked very well also. A rate of 300 GBytes/day was
observed from BNL HPSS to CERN castor, and about 4 TBytes data had been
copied for the Atlas DC1. The file spider worked diligently as usual.
35K DC1 files are available through Magda.

Magda servers were moved and split in this period. The database now is
on \'magda.usatlas.bnl.gov\', which is a dedicated MySQL server.
Significant performace improvements had been reported by our principal
production user Kaushik De - no more MySQL timeout error occurred to him
after the transition. The main Magda web site now is
<http://www.atlasgrid.bnl.gov/magda/dyShowMain.pl>

The command magda_getfile was developed and gridified. It does file
copying economically: it figures out which file instances to retrieve,
and fetches remote instances only if it is necessary. magda_putfile
automatically determines which remote host to contact if the remote host
is not provided as an argument and it is needed; if it is provided, use
the provided host as a means to access storage site, should be hidden
from users if possible.

A new attribute was added to the Magda location. The file spider checks
that attribute to decide whether or not it needs to crawl a location.
This was found to be very useful. In Magda, for a logical file, only one
prime instance is allowed, while multiple replica instances can be
distributed at different locations. If two primary instances exists for
some reasons, the file spider can be disabled to scan one of the
locations until problems are reported and fixed.

The interfaces for the deleting of file records have been developed both
on the web and on command line. A protection of empty input from the web
form was added.

Metadata proposed by the Grappa developers was saved to the Magda
database, can be retrieved with file name from both the web and the
command line.

The integration of Magda with GDMP was tried in Milan:
\'gdmp_register_local_file\' and \'gdmp_publish_catalogue\' were put
into magda_putfile; it seems to be working as expected. The strategy of
managing files which are distributed on local disk of each individual
node of a Linux farm, was discussed and will be implemented soon. A farm
can be seen a special \'Magda\' site, and the location associated with
it will be a directory path preceded by the host name of the node. The
Magda web services were implemented with perl SOAP::Lite module. They
can be invoked to query the database and register files. With web
services, Magda provides one more option to interact with the database.
Magda commands are available by default on BNL Linux farm. Shared disk
cache is supported: users could share the same instance if they need the
same file. This should reduce the traffic with the mass storage.

We set up a new development machine with root privilege located outside
the BNL firewall. The installation of software has been started to do
the RLS test.

A poster about Magda and live demonstrations of file transferring with
Magda are in preparation for SC2002 demo.

The STAR experiment has Magda servers running and is evaluating it.

### WBS 2.2.2 Simulation and Reconstruction Software 

The ATLAS detector simulation infrastructure and production was
supported during Data Challenge 1 phase 1, which was underway during the
period.

ATLSIM has been upgraded with new components (shared libraries) to
provide extra I/O facilities (ROOT, Objectivity) and to load the ATLAS
detector geometry. The description of the ATLAS geometry is taken from
the DICE package. Compared to the physics TDR, several updates have been
made to reflect the design modifications since 1998:

\- Beam pipe: multi-layer beam pipe design.

\- Pixel detectors: symmetrical, insertable layout, more realistic
digitization

\- SCT: tilt angle reversed (to minimize cluster size)

\- TRT: modular design of the Barrel, full readout simulation in
digitization.

\- Inner detector services material updated

\- Realistic field in Inner Detector

\- All End-Cap Calorimeters shifted by 4 cm

\- ENDE: dead material, readout updated

\- TILE: material and readout update

\- HEND: dead material updated

\- FWDC: detailed design with precise rod positions

\- Muon system design corresponds to the AMDB version P.03

During the simulation phase, di-jet events produced by PYTHIA were
analyzed by a filtering routine which looked for a predefined energy
deposition in two neighboring towers in h-f space. Only events selected
by the filter were passed to the simulations step and then written out.

### 

### In the process of the ATLAS Data Challenge we have evaluated the virtual data approach for the production of several datasets. A virtual data catalogue database prototype was deployed for evaluation in the context of the ATLAS Data Challenges. The prototype is being used successfully for data challenge event generation and detector simulation. Production job options for physics event generation and production scripts for detector simulation were recast as parameterized transformations to be catalogued, with the resulting parameterizations represented as derivations. ATLAS DC0 and DC1 parameter settings for simulations are recorded in the virtual data catalogue database.

The production system, based on the virtual data catalogue prototype,
implemented the scatter-gather data processing architecture to enable
high-throughput computing. The production fault tolerance has been
enhanced by the use of the independent computing agents, adoption of the
pull-model for agent tasks assignment (instead of push model typically
used in batch production) and by the local caching of output and input
data. An interesting feature provided by this architecture is the
possibility for the automatic \"garbage collection\" in the job planner
in the following sequence: production agents pull the next derivation
from the virtual data catalogue; after the data has been materialized,
agents register \"success\" in the database; when previous invocation
has not been completed within the specified timeout period, it can be
invoked again.

VDC based production system using Magda for data access and Pacman for
software distribution was used to produce 3,200 fully simulated jet
events and about 10,000 minimum bias events in the pace of DC1.

**WBS 2.2.4 Software Support and QA/QC**

In an important and long awaited development, ATLAS established a
software infrastructure and testing (SIT) team in this quarter
responsible for central and remote software librarian duties, release
management, quality assurance and testing, software compliance with
coding standards, etc. The U.S. is an active contributor to this group
through the U.S. ATLAS Librarian, Alex Undrus.

The ATLAS nightly build facility at CERN was monitored and improved. A
new feature that monitors the build progress and reflects this
information on the nightly build web page was added. With this feature,
users are better notified about the status of the nightly builds.

Nightly builds of ATLAS software were restarted at BNL. These builds
mirror CERN builds and normally use the same packages versions as the
corresponding CERN build. As a result, the U.S. users have local access
to the latest modifications of ATLAS software. The BNL nightly builds
are helpful for all ATLAS software developers in quite frequent cases of
CERN nightly builds failing due to technical problems.

An ongoing program of incorporating greater QA in the automated build
system continued. Several Athena testing jobs were added in the
nightlies with the results reflected on the nightlies web page. That
includes saves and reads Generator transient event store objects in
ROOT, writes and reads of digits event model information, and various
tests of LAr software. These tests are currently described in the
nightlies scripts. They should shortly be moved to a new package
organization for tests that the SIT group is in the process of setting
up with U.S. participation.

The AtlasROOT package was maintained, and the ROOT version changes were
synchronized with corresponding changes in Gaudi (the event processing
framework underlying ATLAS software).

New ATLAS software was promptly installed at the BNL Tier I Center,
usually in one to three days after CERN installation. The BNL software
support page is updated in a timely way with details and examples of
usage of new Atlas Software releases.

The U.S. Atlas MySQL and web server and cvs pserver were maintained and
administered and their services were actively used by the U.S. Atlas
community. The U.S. Atlas MySQL server was moved to a dedicated machine,
db1.usatlas.bnl.gov.

The evaluation of the SCRAM release management tool (at the request of
the LCG) was finished and the memorandum that compares two release
tools, SCRAM and CMT, was made available to the public. The memorandum
was written with Ianna Osborne, CMS representative.

## Project Management

## 

## The XProject project planning software developed in US ATLAS was adopted by the Applications Area of the LCG Project.

## Summary of Major Milestones and Deliverables

### WBS 2.2.1.2 

-   ATLAS and LHCb convergence on shared Gaudi kernel complete

-   Bytestream (DAQ data format) infrastructure in place

-   New 'GeoModel' detector description prototype implemented for muons

-   Python-based shell for new users in place

-   Pre-production pile-up support in Athena 1-detector prototype in
    place

-   Prototype Athena monitoring service in place

### WBS 2.2.1.3

-   NOVA geometry description conversion service prototype in place

-   LCG POOL hybrid data store initial internal release

### WBS 2.2.1.10

-   Third party transfers implemented in Magda

-   DC1 phase 1 simulation dataset (35k files) cataloged in Magda

-   Magda moved to higher performance dedicated DB server

### WBS 2.2.4

-   Nightly builds started at BNL

-   Several Athena tests added to nightly builds

-   SCRAM/CMT evaluation report for the LCG completed

# **Forthcoming Milestones and Deliverables**

### WBS 2.2.1.2

-   Reengineered ProxyProvider complete (Nov)

-   Results from pile-up performance studies (Jan)

-   ByteStream converters in place (Dec)

-   Interval of validity service fully functional (Jan)

### WBS 2.2.1.3

-   Delivery of LCG POOL collection service prototype (Dec)

-   Retirement of Objectivity/DB from releases (Jan)

-   Interval of validity database connection to Athena (Jan)

### WBS 2.2.1.10

-   Production application of Magda in DC1 phase 2 (Nov)

### WBS 2.2.4

-   Migration of Athena and systems tests from nightly scripts to test
    packages (Dec)

-   Incorporation of production chain testing into standard test suite
    (Dec)

**[5. WBS 2.3 US ATLAS Facility Manager's Report (B. Gibbard, R. Baker,
BNL)]{.underline}**

# Introduction

During the fourth quarter of FY '02, facility staffing was maintained at
the level of 4.5 FTE on project plus an additional 0.5 FTE for ATLAS
Grid support funded via PPDG. This is the staffing level that was
presented in the FY'02 plans during the review conducted in November.
During the first two months of the quarter, the facility was fully
engaged in Monte Carlo production for phase 1 of ATLAS Data Challenge 1.
The facility performed extremely well, delivering nearly 7,000 CPU days
and producing 3.4 TeraBytes of data. The US Tier 1 facility out-produced
all other ATLAS facilities except for the prototype Tier 0 center at
CERN.

### WBS 2.3.1 Tier 1 Computing Facility at Brookhaven National Laboratory

### 

### WBS 2.3.1.1 Hardware

During the fourth quarter of FY'02, \$200k of special end-of-year
capital for the purchase of hardware was received. Based on experience
with the first phase of DC1 and the expected needs of DC1 phase 2, an
expansion of the disk storage system from 1.6 TB to 10 TB, and the
addition of a high performance dedicated file server was initiated..
This hardware was purchased and delivered late in the fourth quarter and
will be integrated into the facility during the first quarter of FY
2003.

In order to maximize the CPU resources available for DC1, the facility
began a CPU resource sharing arrangement with the much larger RHIC
Computing Facility (RCF) during the third quarter of FY '02. With this
arrangement, idle US ATLAS Tier 1 CPUs are made available to designated
production managers from RHIC through a special low priority batch
queue. In return, idle RCF CPUs are available for ATLAS production work.
Jobs in the low priority queues are preempted when higher priority jobs
are present, so this arrangement has no negative impact on resources
available to ATLAS. During the fourth quarter, the RCF had significant
idle CPU resources that were used by the ATLAS DC1 production manager.
This was a major factor in making the US Tier 1 production the largest
of any prototype ATLAS Tier 1 facility worldwide.

The level of effort directed to WBS 2.3.2.1.1 during this quarter was
constant at 1.6 FTE.

### WBS 2.3.1.2 Tier 1 Facility Software

During the quarter, continuing maintenance and upgrade work went into
the major commercial products used in the facility including HPSS, AFS,
Objectivity, LSF and Veritas. Much of the effort for this is accounted
for under WBS 2.3.1.2. The level of effort for WBS 2.3.1.2 was at the
planned 1.2 FTE during the quarter.

### WBS 2.3.1.3 Tier 1 Facility Administration and Support

Management efforts were focused on dealing with the evolving funding
realities and the slippage of the LHC startup schedule into 2007. There
was also active in planning for the facility upgrades needed for the
remainder of Data Challenge 1 and the upcoming Data Challenge 2. Work
has started on new funding proposals to be submitted to the NSF later
this year.

The level of effort for WBS 2.3.1.3 was 1.7 FTE, which includes Tier 1
Facility management and planning, oversight of the full US ATLAS
facilities effort and also Tier 1 operation and monitoring support.

## Summary of Major Milestones and Deliverables

This quarter saw completion of one milestone (US ATLAS Online Storage
System Prototype). This deliverable were originally scheduled for
completion in the fourth quarter of FY '01, but the funding for the
required hardware was not available until August 2001. This completed
all of the components of the facility required for Data Challenge 1
phase 1, a major project milestone.

### WBS 2.3.1.2 

### US ATLAS Online Storage System Upgrade (Scheduled for completion 10/1/2001, expected completion 10/31/2002) Hardware procured during Q4 FY '02 will be integrated into facility in October.

# **Forthcoming Milestones and Deliverables**

In addition to the Online Storage System upgrade, the processor farm and
tertiary storage system were scheduled for upgrade in the fourth quarter
of FY '02. Based on reduced available funding, not all of these
milestones could be achieved. Based on experiences with the first phase
of Data Challenge 1 and expected needs of the next phase DC1 and DC2,
combined with the evolving data analysis environment for US physicists,
the disk storage was deemed most critical. A modest upgrade for the
tertiary storage system (upgrading the existing tape drive) is also
likely during the first quarter of FY '03, but significant upgrades to
the CPU and tertiary storage systems will be delayed into FY '04 unless
additional funding is made available. The original plan was to have a
10% Processing Farm Prototype in place for Data Challenge 2 by December
2002. Although the schedule for DC 2 is still evolving, it will almost
certainly begin before FY '04. Although the US managed to out-produce
the actual hardware resources available in phase 1 of DC1, this was
achieved largely through extraordinary personal efforts (notably by P.
Nevski) and borrowed resources. The lack of sufficient CPU may be
alleviated by extremely efficient utilization of existing hardware and
by capitalizing on unused RCF CPUs. Tier 2 facilities and other US
institutions will also be harnessed to produce some of the data
required. The contributions from these other facilities may make up for
some of the Tier 1 capacity shortfall, but the Tier 2 contributions were
already included in the original plans for DC2, and the availability of
unused RCF CPUs is far from certain.

**[6. WBS 2.3.2 Distributed IT Infrastructure (Rob Gardner, Indiana
University)]{.underline}**

# 

Computing for U.S. ATLAS will rely on a distributed information
technology infrastructure, which includes distributed computing
resources and data stores interconnected by high-speed networks. Grid
middleware systems will be deployed to utilize these resources
efficiently. The Distributed IT Infrastructure subproject, WBS 2.3.2, is
organized to meet these requirements for US ATLAS. R. Gardner is the
project manager for WBS 2.3.2, and reports to R. Baker/B. Gibbard, the
Facility Project Managers. The distributed IT project planning page is:

> <http://www.usatlas.bnl.gov/computing/mgmt/dit/>

From this page are links to WBS 2.3.2 with projections into the Grid
Project Planning numbering scheme developed for US ATLAS using XProject.

### 

### WBS 2.3.2.1, 2.3.2.2 ATLAS Requirements and Grid Architecture

Efforts to define the requirements for worldwide ATLAS computing and the

Grid Architecture required to support that computing continued, largely
driven by data challenge activities both within the US and the worldwide
Data Challenge 1 phase 1 which occurred during the second half of
FY\'02. These exercises were instrumental in the formulation of ATLAS
requirements for the second phase of DC1. This exercise will start with
a loosely

coupled worldwide Grid of computing resources and attempt to build

additional commonality, including a first implementation of an ATLAS

Virtual Organization and a single resource broker managing the entire

worldwide ATLAS Grid. The worldwide computing model and architecture
will

certainly evolve based on experience with DC 1 phase 2 and the
subsequent

DC 2.

### WBS 2.3.2.3 Integration of Grid Software

[2.3.2.3.1](http://atlassw1.phy.bnl.gov/XProject/servlet/XProject?wbs=/us/facilities/dit/sw/toolkit&proj=us)         **Toolkit
configuration, deployment and support**

PACMAN news from Saul Youssef (Boston U.):

Continued to Develop Pacman 2 with new features necessary for ATLAS
DC's, SuperComputing 2002 demos and in anticipation of future needs.

Supervised students in using Grappa to help in software development

and testing of the testbed. Continued development of Pacman, worked

with Alain Roy in Wisconson and CMS to re-pacmanize VDT and pacmanize

CMS. Coordinated installation and software upgrades on the US Atlas

testbed.

WBS 2.3.2.3.2    **Grid Workflow Management**

[2.3.2.3.2.3](http://atlassw1.phy.bnl.gov/XProject/servlet/XProject?wbs=/us/facilities/dit/sw/gwf/sched&proj=us)            Distributed
Scheduling

Grappa work from D. Engh U. of Chicago:

the jul-aug quarter more significant improvements in Grappa:

1\) development of a script based job submission framework allowing

interactive and automatic (eg cron based) job scheduling.

2\) addition of monitoring tools allowing interactive viewing of job
info

including diagnostic physics plots, metadata, logfiles and error
messages.

3\) development of US-EDG inter-grid compatibility

4\) initial discussions and tests for a magda web service allowing

development of magda/grappa inter-functionality.

Evaluating the Capabilities of GRAPPA:

The month of August was devoted almost entirely to evaluating the
current

Athena capabilities available through the GRAPPA web-portal. Jerry
Gieraltowski created a

user\'s guide for the existing Athena capabilties:

http://www.hep.anl.gov/grappa/athena

This provided an easy user\'s guide for members of the US-ATLAS grid to

execute

Athena \"atlfast\" and \"atlsim\" jobs using the GRAPPA web-portal.
Jerry worked

closely with Dan Engh (U of Chicago) to debug various flavors of

executing atlfast and atlsim (i.e., locally, boxed and transmitted to a

remote end for execution, or via the afs network). He also worked
closely

with Pawel Nevski of BNL to debug various releases of the atlsim
software

executing in the GRAPPA environment.

Inter-Grid Connectivity:

In early September, the GRAPPA web-portal was used to sucessfully submit

an Athena fast simulation (atlfast) job to a site in the US-ATLAS grid

and, in parallel, to a site in the EDG Testbed1 Production grid. The job

generated 100 events. In the US-ATLAS Grid, the job was executed at

Boston University on a Linux, RedHat 7.2 system. In the EDG Grid, the
job

was first submitted to a I/Gatekeeper server at Argonne, routed to the

CERN Resource Broker and subsequently dispatched by that RB to the CERN

compute resource (lxshare0339.cern.ch) for execution. GRAPPA was able to

monitor the status of both jobs and successfully

returned all output to a destination specified by the user.

Further architectural changes were needed to allow GRAPPA to
successfully

submit boxed jobs to US and EDG nodes running the Condor job scheduler.

These changes were jointly created and tested with Dan Engh (Univ. of

Chicago) and Shava Smallen (Univ of Indiana).T

An comparison of the grid functionalities of the US-ATLAS, EDG, and

NorduGrid grid networks was performed to assess the possibility of

inter-grid connectivity. A report of my findings was presented by John

Huth at the ATLAS Software Week in late September. The report can be

found at:

http://www.hep.anl.gov/gfg/us-edg-interconnect/jerryg-intergrid-comparison.ppt

Members of the US ATLAS grid community

have been working closely with Flavia Donno and others from various grid
projects

to define inter-grid connectivity requirements and ensure that

they are implemented in the US-ATLAS, US-CMS, and DataTAG servers.

[2.3.2.3.3](http://atlassw1.phy.bnl.gov/XProject/servlet/XProject?wbs=/us/facilities/dit/sw/int&proj=us)         **Generic
interfacing to Mass Stores**

Magda:

During this period of time, Magda was tried out for the first time in
the

testbed production for the Atlas Data Challenge 1 (DC1), and was proved
to

be a useful and feasible tool. \'globus-url-copy\' and
\'globus-job-run\' were

integrated into magda_putfile. magda_putfile can be used to do
third-party

transfer, put files to BNL HPSS directly, and register files to the
Magda

database. \'globus-url-copy\' was also used in the bulk data replication

between BNL and CERN. Another trans-Atlantic transfer engine used is

\'bbftp\', which worked very well also. A rate of 300 GBytes/day was

observed from BNL HPSS to CERN castor, and about 4 TBytes data had been

copied for the Atlas DC1. The file spider worked diligently as usual.
35K

DC1 files are available through Magda. Magda servers were moved and
split

in this period. The database now is on \'magda.usatlas.bnl.gov\', which
is a

dedicated MySQL server. Significant performance improvement had been

reported by our user Kaushik De - no more mysql timeout error occurred
to

him after the transition. The main Magda web site now is

http://www.atlasgrid.bnl.gov/magda/dyShowMain.pl The command
magda_getfile

was developed and gridified. It does file copying economically: it
figures

out which file instances to retrieve, and fetches remote instances only
if

it is necessary. magda_putfile automatically determines which remote
host

to contact if the remotehost is not provided as an argument and it is

needed; if it is provided, use the provided. Host as a means to access

storage site, should be hidden from users if possible. A new attribute
was

added to the Magda location. The file spider checks that attribute to

decide whether or not it needs to crawl a location. This was found to be

very useful. In Magda, for a logical file, only one prime instance is

allowed, while multiple replica instances can be distributed at
different

locations. If two primary instances exists for some reasons, the file

spider can be disabled to scan one of the locations until problems are

reported and fixed. The interfaces for the deleting of file records had

been developed both on the web and on command line. A protection of
empty

input from the web form was added. The metainfo proposed by the Grappa

developers was saved to the Magda database, can be retrieved with file

name from both the web and the command line. The integration of Magda
with

GDMP was tried in Milan: \'gdmp_register_local_file\' and

\'gdmp_publish_catalogue\' were put into magda_putfile; it seems working
as

expected. The strategy of managing files which are distributed on local

disk of each individual node of a Linux farm, was discussed and will be

implemented soon. A farm can be seen a special \'Magda\' site, and the

location associated with it will be a directory path preceded by the
host

name of the node. The Magda web services were implemented with perl

SOAP::Lite module. They can be invoked to query the database and
register

files. With web services, Magda provides one more option to interact
with

the database. Magda commands are available by default on BNL Linux farm.

Shared disk cache is supported: users could share the same instance if

they need the same file. This should reduce the traffic with the mass

storage. We finally got a workable machine with root privilege. The

installation of software has been started to do the RLS test. A poster

about Magda and live demonstrations of file transferring with Magda are
in

preparation for SC2002 demo. The star experiment has Magda servers
running

and is evaluating it.

### WBS 2.3.2.3.4 Grid Monitoring

This effort is being led by Dantong Yu of Brookhaven National
Laboratory.

Initial steps in organizing a monitoring effort were taken during this
period. U.S. ATLAS participates in the joint PPDG/GriPhyN effort for
Grid monitoring. Use cases and requirements for a cross-experiment
testbed were developed and collected. Work now focuses on developing
facilities monitors and MDS information providers.

1\. Work on PPDG/SC2002 Grid Monitoring Project.

a\. Deployed US-ATLAS Map Center for SC2002 ATLAS demo. Grid Map Center
has been designed to logically and graphically represent all elements,
applications and services running over grids. It polls grid entities and
services (GridFtp, MDS, Gatekeeper), check their status and builds
aggregated views of difference types of grid entities.

b\. Modify US-ATLAS Map Center for the need of SC2002. The Web site can
be found at:

http://www.atlasgrid.bnl.gov/mapcenter/

c\. Testing GLUE Schema.

### WBS 2.3.2.4 Testbeds

The testbed group continues to hold weekly teleconference

coordination meetings.

A face-to-face meeting of the testbed participants occurred for one

day of the 3 day meeting \"US ATLAS Grid Workshop\" at LBNL July 29-31,

2002\. See http://annwm.lbl.gov/henp/meet/usatlas_grid_jul02/agenda.html

for presentations. A principal focus was a review of the results of

the 1st USAtlas testbed run of Atlas Athena running in production mode

with GRATS. Also readiness of testbed for SC2002. Informal discussion of

having USAtlas Testbed participate in the ATLAS DC1 part 1. Other days

of this meeting were devoted to coordinating grid software with ATLAS

core software and software preparation for SC 2002.

During September, 4 of the 8 sites were used to run (successfully) the

DC1 part1 production codes and contributed significantly to the overall

US effort in DC1 part 1. Results were reported at the Atlas Software

Workshop Sept 17-20 at UK.

Significant upgrades and hardware moves will be made at PDSF and Indiana
U

during Sept.

The testbed continues to host activities associated with the

R&D projects: Atlas Grid, PPDG, Griphyn, and iVDGL.

### WBS 2.3.2.5 Wide Area Network Integration

Presentation on networking at ICFA-SCIC meeting at CERN.

Discontinued network monitoring using Perl scripts. Exploring

alternatives: Ganglia, Nagios and Network Weather Service (NWS).

Network \"monitor and beacon\" boxes being prototyped at Michigan to be

used to provide active \"on-demand\" network test results as well as

passive monitoring to develop baseline network performance metrics.

IXIA network test and monitoring equipment grants to Michigan, Oklahoma

and UTA. Plans for WAN and LAN testing using the boxes.

HENP WG goal pages on the WWW are being developed as a resource for

network issues.

### WBS 2.3.2.7, 2.3.2.8 Prototype Tier 2 Centers

Boston University. Most of our effort during this period was in
deploying the software needed for ATLAS data challenges and the upcoming
Supercomputing 2002 demo. Soul Youssef is managing the work for the
upcoming SC02 demos which include interoperating grid middleware on the
US testbed and European DataTag sites. A large amount of effort during
this period went into establishing and interoperable grid environment
between US ATLAS testbed sites and European DataTag sites. We continue
to delay buying new hardware until it is needed for the participation in
DC2 of ATLAS scheduled for early 2003. We do make use of existing
hardware of the university/s Scientific Computing and Visualization
department to run ATLAS testbed production and test software
development.

Indiana University.

\- Installed Ganglia to monitor site.

\- Updated pacman.

\- Installed grappa demo using pacman.

\- Got trouble ticket system installed.

\- Had \"all-hands\" meeting with computer center personnel doing work
related to

the center.

## 7. Financial Report (Chuck Butehorn, BNL)