**Management sub-group**

1.  What are the plans to test scalability of the DDM?

-   Jan : ATLAS AODs replication

    -   All MC Production AODs files will be replicated to all Tier-1s
        sites. For US ATLAS cloud we will replicate data not only to
        BNL, but also to all Tier-2s

```{=html}
<!-- -->
```
-   Files Catalogues stress tests

    -   Check performance and robustness of File Replicas Catalogs used
        by the experiment (the tests are in progress and they are
        conducted in collaboration with SW developers)

-   Feb : Tests of DDM central datasets catalogs (ORACLE implementation
    and new schema).

    -   Set up DDM test-bed at CERN (already exists for US ATLAS)

-   Feb-May : Regular DDM functional and performance tests

    -   All ATLAS sites. Data distribution functional tests to check new
        version of DDM catalogs.

2.  What is the plan for continuing to provide for analysis in the event
    of US Tier 1 unavailability?

    -   Up to 8 hours: no serious problems

    -   From many days to 1 week or more: we overflow buffers at CERN
        for real data and at T2's for simul. Data.

Depending on the period of unavailability there are various concerns and
differing appropriate courses of action.

If the Tier 1 becomes unavailable for a period of order 8 hours as was
the case during the cyber security stand-down at BNL back at the
beginning of November or even long say 1-2 days, the impact is modest.
The Tier 2's have adequate local disk storage to hold the Monte Carlo
they are generating until the Tier 1 is again available. They also have
local copies of those AOD, TAG, PDP, and ESD sets of most current
interest and so can continue most analyses. The Tier 0 at CERN also has
adequate disk storage to buffer the raw and ESD data destined for BNL
until it is again available.

If the Tier 1 becomes unavailable for a period of many days, a week or
more the problem becomes very serious on many fronts. The Tier 2's can
of course send the Monte Carlo data the are producing to other Tier 1's
for storage and it is also within the capabilities of the distributed
data management system to allow the US Tier 2's to subscribe to receive
the data they need for analysis, AOD, TAG, etc. from other Tier 1's.
More of a problem is the fact that the US Tier 1 is expected to be
storing and then reconstructing 25% of the ATLAS RAW data. CERN only has
adequate disk buffering to hold the data destine for the BNL Tier 1 for
a couple of days. All contingency planning within WLCG for Tier 1
unavailability has assumed down periods of order 24 hours not a week or
more. Given its size, if the US Tier 1 were to go offline for a period
of weeks, it would produce a major disruption to the whole of ATLAS
computing, conceivably even data taking.

At the BNL Tier 1 great effort is being made to assure its cyber
security, even in the eventuality of incidents elsewhere at BNL. To data
this effort has been very effective. The real answer to the above it
that there is need for adequate push back against punitive cyber
security actions to the effect that the services of the US Tier 1's are
scientifically very important and an integral part of international
agreements and so can not be disrupted except in the case of actual
technical justification. This push back needs to come from the
scientific community, its program management and its agency oversight
and be directed at over reaching cyber security establishments at the
institutional or agency levels.

3.  What are the roles and expectations from the tier 3s? How do you
    support the overhead to make Tier 3 resources effective?

Tier 3 centers provide local users with facilities to develop analysis
algorithms plots for publication.

One of the definitions of a T3 in US ATLAS is that it is NOT funded from
the RP. We expect some effort form RP funded people to help setup T3's.
As mentioned this is easiest if the form of that help is to install the
OSG software stack. We think this will be a minor additional workload.
Most support effort for tier 3's will come from the local group it
serves.

4.  What would it take to complete the CDC tasks by August, 2007? What
    are the three critical dates/milestones for the CDC?

CDC has several components and stages. The first stage is to test the
basic conditions database infrastructure, both in terms of deployment
and for access by reconstruction algorithms to both static and time
varying conditions and geometry information. Later stages are to
exercise this in the context of the 24-hour latency that\'s part of the
computing model before the Tier-0 processing of raw data begins.\
\
The first stage has already been performed for all detector subsystems,
although some will not be able to handle time varying information until
release 13 in March. However, all have already demonstrated the ability
to reconstruct using a different statically misaligned geometry from
that which was used to simulate the data.\
\
The major deliverables for release 13 are:\
\
o Complete migration to COOL for conditions DB\
o Completion of support for time-varying rather than static
misalignments\
\
The Tier-0 infrastructure already implements the 24-hour delay and
handling of the calibration processing, albeit today with dummy
algorithms. It is intended that the Tier-0 test scheduled for May will
exercise the real calibration processing infrastructure.

5.  What issues have been uncovered in any end to end tests, such as the
    proposed cosmic test from 2006?

The primary feedback from various standalone or combined cosmics tests
that took place during 2006 and are ongoing in 2007 have been:

-   Exercising the geometry database access with real detector hardware

-   Exercising the conditions (calibration, cabling maps and alignment)
    database infrastructure with real detector-derived information

-   Exercising the event data model from bytestream raw data through to
    ESD/AOD

-   Exercising the detector monitoring and reconstruction algorithms

-   Performing combined monitoring and reconstruction

-   All experience has been fed back into the primary software
    development.

6.  Can you please clarify the milestones and impact of the full dress
    rehearsal on other deliverables?

Because the full dress rehearsal is really a demonstration of full-chain
readiness for data-taking, it imposes few requirements that would not
need to be met for 2008 running in any case. The main risk is that it
could, because of its summer 2007 time frame, push development,
deployment, and testing of functionality not needed for 900 GeV running
ahead of functionality that \_is\_ needed for 900 GeV running, in a
period of already-intense development pressure.  With careful scoping of
the dress rehearsal, this risk can be mitigated. The current strategy of
expanding the ongoing Data Streaming Test to adiabatically meet the
goals of the final dress rehearsal is designed to address these concerns
while still meeting the FDR goals.

The culminating milestone of the dress rehearsal is successful
completion by late summer 2007 of the exercise described in Srini\'s
talk.  Intermediate milestones are the subject of a planning
meeting that will take place on 24 January.  

7.  What are the mechanisms and plans for getting long-term ATLAS
    support for PANDA?

There currently are 3-5 people working on support for the other 2 grid
flavors. As the production system moves over to all PanDA worldwide, we
expect to have these people available to support the ATLAS production
system.

8.  Are the major issues for experiment readiness being sufficiently
    addressed by ATLAS? What mechanisms are used to reassess
    requirements for experiment readiness? Are they effective and
    efficient?

    -   Yes. The SPMB(with broad rep.) and the CMB effectively
        accomplish this.

The major issues for readiness are discussed at either the Software
Project Management Board (SPMB) or Computing Management Board Meetings
(CMB) which are held in alternating weekly meetings. The SPMB, which has
representatives from every major body in ATLAS - such as core software,
databases, sub-systems, trigger, etc., focuses on software deliverables
for major production releases which are driven by major milestones (e.g.
combined cosmics tests, final dress rehearsal, calibration data
challenge, ATLAS low energy running). Each group prepares a list of
milestones for the next production release which are followed to ensure
progress and therefore readiness for the data taking. The CMB focuses on
the global computing model and deployment and operations aspects of
computing.

**Facilities & Grids sub-group**

1.  Quantitatively clarify the effect of the LHC startup delay on the
    T1/T2 facility ramp up. What are the changes in the T1/T2 plans?

The first two tables below describe roughly the incremental cost for the
BNL Tier 1 in adapting to meet the new ATLAS estimate of requirements
first in absolute dollars and then in per cent change. The second set of
two tables attempt to remove most of the effects aside from the actually
running schedule change. You will see that with all effects taken into
account there are small savings in 2007 and 2008 and significant
additional costs in the out years. If only the schedule change is
considered the cost savings are quite substantial in all years except
2010.

![](media/image1.wmf)

![](media/image2.wmf)

2.  The T2 profile is over-target in 2007 and under-target after 2009.
    What funding profile is needed to meet projected resource targets
    (Bruce's slide 10)? What are the expectations that leveraged
    resources will meet this target?

Assuming that of the \$3,000k per year going to Tier 2's approximately
\$600k of it is going into personnel costs (6 FTE's) leaving the
remaining \$2,400k for equipment, it would require \~doubling this
equipment level to produce the target capacities over the three years
2009, 2010, 2011. So the total Tier 2 cost would go in 2009 from its
current \$3.0M/Yr to \$5.4M/Yr to achieve the target capacities. This
assumes the leveraged dedicated capacity would scale with equipment
expenditure. It does not take account of non-dedicated capacities which
might be significant but whose actual level can probably not be
accurately estimated until the first couple of years of actual high
demand operation**.**

3.  What validation process is planned to verify analysis capabilities
    at T2s in advance of data taking?

Validation will be through deployment and operation of Panda/pathena
based analysis at T2s well in advance of data taking. Principal issue is
deployment of AODs to the Tier 2s, which is starting now with UTA\
and will be extended to all Tier 2s over the next \~2 months; once AODs
are at each site, pathena will be activated there.

4.  What is the fraction of pathena (grid-submitted) analysis vs. local
    analysis?

This information is not immediately available. The following can be
said. At the Tier 1 in November, 90,000 PanDA production jobs were run,
27,000 PanDA analysis jobs were run, and 70,000 other ATLAS jobs were
run. It is not possible to determine from the current accounting summary
what fraction of these "other" ATLAS jobs were analysis as opposed to
simulation. Note that this is accounting by job not CPU or wall clock.
Such accounting seems appropriate for typically short running analysis
jobs. It appears that all that can be said is that \~1/3 and perhaps
more likely 1/2 of the analysis jobs being run are using pathena.

5.  How certain is the expectation that event size (ESD & AOD) can be
    reduced sufficiently? What is the plan if event size (ESD & AOD)
    stays much larger than computing model predicted sizes? When will
    the decision be taken to synchronize the model and reality? What are
    the current sizes?

ESD and AOD have expanded in size and in content because physicists have
argued vigorously that they need this information to set up the
experiment. The role of software experts is to help physicists increase
the data content/size ratio (as it has been done with the
transient/persistent separation), to provide tools that will help them
making the necessary hard choices by measuring and monitoring data sizes
down to the data member level, and to limit the impact of this hard
choices on the algorithmic by promoting wherever possible the use of a
common interface between ESD and AOD data objects.

Current ESD size is \~1.5MB (target steady-state size 0.5MB) and current
AOD size is \~250KB of which 40% is MC truth(target size 100KB).

6.  What are the current comparisons between empirical CPU usage and
    computing model predictions?

The following two plots are taken from the WLCG accounting summary as of
the end of November. This information is in the context of Tier 1's
only. There is currently no such comprehensive accounting for Tier 2's
being done by the WLCG. They are first a summary for BNL and then a
summary for the entire WLCG (there is no ATLAS only plot). The blue line
shows the expected (MOU agreed) level of wall-clock time, the green line
shows the installed level and the bars should the actual usage level.
Usage levels can exceed installed capacity because of the possible use
of opportunistic resources and/or because of operation at an efficiency
greater than the assumed efficiency (85%). Usage significantly below the
installed levels can be a result of lack of demand from the experiments
or inefficient use of computing resources by the production management
system. You will see that the BNL Tier 1 has installed its MOU committed
capacity and in the Oct & Nov time frame utilized approximately that
full capacity. The overall WLCG does not quite but is near meeting its
MOU committed installed capacity but has to date only been utilizing
about 2/3's of that capacity.

![](media/image3.png){width="3.875in" height="2.71875in"}

![](media/image4.png){width="3.8958333333333335in"
height="2.7291666666666665in"}

7.  What motivates keeping both ATLAS\' AOD and DPD? Is this a change in
    computing model? How does this comport with previous experience of
    other experiments?

AOD and DPD have been part of the computing model since this was written
down in the Software and Computing TDR. At that time it was thought the
main difference among the two was access speed and direct browsability
of DPD from ROOT. While the gap in access speeds and ROOT functionality
has narrowed, the DPD/AOD difference in scope has grown:

we now expect that physicists will use AOD mainly for high statistics
systematics and reconstruction tuning work and for the more complex
analysis. DPD will be the choice of physicists doing simple
particle-level analysis work. Another important difference is that one
or few AOD will be centrally maintained and produced, while most physics
groups and possibly many individuals will define and produce their own
DPD.

8.  There is evidence that some ATLAS users still eschew use of the
    Grid, and that some ATLAS T2s can be underutilized without apparent
    explanation. How does this compare with your presented demo? How do
    users submit trouble tickets and/or convey these experiences?

> ATLAS has gone through a series of data production exercises -- DC1,
> DC2 CSC11, CSC12. With each exercise, it has become obvious that the
> required scale of data processing in ATLAS can only be achieved
> through the grid. Grid systems contributed \~20% of DC1 production,
> 100% of DC2 and 100% of CSC exercises. Only a small fraction of data
> processing in ATLAS is done outside these exercises, using non-grid
> resources. A few people still eschew the grid -- but they are a very
> small minority of users, with special use cases, or with access to
> large local resources for private use. For the vast majority of ATLAS
> users, the grid is the only choice for doing physics.
>
> The Tier2 utilization rate has been very high with Panda. We do not
> have accounting of idle CPU's from all sites, but the rate of
> production is well matched to available capacity. Exceptions occur due
> to scheduled downtime for upgrades, site problems, data movement
> problems, or availability of jobs. Unused CPU is rarely due to Panda
> problems -- only recent exception being a few days due to Panda
> Database problems.
>
> Consider for example, the following plot from SLAC T2. The red dots
> show average number of running jobs, which fluctuate considerably
> because of shared resources. The green dots show the number of Panda
> pilots waiting in queue. Note it is mostly constant at 50, the limit
> we set. As long as there is any green dot greater than zero in this
> plot, all available CPU at the site is being used. There were some
> exceptions during the holidays, when we had DQ2 problems:
>
> ![](media/image5.png){width="6.666666666666667in" height="5.0in"}
>
> Panda is still a relatively new system for users. Few people (\~75
> analysis users compared to \~1800 collaborators) have tried Panda.
> User feedback has been mostly positive. We have set up wiki pages, a
> Savannah bug reporting system, a very active listserve, and a RT
> trouble ticket system for Panda.

9.  Show Tier2 utilization for the last several months.

![](media/image6.wmf){width="5.997916666666667in"
height="3.454861111111111in"}

![](media/image7.png){width="4.114583333333333in"
height="1.53125in"}![](media/image8.png){width="4.114583333333333in"
height="1.53125in"}

**Software sub-group**

1.  The Full Dress Rehearsal seems unobtainable with currently allocated
    resources. Is this true of other important tests/milestones in
    2007/2008? What are the minimum acceptable required tests for
    success:

    -   with first beams in 2007?

    -   with the first physics run in 2008?

    -   IE. In triage-mode, what would be deferred on the current list
        of responsibilities?

Sufficient resources exist for the completion of other tests (listed
below) and the strategy for FDR s being re-worked to not overburden
these resources. These exercises must be completed by Summer 2007, do
not foresee any major exercises after the engineering run.

The other important tests/milestones in 2007 include:

\- The Computing System Commissioning

\- The Calibration Data Challenge

\- The integrated cosmic ray run

\- The TDAQ Large Scale Tests

\- 900 GeV engineering run.

All of them are ongoing and planned for completion by Summer 2007. Many
of the software infrastructure components (framework, conditions
database, schema evolution, I/O optimization, data replication) are
common issues to all of them. These are being addressed by dedicated
group of people (U.S. plays a major role in framework, Event Database
and production software) with all these exercises in mind. Their work on
this is not five-fold or even doubled because of several tests. The
actual tests are decouple in many cases - decouple implying that
different groups are addressing different tests. For example the cosmic
ray run is mostly handled by dedicated people within sub-systems, the
TDAQ LST by the trigger community, etc. Hence we believe that these
tests have reasonably equipped resources to achieve successful
completion.

As noted in the response to item 6 in the Management section, the
strategy for the final dress rehearsal is to adiabatically extend the
existing Data Streaming tests and to couple it to other tests such as
the Tier-0 scaling and distribution test scheduled for May 2007. We
believe this will minimize the disruption to the ongoing ATLAS
commissioning activities, while maximizing the usefulness of FDR.

The time between the 900 GeV run and the 14 TeV run is a critical phase
that will essentially be devoted to code optimizations and to rectify
many of the problems we will encounter during the 900 GeV run - from
pathological detector readouts to algorithm optimizations. It is not
possible to foresee dedicated large scale exercises to be carried out
during this critical phase. Smaller component tests could be carried out
to demonstrate resolution of problems that we would have encountered
during the 900 GeV running.

2.  Please clarify the effort required to establish the DPD and outline
    the boundaries of your responsibility. How do you leverage the
    experience gained in existing experiments?

Structured Athena-Aware Ntuples can be (and are) written out using the
current production release. We expect them to be used by physicists for
their physics analysis work on CSC Monte Carlo sample. The work which
remains to be done between now and this spring (release 14) is to
complete the Transient/Persistent separation of AOD data objects and
then to merge the classes used for the persistent representation of AOD
with the corresponding SAN ones.

Extensive input from other experiments was obtained by the Analysis
Model working group as can be seen by the agenda here:
<http://indico.cern.ch/conferenceDisplay.py?confId=6866> where talks
were given by experts from D0, CDF, BaBar, H1, CMS.

3.  Did you perform a requirements analysis for the development of
    vATLAS? If yes, what are the requirements? Did your resource
    estimate result from these? Demonstrate that 1 FTE is sufficient.

The requirements were drafted at a joint meeting between USATLAS and
ATLAS management and the outcome of that workshop was to establish 1 FTE
to provide the requested integrated framework view. Sub-system
non-experts will be able to provide "plug-ins" for their individual
sub-system views.

A meeting in Brookhaven in May 2006 included the Technical Coordination
Leader and representatives of USATLAS. The meeting identified specific
areas in which the existing 3D graphics were insufficient for the
purpose of a monitoring event display. The chief difficulties were the
simultaneous availability of several standardized detector

views, and the requirement of a high degree of coordination between two
dimensional and three dimensional views. A plan was developed to provide
the missing functionality. It entails additional work at the level of
the framework and the GUI (both operating within ATHENA). A three month
period was envisioned for the design phase (Boudreau and Kittelman).
About one month after that (4 months into the project) specific displays
for one or two detector subsystems are foreseen to be available. One
year into the project, good displays of all the major systems are
working within the monitoring framework. Two years into the project, the
subsystem-specific displays have been improved after exposure to a year
of actual detector operations.

The price of this extra functionality is the additional manpower that we
requested, which is tailored to the requirements requested by ATLAS
technical coordination. The system is foreseen to be open so that
contributions from nonexperts could be easy to sollicit and incorporate,
once the new framework was mature enough. However the framework itself
and a reasonable level of sophistication in the display can be assured
by a combination of Boudreau, Kittelman and Tsulaia.

4.  Will International ATLAS provide sufficient user support so that US
    ATLAS support will not be unfairly burdened? What is the quality of
    the user workbook? Is it sufficient to cover standard user
    questions?

Various UK groups have official responsibilities in this area (workbook,
tutorials) and they are putting quite some effort into user support
especially for what concerns beginners. The workbook is indeed a
valuable tool for beginners and together with the tutorials, goes a long
way into addressing typical beginners questions. US ATLAS leads  the
work in core software (framework and event store), and physics analysis
tools, hence US ATLAS developers tend to field many of the support
requests from advanced users or from other developers.

5.  What is the status of the calibrations database? Is it part of the
    US responsibilities?

The calibrations database is well advanced, and is being stress tested
in the calibration and alignment data challenge. The underlying temporal
database infrastructure comes from an LCG common project (COOL). 
Replicas of the content of the calibration database will be deployed in
the U.S., but the database itself is not a U.S. deliverable.

6.  Quantify job & software failure rates. How do they compare to the
    established targets? What are those targets?

> The requirements coming from physics (measurement of integrated
> luminosity) is that untraceable event losses due to software failure
> not exceed few percent (based on Tevatron experience). We are not
> there yet after every release -- however this rate is achieved after
> an extensive software validation process.
>
> Every software release undergoes validation, over many months. During
> this period, various bugs are fixed. The release is considered to be
> ready for production when the error rate reaches less than a few
> percent due to software errors. An intensive robustness and
> performance period is planned for mid 2007, aimed at reducing
> software-induced job failure rates, improving the cpu & memory
> performance, and reducing e.g. memory leaks. Although these are all
> ongoing activities, the intention is to make these the highest
> priority for a preiod of several months, even at the expense of
> improved functionality. Examples of decreasing failure rates after
> each validation release (for the Release 12 series) can be seen in the
> plot below (showing successful vs failed walltime):

![](media/image9.png){width="8.001388888888888in"
height="4.191666666666666in"}

7.  How long would it take to generate 10M dress-rehearsal events with
    the current system?

> Our current ATLAS-wide production rate is \~2M events per week. This
> includes a factor of two increase in rate achieved this past winter.
> At this rate, we can generate the dress-rehearsal sample in 5-6 weeks.
> However, we expect to achieve another increase of factor of two by
> summer 2007. If achieved, the dress-rehearsal sample can produced in
> 2-3 weeks.

**Q: \"What are US ATLAS\' dependencies on OSG and how is OSG bringing**

**added value to your Software and Computing program?\"**

US ATLAS depends crucially on the OSG infrastructure. All our facilities
have the OSG software stack as the base upon which we install the ATLAS
software system. In moving to the PanDA system over one year ago, we
eliminated our dependency on SOME OSG software components, limiting our
dependency to only those components that had been shown to be robust and
scalable. PanDA was designed to be modular enough to be able to plug in
software components from middleware projects, like the OSG, as these
components pass scalability tests.

We\'ve sought to minimize grid middleware dependencies in general for
well known reasons. Where we still have middleware dependencies, we
often still have problems. The OSG has been helpful in mitigating or
minimizing the problems. Examples:

-   GRAM dependency in CondorG submission of pilots. Not sufficiently
    scalable to support analysis in Panda. OSG program includes a
    \'pilot factory\' effort to work around this by doing site-local
    pilot submission without every pilot seeing the gatekeeper and GRAM.

-   glexec. This has been validated and deployed for production by
    FNAL/CMS. It will be added to the OSG software stack so we can pick
    it up easily for integration ourselves via the OSG stack.

-   OSG-standard site configuration. Provides a \'known\' environment on
    OSG WNs for more homogeneity in execution environment. Still a bit
    rough but will be more useful as it improves. This lessens the
    application-level work of establishing this homogeneity.

-   Tools for resource discovery. The tools we have are provided by OSG,
    and while they need improvement, we\'d be in much worse shape
    without them; it would be very difficult to gather the info on
    resource availability, health, and access rights that is required to
    run a distributed system on all the resources available.

> A new OSG component that we are particularly interested in is the
> Workflow Management System which we (T. Wenaus) are working on now
> with the OSG. This is an important piece of PanDA and we have a
> temporary solution now which will work when data first starts to flow
> from the LHC. Thus our dependency on this component is not critical,
> but we would really benefit from a common OSG developed solution. We
> are working to make this happen and will deploy it as soon as it is
> ready.
>
> Another area in OSG we are very interested in is the attempt being
> made in the OSG in the extensions area and elsewhere to have OSG \'add
> value\' in the area of storage management. For example, by taking a
> close look at dCache, code and performance, to look for ways to
> improve it; and by developing a monitoring system providing uniform
> grid-wide monitoring of dCache systems. At BNL we (ATLAS and STAR) are
> also interested in getting xrootd into the mix. The OSG may not have
> much impact on SRM 2.2 deployment timeline and quality, but even if it
> doesn\'t, there are other ways in which we\'re getting value out of
> OSG.
>
> We will also benefit from the new OSG-wide accounting services being
> introduced in the next release of OSG, as well as the information
> services that will provide OSG site information in both Condor ClassAd
> and ldiff (for LCG interoperability) formats.
>
> We benefit and rely on the VDT and OSG packaging, installation, and
> configuration processes that lead to a well-documented and easily
> deployable OSG software stack, and the OSG integration and validation
> processes that precede incorporation of new services into the VDT.
>
> We also benefit from and rely on the infrastructure maintenance
> aspects of the OSG such as the GOC and others that keep the OSG
> facility as a whole operational. We have PanDA running on ALL OSG
> sites (not just ATLAS) are poised to take advantage of opportunistic
> resources on the OSG to do further Monte Carlo simulation for ATLAS.

We should point out that the OSG \'dependencies\' could be a bit
misleading because the dependencies don\'t come from OSG, they come from
the grid middleware and facilities underneath, and the OSG is a layer in
between where the experiments work together and with the grid projects
to mitigate the problems and the risks coming from the layer below. The
signs so far are that that is working.