**Answers to CAP questions 11/13/2023**

**Grid:**

1.  List people working in this area, and identify their sources of
    funding (to help understand cost/benefit of grid involvement)

People working at the Tier 1 for ATLAS on Grid related activities other
than production running or network engineering are: (this is all Grid
work not limited to OSG)

50% of Dantong Yu

100% of Gabriele Carcassi (PPDG)

100% of John Hover

100% of Jason Smith

50% of Zhenping Liu

50% of Xin Zhao

25% of Yingzi Wu

50% of Razvan Popescu

25% of Bruce Gibbard

This gives a total of \~5.5 FTE's of which only 1 FTE comes from PPDG
and the rest is cover by ATLAS program funds. Note that this number is
greater than you would guess from looking at the WBS item 2.3.1.5 Wide
Area Services in the table below because it includes contributions from
the management WBS line and also describes the current situation with
11.5 heads rather than the 10.5 FTE average of 2005, where the
additional 1 is doing Grid work.

Outside of T1:

100% Marco Mambelli (iVDGL)

75% Nucan Ozturk (UTA, core)

100% Mark Sosebee (RP)

50% of Research Time K. De (UTA core)

100% R. Gardner (U Chicago, core)

50% S. Youssef (RP, T2 funds)

30% S. McKee (Core)

30% H. Severini (OU Core)

100% G. Gieraltowski (ANL RP)

Clearly all of our sites, Tier 2's and Tier 3's, of which there are
perhaps 10-12 have someone doing Grid system admin and middleware
installation perhaps typically at the ½ FTE level. For the official Tier
2's this may be out of either RP or iVDGL funds, but is transitioning to
all RP funds, while for the Tier 3's it is out of base program funds.

**Management:**

2.  To what extent does management believe that software tasks are well
    defined so that manpower estimates are well understood? What
    additional work needs to be done in WBS elaboration? Please provide
    "typical" documentation of software task definitions?

We think the tasks are very well defined. We are sometimes shifting
priorities, putting manpower on different tasks accordingly. We think we
need this flexibility to react to overall ATLAS shortfalls and to make
sure we are answering the needs of U.S. ATLAS physicists.

One example of software task definitions is what we have for athena in
[savannah](https://savannah.cern.ch/task/index.php?go_report=Apply&group=athena&func=browse&set=custom&msort=0&report_id=180&advsrch=0&category_id=0&assigned_to=0&status_id=1&resolution_id=0&history_search=0&history_field=0&history_event=modified&history_date_dayfd=6)
.

We also have a WBS dictionary which is being entered into the Access db,
here is an excerpt:

  ------------- --------------------- -----------------------------------------
  **2.2.2.3**   **Detector            Detector description includes the design
                Description** (J.     and implementation of a system through
                Boudreau)             which material geometry, readout
                                      geometry, and alignment information is
                                      accessed throughout all Atlas
                                      applications that are concerned with
                                      geometry. The implementation of indi

  2.2.2.3.1     Detector Element      ATLAS detector elements are known to the
                Identifiers           detector description software by unique
                                      identifiers. This item covers the design
                                      and development of the software
                                      supporting detector element IDs.

  2.2.2.3.2     Atlas Geometry Tool   Main infrastructure package of geometry
                                      description in Atlas. It is a data layer
                                      designed for low memory consumption
                                      through shared instancing, compressed
                                      transformations, an embedded symbolic
                                      language for parametrizing
                                      transformations, and boolean shape

  2.2.2.3.3     Geometry Browser      An interactive 3D display for the
                                      geometry, the Browser displays precisely
                                      the same transient representation that
                                      simulation and reconstruction will use.
                                      It builds and displays detail upon demand
                                      for memory optimization. The browser is
                                      an application whi

  2.2.2.3.4     DB support            The database infrastructure is not the
                                      responsibility of the detector
                                      description team, but the application of
                                      this infrastructure to the geometry
                                      description is its responsibility. This
                                      item covers both time-independent and\
                                      time-dependent geometry infor

  2.2.2.3.5     Geom. versioning &    A mechanism must be put in place to
                configiration         enable the configuration of geometry.
                                      User-driven configuration information
                                      must be read in order to configure the
                                      geometry, and then in addition passed
                                      persistently from one executable to the
                                      next for consistency.

  2.2.2.3.6     Integration suport    Clients of the geometry kernel include
                                      all of the subsystems engineers who
                                      describe their geometries in terms of
                                      geometrical primitives, and also to
                                      simulation and reconstruction people who
                                      use these geometries within their
                                      respective physics projects. A

                                      
  ------------- --------------------- -----------------------------------------

We will post the whole dictionary on the cap review web site.

The software wbs has just been updated (Dec 2005). The facilities has
been kept up to date throughout the duration of the research program.
The one that needs the most work and is getting our attention now is the
Grid Tools and Services area. We are probably going to rename this work
area and are aligning the work with our PanDA development and the
emerging OSG program of work.

Software:

3.  What would the PanDA competitors present as their argument for not
    using PanDA? How would you respond to those?

First a comment:

The coffee conversations about Panda with people like the ATLAS
computing coordinator, CERN ATLAS computing group, ATLAS distributed
analysis coordinator have been supportive of Panda, even \"can you
install Panda here at CERN on LCG\", more than \"why Panda\".

Specifically:

Panda is still under early stages of development and has not been
released for general use. There is interest among many ATLAS communities
to try Panda. We will release Panda in the next few months, when we have
a fully debugged production quality system. We expect wide spread
adoption when we demonstrate how automated and easy production with
Panda can be.

On the question:

\"It\'s a dilution of scarce manpower we have for ATLAS prodsys

development, people should be working together.\"

Ans: We agree. We advocated an architecture that greatly increases the
commonality in the prodsys software across the three grids \-- by
\'deepening\' the presently thin \'supervisor\'

layer at the expense of thick grid-specific \'executor\' layers\-- and
so uses manpower more efficiently; Panda is our realization of this
architecture. We should all be working on an architecture that uses our
resources most efficiently. Panda also puts heavy emphasis on automation
and low operations manpower requirements with the objective of saving
substantial manpower among both developers and shift takers. Finally,
Panda has contributed enormously to prodsys development through its
early adoption of DQ2: all other executors are learning from the Panda
experience of the past 3 months.

\"It fails to leverage middleware, eg. resource broker, that a great
deal of effort has been put into.\"

Ans: Resource broker middleware has a track record in production ATLAS
usage, and a bad one. Furthermore, RB middleware does not readily
support data driven, dataset based

workflow organization. With a tiny fraction of the RB middleware effort,
we have implemented in Panda the brokerage infrastructure that ATLAS
needs; flexible, easy to extend/tune/optimize, and fully integrated with
ATLAS DDM.

\"It\'s grid specific. We aren\'t using OSG.\"

Ans: It isn\'t grid specific. It\'s specifically designed to make
deployment spanning multiple \'pilot job management mechanisms\' \--
different grids, job submission middleware, batch systems, \... \--
easy, by encapsulating these environment specifics in the scheduler
component.

\"It hasn\'t been demonstrated to be any better than what we have.\"

Ans: Touché! Give us another month or so, it\'s the product of only 3-4
months development, and its development has involved a substantial
amount of DQ2 debugging and tuning for production that ATLAS prodsys
hasn\'t seen because ATLAS prodsys isn\'t using DQ2 yet.

4.  If move to DQ2 is delayed within international Atlas by a year after
    startup, what will the US do?

First a point not made clearly in the presentation: International ATLAS
\_is\_ using DQ2, it is the production system that isn\'t. In particular
you heard about the SC3 T0 test, exercising the newly developed Tier0
production management system, called T0MS; DQ2 is fully integrated with
T0MS, is established as the basis for Tier 0 data management and T0-T1
data movement, and was very successful in the SC3 tests. The prodsys DQ2
usage issue really is a localized issue arising primarily from the fact
that in order to use DQ2, the ATLAS prodsys software coordinator has to
make significant changes to the production executor he developed for DC2
and is now using in the new prodsys, and he\'s been in no rush to do
that. This is utterly contrary to an ATLAS policy to use DQ2 that has
been expressed not just in words but in the full adoption of DQ2 for
T0MS, with the next step for SC4 being to incorporate also DQ2-based
online-\>Tier 0 data movement.

But on the specific question, if international ATLAS is not using DQ2 in
the production system by SC4 (May), the only ATLAS production system
compatible with the DQ2-managed data from T0MS and commissioning will be
Panda, and what the US will do is probably succeed in replacing the
present ATLAS prodsys with Panda.

However as Ian suggested today, the present DDM incompatibility is
likely to come to a head quickly. Despite the fact that ATLAS has
developed an ATLAS-wide, grid-neutral DDM system, we have today the
untenable situation of LCG-produced data not being readily accessible in
the US and vice versa.

If ATLAS abandoned DQ2 we would abandon it too, but there\'s no sign of
that; quite the reverse, with T0MS, ATLAS commissioning/TDAQ, file-based
conditions data management, Panda, NorduGrid, ATLAS operations, all
signed on to DQ2 and at various stages of adopting/integrating it.

5.  What is the set of s/w components that you see are the preferred set
    for building the US physicists analysis environment? How do these
    components fit together into a coherent environment? Why will users
    use this instead of building their own N-tuples?

In addition to the AOD which has been developed in the last 18 months,
the ATLAS Physics Analysis Tools group has been developing a number of
other components to the analysis framework and a coherent \"analysis
model\".  The most important aspects of these developments are the
EventView and \"Athena Aware Ntuples\".  The EventView is an analysis
helper class which facilitates the sharing of analysis code based on the
AOD and the creation of small, customized, analysis-specific ntuples
(known as Derived Physics Data in the Computing Model).  The EventView
class and these small ntuples can be accessed in a native Root
environment and have Pool references to their parent AOD files (ie. they
are \"athena aware\").  This gives users the fast interactive access
they like in Root, while maintaining consistency with the rest of the
Atlas computing model.  By following this path, we aim to improve the
analysis environment in Athena and allow for code to be easily migrated
to and from Root and to be shared amongst a large number of clients.\
\
In addition, the development of a pyroot based c++ interoperability
layer will allow clients to call athena based C++ algorithms to be
invoked from python thus allowing use of ROOT functinalities while
maintaining the algorithms and data structures in an athena based
environment.

Facilities:

6.  What impact will the 30% overhead at BNL have on the Tier 1 in FY08
    and beyond?

If there were an addition of 30% to the cost of each FTE in the out
years, 2008 and beyond, and that loss was absorbed into staffing only it
would reduce the Tier 1 staff from 20 FTE's to 16 FTE's. If that loss, a
total of approximately \$900k were instead distributed uniformly across
the facility it would reduce the staff to 18 FTE's and the overall
facility capacity by approximately 15%. If the full central ATLAS
capacity commitment were met this would reduce the US ATLAS specific
part of the facility capacity by nearly 50%.

If this 30% increase in labor cost were to occur the facility would
appeal for management reserve to cover at least part of it and
management would have to decide how to respond but would presumably at
some level relieve the worst case scenarios described above.

7.  Please provide the FTE breakdown for the existing 10 people at the
    tier 1 (where is effort going)?

  --------- ------------ ---------------------------------------------------
            **FY \'05**   

  **WBS**   **FTE\'s**   **Description**

  2.3.1     10.5          

  2.3.1.1   2.0          Mgmt/Admin

  2.3.1.2   1.0          Tier 1 Fabric Infrastructure

  2.3.1.3   0.9          Tier 1 Linux Systems

  2.3.1.4   1.0          Tier 1 Storage Systems (Excluding dCache)

  2.3.1.5   4.0          Tier 1 Wide Area Services (Grid & Network plus
                         dCache)

  2.3.1.6   1.6          Tier 1 Operations (User and production support)
  --------- ------------ ---------------------------------------------------

8.  What software development or procedural plans do you have to
    maximize tape bandwidth utilization?

Past experience with HPSS in the context of RHIC experiments revealed
two main sources of tape access inefficiencies:

-   Excessive time spent preparing for access (mounting/dismounting
    volumes and positioning inside the volume) *due to reduced ratio of
    accessed data volume per total volume capacity*. A very random
    pattern of file accesses relative to the file layout over tape
    cartridges may generate the need to mount and dismount the same
    volume in order to read multiple files co-located in the specified
    volume. To a lesser extend, inefficiencies exists when access to
    multiple files is not ordered following a monotonic function of file
    position, generating repeated rewinding and fast-forwarding of the
    tape. Another source of inefficiency is overall reduced volume of
    accessed data relative to the cartridge volume.

-   Slow read but mostly slow write performance due to fragmentation of
    file structure inside tape volumes by small file sizes.

In order to improve the overall tape bandwidth by optimizing the read
access patterns we employ a dedicated service that acts as buffer before
the mass storage system, collecting the read requests, queuing and
reordering them so tapes are mounted once for reading all relevant files
in the given volume. dCache for both RHIC and USATLAS access HPSS via
this optimizer only. The reordering algorithm considers also the inside
volume location sequence so unnecessary rewinds are completely
eliminated. In order to improve the ratio of access data to total volume
cartridge we could only make sure that the optimizer is capable to
buffer as many request as possible before a single cartridge content is
processed. The rest is controlled by the data requests pattern.

The ATLAS file management software (DQ2) organizes files in groups
(\'data blocks\') in an automated way for managed production and
analysis, and will offer datablock based organization to end users.
Blocks can be divided into sub-blocks to support different granularities
needed in different contexts. We have a mechanism to zip files within a
block or sub-block together into one file, thus giving us flexibility in
the file sizes we actually write to tape.

Non US Atlas:

9.  What are the risks of shortfalls within international Atlas software
    efforts that you are most concerned about? Which of these most
    effects CSC?

\- Distributed production aggregate rate needs to scale from \~1k jobs 
per day now to \~100k by end of 2006 to \~1000k by end of 2007. By any
measure that\'s a large scale up. Needs adequate monitoring, diagnostic
& error recovery tools as well as low failure rate & short  recovery
time from failure.\
\
- Transient memory needs for calibration & alignment are not
well estimated. Magnetic field map will certainly be a large part of
this,  but preliminary estimate is \~500MB. Current reco jobs require 
\~1000-1200MB so we need to take a close look at our overall usage 
(problem isn\'t memory leaks per se, since these are solvable using 
tools, but overall memory footprint.\
\
\
-AMI, the long term ATLAS Meta-data Interface for providing meta-data
information for physicists and other software clients such as the
production system is under-developed. It is not in a mature state for
use with the production system today. The ATLAS operations group has
therefore decided to adopt a short term solution using mySQL to provide
the required meta-data information. PANDA is closely working with this
solution. However, serious concerns remain on the viability of AMI as
the solution for ATLAS.

\- Simulation cpu time too large per event, even based on relatively
small fraction of events to be simulated. Effort needed in
shower parameterization.\
\
- Schedule for closed-loop calibration & alignment is aggressive.  Needs
careful attention from detector subsystems. Similarly for  migrating all
conditions data to COOL.\
\
- General transition from development to production. Operations needs  a
solid core of \"permanent\" staff with shift crew. Priority for the
shift  crew needs to be at same level as for online, and need to be put
into  place earlier.\
\
- Planning for full system integration is still incomplete. Need 
dedicated integration periods during overall ATLAS commissioning 
schedule.