**DØ Computing and Software Operations and Plan**

**The DØ Collaboration**

**September 2005**

Abstract

This document reports on DØ Computing and Software operations as well as
the current evolution plan for the next several years. It updates DØNote
4616, produced for last year's Run II computing review. It includes
scope and cost estimates for hardware and software upgrades, with a
detailed equipment spending description for the next few years. In view
of the amended charge, particular discussion is devoted to the
methodology and resources necessary to ensure a successful completion of
Run II.

**Contents**

[1. Introduction 3](#introduction)

[1.1 Overview 3](#overview)

[2 Computing Needs 5](#computing-needs)

[2.1 Production Executables 5](#production-executables)

[2.2 Data Analysis 7](#data-analysis)

[3 Computing Systems 9](#computing-systems)

[3.1 Reconstruction Farm 9](#reconstruction-farm)

[3.2 Central Analysis Systems 10](#central-analysis-systems)

[3.3 Remote Computing 11](#remote-computing)

[3.4 Online Systems 16](#online-systems)

[4 Databases 19](#databases)

[4.1 Offline Calibration Database 19](#offline-calibration-database)

[4.2 Luminosity Database 19](#luminosity-database)

[4.3 Trigger Database 20](#trigger-database)

[5 Data Handling and Storage 22](#data-handling-and-storage)

[5.1 SAM Overview 22](#sam-overview)

[5.2 Robotic Storage 24](#robotic-storage)

[5.3 Physics Considerations 25](#physics-considerations)

[5.4 Cost Projections 26](#cost-projections)

[6 Networking 28](#networking)

[6.1 Infrastructure 28](#infrastructure)

[6.2 Utilization and Prospects 28](#utilization-and-prospects)

[7 Data Handling, Resource Management and Tools
29](#data-handling-resource-management-and-tools)

[7.1 SAM-Grid 29](#sam-grid)

[7.2 Tools 33](#tools)

[8 Manpower Requirements and Budget Summary
35](#manpower-requirements-and-budget-summary)

[8.1 Requirements 35](#requirements)

[8.2 Yearly Cost 38](#yearly-cost)

[8.3 Virtual Centre Value 41](#virtual-centre-value)

[9 Summary 43](#summary)

[10 Appendix 1 44](#appendix-1)

[11 References 47](#references)

# 1. Introduction {#introduction .unnumbered}

This document reports on DØ Computing and Software operations as well as
the current evolution plan for the next several years. It updates DØNote
4616, produced for last year's Run II computing review. It includes
scope and cost estimates for hardware and software upgrades, with a
detailed equipment spending description for the next few years. In view
of the amended charge, particular discussion is devoted to the
methodology and resources necessary to ensure a successful completion of
Run II.

## Overview

DØ\'s computing operation continues to run rather. Reconstruction has
kept up with the accelerator\'s excellent performance and the
detector\'s high data-taking efficiency, data handling (SAM) works
extremely well, remote Monte Carlo production has doubled since last
year, we are reprocessing most of the Run II data set remotely (on
SAM-Grid) and analysis cpu power has generally been sufficient.

However, maintaining this state will require significant effort since
the dataset is expected to roughly double on a yearly basis, while
resources are expected to increasingly be diverted to the LHC
experiments. Our computing model was based on distributed computing from
its origin, with progressive evolution to the use of standard common
tools on the grid, allowing us to use shared resources. Virtually all
resources used for production activities, namely Monte Carlo and
reprocessing, are already indeed at shared facilities. A data grid (SAM)
has been used since the start of Run II as the sole means of data
transport, enabling local / remote production tasks as well as
FNAL-based / remote analysis (with local job submission). The focus for
the computational grid (SAM-Grid[^1]) so far has been on production
activities, leading finally to user analysis as the most complex
activity. All of the current reprocessing, including use of spare cycles
on the central FNAL farm, is carried out via SAM-Grid. Full integration
(as opposed to the current co-existence) of SAM-Grid with other grids,
e.g. LCG and OSG is an ongoing project. At FNAL we are migrating the
reconstruction farm to use SAM-Grid rather than custom scripts, and will
then convert this farm, along with our central analysis resource (CAB)
to OSG.

Whilst believing that this "grid path" is the necessary approach, there
is still considerable ongoing development to make the experiment
grid-compatible, particular as the LHC computing model (and associated
grids) are also under evolution. DØ is one of the first experiments to
follow the grid path, so despite developing shared solutions wherever
possible (SAM is used by three FNAL based experiments), as a running
experiment, unique solutions had, at times, to be implemented. Thus we
are now, naturally, critically dependent on SAM-Grid and its ongoing
evolution to make it compatible with other grids.

As well as the technological developments necessary for the 'grid path'
it is also necessary to recognise the in-kind contribution represented
by the provision of significant computing resources. After considerable
discussion we settled several years ago on the concept of the 'virtual
centre'. The 'virtual' cost of carrying out all computing tasks at FNAL
is evaluated, using standard FNAL costings. A country's fractional
contribution, as measured by events produced rather than nominal cpu
provided, is used as the input to determine their in-kind contribution,
in turn determining any common fund reduction. Using a model driven by
actual contribution rather than nominal cpu provided has proven very
successful. This model will be used as the basis for the costings
carried out in Section 8.3.

### Managerial Structure and Document Layout

The DØ computing management structure remains basically unchanged, with
the additional creation of a deputy co-ordinator with primary
responsibility for remote computing activities. Additionally there have
been several replacements, and the up-to-date organisation chart can be
found at <http://www-d0.fnal.gov/d0_org.html>.

The document follows the same structure as last year's, with particular
emphasis on: remote activities, namely Monte Carlo and reprocessing, in
Section 3; SAM-Grid development in Section 7 and manpower and budget
issues in Section 8.

# Computing Needs

## Production Executables

In the previous years, the simulation and reconstruction has seen a
number of successive production releases, often several a year. As the
algorithms have stabilized, the efforts have shifted towards a better
understanding of the detector and the differences between real data and
simulated data. These efforts demand long and complex studies and for
this reason only one major production release, p17, was introduced this
year. Since January, the reconstruction version of p17 has been
available, and we have recently released the improvements to the
simulation code. All aspects of the code have been improved; the details
will be highlighted below. The focus has now shifted to preparations for
a first 'fixing' pass which will apply detailed corrections to the
calorimeter calibration and the material in the tracker. A second
primary focus is the preparation of the algorithm and infrastructure
modifications to accommodate the Run IIb upgrades to the trigger and
tracker systems.

### Reconstruction

For the reconstruction code, the p17 release has included the detailed
calibration of the electromagnetic portions of the calorimeter which
have been determined for all data-taking periods in Run II. As of July,
the detailed calibration of the hadronic layers of the calorimeter has
also been available. The offline calorimeter calibration database has
been brought into production running for the first time in Run II,
enabling a reprocessing of all of the Run II data, which should be
completed during October. The hadronic corrections will be applied
during a further \`\`fixing\'\' pass through the data since they were
not ready when the full reprocessing began in order to provide the full
dataset by October. Sufficient information is stored in the thumbnail
(most compact) data to allow this fixing to be done at this high level.

A new treatment of the material in the tracking volume will also be
incorporated into the fixing pass. This will use the new material
description of the detector developed over the past year, and will
employ the novel feature of track refitting using the information stored
in the thumbnail data tier.

As mentioned last year, the previous reconstruction version (p14) was
reaching its limitation in terms of speed, as can be seen in Figure 1,
at instantaneous luminosities around 10^32^cm^-2^s^-1^. A significant
effort was initiated with help from the Computing Division to address
this problem from the computing point of view (speed of the current
algorithm) and the algorithm and physics points of view (tuning of
algorithm versus physics trade-off). The results of "computing"
improvements can be seen in the Figure, where the p17 reconstruction
time versus instantaneous luminosity is compared with the previous
version. The improvement varies from \~20% at low initial luminosities
of 0.2 x 10^32^cm^-2^s^-1^ to more than a factor of two at luminosities
of 0.8 x 10^32^ cm^-2^s^-1^. Currently algorithmic improvements are
under study; additional, significant gains seem possible.

![](media/image1.png){width="3.75in" height="2.625in"}

Figure 1: Average reconstruction time per event for p14 and p17 as a
function of initial instantaneous luminosity for the run.

### Simulation

It is essential for the DØ physics program to provide a realistic
simulation of the detector. First, an improved description of the
material within the tracking volume has been implemented in the
simulation (and in the reconstruction). Second, studies have shown that
overlaying real data events over generated signal events to simulate the
effect of pileup and multiple interactions significantly improves the
agreement between data and simulation of the tracking performances.
These capabilities are now available for large scale Monte Carlo (MC)
production; large datasets of zero-bias events have been processed to
provide a proper luminosity- and time-weighted distribution of occupancy
in all detectors. Significant effort has also gone into updating the
existing MC generators, incorporating new PDF libraries, and adding new
generator packages to the release. Effort continues on understanding the
detailed description of the interactions in the tracking detectors and
the calorimeters in order to improve the agreement between data and
simulation.

### Prospects

Sometime this Fall/Winter, the Run IIb upgrades will be installed in DØ.
This will include a significantly altered Level 1 Trigger system and an
additional inner silicon layer for the tracker. All of these components
need to be properly read-out and incorporated into the reconstruction
code. Their performance will also need to be simulated, both to develop
the algorithms necessary for their incorporation into the reconstruction
code and to optimize their performance. This effort is rapidly
approaching maturity. The trigger simulation code is nearly complete,
and has already enabled detailed rate studies during the design process
of new trigger terms. For the new silicon Layer 0, GEANT and detector
geometries exist, hits can be produced in simulation, and tracking
algorithm development is well underway. We estimate that final
algorithmic software work on both systems will be complete during the
next month, after which the simulation can be used to optimize
performance. Development of code for the upgrade of the central fibre
tracker electronics has also begun; delivery of this system is expected
mid-way through 2006, so time pressure is less severe in this case.

## Data Analysis

### Post-processing

After the data have been reconstructed on the farm, the Common Samples
Group (CSG) runs it through the so-called "fixing" and "skimming"
processes. The fixing applies corrections for improvements selected
after the production release was cut or important algorithm
modifications. It consists of unpacking the thumbnail, correcting some
calorimeter cell energies, re-running jet, electron and missing E~T~
algorithms, refitting tracks, and then repacking the thumbnails. The
skimming selects events, based on reconstructed physics objects, and
writes them to corresponding streams. One or more of these \`\`skims\'\'
forms the base data sample for the vast majority of the physics
analyses. We are currently in the process of fixing the entire Run II
dataset, data that have been reprocessed with p17 but missing key
corrections like the hadronic calorimeter calibration and fuller
treatment of the material in the tracker. We anticipate that this will
finish shortly after the reprocessing. Crucial datasets, such as that
used for the determination of the Jet Energy Scale in the calorimeter,
have been processed first to allow fast analysis of the final event
sample.

### Data Format

First we provide a brief history. Up to now, different physics groups
and individuals have taken different approaches to using the thumbnails.
Some have decided to work within the framework, using the thumbnails
directly. Since the unpacking of the thumbnails depends on so much of
the DØ code, however, linking of the executables required very
significant amounts of memory, and remained slow even on well-equipped
machines. The recently introduced possibility to used shared libraries
has improved this situation dramatically, but this development was
sufficiently late in the analysis history that thumbnail-based analyses
had already been rejected by most groups in favour of analysis on
root-based datasets. There exist multiple root formats however, used by
different groups. Producing these, and keeping the data on disk proved
to be a strain on analysis resources, and a Common Analysis Format (CAF)
was developed. This is a root-based format that contains much of the
information in the existing thumbnail. As well as reducing the required
resources, a common format brings numerous efficiency gains, including
easier sharing of data and analysis algorithms between physics groups
and reducing the development and maintenance effort required by the
groups. Such centralization should also lead to faster turn-around
between data taking and publication. In addition to CAF, the CAF
Environment (CAFÉ) has been developed; this very effectively provides a
single, user-friendly, root-based analysis system, forming the basis for
the common tools being developed. The analysis groups are currently
developing such common tools for standard analysis procedures, such as
trigger selection, object-ID selection and efficiency calculation, etc.,
so that all groups can benefit from shared effort. This development
effort and the conversion of analysis code are expected to be completed
as the reprocessed and fixed data from p17 become available.

### Resources

The computing power available at Fermilab for data analysis has proven
adequate. As the dataset grows however, disk and cpu demand is likely to
increase in proportion. While we expect to be able to supply the
necessary resources in the short term at least, as discussed we are
working towards enabling the use of grid resources for individual user
analysis jobs as well as production tasks. To achieve this without
introducing a significant learning curve, the necessary tools and
interfaces are being built into *d0tools*, with which the majority of DØ
users are already familiar -- see Section 7.2.

# Computing Systems

## Reconstruction Farm

### Current Status

The current DØ reconstruction farm consists of 448 dual processor worker
nodes, 12 dual processors for input staging and an 8 processor SGI
Origin system which is used as a disk server to the workers as well as
an output buffer and stager. The worker nodes are a mix of 1GHz PIII,
1.67GHz Athlon, and both 2.6 and 3.0GHz Xeon class machines. The total
compute power of the system is approximately 1550 GHz in PIII equivalent
units. In the last year one node has been set up to act as a SAM-Grid
head node for running part of the farm as a Grid operation. Buffer space
for the Grid operation is provided by the Origin machine.

At this point 240 of the worker nodes are located in New Muon Lab (NML)
and 176 nodes in the High Density Computing Facility (HDCF). Operation
of the farm with workers distributed at multiple locations has so far
worked well, however stability of both power and cooling at NML have
been problematic over the past year. We currently expect to move all
worker nodes to HDCF this Fall. The current plan is to schedule the move
concurrent with electrical work to be done at HDCF in the Fall of 2005.

### Performance

Data taken after the Fall 2004 shutdown with the V13 trigger version
have been processed on the farm with the p17.03.03 version of the
reconstruction program (*d0reco*). This version has performed at a level
comparable to p14.06 with respect to robustness. Data taken with the V14
trigger version have been processed with the p17.05.01 version of the
reconstruction program.

Significant effort was put into fixing the residual failure modes of
*d0reco* in the p17.05.01 version. We currently have processed about two
months worth of new data with p17.05.01. So far we have no processing
losses of data due to *d0reco* crashes except for ones attributable to a
known corruption problem in the raw data.

Performance of the p17 version of *d0reco* has been significantly
improved with respect to the p14 version. Figure 1 shows the performance
of the reconstruction program under both p14 and p17. On its busiest
days the DØ detector records abut 3.5 million physics events which go to
the reconstruction farm. Assuming an 80% operating efficiency for the
farm and 1550 GHz of compute capacity we can see from the plot that the
farm can keep up with a detector running at 100% duty cycle if the
average initial luminosity is no more than \~0.6 x 10^32^cm^-2^s^-1^. As
the average luminosity grows beyond that we become dependent on the
limited duty cycle of the accelerator and detector to keep up with data
acquisition. With an accelerator and detector duty cycle of 33% we could
survive up to average initial luminosities of at least 1.2 x
10^32^cm^-2^s^-1^. Note this is a lower limit and not an extrapolation.
Current data do not give us enough information to clearly extrapolate to
the exact cut-off point. Thanks to the highly efficient operation of the
farm we have some spare capacity, and this is currently used for
reprocessing via SAM-Grid. Note the above assumes that all cycles are
available for reconstruction processing and that the data rate coming
from the online system does not change. At present this rate to tape is
limited by offline computing capabilities to 50 Hz. Our request to
increase this, as originally planned for Run IIb, is discussed in
Section 8.

### Planned Expansion

The functionality of the SGI server node on the farm will be distributed
to linux nodes over the coming year. The farm SAM station will be moved
to an existing 2 processor worker node. A 2 TB fibre connected raid
array has been purchased to replace the ageing disks on the SGI node.
These disks will be served out by 4 existing dual processor worker
nodes. This new disk array is already in place although the
functionality has not yet been moved to the new array.

There will also be an expansion of the capability of the SAM-Grid
headnode. This will be accomplished by distributing the head node
functionality across three separate dual processor nodes. One node will
handle the standard job submission actions, one node will be used for
XML DB logging, and one node for I/O operations. Existing worker nodes
will be used for this redeployment.

Current plans are to add 140 dual processor nodes to the reconstruction
farm by the beginning of calendar year 2006. These are expected to be
Opteron nodes equivalent to 3GHz PIII processors. This will bring the
total capacity of the farm to 2390GHz. However, 240 existing Athlon
nodes will fall off warranty in November 2005. These nodes will not be
repaired as they fail so the average capacity over the 2006 calendar
year will be somewhat less than 2390GHz. It is also expected that
additional nodes will be added in FY 2006, see Section 8 for overall
budget request.

## Central Analysis Systems

As planned DØmino was retired at the end of CY04 and has been replaced
by a linux-based login pool, with the mail server moved to the
imapservers. In a similar way disk space is served from linux servers.
We have \~ 60TB of scratch and project space and 130 TB of SAM cache
served in this way. These disk servers are located in FCC and linked to
the DØ network by at least gigabit connections.

Both CAB (Central Analysis Backend) and clueDØ (our cluster of institute
maintained desktop linux PCs) continue to expand as planned and so only
very brief updates are given below. CAB (together with clueDØ) have
provided sufficient analysis power during the last year.

### CAB

CAB is centrally provided and actually consists of two sets of nodes,
cabsrv1 and cabsrv2; these now consist of 192 and 344 worker nodes
respectively. These are linked to \~130TB SAM cache via \~20 server
nodes. The benefits of this increased SAM cache are discussed in Section
5.1.1. As part of our 'grid' evolution CAB will be converted to the
FermiGrid / OSG infrastructure during the next year.

### CluedØ

ClueDØ now numbers over 420 nodes, providing \~ 450 batch slots. Disk
space continues to be provided as different volumes under /rooms/..;
there are now some 120 rooms providing \~ 100TB of disk space. These IDE
raid arrays are centrally located on the second floor in the DØ assembly
building. System administration continues to be provided by a local
administrator from each providing institute and a core team of 7-8
people. Recruitment can at times be problematic.

## Remote Computing 

There has been considerable activity in this area in the last year. A
particular success is the ongoing reprocessing of virtually the full Run
II data set (\~10^9^ events); this has been carried out entirely using
SAM-Grid and has been possible through the very significant advances
made to this project over the last year. In parallel we have produced
some 75 million Monte Carlo events. We preserve here the 'historical'
separation of the two tasks, though with the increasing use of SAM-Grid
both should be considered simply as production computing.

### Monte Carlo Production

In the last year the DØ experiment has produced some 75 million Monte
Carlo (MC) events, twice the number produced in the year before. See
Table 1 for a breakdown by site. The two largest producers have been
IN2P3 and 'Nikhef'. All Nikhef production is actually carried out on LCG
sites, accepting the DØ V0. This 'first step' of LCG interoperability
has been in operation for several years, and is in the process of being
fully automated / extended -- see Section 7.1. The number of MC events
could be larger, but for two major factors. The start of p17
reprocessing in the spring of 2005 reallocated very significant
resources from MC production to reprocessing. In conjunction with this
has been the transition from p14 to p17 MC over the summer. The
transition has resulted in a relatively low number of available MC
requests. These two factors are reflected in the drop off in MC
production since spring. Over the first 8 months of the past year MC
production averaged 13.6 million events per month. The last 4 months,
after reprocessing had gotten underway in earnest, the average MC
production was 2.1 million events per month. The bulk of production in
these last 4 months has been from Nikhef because, as a non-SAM-Grid
site, it is not participating in the p17 reprocessing.

  ------------------------------------------------------------------------
  Site                            Events             Size (MB)
  ------------------------------- ------------------ ---------------------
  GridKa/Wuppertal                4552800            222052

  LTU                             501750             24471

  LU                              863263             46651

  OU                              1618000            86907

  SPRACE                          3687155            191528

  Tata                            793800             41997

  UTA                             2691941            147193

  Wisconsin                       12778              771

  CCIN2P3                         32066167           1939765

  FZU                             7740563            385985

  Lancaster                       4320975            176929

  Manchester                      100500             6276

  Nikhef/LCG                      17148986           883450

  TOTAL                           76098678           4153975
  ------------------------------------------------------------------------

Table 1 Monte Carlo production by site.

As part of the ongoing migration to common tools, the DØSAR sites have
been converting to SAM-Grid based production. LUHEP, OUHEP, and LTU have
or will shortly convert from the McFarm jobmanager to the SAM-Grid
jobmanager. SPRACE has converted from McFarm non-grid production to
SAM-Grid based production. Production at Tata, Lancaster, and Manchester
has decreased due to hardware upgrades and limited manpower at these
sites.

As reprocessing winds down in the Fall, very significant resources will
become available for MC generation, including both existing and new
sites. Switch-over to MC from reprocessing requires very limited effort
for these sites as the SAM-Grid infrastructure handles both types of
jobs. Based on the reprocessing experience it appears possible to use
non-DØ resources for MC production beyond the LCG activity mentioned
above. The CMS-FNAL farm and the Wisconsin cluster are being operated
via the grid, whilst OSG/SAM-Grid "co-existence" was successfully
demonstrated for the first time on the OSCER and CMS-FNAL clusters. The
ongoing programme of work relating to making SAM-Grid interoperable with
LCG and OSG is discussed in Section 7.1. In addition, new resources are
expected to come online and be available for MC generation, including
new clusters at OU, in Rio De Janeiro, at LSU and University of
Mississippi. It should be noted however, that the bulk of our MC
production takes place at large, shared non-US Tier-1 sites.

Another positive development is the increase in bandwidth that is being
put in place in the US and internationally. Many DØ institutions will be
taking advantage of connections to the National Lambda Rail with similar
improvements in Europe and South America underway. This additional
network capacity will increase the efficiency of DØ\'s grid based MC
production.

In summary, despite the reduced production numbers of recent months, the
outlook for MC production appears bright. The reasons for the limited
production are understood. Very significant resources will be available
when reprocessing ends and new grid installations are planned for the
near future. DØ MC has been a pioneering application in the use of grid
technology and by further embracing and extending grid based production
should be able to provide adequate simulated events for physics data
analysis.

### P17 Reprocessing

The investigation of real data leads to an improved understanding of the
actual detector performance and thus improved reconstruction algorithms.
Some of these improved algorithms can only be performed on the original
raw data format, requiring data base access. To make these improvements
available for data originally reconstructed with older software
releases, the data have to be re-reconstructed. The most notable
improvement of the new p17 software release is a new calorimeter
calibration which is implemented with a database.

Before the November 2004 shutdown, after which the p17 software version
became available, DØ had collected and reconstructed around 470pb^-1^
corresponding to 1000 million events. This amounts to 250TB of raw input
data, to be processed into 70TB of TMB output files. At an expected
processing time of 50s/event[^2] the effort requires 20,000 cpu months
or 1600 cpu years on 1GHz Pentium III processors. Given that the
experiment required the task to be completed within 6 months, the number
of required cpus was equivalent to 3400 1GHz Pentium III. As it was
assumed that the d0farm was used to its capacity, or close to, by
processing the incoming data, the complete cpu resources for this effort
had to come from remote sites.

The TMB files produced as output by *d0reco* are much smaller than
*0*(1GB) input files. As tape access is only efficient for large files,
multiple output files must be merged to reach a file size of 1-2GB. A
small number of files below this goal is acceptable. In addition, the
files that are merged are required to stem from the same physics run and
must be reconstructed with identical *d0reco* versions. This requirement
leads to a two-step application flow. In the first step, raw data files
are fed into *d0reco* to reconstruct the physics event and to create TMB
output files. During this step, access to the calibration database is
needed. After all individual files have been reconstructed, they are
then merged into larger files of the same format in the second step.
Only these merged files are then stored to the tape system (enstore).

The entire effort was carried-out using SAM-Grid. Besides providing a
common runtime environment, a grid also provides a common interface for
the job submission, allowing the development and use of common tools for
job submission, merging and recovery. SAM-Grid enables a submitted grid
job to be performed by multiple parallel batch jobs on separate cpus.
This feature is used to do the production step (step 1). Each file
within the dataset of raw files associated with a single grid job is
processed by a separate batch job. The resulting thumbnails are stored
to a \"durable\" SAM location associated with the cluster on which the
grid job is running. In order to distinguish these files from the final
merged thumbnails, a specialised datatier, \"unmerged-thumbnail\", was
created within SAM. The second step (merging) was implemented as a
separate grid job. Even though a parallelisation would have been
possible this wasn\'t pursued. Greatest effort was put into avoiding
data duplication i.e. assuring that results from one and the same raw
input file can\'t be merged into more than one output thumbnail. Both
the SAM-Grid client (submission commands) and the scripts actually
performing the merge verified that no other thumbnail produced from the
same raw data file with the same application_name and the same version
had been previously merged. The former check was implemented to create a
fast job rejection in case of a conflict; the latter check is needed to
minimise the possibility that such a conflicting file is produced
between submission and completion. The intermediate unmerged thumbnails
are not needed after the merged files are successfully stored to enstore
and so are removed from disk and the SAM locations are erased. They
remain as virtual files in the SAM database.

Inevitably, in such a big production effort, one has to deal with errors
such as disk failure, network interruptions, server crashes, etc. In
contrast to MC production, special care needs to be taken to avoid any
data duplication or data loss. Reliable book-keeping is necessary. An
efficient way of recovering from failures is crucial for an efficient
production, and a two prong approach was used. The SAM database keeps
track of all stored files. The associated metadata allows the
determination of the application_name and version used for production as
well as the (list of) parent files. Thus the SAM database was chosen to
determine the completion of tasks, to avoid data duplication and to
construct recovery datasets. In addition, the JIM[^3] component of
SAM-Grid provides job monitoring information on the grid- and the
batch-job level in an XML database that is also accessible through a Web
interface. The information fed into this database was tuned to contain
the main failure messages needed to allow fast diagnostics for system
problems and thus a fast recovery of service.

As mentioned, the use of grid mechanisms for job submission provides a
uniform interface for operators at all sites, allowing for the use of
common operational tools. The tool developed, *d0repro*, consists of
three layers. The lowest layer shields the creation of job description
files (jdf) that describe the desired job. Only the dataset name and the
*d0release* to be used need be specified. Other parameters needed by
SAM-Grid are constructed from these or taken from the local
configuration. This layer shouldn\'t be needed by the operators. The
second layer adds the ability to verify the completion status of grid
jobs. This information is stored in local files, imitating the behaviour
of a request system. Based on the status information, the operator is
protected from various common mistakes, such as submitting the same
request twice or submitting a request without a sufficiently long-lived
grid certificate proxy. This layer is used by the operators for manual
operations. On top of these commands, a so called auto-pilot layer is
built. The autopilot aims to suggest the most obvious next step and
creates a script that can be used to automatically perform these
suggestions e.g. for complete production jobs the corresponding merge
job is suggested. The most recent version adds a feature to approve (and
later submit) a production request for each merge job automatically,
thereby allowing one to maintain a constant production load without
intervention. This layer aims to reduce the workload on operators by
automating reoccurring standard tasks but in its current form still
requires manual supervision. The long term plan is to automate all
production tasks, with the off-line shifter being responsible for
initial job submission and monitoring.

Before the production effort was started, all aspects of the software
and the infrastructure were thoroughly tested. The *d0reco* executable
was deployed on the d0farm and 20pb^-1^ were processed with the standard
procedures and then given to the physics and algorithm groups for
verification. Each version of the *d0release* tar-ball for SAM-Grid was
certified by comparing the plots from results obtained on the d0farm
with SAM-Grid mechanisms to the equivalent plots obtained from
thumbnails produced by the standard procedures. These plots were
generated and overlaid using the DØ recocert package. To confirm that
the grid infrastructure was operating correctly on all participating
clusters the production and the merge step had to be certified by each
site. The production certification was achieved by producing thumbnails
for a specified dataset of raw input files and comparing the histograms
produced to the reference histograms. The merge certification was done
on separate datasets for each site. Plots obtained from the
unmerged-thumbnails were compared to those obtained from the merged
thumbnails. Both sets of comparison plots had to be approved by the
algorithm convenors before a site was allowed to start official
production.

The significant resources required for this undertaking were mainly
provided by shared computing centres that provide their services to
several experiments / user-groups with different requirements. To
participate in the data-reprocessing, each site was required to provide
one or more nodes on which some DØ specific services could be installed.
The required services were the SAM-Grid job-manager for Globus to accept
and manage jobs, an XML database server for monitoring information, a
SAM station to provide data management, and a database proxy to cache
information from the DØ calibration database. Participating sites are
the CMS-FNAL farm, D0SAR-OSCAR, D0SAR-SPRACE, D0SAR-UTA, GridKa, IN2P3,
Prague, UK (several sites), WestGrid and Wisconsin. The CMS-Farm at
Fermilab was special in that the SAM-Grid job-manager was integrated
onto the existing OSG environment and that the SAM services were
provided by DØ nodes elsewhere on site.

The production started on 25th March 2005. The first sites to be
certified and start production were Lyon and WestGrid. Others followed
over time. Due to the improvements in the speed of *d0reco* it was also
possible to use spare cycles on the d0farm, again using SAM-Grid. After
5 months nearly 80% of the data are reprocessed, see Figure 2.
Continuing at the current speed the bulk of the project should be
completed in October. Fuller details, including monitoring of both
overall status and the site efficiencies, can be found at
http://www-d0.fnal.gov/computing/reprocessing/ and in ref [^4]. We
intend to produce and publish in a referred journal a complete write-up
of this activity.

![](media/image2.png){width="6.003472222222222in"
height="1.9833333333333334in"}

Figure 2: Status of p17 reprocessing as of 1^st^ Sept 05.

## Online Systems

The DØ online systems support data acquisition from the detector readout
elements, slow control systems, data quality monitoring, and control
room activities. The Level 1 hardware trigger for DØ accepts events at a
rate of \~1.6 KHz. A Level-2 system, a hybrid of dedicated hardware and
software elements, reduces the accepted event trigger rate to \~800 Hz,
for which the entire event information is transferred to a Level 3
software filter farm. The event data readout path begins at \~80 VME
readout crates, each containing a Single Board Computer (SBC). Under the
guidance of a specialized SBC, the routing master, each SBC transfers
event fragments via Ethernet through a Cisco Catalyst 6509 switch
directly to a selected node in the Level 3 filter farm. The Level-3
software trigger reduces the accepted event rate to one which is deemed
manageable for offline analysis, currently \~50 Hz. Accepted events are
passed by Ethernet via a 2nd Cisco Catalyst 6509 switch to a chain of
Host system applications: a Collector process which acts as the target
of Level-3 transfers and which forwards the data stream to Data Logger
processes and a Distributor process. The Distributor process distributes
event data to data monitoring applications. The event data is buffered
on local disk arrays, then transferred to the Feynman Computing Center
(FCC) via the enstore and SAM systems. The slow control system consists
of \~110 embedded VME processors, in both the DAQ readout crates and
crates dedicated to controls. DØ employs the EPICS control system, and
has expanded its functionality to provide a structured detector
configuration architecture, a complete significant event (alarm) system,
and a variable archiving and retrieval system. The Host system supports
the \"back end\" of the DAQ task for logging and monitoring of event
data. The Host system also supports detector control and configuration
tasks, and houses an Oracle database for those purposes. The general
online computing infrastructure is also supported by Host file servers,
application servers, and web servers.

The bulk of recent additions to the Online system have been associated
with the Run IIb upgrade project. There were three components of the
upgrade associated with the Online: a Level-3 farm upgrade, a Host
system upgrade, and a slow control system upgrade. Online network
additions associated with the RunIIb upgrade are discussed elsewhere.
With the expected increase in instantaneous luminosity expected in
RunIIb, and the corresponding increased event complexity, it is
necessary to expand the computing capacity of the Level-3 farm. In July,
2005 a 128-node (dual 2.8 GHz Xeon processor) addition was made. This
brought the total Level-3 farm to 274 nodes (corresponding to
\~1000GHz-equivalent processors). For RunIIb, the Host system was
changed from one based upon a cluster of three AlphaServers to a much
more flexible Linux cluster configuration. There are now four
high-availability clusters sharing a Fibre Channel Storage Area Network
(SAN). The SAN contains \~4 TB of JBOD disk for event data buffers, and
\~6 TB for detector support functions. The linux clusters support the
DAQ services, NFS file services, Oracle database services, and various
Online application services. The upgrade to the DAQ servers now allows
highly parallelizable data logging, with a rate capacity exceeding \~250
Hz. The final component of the Run IIb Online upgrade was the slow
control system. Approximately 50 of the latest generation of PowerPC
boards were purchased to upgrade the older, slower control system
processors. A number of the replaced processors are now maintained as
spares, with the expectation that there are sufficient numbers to
satisfy needs for the remainder of the Tevatron running period. The
newly installed upgraded processors allow a number of monitoring
functions to now be enabled, providing for closer monitoring of detector
behavior.

The funds for the Run IIb upgrade were expected to provide the online
computing capacity sufficient for the duration of the DØ detector
operation at the Tevatron. The one exception to this expectation was the
Level-3 software filter farm. The 128-node addition for Run IIb was
sized to meet the filtering application processing time requirements for
a Level 2 accept rate (Level 3 input rate) of \~800 Hz at an
instantaneous luminosity of \~2 x 10^32^cm^-2^s^-1^. The uncertainty in
the extrapolations from current conditions used in this calculation,
plus the expectation that the instantaneous luminosity will increase to
\~ 3 x 10^32^cm^-2^s^-1^before Run IIb is complete, leads to the
expectation that the Level-3 farm will need further enhancement. The
expansion of the Level-3 farm is limited by several factors: funds, the
number of network ports required to maintain the Ethernet based DAQ, and
the DØ site power and cooling infrastructure. The timing of any
expansion is also affected by the capabilities of the processors that
can be purchased at any time. The introduction of multi-core processor
architectures will ease several of the constraints. The DØ plan is to
add Level-3 processing as needed and as late as possible. We expect to
retire older systems to make way for new. The funding for expansion must
come from operating funds - with an expected need of \~\$50K per 32-node
(multi-core, dual processor) addition. There may be need for several
such additions if the accelerator performs as expected.

# Databases

## Offline Calibration Database

The calibration constants for the tracking detectors and the calorimeter
are served to the reconstruction processes on the Fermilab
reconstruction farm and to the jobs that users run for their own
purposes via the DAN client-server system based on the CORBA technology
and developed by the Computing Division and DØ. This system can also
deliver calibration constants for the muon subsystems to *d0reco*. Up to
now the muon group has not used this facility for *d0reco* processing,
but is planning to use it in the near future, with updated database
tables.

The database has been in use since June, 2003 and is fully reliable.
There is one dedicated linux node for running the calibration DB servers
for the farm, and another for the non-farm users. Since these nodes are
not supported 24/7, there is a failover node and a procedure in case of
troubles with the server nodes. However these nodes are three years old
and will need to be replaced in the near future.

The DØ farm runs \~600 processes and requests a new set of calibration
constants on average every 100 seconds. It takes about 90 seconds of
wall-clock time for a *d0reco* process to receive one whole set from the
servers, including the time for marshalling data on the servers and
un-marshalling them on the client. The cpu usage of the farm server node
is about 5-10% and the network usage out of the server node is about
30%. This implies that up to a three-fold increase in the number of the
farm nodes can be handled without upgrading the network interfaces.

The p17 remote reprocessing started in late March and the calibration
constants have been delivered to the remote processes through the remote
proxy servers since then. There are now \~10 participating sites. The
system is reliable and we have not had problems with serving calibration
constants to the remote institutions.

One possible future problem with the system is that the interface to the
Oracle database is based on a freeware python package that is no longer
supported by the developer. It is preferable to be prepared to switch to
a different, supported, interface.

## Luminosity Database

Many existing components of the luminosity software are based on ASCII
files, a system which will become unmanageable in the life span of the
experiment. In order to address this issue for the long term, the DØ
Luminosity Group and the DØ Database Group have been developing the
luminosity database application. Transferring the existing information
to an Oracle database will allow us to profit from the better tools
available with a modern database system.

This application is a large scale and complex project which, at its
core, relies upon a set of 37 database tables and 2 views stored in a
new offline production database instance on a new machine that is now
being set up. The database itself contains information from many
subsystems: the luminosity monitor detector, the trigger framework, the
L3 system, the Tevatron accelerator and information about offline data
processing (from SAM). The application includes components that will
satisfy the following essential operations for the DØ experiment:

-   Calculate and deliver a luminosity normalization for any given data
    sample

-   Improve error tracking and characterization by making fundamental
    luminosity information available to offline programs

-   Produce operations reports and graphs

-   Produce quality reports by luminosity block[^5] number

-   Verify and certify data integrity for normalization purposes

-   Additional customized reports also become possible using this
    system.

The database schema has been stable for a year now. Nearly all of the
data taken by the experiment through 2004 have been loaded into the
database tables. These data will be transferred to the production
machine when the machine is available. We are in the process of loading
data for 2005. The application that loads the data from the online
system as the data are recorded is currently in development. With much
of the data loaded, large scale testing of all the client applications,
as listed above, is nearly completed. More detailed testing is underway
with realistic operations use cases and actual analyses to bring each
component to \'production\' quality. To access the luminosity database
from the analysis programs, we have developed a client-server system
similar to the calibration system. The database server was generated
using the same code generation system as the calibration system, and the
client code (lmAccess) has been converted to use the CORBA protocol
instead of reading the ASCII files. At this time, the new lmAccess
performs as fast as, or faster than, the old system. We plan to move
this application into production before the end of FY05.

## Trigger Database

The trigger database stores trigger configuration parameters and
settings entered by trigger experts including information from all
trigger subsystems at each level of the trigger. These settings are
downloaded at run time for physics, test and calibration data taking and
used by the trigger simulator. The system includes interfaces to report
this information to those physicists who are involved in optimizing data
collection and/or performing offline physics analysis.

The trigger database application has been in use since the start of Run
II. The trigger data base entry system and some command line interfaces
rely on a database server that is similar to the SAM servers. In the
last year, with personnel from the CD Database Software and the CD Run
II Computing and Analysis Groups, major progress has been made to bring
the web client and database server components to completion. The
progress has been facilitated and organized according to the trigger
database project plan which describes all components of the application
and includes a detailed, prioritized description of the work that needs
to be completed [^6]. The work, as listed in the existing plan, should
be completed by the end of the calendar year. New trigger systems will
come online in Run IIb and many of the new trigger features will be
programmable through the existing system without changes to the
database, client or server. Changes to the system may be required for
some new trigger features, but how those features will be configured has
not been completely specified yet.

Both the luminosity and trigger data base projects are challenging, yet
rely on common key manpower. After an internal review over the summer,
the experiment is working closely with the Computing Division to bring
these projects to timely completion, in such a way as to provide the
required functionality and maintainability.

# Data Handling and Storage

## SAM Overview

SAM (Sequential Access to Metadata) provides the only data handling
mechanism for DØ, and was originally started as a joint DØ-CD division
project in 1997 and has since been taken up by CDF and MINOS. A
description of SAM (and SAM-Grid) and the project status are described
in a different report to the review committee. This document will focus
on information specific to SAM-Grid operations at DØ. In the last year,
\~2.5Pbytes (\~50 billion events) of (DØ) data have been shipped around
the world, at up to 250TB/month. Figure 3 shows the number of gigabytes
consumed per month on various stations, with Figure 4 showing the same
in terms of number of events.

In addition to the SAM cache, we supply the physics and analysis groups
with project space. This space is intended for highly specialized
samples, code development, documentation and other group specific needs.
For budgeting purposes, we assume that the amount of project disk needed
is one sixth of the amount of space estimated for disk resident analysis
samples. There were several important changes in the data handling
infrastructure that occurred in the review period, and they are
described in the following sections.

![](media/image3.jpeg){width="4.122916666666667in"
height="2.877083333333333in"}

Figure 3: Gigabytes consumed per month on various stations, including
central-analysis (cab, cabsrv1 & cabsrv2) and various remote sites (of
which there are now, in total, \~ 50).

![](media/image4.jpeg){width="4.122916666666667in"
height="2.877083333333333in"}

Figure 4: Events consumed per month on various stations, including
central-analysis (cab, cabsrv1 & cabsrv2) and various remote sites (of
which there are now, in total, \~ 50).

### Analysis systems

DØ discontinued use of the central d0mino system in January 2005. D0mino
hosted the *central-analysis­* SAM data handling station, which served as
the large SAM cache for the experiment. At the same time, it was noticed
that the two analysis farm SAM stations (*cab* and *cabsrv1*) were
pulling the majority of their files from tape, causing a large load on
the tape plant.

Reorganizing the analysis farms mitigated the tape load problem. We
added several *cache server* nodes with large RAID arrays to the
*cabsrv1* station and route all external file deliveries through these
nodes. SAM sees the server node caches as a large pool of cache, and
thus files there have a much longer lifetime than files in the
individual worker node caches. All external file transfers (e.g. from
tape) are first routed through the server node cache and then delivered
to the worker node. While this operation involves an extra disk copy
than previously, the fact that files live much longer on the server
cache means that subsequent requests for a file will more likely be
serviced by the cache and not by tape. A quantitative analysis of the
benefits is below.

Adding over 100 nodes to the analysis farm triggered another
reorganization, leading to renaming the *cab* station to *cabsrv2* (for
consistency). Instead of splitting the large server cache between the
two stations, it was simpler to keep the cache on cabsrv1 and have that
station act as a router for cabsrv2. That is all external file
deliveries for jobs on cabsrv2 are routed through cabsrv1 and its large
cache. The load on cabsrv1 for this routing is minimal and it ensures
that files that are in demand by the entire analysis farm populate the
large cache.

As of August 2005, cabsrv1 has 192 worker nodes and 22 server cache
nodes. The total server cache is 133 TB. The cabsrv2 station has 344
worker nodes. To gauge the benefit of the large cache and the new
routing policy, data file transfer statistics were examined for
transfers occurring on the analysis farm stations between June 1, 2005
and August 22, 2005. Out of 842,000 file transfers, 32% were from tape,
38% were worker node to worker node transfers, and 30% were server cache
to worker node transfers (this value does not include the extra copy
required for the tape copies). Without the server cache (as was the case
previous to January 2005), those latter transfers would have come from
tape. Therefore, the large server cache decreases the load on the tape
system by about a factor of two. This is an extremely positive
development. We are in the process of evaluating if the percentage of
transfers from tape to the analysis systems is at the steady-state level
or not. One area under investigation is group caches with quotas to
isolate data that should have short cache lifetimes such as picked raw
data from samples that should have long cache lifetimes such as the
skimmed tmb++ samples.

### Upgrade to SAM v7

DØ had been using SAM v5 for several years, but much of the SAM dbserver
and client code were rewritten for improved maintainability and
performance and to include features essential to CDF as well as new
features for DØ. This new code comprised SAM v6. Because this new
version involves many more python libraries than v5 and because DØ
accesses code from its d02ka NFS server, performance of SAM commands was
very slow. The cause was traced to python\'s extremely inefficient
strategy for loading external libraries (for each library, python
attempts to open many files to determine the library type and if
recompilation is necessary). Introducing a *frozen* SAM release, which
we call v7, mitigated this problem. In the frozen release, the python
executable and the SAM libraries are packaged in one executable file.
This packaging reduced the SAM command time by approximately a factor of
ten in some cases and has some nice side benefits, including divorcing
the SAM python version from the DØ code python version and easier
deployment of the SAM client code. In conjunction with the experiment
schedule, the SAM team deployed v7 in stages over a three month period
ending in July 2005.

## Robotic Storage

The DØ data management system relies heavily on the hierarchical storage
management system, enstore. Currently we write-out different types of
200 GB cartridges in three different robots. We use two STK Powderhorn
silos (each with 5500 tape slots) currently writing with 9940b drives,
and one tower (3500 tape slots) in an ADIC AML2 (a 3-tower system shared
with SDSS who are using DLT media) writing with LTO II drives. DØ has
enjoyed excellent performance and negligible data loss with the 9940b
and LTO II series drives. In principle, LTO III drives could be put in
the AML2; however, vendor support for that combination has not been
secured as the AML2 is reaching "end-of-life". The FNAL CD will be
purchasing robot storage equipped with LTOIII drives in FY2006 and
stakeholders can purchase slots for an estimated \$50 per slot (where
each slot will hold a 400GB tape). This is not attractive to DØ at the
moment given the large number of available slots in the AML2. At \$8.50
per slot for the 200 GB tapes and the existing drive plant of LTO II,
the additional cost of tape of staying with LTO II tapes would mean an
increase of operating funds needed for DØ data to supply tapes, however
with less in equipment funds to purchase new drives and robots. For the
moment, we plan to use LTO IIs until the end of life of the AML2, and
plan to add 4 or 5 drives per year. We do assume that the tape cost for
LTO IIs (currently \$50/tape) will fall by 20% per year. Indeed,
currently it is possible to purchase TDK media for 20% less per
cartridge; however the operations group insists on FUJI media, due to
some concerns about the TDK media that were subsequently traced to a
firmware problem. In FY2005, the DØ experiment traded some under-used
9940b drives to the STKEN system in exchange for the purchase of 4 LTO
II drives. We will likely continue to use 9940b drives as an offset for
other costs. Tape costs are summarized in Table 3 below.Need to include
tape costs -- I'd suggest as a separate table after table 10.

## Physics Considerations

In order to project our storage needs, we assume a data collection rate,
a size for the various formats, and that all of the formats scale with
the raw data rate. Whenever possible, we use our current experience as a
guide. We assume that the disk cache under SAM control should be large
enough to hold the most frequently accessed data samples with 100%
over-capacity for other samples.

The experiment has discarded the DST format, storing additional
information into the tmb+ data format, yielding tmb++. The p17 tmb++
size is 70 kBytes/event for data and 100 kBytes for MC to accommodate
the truth information. Additionally, the new centrally supported
root-based user format (CAF- 40 kBbytes/event) is now being deployed.
For a variety of reasons we plan to keep the final step of the data
processing available in both centrally supported formats (ie tmb++ and
CAF) and that the SAM cache is sized to hold both. Past experience
indicates that we'll need to reprocess the Run IIb data set once per
year (though this may well be more frequent initially after the
start-up), and run a fixing pass on the reconstructed data once per
year, with a tmb++ output format. After that, the data are skimmed into
samples for physics group use, with tmb++ output and root analysis
format outputs. Current experience indicates that the ratio of the
output to input of the skimming step is 1:1; we have asked the physics
groups to reduce this to 3:4 by working towards more common samples in
order to reduce duplicated events. on what basis can we assume ? if we
don't does rate disk requirement simply scale by 4/3?

We assume that the peak event rate increases from 50 to 100 Hz in a two
stage process, discussed in detail in Section 8.1 (and ref [^7]).
Folding in data sizes and scaling (should say in a footnote what these
are) factors Table 2 shows the estimated amount of data for tape and
disk storage. We purchase at the end of the fiscal year, so purchases in
FY05 cover 06 running. We estimate the need for cache disk as a
percentage of the raw data collected in a year. For *d0reco*
development, testing, fixing and special analysis, we assume that we
need cache space for 10% of the raw data (250 kBytes/event) and 30% of
the TMB++ and for analysis, we plan to accommodate the full skims in
Common Analysis Formats and TMB++ formats as well as a modest amount of
Monte Carlo data. We anticipate that the use of groups in SAM cache will
enable us to isolate the analysis cache from the raw data cache. We
assign a contingency of 40% on the cache space. The physics group
project space is estimated at 1/6 of the cache space. In addition,
university groups purchase fileservers for use on cluedØ.

  ------------------- ------------- --------- ----------- ----------- ----------- ----------- -----------
  Data samples                                                                                
  (events)                                                                                    

                      **Current**             **2005**    **2006**    **2007**    **2008**    **2009**

                      ** **                   ** **       ** **       ** **       ** **       

  Events collected    1.00E+09                8.16E+08    1.09E+09    1.09E+09    1.09E+09    1.09E+09

  Total events                                1.82E+09    2.90E+09    3.99E+09    5.08E+09    6.17E+09

  Geant events                                1.22E+08    1.09E+08    1.09E+08    1.09E+08    1.09E+08

  PMCS events                                 0.00E+00    1.09E+08    1.09E+08    1.09E+08    1.09E+08

  Tape data                                                                                   
  accumulation (TB)                                                                           

  **Yearly storage                  **731**   **627**     **888**     **1,102**   **1,308**   **1,515**
  (TB)**                                                                                      

  **Total storage                   **731**   **1,357**   **2,245**   **3,347**   **4,655**   **6,171**
  (TB)**                                                                                      

  Disk data                                                                                   
  accumulation (TB)                                                                           

  **Total storage     **88**                  **140**     **186**     **186**     **186**     **166**
  (TB)**                                                                                      
  ------------------- ------------- --------- ----------- ----------- ----------- ----------- -----------

Table 2: Data sample sizes and consequent tape and disk requirements

## Cost Projections

The cost of tape is estimated in Table 3, assuming all of the year's
data are written to blank tape. We do not include the cost of
duplicating the data, nor the cost of migrating any data until 2009 when
we assume that a 1TB technology will be available. A possible economy is
the reuse of tape as older reconstruction versions become obsolete.

  ------------ --- ----------- ----------- ----------- ----------- ----------- -----------
                   2004        2005        2006        2007        2008        2009

  DataVolume       731         627         888         1,102       1,308       1,515

  \# to retire     0           0           0           0           0           10000

  Tape Cost        \$182,650   \$156,650   \$177,680   \$192,815   \$229,005   \$239,260
  ------------ --- ----------- ----------- ----------- ----------- ----------- -----------

Table 3: Tape cost estimate. The 2004 number shown is not the 2004
increment, but the total amount of data DØ currently has on tape.

For the FY2005 project and cache disk purchases, we have ordered 15 TB
useable disk space ATABeasts systems at a cost of \$20,500 k per system.
For out-year cost projections, we assume continuing with the ATABeast
systems, and scaling up the size of the useable space by 20% per year.
In the past, we purchased different types of systems for cache and
project space in order to provide more reliability for project space
while keeping the costs lower for cache space. The ATABeasts have the
performance and reliability and features we desire for project space,
with competitive costs for cache disk, so we are using this for all of
our needs. The cost of disk storage is given in Table 4, with the data
volumes coming from Table 2. The contingency is that used in the past
and covers such things as the evolution of format size with luminosity
and uncertainty in the use of cache space. (The associated costs for
file servers are shown in Table 9.) The 2005 number is projected from
the spreadsheet and is in reasonable agreement with the 2005 pending
requisitions.

  ----------------- --- ----------- ----------- ----------- ----------- -----------
                        2005        2006        2007        2008        2009

  Data Volume (TB)      140         186         186         186         166

  Project Volume        23          31          31          31          28

  Total volume          164         217         217         217         193

  Contingency           40%         40%         40%         40%         40%

  Cost                  \$328,000   \$348,500   \$328,000   \$307,500   \$307,500
  ----------------- --- ----------- ----------- ----------- ----------- -----------

Table 4 Disk cost estimate.

# Networking

## Infrastructure

The principal elements of the network infrastructure supporting DØ
Computing are noted in Figure 5 in Appendix 1. These elements and the
functions they provide were described in detail in last year\'s review
document. Significant changes during the previous year include:

-   Two switches, s-d0-gcc-1 and s-d0-gcc-2, have been added to support
    nodes at the Grid Computing Center (GCC, aka HDCF). The gcc-2 switch
    currently has \~300 free ports, and is targeted for support of
    recent node acquisitions.

-   The Feynman Computing Centre (FCC) CAB switch, s-d0-fcc2-cab, will
    soon also move to GCC. It is currently \~2/3 populated, and will
    provide ports for an additional \~100 nodes.

-   The switch supporting the DØ Online system, s-d0-dab2cr-online, will
    soon have an upgrade of its router module and an addition of a
    48-port Gb blade. This will allow it to serve a larger number of
    Gb-enabled Linux servers.

-   The switch supporting the Level 3 system, s-d0-dab2cr-l3, has been
    fully outfitted with 48-port 100Mb blades in order to support the
    increased number of Level 3 filter farm nodes. A planned supervisor
    module upgrade will allow another 48-port blade to be utilized.

A Gb connection is now in place allowing direct communication between
the CAS switch and Starlight (ref
<http://netweb.fnal.gov:8080/starlight.html>). This path has been used,
for example, in data transfers to WestGrid.

## Utilization and Prospects

Figure 6, Appendix 1, shows recent network utilization, in bits per
second, on the principal switch interconnects within the DØ network and
to the external network. There are several links which show peak
utilizations near the link capacity, primarily those serving the
reconstruction and analysis farms in FCC, GCC, and NML (New Muon Lab).
The external links to Starlight and the FCC core router are also heavily
utilized. The network infrastructure appears adequate for DØ needs.
Approximately 400 ports are available to support planned additions to
reconstruction and analysis farms. The primary network interconnects are
mostly operating comfortably within capacity. The largest anticipated
growth in DØ network traffic is on the links to the new farms in GCC, to
analysis operations at DAB, and off site via the Starlight connection.
DØ is thus planning to upgrade each of these links to 10Gb, at a cost of
\~\$15K per link.

# Data Handling, Resource Management and Tools

DØ uses the SAM-Grid data handling system at all levels in its online
and offline systems, to move files, manage resources, to perform
production computing & user analyses and handle process bookkeeping.
SAM-Grid is supplied by the Fermilab Computing Division via a joint
project with participation from the Run II Computing and Analysis
Department, the CCF Department, and the CEPA Department, as well as
external effort from the GridPP project and other collaboration effort.
SAM provides the data handling mechanism and JIM (Job Information and
Monitoring) the job submission and monitoring. Taken together, they are
referred to as SAM-Grid. A fuller description of SAM-Grid and the
project status is described in a different report to the review
committee. This document will focus on information specific to SAM-Grid
operations at DØ. The experimentally supplied tools which interface to
SAM-Grid are undergoing a rolling programme of integration /
standardization to provide greater commonality and maintainability.

## SAM-Grid 

### SAM-Grid Status and Plans

During the past year, the SAM-Grid infrastructure has been enhanced to
meet the computational requirements of the reprocessing project and to
continue the effort of integration with other grid systems, in
particular LCG and OSG.

The challenges of the reprocessing project stem from the volume of data
and the intensiveness of the computational task. The goal of the project
was to reprocess one billion events, organized in 250 Terabytes of input
data, generating 70 Terabytes of output. The estimated processing power
was 1600 GHz-cpu year, almost all of which had to be provided by remote
institutions. In addition, the target duration of the project was
approximately 6 months, so requiring some 3THz-cpu for this period. To
achieve these goals, SAM-Grid had to be able to adapt to the available
remote hardware configurations in order to coordinate and optimize the
use of local resources, such as computing nodes (gateways and cluster
worker nodes), storage, and network.

The Job and Information Management component of the SAM-Grid has been
enhanced from the version used for Monte Carlo production in 2004. The
major additions are the optimization of the resources at the gateway
node of the site, the use of data queues to access storage resources,
and the ability to optimize site resources according to
application-specific parameters. These new features allow the control of
the load of key resources, such as gateway nodes and data servers,
resulting in higher efficiency and scalability.

In more detail, the standard implementation of the grid-to-fabric
interface, available from the Globus Toolkit, starts up a proxy process
(job-manager) at the gateway for every batch job. This limits the number
of concurrently running jobs to about 200 for a gateway deployed on an
average modern commodity cpu. SAM-Grid has modified the standard
grid-to-fabric interface in order to split grid jobs into multiple
parallel instances of batch jobs. For the typical DØ application, this
reduces by two orders of magnitude the number of job-managers running at
the gateway with respect to the standard implementation. In addition,
job aggregation eases job management for the users, who have to track a
few grid jobs instead of hundreds, and facilitates the optimization of
resources. An example of such optimization occurs when reporting the
aggregated status of the job to the grid. Caching the aggregated job
status limits the interaction of the grid-to-fabric interface with the
batch system in order to retrieve the status of hundreds of jobs, thus
reducing the load on the gateway node. Another example is the usage of
aggregate characteristics of the jobs, such as the "application type",
in order to enforce local policies and optimization strategies. In
particular, different application types are configured to run on
different batch system queues and access storages via different data
queues. This allows the prioritization of the jobs and the control of
the load on the storage servers.

The enhanced SAM-Grid can manage continuously at least one thousand data
intensive jobs concurrently running per cluster. This corresponds to a
total average throughput of ten million events per day with an
efficiency of 90%. The efficiency is defined as the ratio between the
number of actual output files from the reprocessing activity over the
number of expected files from the reprocessing request, at 1^st^ try.
The 10% inefficiency includes failures at the level of the application,
of the fabric services and resources, and of the SAM-Grid
infrastructure. More detailed studies would be necessary to discriminate
the contribution of each of them to the inefficiency. We expect the
final error rate to be at the 0.05% level. In October 2004, before many
of the improvements to the SAM-Grid system, the inefficiency due to the
grid infrastructure itself was measured to be less than 5%. The
throughput and efficiency monitoring can be seen at: [http://SAM-Grid
.fnal.gov:8080/cgi-bin/plot_efficiency.cgi](http://samgrid.fnal.gov:8080/cgi-bin/plot_efficiency.cgi).
For further details, see reference [^8].

The past year has seen an increase in the work to make SAM-Grid
interoperable with other grid infrastructures, focusing in particular on
the Open Science Grid (OSG) and the LHC Computing Grid (LCG). A main
motivation for the focus on interoperability is enabling SAM-Grid to
access a large number of shared computing resources without the need to
deploy SAM-Grid services on all remote sites. To meet this goal, the SAM
data handling system had to be enhanced to lift the assumptions on the
locality of the computational activity. The SAM station was modified to
service requests coming from jobs running on external network domains
with restricted incoming access. This was achieved by changing the
notification interfaces, by implementing call-back instead of polling
semantics.

The SAM-Grid team have deployed a proto-production system that allows
jobs prepared by SAM-Grid to run on LCG computing clusters. The
architecture of the system presents LCG to the SAM-Grid as if it were a
large computing cluster. SAM-Grid submits jobs to a gateway, simply
viewed as one of the possible computing resources, which submits the
jobs in turn to LCG, acting as a "forwarding" node. Adaptation to LCG is
provided at the gateway/"forwarding" node at the level of the SAM-Grid
grid-to-fabric interface. In general, the interface coordinates the
interaction with multiple fabric services (SAM for data pre-staging,
data base servers for query aggregations, push-based monitoring, job
environment preparation, etc.), including the batch system. The
interface interacts with the batch system through a layer of abstraction
called the "batch adapter". Adaptation to LCG was provided by
implementing an LCG-specific "batch adapter". With this schema, the
"forwarding" node prepares the job for execution on any "hostile"
computing environment (using the job environment preparation service),
then uses the batch adapter to schedule the execution on one of the LCG
clusters. More information can be found in reference [^9].

The architecture achieves scalability by allowing multiple forwarding
nodes, each interfacing a different region of the LCG system. The
current proto-production system submit jobs to LCG using a "forwarding"
node deployed at the University of Wuppertal, Germany, executing on the
CCIN2P3 cluster in Lyon, France, and the LPC cluster in
Clermont-Ferrand, France. The system uses a SAM station for data
management deployed at the CCIN2P3 cluster.

The focus for next year in terms of development will be on moving the
LCG interoperability infrastructure to production and to create a
similar system for the OSG resources, with immediate emphasis on
production tasks, such as Monte Carlo generation and reprocessing. In
order to move the LCG interoperability proto-production system to
production, the following steps are planned. First, implement a
dynamically deployable scratch management system, which could choose the
best file system for execution once at the LCG worker nodes, considering
the specific I/O requirements of the DØ applications. Second, integrate
the output sandbox services of LCG with SAM-Grid, in order to provide
the users with greater capabilities for analysing job failures. Third,
limit the range of ports required by the SAM station, making new
deployments easily compliant with site network security requirements. In
parallel we will expand the test bed certifying new LCG clusters for the
DØ production applications. More details can be found in reference
[^10].

The integration with OSG is planned in two phases. First, work with the
OSG community to deploy a resource selection service for the OSG
resources. Second, implement a SAM-Grid interoperability test bed using
the forwarding technique developed for the LCG integration. The first
step is a necessary pre-requisite for the second. The presence of a
resource selector in LCG was one of the reasons for the priority given
to LCG in the interoperability effort.

Effort will also be devoted to assisting new sites with deployment. In
the longer term user analysis jobs will also be incorporated into the
full grid environment. Note, off-site analysis, with locally submitted
jobs, has been underway for several years as SAM provides a complete
remote data handling system.

### SAM-Grid Effort

While significant progress has been made, limited manpower remains a
major issue. In the long term, the SAM-Grid roadmap for DØ is to
continue supplying the experiment with a performant and robust data
handling system able to respond to the changing grid environment and so
provide access to the additional resources needed as the Run II datasets
multiply. To achieve this, sufficient manpower is needed to:

-   Maintain production level performance with existing functionality.

-   Improve / extend functionality e.g. scalability improvements
    associated with reprocessing, extension to user analysis.

-   Completion of integration projects with other grids, in particular
    LCG and OSG.

-   Assistance with SAM-Grid deployments at new sites.

Manpower for the DØ "part" of the SAM-Grid project in the main follows
from two routes: the core SAM team provides the lead, with significant
support (in particular testing) from D∅ collaborators associated with
facilities with significant computing resources (for the most part
shared with LHC experiments). Effective interplay between the central
team and institutional based effort has been essential for progress. It
is the institutional based effort that is responsible for installing,
maintaining and running the DØ grid software, and delivering the Monte
Carlo and reprocessed data to the collaboration. To ensure that the
developments are well integrated with / matched to the experiment we
have needed post-doc and guest scientist support from FNAL CD, as well
as benefiting from the overall project integration / management provided
by those at the associated scientist level. Additional manpower is
required to ensure that sufficiently timely progress can be made on all
fronts listed above.

### SAM-Grid Performance

The use of the SAM data handling system, as measured by the volume of
data consumed, grew modestly compared to the previous review period (now
= 8/2004-8/2005; previous = 8/2003-8/2004). There were 106,195 projects
run on fifty SAM stations world wide. The top ten most active stations
are shown in Table 5.

  ----------------------- ----------------------- -----------------------
  Station                 \# Projects this period \# Projects previous
                                                  period

  cabsrv1                 36,111                  12,180

  cabsrv2                 16,942                  (not present)

  clued0-v5               10,665                  12,094

  fnal-farm               7,828                   8,034

  ccin2p3-analysis        5,233                   7,206

  central-analysis        5,225                   16,326

  cab                     4,938                   26,615

  westgrid-ubc            4,318                   (not present)

  clued0                  3,754                   (was clued0-v5)

  ccin2p3-grid0           2,060                   (not present)
  ----------------------- ----------------------- -----------------------

Table 5: Top ten active stations from 8/1/04-8/1/05 (this period).

The comparison to the previous period is somewhat confused by the
reorganization of the SAM stations (e,g, central-analysis was
discontinued, cab was renamed to cabsrv2), but it is clear that the use
of the Fermilab analysis farms has increased from the previous period by
\~50%. ccin2p3 and westgrid are remote stations. The data consumption
statistics are shown in Table 6.

  ----------------- ----------------- ----------------- -----------------
  Station class     \# of Files       Billions of       PetaBytes of Data
                                      Events            

  Fnal-analysis     3,794,279         52.9              2.2

  Fnal-farm         392,957           1.1               0.2

  Remote            644,316           2.1               0.3
  ----------------- ----------------- ----------------- -----------------

Table 6: Data consumed from 8/1/04-8/1/05

While the consumption on the analysis and reconstruction farm
(Fnal-farm) increased during this period compared to the previous
period, the consumption on the remote sites dropped significantly (about
half the number of events compared to the previous period). This drop
could be due to a more efficient reprocessing effort and more effort
going into MC production (which consumed few files). File consumption
errors were analyzed in the report for the previous period. There was
little change in those values for this year.

## Tools

As well as suitable data handling and grid infrastructures we require
the appropriate tools to manage DØ-specific applications and their
interactions with this infrastructure, including applications such as
reconstruction software aimed at a production environment as well as
user applications. Here we briefly describe the ongoing programme of
tool development & integration.

### OS Compatibility, Run Time Environment and Sandboxing

To use a grid environment we need to need to package and export our
software suitably, as well as ensuring that it will run in the remote
environment, despite "local" differences. To address these needs DØ has
developed an OS-compatibility system, a developer-level system to
publish lists of required files (*d0rte)* and a sandboxing mechanism to
package the user environment and pass it to SAM-Grid.

The OS compatibility system consists of UPS/UPD products that accompany
each release, providing enough system-level code with each release to
allow executables to run on many flavours and versions of linux.
*D0rte*, already used to create the Monte Carlo (MC) tarballs
distributed to remote production farms, provides developers with a
straightforward mechanism to publish a list of required libraries and
runtime files for their component or executable. In considering how best
to integrate *d0rte* with DØRunjob (see next section), the DØ sandbox
product was created as part of the runjob project. This allows for the
creation of a hierarchy of environments starting from the general DØ
release and ending with a small \"user\" tarball that can be quickly and
easily distributed to many nodes in a farm. Sandboxing of DØ executables
is now being tested using this new infrastructure.

### DØ Tools, DØRunjob and Runjob

Historically people in DØ have used two independent packages to run
their executables: *D0tools* and MCRunjob. Whilst the former's
traditional role was code testing and debugging with local job
submission it organically grew such that it became un-maintainable and
could not be extended to the grid environment. The latter was primarily
used for production activities, originally MC (and now reprocessing) on
remote farms as its strength is workflow management i.e. chaining
together appropriately the various executables. While it was originally
referred to as MCRunjob it has since been rewritten, generalized and
adopted as a Fermilab Computing Division project known simply as Runjob.
The DØ-specific interface to Runjob is referred to as DØRunjob. For ease
of long term maintenance we decided to consolidate these two executables
by re-writing *d0tools* as a command line user interface to DØRunjob. As
a first step DØ tools has been re-implemented in python. As planned
originally it could then effectively become a macro-generator for
DØRunjob while maintaining the existing *d0tools* UI, so addressing user
concerns about the user interface to DØ Runjob (users strongly dislike a
macro language approach to simple tasks) while allowing support for grid
tools to be concentrated in Runjob / DØRunjob. Support for Runjob and
DØRunjob was provided by the Fermilab Computing Division and UK
personnel (GridPP) respectively; however the long term future of the
common runjob project is under question, a cause for concern.
Furthermore the original plan was for DØRunjob to be available, with the
interfaces for *d0tools* and the new sandboxing mechanisms described,
for the current p17 reprocessing. Unfortunately the release is now
expected for autumn of this year. The experiment is critically dependent
on individual effort in this area at the interface between grid and
experiment developers and the delays to DØRunjob serve as typical
example. To help remedy this, DØ carried out an internal review of the
DØRunjob project over the summer. The reviewers agreed to the revised
schedule, but strongly commented that the extension of the project to
user analysis would require extra effort for it to be accomplished in a
timely manner.

# Manpower Requirements and Budget Summary

In view of the amended charge this year we have extended this section to
include areas of "vulnerability" and to list our "requirements" both in
terms of manpower and resources to ensure a full success of Run II.
These are discussed in section 8.1.

In sections 8.2 and 8.3 we present our usual two budgetary concepts;
namely the \"yearly cost\" targeted towards estimating the best use of
the FNAL equipment budget and the \"virtual center\" that would be
required to meet all of DØ\'s computing needs. As in previous years we
budget for an increased rate to tape in Run IIb as in the original Run
II plans. To mitigate the incremental costs we split this increase
across two years, increasing the L3 rate to tape to 75Hz in FY06 and
100Hz thereafter, to follow the expected luminosity profile. The physics
motivation for such an increase is discussed in section 8.1.

## Requirements

### Hardware

The original plan for Run IIb called for an increased rate to tape to
match the increased instantaneous luminosity. The experiment has
recently produced a separate document detailing the continued need for
such an increase, see ref ^3^ for details. A brief summary is given
here.

The DØ collaboration has prudently handled data manipulation and
economically managed computing resources, resulting in efficient use of
resources and successful publication of physics analyses. By way of
examples: Data collection efficiency has averaged 84% over Run IIa and
now averages close to 90%, \~70% of all data taken are finally used in
at least one physics skim, processing keeps pace with data collection
and in some analyses over 0.6 fb^-1^ of the 0.8 fb^-1^ of data recorded
to tape have already been incorporated. At present the data rate to tape
is limited by offline computing capabilities to 50 Hz. Since 2001, the
collaboration has planned an integrated approach to the expected Run IIb
increase in luminosity by improving trigger rejection and doubling the
data rate to tape. These upgrades are necessary to maintain current
measurement and discovery sensitivity for the experimental priorities
such as precision measurement of the W mass, discovery of single top
production, investigation of the B-sector, and searches for SM and SUSY
Higgs. Any reduction in the offline capabilities effectively reduces the
Tevatron luminosity.

Studies carried out by all physics groups are detailed in ref ^3^. At a
luminosity of 1 x 10^32^cm^-2^s^-1^ and with current trigger
requirements the highest energy jet triggers, the W mass measurement,
the high priority top and B-physics programs, and the searches for SM
and non-SM Higgs already saturate the 50 Hz bandwidth. Reasonable
adjustments of the thresholds to maintain 50 Hz at 2x10^32^cm^-2^s^-1^
will entail a 20% loss in trigger efficiency for the W mass, efficiency
losses of at least 15% for the top trigger suite, statistics limited
b-quark measurements, and up to 20% loss in the high priority Higgs
searches. This essentially translates into a loss of \~1-2 fb^-1^ for
the remainder of the run. Furthermore remaining at 50Hz will require
increased manpower to develop the necessary trigger algorithms and to
deal with the additional trigger complexity in the offline analyzes. We
thus request funding to increase the rate to tape for Run IIb as
originally planned. In an attempt to mitigate the additional cost and
match the expected luminosity profile we request incremental
restorations to the computing equipment budget to allow us to increase
the rate to 75Hz in FY2006 and to 100Hz in FY2007. Fuller costings are
given in Section 8.2.

### Manpower Issues

As part of the Directorate's Run II manpower study the CDF and DØ
experiments along with the Computing and Particle Physics Divisions have
produced a report detailing manpower requirements in the offline
computing area. Areas of vulnerability and recommendations to overcome
them have been made. They can be broken down into two areas: those
pertaining to areas of immediate vulnerability, and those that apply in
particular to future strategies for dealing with the increasing data
volumes associated with the increasing luminosity, namely a full,
efficient use of the grid. The taskforce results will be discussed
separately and here focus is given to the DØ aspects, though it should
be noted that the taskforce's conclusions are basically
cross-experiment.

Taking the non-grid issues first; such vulnerabilities may arise through
personnel problems, with limited suitably qualified experts available
from either the experiment or CD. The following areas fall into this
category:

-   Database support: Effort is required to bring the luminosity and
    trigger data base projects to completion, and to provide suitable
    support for long term operation of the various database projects at
    DØ.

-   Online system: Currently, support for the critical online systems is
    carried out largely by a single individual.

-   Offline code management, build and distribution systems. This
    important task has long been staffed by a single individual, from
    CD, who has had now to be primarily redeployed.

Additionally there are areas where central consolidation of hardware
support could reduce overall manpower needs. These include:

-   Level-3 trigger hardware support: Currently, collaborating
    institutions support and maintain both the service-level and the
    computing farm hardware.

-   CluedØ support: This is currently provided by members of the
    collaboration.

These issues are under discussion with the Computing Division. Solutions
which both satisfy the immediate and longer term are being sought,
building where appropriate on CD's stated intention to increase the
consolidation of key central tasks. Such consolidation is one of the
taskforce recommendations.

Increased use of shared resources using grid technologies offers the
most viable route to the increased computing resources required as Run
II progresses. This route has long been the plan of both DØ and the
Computing Division, and there is strong collaborative effort. Using
common solutions wherever possible, significant progress has been made.
Some of the most recent is detailed in this document. Despite this
progress significant vulnerabilities remain.

-   Deployment of SAM-Grid and grid technologies: Deployment of the
    required products remains specialized and a labour intensive
    activity. Achieving such deployments in a manpower effective way
    will become increasingly important as the need for grid-based
    services grows.

-   SAM-Grid interoperability: The core feature of grid technologies is
    interoperability between grid sites, enabling a V0 (in this case the
    DØ collaboration) to access sufficient shared resources. The
    Computing Division and the DØ experiment have invested effort into
    making SAM-Grid interoperable with existing LHC grid infrastructure
    (LCG and OSG). Completion of this project will be required for the
    long-term success of DØ computing.

-   User analyses on the grid: We have concentrated thus far on
    production activities, i.e. Monte Carlo and data reprocessing.
    Primarily this has been because these areas present our most
    pressing need; SAM provides worldwide data-handling and cpu local to
    sites or at FNAL has proven adequate so far. Individual data
    analysis jobs on the grid represent the most challenges, and so to
    achieve this in timely fashion will require additional effort.

To address these issues the taskforce recommended:

-   "Grid deployment team. The Computing Division should form a grid
    deployment team charged with the responsibility to assist remote
    sites deploy the products required by the experiments to better
    exploit a grid-based computing model. The additional personnel will
    allow the existing developer base to focus on such tasks as creating
    viable grid-based analysis models for the Run II experiments."

-   "Interoperability. Fermilab should aggressively pursue the
    implementation of interoperable solutions for distributed computing
    by the Run II experiments. The first goal should be to make SAM-Grid
    fully interoperable on OSG and LCG computing resources. Funding for
    these projects should be sufficient to ensure success in a timely
    manner."

The taskforce also recommended increasing the number of guest scientists
and the number of Associate Scientists and Research Associate positions.
We would like to confirm our support for this, as these persons playa
key role within the experiment's computing, particularly topical are
those who ensure the effective integration of grid technologies into the
experiment.

## Yearly Cost

There are four major costing areas used for equipment budget planning;
worker nodes for processing tasks, disk and associated servers,
components for the mass storage and infrastructure hardware needed to
support the computing and user access. To the greatest extent possible,
we use our knowledge about the past use of the system and past hardware
costing trends to estimate future need and equipment cost. In general,
we purchase the bulk of the equipment at the end of the fiscal year in
order to get the most computing value for the budget. For out-year
projections, we assume that the hardware cost per unit for compute nodes
and fileservers remains constant, while scaling up performance based on
our understanding of history and trends. This scaling remains
speculative; in FY2005; the performance per cost varied by 20% between
available choices based on DØ benchmarks. The benchmark performance for
CDF and DØ code on a Xeon 3.2 differed by 25%. Our best estimate at this
time is to scale up performance by 25% per year, using this year's
purchase as a baseline.

In our experience, the data collection rate (Table 7) and total number
of events (Table 2) are useful estimators for scaling usage as a
function of time. We assume an increased rate for Run IIb, though
stepped, rather than a direct increase to 100Hz as originally planned,
bearing in mind that we purchase computing at the end of the fiscal
year---thus the 2005 purchases cover 75 Hz in 2006.

The estimate of computing cycles needed for a task (reconstruction,
Monte Carlo generation, user analysis) will depend on the efficiency at
which a facility can run a task, the rate at which the task needs to be
accomplished and the processing time per event. We also add some
contingency to the estimate.

  --------------- ---------- ----- ---------- ---------- ---------- ---------- ----------
                                   **2005**   **2006**   **2007**   **2008**   **2009**

  **Peak event                     75         100        100        100        0
  rate**                                                                       

  **Average event                  25.86      34.48      34.48      34.48      0
  rate**                                                                       

  **Weekly                         37.5       50         50         50         0
  average**                                                                    

  **Geant MC                       3.88       3.45       3.45       3.45       3.45
  rate**                                                                       

  **PMCS MC                        0.00       3.45       3.45       3.45       3.45
  rate**                                                                       

  **Events        1.00E+09                                                     
  collected**                                                                  
  --------------- ---------- ----- ---------- ---------- ---------- ---------- ----------

Table 7 Data rates in Hz. Note that the data rate in the 200x column is
used to calculate the computing needs at the beginning of the following
year as we generally purchase equipment as late as possible. For
example, the data rate in the 2005 column refers to an increase in the
rate in 2006. We assume for this document that data collection ends in
2009, covered by the 2008 purchases and do not present projections for
2009.

For the FNAL farm, we assume that the facility keeps up with data
collection. The current p17 timing appears linear in the current
luminosity regime (see Figure 1), reducing our ability to extrapolate to
higher luminosities at this time. Using Figure 1 we assume 85 GHz\*sec
per event in 2006 budget calculations with 100 GHz\*sec per event after,
consistent with past estimates. Keeping to this linear extrapolation
will be very challenging, requiring continued effort on *d0reco*, and
may require contingency at the 20% level. We calculate the total number
of cycles needed to keep up and subtract off the existing system, and
then estimate the number of nodes to purchase. Because the existing
plant has an effect on the out-year projections, the purchase of
dual-core Opteron 265 systems in FY-2005 has a positive effect. Those
systems have been benchmarked on DØ reconstruction code as 2.45 times
the speed of PIII per processor. This enabled us to purchase 20% more
computing cycles for the same cost than we initially planned, putting us
in a good position for the rate increase. Table 8 details this process
step by step.

  ----------------- -- ------------- -------------- --------------- ---------------
  **Year**                           **2006**       **2007**        **2008**

  **Average Rate**                   34.48          34.48           34.48

  **Efficiency**                     80%            80%             80%

  **Contingency**                    20%            20%             20%

  **Reco time**                      85             100             100

  **Required CPU**                   2092759        2462069         2462069

  **Existing                         902761         1704485         2025993
  system**                                                          

  **Nodes to                         208            106             49
  purchase**                                                        

  **Node Cost**                      \$666,665      \$339,534       \$156,352
  ----------------- -- ------------- -------------- --------------- ---------------

Table 8: Primary reconstruction cost estimate.

The same procedure is followed for the FNAL based analysis system, using
the average weekly data consumption and past system usage as estimators.
On the analysis system, the usage pattern is less steady than on the
farm, however, the average usage scales with the number of events
collected and reconstructed. Mechanically, we maintain a series of
linked Excel spreadsheets that contain the assumptions, and
calculations. Current versions of these spreadsheets are available at
ref. [^11]. For the FNAL farm estimate, in addition to the primary
reconstruction, an estimate for the number of computing cycles needed
for the post-production fixing was made, and found to be within the
contingency. The estimates for disk are detailed in Section 5.4.

Mass storage estimates focus on the amount of tape to buy and insuring
that we have enough slots in the existing robots to accommodate them. We
do not yet have a good data rate driven model for estimating the number
of drives to purchase based on fundamental parameters, and typically use
the capacity of the system at peak as an indicator that more are needed
(or not).

The other large spending category is infrastructure, which covers
equipment for database support, interactive use, networking, and I/O
support for the farm and analysis cluster. Based on the estimated number
of nodes and fileservers to purchase at FNAL, we make a crude estimate
of the networking costs, which is refined and then added to the
infrastructure total. For other items, we look at the aging elements of
the system and work out strategies for replacement. In the past year, we
replaced most major components, at a lower than estimated cost. The
Network Appliance that will replace the SGI machine for serving home
areas cost \$63K instead of the estimated \$120K. Additionally, we were
able to avoid the purchase of a new switch by working to judiciously
reuse components and by purchasing dual core worker nodes. At this
point, we estimate that we'll average around \$100K/year in
infrastructure costs, primarily disk for the database machines,
networking components and disk for the home areas.

As a point of interest, comparisons between the estimates for 2004 and
2005 (\$1.5M guidance) and actual purchases are shown in Table 9. In
general, within a constant budget, DØ has typically chosen to purchase
worker nodes over infrastructure to maintain rate to tape. In past year,
we have addressed the aging infrastructure, which came in well under the
budget.

  ---------------- ------------- ------------- ------------- -------------
                   2004                        2005          

                   Projected     Purchased     Projected     Purchased

  FNAL Analysis    \$339,000     \$277,000     \$420,000     \$400,000
  cpu                                                        

  FNAL             \$83,000      \$370,000     \$450,000     \$400,000
  Reconstruction                                             

  File             \$490,000     \$350,000     \$360,000     \$325,000
  Servers/disk                                               

  Mass Storage     \$230,000     \$254,700     \$40,000      \$20,000

  Infrastructure   \$290,000     \$140,000     \$500,000     \$276,171

  FNAL Total       \$1,432,000   \$1,391,700   \$1,770,000   \$1,421,171
  ---------------- ------------- ------------- ------------- -------------

Table 9: Projected and spent budgets for FY2004 and FY2005.

The current cost projections for 2006-2008 are shown in Table 10. Even
with the data collection rate increase to 100 Hz, we are only slightly
above the guidance of \$1.5M per year for 2006 and below for 2007 By
2008, we fit within the projected \$1M guidance. This represents a
difference relative to the figures that we presented in the 100 Hz
document (ref ^3^), There are several factors at work. The first is a
reassessment of the infrastructure and tape drive costs; we are not
planning on purchasing LTOIII drives or other major hardware, which
represents a difference of \$200K. Additionally, we have assumed that
the analysis programs will become faster by 20%, through the change of
format. Finally, the 2005 purchases had a 25% improvement in performance
for cost for DØ with the Dual Core Opterons. Purchasing more computing
cycles than anticipated in 2005 reduces the amount of computing needed
in 2006 for a fixed set of assumptions, and the 2005 costs are used as a
baseline for estimating the performance for cost for the out-years.

If the guidance is reduced to \$1.25M for 2006, DØ experiment would
either have to reconcile the physics program with a 75 Hz output rate
or, more likely, would aggressively work to further reduce the analysis
cost for worker nodes and disk. We do not fully understand the effects
of the Common Analysis Format however it is a reasonable assumption that
the efficiency and speed will enable us to save in the area of analysis.
Additionally, there are options to pursue offsite analysis or to use
FermiGrid as a way of levelling out the analysis needs to steady-state
but this will require additional manpower to be achieved quickly.

  ----------------------- --------------- --------------- ---------------
                          Purchase 2006   Purchase 2007   Purchase 2008

  FNAL Analysis CPU       \$449,308       \$475,835       \$404,720

  FNAL Reconstruction     \$666,665       \$339,534       \$156,352

  File Servers/disk       \$ 348,500      \$328,000       \$307,500

  Mass Storage            \$57,000        \$97,500        \$97,500

  Infrastructure          \$100,000       \$100,000       \$100,000

  FNAL Total              \$1,621,473     \$1,340,869     \$1,066,072
  ----------------------- --------------- --------------- ---------------

Table 10: Projected budgets for FY2005 through FY2008.

## Virtual Centre Value

As well as the technological developments necessary for distributed /
grid computing it is also necessary to recognise the in-kind
contribution that the providing of significant computing resources
represents. Our calculation of the value of the Virtual Centre is
designed to recognize that non-FNAL computing contributions can be used
to offset contributions to the DØ operating fund. The value of the
centre is calculated by assuming that all necessary nodes and
fileservers to do all production activities (including reprocessing and
MC production) and FNAL analysis would be purchased in the current year.
In this way, for past fiscal years, we can calculate a cash value based
on a country\'s fractional contribution to a given activity as measured
by the number of events delivered[^12]. The 2005 value is calculated
from the observed production, including the six months of intense use of
the computing for the reprocessing. (not sure I understand this sentence
as MC was more before repro started, and are these FYs anyway?) We
assume that we will continue to reprocess at the remote centres in the
out-years---fewer events, but with a shorter duration, which leads to a
comparable value as DØ would request a large amount of computing for a
relatively short time while the rest of time that computing would be put
to other uses.

For future years, we estimate the value in a way similar to that
described above. Currently, analysis at remote facilities is not
considered to be part of the virtual centre; however, that is likely to
change once SAM-Grid is fully deployed. Note that the concept of "value"
is one that we are still developing. The estimates of the infrastructure
value are difficult to make for systems that are purchased and function
for many years at time. We do not currently assign a value for
infrastructure. For the FNAL mass storage system, we have calculated the
value in the past based on pro-rating drives and the libraries. At this
point, we assign that a constant value of \$800K, rather than attempt to
adjust yearly for a small number of added drives. The virtual centre
cost estimate is given in Table 11.

![](media/image5.wmf)

Table 11: Virtual centre cost estimates for FY2005 through FY2009.

.

# Summary

As can be seen DØ computing continues to operate well. We have continued
to make progress along our long term plan of migration from distributed
computing to grid computing, most noticeably with the success of the p17
reprocessing using SAM-Grid. This is the largest such grid activity in
HEP.

Despite the significant progress / success, DØ computing however
continues to be manpower limited, and in the last year vulnerabilities
due to this have become particularly apparent. These relate both to the
immediate situation and to the continued development of grid
technologies needed to provide access to the larger resources required
as the Run II data sets increase. Overcoming these issues is essential
for the success of DØ.

In addition we strongly request the restoration of sufficient funds to
allow us to write to tape at 100Hz in Run IIb. We can achieve this with
funding which fits the \$1.5M guidance for the next two years, and
subsequently below the \$1M projected guidance for later years. The tape
budget needed to achieve this is very reasonable. These modest
additional sums are entirely motivated by the effective luminosity
saving of order 1-2fb^-1^ that they represent.

# Appendix 1

![](media/image6.jpeg){width="5.041666666666667in"
height="6.517361111111111in"}

Figure 5: Principal elements of the network infrastructure supporting DØ
Computing.

![](media/image7.jpeg){width="1.560236220472441in"
height="0.44448818897637793in"}![](media/image8.png){width="2.3437007874015747in"
height="0.6311023622047244in"}![](media/image9.png){width="2.3437007874015747in"
height="0.6307086614173228in"}![](media/image10.png){width="2.3437007874015747in"
height="0.6307086614173228in"}![](media/image11.png){width="2.3437007874015747in"
height="0.6311023622047244in"}![](media/image12.png){width="2.3437007874015747in"
height="0.6307086614173228in"}

![](media/image7.jpeg){width="1.588976377952756in"
height="0.452755905511811in"}![](media/image13.png){width="2.3874015748031496in"
height="0.6429133858267716in"}![](media/image14.png){width="2.3874015748031496in"
height="0.6429133858267716in"}![](media/image15.png){width="2.3874015748031496in"
height="0.6425196850393701in"}

Figure 6: Network utilization, in bits per second, within DØ.

#  Need these xx's

# References

[^1]: SAM-Grid is supplied by the Fermilab Computing Division via a
    joint project with participation from the Run II Computing and
    Analysis Department, the CCF Department, and the CEPA Department, as
    well as external effort from the GridPP project and other
    collaboration effort.

[^2]: It should be noted that the above number contains contingencies
    that account for file delivery and inefficiencies occurring in
    cluster operations. Thus the actual running speed of d0reco is much
    faster.

[^3]: JIM, Job Information and Monitoring, provides the job submission
    and monitoring aspects for SAM-Grid.

[^4]: The p17 data reprocessing webpage:
    <http://www-d0.fnal.gov/computing/reprocessing/p17/> ;

    Fermilab Press release (26 Apr. 2005,
    http://www.fnal.gov/pub/presspass/press_releases/dzero_datagrid.html)

    published in FermiNews (28 Apr. 2005,
    <http://www.fnal.gov/pub/today/archive_2005/today05-04-28.html>);

    Science Grid (29 Apr. 2005,
    <http://www.interactions.org/sgtw/archive/20050428.html>), Science
    Daily (5 May 2005,
    <http://www.sciencedaily.com/releases/2005/05/050504175948.htm>).

[^5]: The luminosity block is the fundamental unit of time for the
    luminosity measurement.

[^6]: <http://d0db-dev.fnal.gov:8514/atrig/TDB_Plan.html>

[^7]: <http://d0server1.fnal.gov/projects/Computing/Reviews/Sept2005/Index.html>

[^8]: A. Rajendra, \"Integration of the SAM-Grid Infrastructure to the
    DØ Data Reprocessing Effort\", Thesis of Master in Computing
    Science, The University of Texas, Arlington, Dec. 2005.
    <http://www-d0.fnal.gov/computing/grid/papers/Thesis-A.Rajendra.pdf>

    A. Nishandar, \"Grid-Fabric Interface For Job Management In
    SAM-Grid, A Distributed Data Handling And Job Management System For
    High Energy Physics Experiments\", Thesis of Master in Computing
    Science, The University of Texas, Arlington, Dec. 2004.
    <http://www-d0.fnal.gov/computing/grid/papers/Thesis-Aditya.pdf>

    S. Jain, \"Abstracting the hetereogeneities of computational
    resources in the SAM-Grid to enable execution of high energy physics
    applications\", Thesis of Master in Computing Science, The
    University of Texas, Arlington, Dec. 2004
    <http://www-d0.fnal.gov/computing/grid/papers/Thesis-Sankalp.pdf>

    A. Nishandar, D. Levine, S. Jain, G. Garzoglio, I. Terekhov,
    \"Extending the Cluster-Grid Interface Using Batch System
    Abstraction and Idealization\", to appear on Cluster Computing and
    Grid 2005 (CCGrid05), Cardiff, UK, May 2005.
    <http://www-d0.fnal.gov/computing/grid/papers/CCGrid2005-sj.pdf>

    A. Nishandar, D. Levine, S. Jain, G. Garzoglio, I. Terekhov, \"Black
    Hole Effect: Detection and Mitigation of Application Failures due to
    Incompatible Execution Environment in Computational Grids\", to
    appear on Cluster Computing and Grid 2005 (CCGrid05), Cardiff, UK,
    May 2005.
    <http://www.d0.fnal.gov/computing/grid/papers/CCGrid2005.pdf>

[^9]: <http://www-d0.fnal.gov/computing/grid/doc/SAMGrid-LCG-integration.pdf>

[^10]: <http://www-d0.fnal.gov/computing/grid/doc/SAMGrid-LCG-integration-Lyon-report.pdf>

[^11]: <http://d0server1.fnal.gov/projects/Computing/Reviews/Sept2005/Index.html>

[^12]: <http://d0server1.fnal.gov/projects/Spokes/FinancialCommittee/Scrutiny.html>

    #  {#section .unnumbered}