Distributed IT Infrastructure Plan

WBS 2.3.2

Subproject of Facilities

Software and Computing Project

U.S. ATLAS Collaboration

7/31/2001

Version 1.4

Table of Contents

[**1** **Introduction** **5**](#introduction)

[2 ATLAS Requirements 5](#atlas-requirements)

[2.1 Definitions and Simulation 5](#definitions-and-simulation)

[2.2 Use Cases 6](#use-cases)

[2.3 Requirements 6](#requirements)

[2.3.1 Uniform Requirements Definition
6](#uniform-requirements-definition)

[2.3.2 Communication, Liaison 6](#communication-liaison)

[3 Grid Architecture 6](#grid-architecture)

[3.1 Architectures Design 6](#architectures-design)

[3.2 Protocols and Standards 6](#protocols-and-standards)

[4 Grid Software Services 6](#grid-software-services)

[4.1 Grid Software Environments 6](#grid-software-environments)

[4.1.1 Grid Toolkit Support 6](#grid-toolkit-support)

[4.1.1.1 Globus Toolkit Liaison 6](#globus-toolkit-liaison)

[4.1.1.2 Condor Toolkit Liaison 6](#condor-toolkit-liaison)

[4.1.1.3 PPDG Toolkit Liaison 6](#ppdg-toolkit-liaison)

[4.1.1.4 DATAGRID Toolkit Liaison 6](#datagrid-toolkit-liaison)

[4.1.1.5 GriPhyN Virtual Data Toolkit Liaison
6](#griphyn-virtual-data-toolkit-liaison)

[4.2 Grid Information Infrastructure
6](#grid-information-infrastructure)

[4.2.1 Resource Specification, Expression
6](#resource-specification-expression)

[4.2.2 LDAP Directories 6](#ldap-directories)

[4.3 Grid User Environments 6](#grid-user-environments)

[4.3.1 Grid Portals 6](#grid-portals)

[4.4 Grid Security Systems 7](#grid-security-systems)

[4.5 Grid Workflow Management 7](#grid-workflow-management)

[4.5.1 Architecture Definition 7](#architecture-definition)

[4.5.2 Co-allocation of Resources 7](#co-allocation-of-resources)

[4.5.3 Distributed Scheduling 7](#distributed-scheduling)

[4.5.3.1 Connecting the Atlas Grid Sites
8](#connecting-the-atlas-grid-sites)

[4.5.3.2 Prototyping 8](#prototyping)

[4.5.3.3 Site Management Applications 9](#site-management-applications)

[4.5.4 Uniform Interfaces to Resource Managers
9](#uniform-interfaces-to-resource-managers)

[4.5.5 Grid Policy Management 9](#grid-policy-management)

[4.6 Grid Data Management 9](#grid-data-management)

[4.6.1 Generic Interfaces for Mass Storage
9](#generic-interfaces-for-mass-storage)

[4.7 Grid Monitoring Services 10](#grid-monitoring-services)

[4.7.1 Grid Monitoring Architecture Definition
10](#grid-monitoring-architecture-definition)

[4.7.2 Telemetry Databases 10](#telemetry-databases)

[4.7.3 Grid Performance Monitoring Toolkits
10](#grid-performance-monitoring-toolkits)

[4.7.4 Grid Resource Management Information Systems
10](#grid-resource-management-information-systems)

[4.7.5 Grid Operations Centers 10](#grid-operations-centers)

[4.7.5.1 Requirements and R & D 11](#requirements-and-r-d)

[4.7.5.2 Trouble Ticket System 11](#trouble-ticket-system)

[4.7.5.3 Certificate Authority Services
11](#certificate-authority-services)

[4.8 Grid Administration Infrastructure
11](#grid-administration-infrastructure)

[4.8.1 Grid User Registration System 11](#grid-user-registration-system)

[4.8.2 Grid Automated Account Management Systems
11](#grid-automated-account-management-systems)

[4.8.3 Grid Accounting Systems 11](#grid-accounting-systems)

[5 Grid Testbeds 11](#grid-testbeds)

[5.1 Lessons Learned for Building Large-Scale Grids
11](#lessons-learned-for-building-large-scale-grids)

[5.2 Requirements 15](#requirements-1)

[5.3 ATLAS-DataGrid Testbed 15](#atlas-datagrid-testbed)

[5.4 ATLAS-iVDGL Testbed 15](#atlas-ivdgl-testbed)

[5.5 ATLAS-PPDG Testbed 15](#atlas-ppdg-testbed)

[6 Tier 2 Regional Centers 15](#tier-2-regional-centers)

[6.1 Tier 2 Facility Hardware 15](#tier-2-facility-hardware)

[6.1.1 Physical Infrastructure 15](#physical-infrastructure)

[6.1.2 Design Tier 2 Physical Infrastructure
15](#design-tier-2-physical-infrastructure)

[6.2 Tier 2 Facility Software 15](#tier-2-facility-software)

[7 Tier 3 and Tier 4 Facilities 15](#tier-3-and-tier-4-facilities)

[7.1 Tier 3 (Institute) Facilities 16](#tier-3-institute-facilities)

[7.2 Tier 4 (Desktop) Systems 16](#tier-4-desktop-systems)

[8 Collaborative Services 16](#collaborative-services)

[8.1 Web Services 16](#web-services)

[8.2 E-Mail 16](#e-mail)

[8.3 Document Distribution 16](#document-distribution)

[8.4 News Distribution systems 16](#news-distribution-systems)

[8.4.1 Usenet news or similar (asynchronous distributed)
16](#usenet-news-or-similar-asynchronous-distributed)

[8.4.2 Hypernews and the like (central repository)
16](#hypernews-and-the-like-central-repository)

[8.4.3 listservers and other mailing lists
16](#listservers-and-other-mailing-lists)

[8.5 Software Management and Distribution
16](#software-management-and-distribution)

[8.5.1 Source code distribution for developers
16](#source-code-distribution-for-developers)

[8.5.2 Distribution tools (eg rpm, PACman, AFS)
16](#distribution-tools-eg-rpm-pacman-afs)

[8.5.3 Distribution of ATLAS software environment
16](#distribution-of-atlas-software-environment)

[8.5.4 Documentation 16](#documentation)

[8.6 Web Lectures 17](#web-lectures)

[8.6.1 Archive 17](#archive)

[8.6.2 Indexing 17](#indexing)

[8.6.3 Producing Web Lectures 17](#producing-web-lectures)

[8.6.4 Software development 17](#software-development)

[8.6.5 New developments in this technology
17](#new-developments-in-this-technology)

[8.7 Video Conferencing 17](#video-conferencing)

[8.7.1 Room sized systems (eg Access Grid)
17](#room-sized-systems-eg-access-grid)

[8.7.2 Desktop systems (eg VRVS) 17](#desktop-systems-eg-vrvs)

[8.7.3 Vendor Systems (eg Polycom) 17](#vendor-systems-eg-polycom)

[8.7.4 Hybrid systems (eg phone bridge & VRVS w/o audio)
17](#hybrid-systems-eg-phone-bridge-vrvs-wo-audio)

[8.7.5 New Developments 17](#new-developments)

[8.8 Tele-immersion Environments 17](#tele-immersion-environments)

[8.8.1 Single site (asyncronous use) 17](#single-site-asyncronous-use)

[8.8.2 Interaction between Remote site (syncronous)
17](#interaction-between-remote-site-syncronous)

[8.8.3 Interoperability between different systems/software
17](#interoperability-between-different-systemssoftware)

[8.8.4 New Developments 17](#new-developments-1)

[8.9 Whiteboard / Notepad 17](#whiteboard-notepad)

[8.9.1 Remote whiteboard collaboration,
17](#remote-whiteboard-collaboration)

[8.9.2 Notepad device 17](#notepad-device)

[8.10 Instant Messaging 17](#instant-messaging)

[8.11 New Collaborative Technologies
17](#new-collaborative-technologies)

[8.12 Liason 17](#liason)

[9 Network Systems 17](#network-systems)

[9.1 Network Connectivity Requirements
18](#network-connectivity-requirements)

[9.1.1 Data Distribution Models 18](#data-distribution-models)

[9.1.2 Tier 1 Connectivity 18](#tier-1-connectivity)

[9.1.3 Tier 2 Connectivity 18](#tier-2-connectivity)

[9.1.4 Tier 3 and Desktop (Tier 4) Connectivity
19](#tier-3-and-desktop-tier-4-connectivity)

[9.1.5 Uniform Requirements Definition
19](#uniform-requirements-definition-1)

[9.2 Network Services 19](#network-services)

[9.2.1 Security and Authentication Services
19](#security-and-authentication-services)

[9.2.2 QoS/\"Service Differentiation\" 19](#qosservice-differentiation)

[9.2.3 Bandwidth Brokering 20](#__RefHeading___Toc528642250)

[9.2.4 Monitoring and Measuring Services
21](#monitoring-and-measuring-services)

[9.2.5 Multicast (Possible uses for grid computing, concerns and
viability)
23](#multicast-possible-uses-for-grid-computing-concerns-and-viability)

[9.3 End to End Performance 24](#end-to-end-performance)

[9.3.1.1 Introduction 24](#introduction-1)

[9.3.1.2 Related Work 28](#related-work)

[9.3.1.3 Web100 29](#web100-1)

[9.3.2 Requirements 30](#requirements-2)

[9.3.3 Host-based Diagnostic Toolkits
30](#host-based-diagnostic-toolkits)

[9.3.4 Network-based Diagnostic Toolkits
30](#network-based-diagnostic-toolkits)

[9.4 Local Site Infrastructure Development
30](#local-site-infrastructure-development)

[9.4.1 Requirements 31](#requirements-3)

[9.4.2 Hardware Infrastructure 31](#hardware-infrastructure)

[9.4.3 Liaison and Support 31](#liaison-and-support)

[9.5 Operations and Liaison 31](#operations-and-liaison)

[9.5.1 Network Operations Centers 31](#network-operations-centers)

[9.5.2 Liaison 31](#liaison)

[9.5.2.1 CERN Liaison 31](#cern-liaison)

[9.5.2.2 Tier 1 Center Liaison 31](#tier-1-center-liaison)

[9.5.2.3 Tier 2 Centers Liaison 31](#tier-2-centers-liaison)

[9.5.2.4 Internet2 Liaison 31](#internet2-liaison)

[9.5.2.5 ESnet Liaison 31](#esnet-liaison)

[9.5.2.6 International Networks Liaison
31](#international-networks-liaison)

[9.5.2.7 HENP Networking Forum Liaison
31](#henp-networking-forum-liaison)

[9.5.2.8 Grid Forum Networking Group Liaison
31](#grid-forum-networking-group-liaison)

[9.6 Network R&D Activities 31](#network-rd-activities)

[9.6.1 Protocols 31](#protocols)

[9.6.2 Network Performance and Prediction
31](#network-performance-and-prediction)

[9.6.3 Network Security 31](#network-security)

[9.6.3.1 General Security Requirements
31](#general-security-requirements)

[9.6.3.2 Policy and Standards 31](#policy-and-standards)

[9.6.3.3 Network Device Security (Routers and switches)
32](#network-device-security-routers-and-switches)

[9.6.3.4 Firewalls 32](#firewalls)

[9.6.3.5 Current Events / Exploits / Advisories
32](#current-events-exploits-advisories)

[9.6.3.6 Web Security 32](#web-security)

[9.6.3.7 Encryption 32](#encryption)

[9.6.3.8 Security Monitoring (passive) 32](#security-monitoring-passive)

[9.6.3.9 Security Testing (active) 32](#security-testing-active)

[9.6.3.10 Software Tools 33](#software-tools)

[9.6.3.11 Software Updates 33](#software-updates)

[9.6.3.12 Node Security 33](#node-security)

[9.6.3.13 Authentication/Authorization 33](#authenticationauthorization)

[9.6.3.14 Specific Protocol Security Issues
33](#specific-protocol-security-issues)

[9.6.3.15 Liaison 33](#liaison-1)

[9.6.4 Virtual Private Networks 33](#virtual-private-networks)

[9.6.5 Technology Tracking and Evaluation
33](#technology-tracking-and-evaluation)

[9.6.6 IETF Liaison 33](#ietf-liaison)

[9.7 Network Cost Evaluation 33](#network-cost-evaluation)

[9.7.1 International Networks 34](#international-networks)

[9.7.2 Domestic Backbone Use 34](#domestic-backbone-use)

[9.7.3 Tier 1 Cost Model 35](#tier-1-cost-model)

[9.7.4 Tier 2 Cost Model 36](#tier-2-cost-model)

[9.7.5 Next Steps in Network Planning
36](#next-steps-in-network-planning)

[10 APPENDIX A: Network Notes 38](#appendix-a-network-notes)

[11 APPENDIX B: Relevant MONARC Information
39](#appendix-b-relevant-monarc-information)

# Introduction

Computing for U.S. ATLAS will rely on a distributed information
technology infrastructure, which includes distributed computing
resources and data stores interconnected by high-speed networks. Grid
middleware systems will be deployed to utilize these resources
efficiently.

# ATLAS Requirements 

## Definitions and Simulation

LHC analysis definitions, anticipated activities, and access patterns
have been studied and reported in the MONARC Phase 2 Report[^1]. The
MONARC report focused mainly on transatlantic links, and the links
between Tier 0 and Tier 1 regional centers.

Table 1. Characteristics of the main analysis tasks (MONARC)

+----------+---------------------+------------------+-----------------+
| :::      | ::: Definition-Term | :::              | :::             |
|  Definit | Re-Define AOD,      |  Definition-Term | Definition-Term |
| ion-Term | based on event TAG  | Define Group     | Physics         |
| [        | :::                 | datasets         | Analysis Jobs   |
| ]{.mark} |                     | :::              | :::             |
| :::      | ::: Definition-Term |                  |                 |
|          | (Tier 0,1)          | :::              | :::             |
|          | :::                 |  Definition-Term | Definition-Term |
|          |                     | (Tier 1)         | (Tier 2)        |
|          |                     | :::              | :::             |
+----------+---------------------+------------------+-----------------+
|          | ::: Definition-Term | :::              | :::             |
|          | Range               |  Definition-Term | Definition-Term |
|          | :::                 | Range            | Range           |
|          |                     | :::              | :::             |
+----------+---------------------+------------------+-----------------+
| :::      | ::: Definition-Term | :::              | :::             |
|  Definit | 0.5-4/month         |  Definition-Term | Definition-Term |
| ion-Term | :::                 | 0.5-4/month      | 1-8/day         |
| F        |                     | :::              | :::             |
| requency |                     |                  |                 |
| :::      |                     |                  |                 |
+----------+---------------------+------------------+-----------------+
| :::      | ::: Definition-Term | :::              | :::             |
|  Definit | 0.1-0.5             |  Definition-Term | Definition-Term |
| ion-Term | :::                 | 10-50            | 1-5             |
| C        |                     | :::              | :::             |
| PU/event |                     |                  |                 |
| (        |                     |                  |                 |
| SI95\*s) |                     |                  |                 |
| :::      |                     |                  |                 |
+----------+---------------------+------------------+-----------------+
| :::      | ::: Definition-Term | :::              | :::             |
|  Definit | ESD                 |  Definition-Term | Definition-Term |
| ion-Term | :::                 | DB query         | DB query        |
| Input    |                     | :::              | :::             |
| data     |                     |                  |                 |
| :::      |                     |                  |                 |
+----------+---------------------+------------------+-----------------+
| :::      | ::: Definition-Term | :::              | :::             |
|  Definit | 0.02-0.5 PB         |  Definition-Term | Definition-Term |
| ion-Term | :::                 | 0.02-0.5 PB      | 0.001-1TB (AOD) |
| Input    |                     | :::              | :::             |
| size     |                     |                  |                 |
| :::      |                     |                  |                 |
+----------+---------------------+------------------+-----------------+
| :::      | ::: Definition-Term | :::              | :::             |
|  Definit | DISK                |  Definition-Term | Definition-Term |
| ion-Term | :::                 | DISK             | DISK            |
| Input    |                     | :::              | :::             |
| medium   |                     |                  |                 |
| :::      |                     |                  |                 |
+----------+---------------------+------------------+-----------------+
| :::      | ::: Definition-Term | :::              | :::             |
|  Definit | AOD                 |  Definition-Term | Definition-Term |
| ion-Term | :::                 | Collection       | Variable        |
| Output   |                     | :::              | :::             |
| data     |                     |                  |                 |
| :::      |                     |                  | :::             |
|          |                     |                  | Definition-Term |
|          |                     |                  | (DDO)           |
|          |                     |                  | :::             |
+----------+---------------------+------------------+-----------------+
| :::      | ::: Definition-Term | :::              | :::             |
|  Definit | 10 TB               |  Definition-Term | Definition-Term |
| ion-Term | :::                 | 0.1-1TB (AOD)    | Variable        |
| Output   |                     | :::              | :::             |
| size     | ::: Definition-Term |                  |                 |
| :::      | (AOD)               |                  |                 |
|          | :::                 |                  |                 |
|          |                     |                  |                 |
|          | ::: Definition-Term |                  |                 |
|          | 0.1-1TB             |                  |                 |
|          | :::                 |                  |                 |
|          |                     |                  |                 |
|          | ::: Definition-Term |                  |                 |
|          | (TAG)               |                  |                 |
|          | :::                 |                  |                 |
+----------+---------------------+------------------+-----------------+
| :::      | ::: Definition-Term | :::              | :::             |
|  Definit | DISK                |  Definition-Term | Definition-Term |
| ion-Term | :::                 | DISK             | DISK            |
| Output   |                     | :::              | :::             |
| medium   |                     |                  |                 |
| :::      |                     |                  |                 |
+----------+---------------------+------------------+-----------------+
| :::      | ::: Definition-Term | :::              | :::             |
|  Definit | 5-15 days           |  Definition-Term | Definition-Term |
| ion-Term | :::                 | 0.5-3 days       | 2-24 hours      |
| Time     |                     | :::              | :::             |
| response |                     |                  |                 |
| (T)      |                     |                  |                 |
| :::      |                     |                  |                 |
+----------+---------------------+------------------+-----------------+
| :::      | ::: Definition-Term | :::              | :::             |
|  Definit | 1                   |  Definition-Term | Definition-Term |
| ion-Term | :::                 | 1/Group          | 10-100\         |
| Number   |                     | :::              | / Group         |
| of jobs  |                     |                  | :::             |
| in T     |                     |                  |                 |
| :::      |                     |                  |                 |
+----------+---------------------+------------------+-----------------+

## Use Cases

## Requirements

### Uniform Requirements Definition

### Communication, Liaison

# Grid Architecture

## Architectures Design

## Protocols and Standards

# Grid Software Services

## Grid Software Environments

###  {#section .unnumbered}

### Grid Toolkit Support 

#### Globus Toolkit Liaison

#### Condor Toolkit Liaison

#### PPDG Toolkit Liaison

#### DATAGRID Toolkit Liaison

#### GriPhyN Virtual Data Toolkit Liaison

###  {#section-1 .unnumbered}

## Grid Information Infrastructure

### Resource Specification, Expression

### LDAP Directories

## Grid User Environments

### Grid Portals

Discuss GRAPPA, etc.

## Grid Security Systems

## Grid Workflow Management

###  {#section-2 .unnumbered}

### Architecture Definition

Define/adopt and implement a suitable architecture for distributed
scheduling and resource management in a grid environment.

### Co-allocation of Resources

Integrate grid tools that ensure optimal co-allocation of data, CPU and
network for specific grid-network-aware jobs.

### Distributed Scheduling

Condor has been one of the most successful remote job schedulers over
the past ten years\[1\]. Condor remains in active development and the
Condor team is intimately involved in grid planning for the LHC
experiments and so Condor is a natural candidate for the job scheduling
component of the Atlas grid architecture. At one level, the task is to
integrate a working scheduler with Atlas software (such as Indiana's
GRAPPA\[2\]) and to use additional remote file addressing capabilities
as developed within the Condor team. At a more ambitious level, though,
it may be particularly interesting to make use of the Condor "standard
universe." In the Condor standard universe, system calls from a remotely
running batch job are captured and actually executed on the submitting
machine. This means that, in effect, the remove batch job "sees" the
environment of the submitting machine. In particular, input files are
read and output is written on the submitting machine. If this can be
made to work within Atlas, it may allow us to avoid some problems which,
although mundane, are also potentially very serious. In particular

1.  There would be no requirement for Atlas software to be installed at
    the remote site.

2.  We would be less sensitive to the exact OS level of the remote
    machine. Only binary compatibility would be required.

3.  There would be less need to arrange for disk space at the remote
    site.

4.  By using Condor "flocking," there would be no need for a user to
    have an account at the remote site.

Point 2) may be a serious concern. At a typical university or
laboratory, it will be tempting to pool administrative resources and
have facilities which are not Atlas only. This means that Atlas may not
be able to dictate OS level or even what's in /usr/bin. In any case
minimizing requirements in this area is likely to lead to greater
available resources and more "grid-like" operation.

Point 3) may also be a serious concern. Consider, for example, using 500
Linux nodes at NCSA for an Atlas application. It may be quite possible
to get large CPU resources this way, but much harder to get a long term
commitment of disk space and even harder to convince NCSA to go from Red
Hat 7.1 to Red Hat 6.2. If we can transparently make use of such
machines, it will be an important practical advance and an important
step towards the Grid concept.

The solution above may seem problematic because one is always doing i/o
over the network while a job is running. However, this is no extra
burden on the network *provided that the results of a job are analyzed
or further processed at the point of submission*. One then simply
chooses to submit from the location where further analysis is to be
done. If this is not practical, or if there is no need for the standard
universe because, for example, all jobs will run at one site, one can
choose the "vanilla" universe at job submission time to get an ordinary
job scheduler.

#### Connecting the Atlas Grid Sites

If the scheme outlined in section A can be made to work for Atlas
applications, the next issue is how to optimally connect the Condor
"pools" from the various Atlas sites. Here it is tempting to use the
already established Tier 0, Tier 1, Tier 2, ... tree structure. To
simplify things at first, assume that there is one condor pool with one
central manager machine at each node of the tree of Tiers. Let each node
"flock_from" the pool above it in the tree and "flock_to" each of it's
children. With an easy modification, this can be done so that each Tier
has flocking access to all the nodes below it in the tree recursively.
Now we get to take advantage of one of the good ideas in Condor. By
controlling the local configuration file, each node in the tree can
maintain control of its resources just as a workstation owner has
control over how their workstation gets used. This sort of arrangement
has many potential attractive features:

1.  By providing easy local control of a Tier's resources, we make it as
    attractive as possible to provide useful resources for Atlas
    computing.

2.  The administrative load is distributed down the tree. At the top
    level, there are huge collaboration-wide jobs that need to be run.
    Arranging for a job like this from Tier 0 would only require
    arrangements with the Tier 1 contacts. The Tier 1 contacts would
    contact Tier 2 as necessary, and so on. This avoids a situation
    where it takes a huge administrative effort to arrange for a
    collaboration-wide computing job.

3.  Load control can be maintained without bureaucratic overhead for
    small jobs. Small jobs get run on someone's PC, larger jobs can be
    run on the local Condor pool with only local permissions required.
    Extra bureaucracy only comes into play when a job is too large for
    the local pool. In this case, the normal procedure would be for a
    user to either get an account or get flocking access to a Tier 0,1,2
    center higher in the tree, depending upon where analysis or further
    processing of the output would take place.

4.  The tree would make it administratively easier to prevent the system
    from being saturated. Just as with scratch disks and other
    resources, the fact that the people who use the local pool know each
    other will prevent their system from being saturated. It is well
    known that an ordinary batch scheduler alone will not prevent
    saturation.

5.  If many CPUs are available at universities, this may allow us to be
    more effective with hardware purchases since Tier 0,1,2 centers can
    concentrate on providing effective submitting machines, mass storage
    and network bandwidth -- just what is often missing at universities.

6.  Ideally, this setup would scale both in the sense that a small job
    on 10 nodes runs unmodified as a large job on 1000 nodes and in the
    sense that adding Tiers to the tree requires no change in job
    submission procedures.

Of course, there are quite a few issues to investigate and tests to make
before knowing whether these ideas will work. Section C outlines a
program to test the feasibility of these ideas.

#### Prototyping

In order to measure the feasibility and performance of the ideas
outlined in sections A and B, we need to embark on a series of
demonstrations involving several sites and using a variety of the main
Atlas applications.

**Plan:**

1.  Create Linux x86 condor pools at several sites including the
    prototype Tier 1 and Tier 2 centers in the U.S. Link the sites with
    tree-wise flocking.

2.  Prepare a selection of Atlas applications for running in the
    standard condor universe.

3.  Create a convenient procedure for condor_compile linking of Atlas
    applications within the standard build framework.

4.  Construct two or three typical batch applications for use with
    Condor. This should include at least Atlsim, Atlfast and GRAPPA in
    both the vanilla and standard universes.

5.  Identify any constraints on future development of Atlas applications
    which are to run in the standard universe.

6.  Test the performance of the standard applications measuring at least

    a.  Network performance

    b.  Saturation of the submitting machine in the standard universe

    c.  Test flocking with variable "x" in x86.

    d.  Test flocking across Red Hat OS versions.

    e.  Identify any problems with GRAPPA.

7.  Evaluate the convenience of the Condor control of the overall Tier
    resources.

Since the Condor team itself is working on these problems, we expect to
take advantage of their work whenever possible.

#### Site Management Applications

We propose that the testing program outlined in section C should proceed
in parallel with development and testing of "site management software."
By site management software, we mean an integrated collection of
applications which allow

-   Trivial installation and management of both Atlas software and
    external software needed for Grid operation.

-   Uniform access to information about the site: contact person, condor
    pool information, hardware information, current use, software
    installed, data available.

-   Construction of summaries of how the site has been used.

The idea here is that we would establish an easy procedure for creating
a new Atlas Tier node. This would proceed by first installing a site
management (SM) application. From this point on, installing and
maintaining Atlas software must be trivial operations. SM applications
would then keep track of Tier/Web/Condor pool connectivity to other
sites and will generate uniform web pages to display this.

Concerning point 2), we propose this as a way of unifying a chaotic web
situation. The SM application would generate standard web pages at each
site which can then be included as a sub-page of an individual sites
main page. The point is to provide a uniform SM generated subset of web
pages so that a site may have their own pages, but also provide a
uniform set so that Atlas users can traverse the entire Grid and see the
essential information displayed in the same way at each site.

### Uniform Interfaces to Resource Managers

Adapt uniform interfaces to various local resource managers necessary
for grid-endabled ATLAS software programs.

### Grid Policy Management

Develop and implement tools that enforce policies for CPU, data, and
network resource usage.

## Grid Data Management

### Generic Interfaces for Mass Storage

Perform development and/or integration of generic, uniform interfaces to
heterogeneous mass storage management systems. A Replica Manager manages
file and metadata copies in a distributed and hierarchical cache. Data
movers transfer files from one storage system to another one. The
interfaces should encapsulate the details of the local file system and
mass storage systems such HPSS and others. Evaluate storage managers
such as SRB: Storage Resource Broker, an system which encapsulates the
details of the local file system and mass storage systems such HPSS.

## Grid Monitoring Services

### Grid Monitoring Architecture Definition

### Telemetry Databases

### Grid Performance Monitoring Toolkits

### Grid Resource Management Information Systems

### Grid Operations Centers

Discuss network of grid operations centers. How are users expected to
interact with these centers? Discuss iGOC: International Grid Operations
Center to be developed for the iVDGL.

The effective operation of a distributed system requires certain global
services and centralized monitoring, management, and support functions.
These functions will be coordinated by the iVDGL Grid Operations Center
(iGOC), with technical effort provided by iGOC staff, iSite staff, and
the CS support teams. The iGOC will operate iVDGL as a NOC manages a
network, providing a single, dedicated point of contact for iVDGL
status, configuration, and management, and addressing overall robustness
issues. Building on the experience and structure of the Global Research
Network Operations Center (GNOC) at Indiana University, as well as
experience gained with research Grids such as GUSTO, we will
investigate, design, develop, and evaluate the techniques required to
create an operational iVDGL. The following will be priority areas for
early investigation.

-   [Health and status monitoring]{.underline}*.* The iGOC will actively
    monitor the health and status of all iVDGL resources and generate
    alarms to resource owners and iGOC personal when exceptional
    conditions are discovered. In addition to monitoring the status of
    iVDGL hardware, this service will actively monitor iSite services to
    ensure that they are compliant with iVDGL architecture
    specifications.

-   [Configuration and information services]{.underline}. The status and
    configuration of iVDGL resources will be published through an iVDGL
    information service. This service will organize iSites into one or
    more (usually multiple) "virtual organizations" corresponding to the
    various confederations of common interest that apply among iVDGL
    participants. This service will leverage the virtual organization
    support found in MDS-2Error: Reference source not found.

-   [Experiment scheduling]{.underline}*.* The large-scale application
    experiments planned for iVDGL will require explicit scheduling of
    scarce resources. To this end, the iGOC will operate a simple online
    experiment scheduler, based on the Globus slot manager library.

-   [Access control and policy]{.underline}. The iGOC will operate an
    iVDGL-wide access control service. Based on the Globus CAS, this
    service will define top-level policy for laboratory usage, including
    the application experiments that are allowed to use the laboratory.

-   [Trouble ticket system]{.underline}. The iGOC will operate a
    centralized trouble ticket system to provide a single point of
    contact for all technical difficulties associated with iVDGL
    operation. Tickets that cannot be resolved by iGOC staff will be
    forwarded to the support teams of the specific software tool(s).

-   Strong cost sharing from Indiana allows us to support iGOC
    development at a level of two full-time staff by FY2003.
    Nevertheless, sustained operation of iVDGL will require a
    substantially larger operation. To this end, we will establish
    partnerships with other groups operating Grid infrastructures, in
    particular the DTF, European Data Grid, and Japanese groups. We will
    also seek additional support. In addition, we are hopeful that some
    degree of 24x7 support can be provided by the Indiana GNOC, however
    further discussion is required to determine details.

#### Requirements and R & D

#### Trouble Ticket System

#### Certificate Authority Services

## Grid Administration Infrastructure

### Grid User Registration System

### Grid Automated Account Management Systems

### Grid Accounting Systems

# Grid Testbeds

## Lessons Learned for Building Large-Scale Grids

##  {#section-3 .unnumbered}

Operational infrastructure

Grid technology scaling issues

Steps for Building a Multi-site, Computational and Data Grid

1)  Establish an Engineering Working Group that involves the Grid
    deployment teams at each site

2)  schedule weekly meetings / telecons

3)  involve Globus experts in these meetings

4)  establish an EngWG archived email list

5)  Identify the computing and storage resources to be incorporated into
    the Grid

6)  Set up liaisons with the systems administrators for all systems that
    will be involved (computation and storage)

7)  Build Globus on a test system and validate the operation of the
    GIS/MDS at multiple sites

8)  use PKI authentication and Globus or some other CA issued
    certificates for this test environment

9)  can use OpenSSL CA to issue certs manually

Steps for Building a Multi-site, Computational and Data Grid

1)  Determine the model of operation for the Grid Information Service
    (MDS)

2)  Decide on Netscape LDAP hierarchy (\"classic model\") vs. Globus
    OpenLDAP model this may be determined by how large a Grid you plan
    top build larger Grids may use Netscape of a meta-directory servers
    at the higher levels (above the GIISs)

3)  Establish the GIS/resource namespace\-\--be very careful about
    this.Try and involve someone who has some X.500 experience

4)  Don\'t use colloquial names for institutions - consider their full
    organizational hierarchy when naming many Grids use o=grid as the
    top level

5)  plan for a GIS sever at each distinct site with significant
    resources

6)  get the GIS operational

Steps for Building a Multi-site, Computational and Data Grid

1)  Grid Security Infrastructure (GSI) (assuming PKI based)

2)  Set up or identify a Certification Authority to issue Grid X.509
    identity certificates

3)  Issue host certificates for the resources

4)  count on revoking and re-issuing all of the certificates at least
    once before going operational

5)  validate correct operation of the GSI libraries, GSI ssh, and GSI
    ftp

6)  Establish the conventions for the Globus mapfile maps user Grid
    identities to system UIDs - this is the basic authorization
    mechanism for each individual platform - compute and storage

7)  Establish the connection between user accounts on individual
    platforms and requests for Globus access on those systems (initially
    a non-intrusive mechanism such as email to the responsible sys
    admins to modify the mapfile is best)

Steps for Building a Multi-site, Computational and Data Grid

\* Validate network connectivity between the sites and establish
agreements on firewall issues

\- Globus can be configured to use a restricted range of ports, but it

still needs ten, or so (depending on the level of usage of the resources

behind the firewall), in the mid 700s

\- GIS/MDS also needs some ports open

\- CA typically uses a secure Web interface (port 443)

\* Establish user help mechanisms

\- Grid user email list and / or trouble ticket system

\- Web pages with pointers to documentation

\- a Globus \"Quick Start Guide\" that is modified to be specific to
your

Grid, with examples that will work in your environment

Steps for Building a Multi-site, Computational and Data Grid

\* At this point Globus, the GIS/MDS, and the security infrastructure

should all be operational on the testbed system(s). The Globus
deployment

team should be familiar with the install and operation issues, and the

sys admins of the target resources should be engaged.

\* Next step is to build a prototype-production environment.

Steps for Building a Multi-site, Computational and Data Grid

\* Deploy and build Globus on at least two computing platforms at two

different sites.Establish the relationship between Globus job submission

and the local batch schedulers (one queue, several queues, a Globus
queue, etc.)

\* Validate operation of this configuration

Steps for Building a Multi-site, Computational and Data Grid

\* Establish the model for moving data between the Grid systems.

\* GSIftp / GridFTP servers should be deployed on the computing
platforms and on

the data storage platforms

\- It may be necessary to disable the Globus restriction on forwarding
of

user proxies by third parties in order to allow, e.g., a job submitted

from platform_1@site_A to platform_1@site_B to write back to a storage

systems at site A (platform_2@site_A)

\- Determine if any user systems will manage user data that is to be use

in Grid jobs. If so, the Grid ftp server should be installed on those

systems. (So that data may be moved from user system to user job on the

computing platform, and back)

\- Validate that all of these data paths work correctly

Steps for Building a Multi-site, Computational and Data Grid

\* Establish a Grid/Globus application specialist group

-they should be running sample jobs as soon as the prototype-production

system is operational

-they should serve as the interface between users and the Globus system

administrators to solve application problems

\* Identify early users and have the Grid/Globus application specialists
assist

them in getting jobs running on the Grid

-Decide on a Grid job tracking and monitoring strategy

-Put up one of the various Web portals for Grid resource monitoring

A focus on

Directory Services Technology Issues for Large-Scale Grids

Technology Issues for Large-Scale Grids

• Deployment of sizable prototype Grids is revealing many issues for
scalability, and R&D will have to address those issues

• Two areas will be discussed here:

Grid Directory Services and security

\* Grid Directory Services

• The Grid will be a global infrastructure, and it will depend heavily
on the ability to locate information about computing, data, and human
resources for particular purposes, and within particular contexts.

• Most Grids will serve virtual organizations whose members are
affiliated by a common

-- administrative parent (e.g. the DOE Science Grid and NASA\'s
Information Power Grid)

-- long-lived project (e.g. a High Energy Physics experiment)

-- funding source, etc.

• Grid Directory Services refer to the implementation of the large-scale
directory hierarchies needed for global scale Grids (as opposed to the
GIS components and protocols that manage and make available information)

Grid Directory Services: User Requirements

• Searching

-- The basic sort of question that a GIS must be able to answer is for
all resources in a virtual organization, provide a list of those with
specific characteristics.

-- For example:

-- \"Within the scope of the Atlas collaboration, return a list of all
Sun systems with at least 2 CPUs and 1 gigabyte of memory, and that are
running Solaris 2.6 or Solaris 2.7.\"

-- Answering this question involves restricting the scope of the search
to a virtual org. attribute and then \"filtering\" resource attributes
in order to produce a list of candidates.

Grid Directory Services: Virtual Organizations

• The Grid Directory Service (GDS) should be able to provide \"roots\"
for virtual organizations. These nodes provide search scoping by
establishing the top of a hierarchy of virtual org. resources, and
therefore a starting place for searches.

-- Like other named objects in the Grid, these virtual org. nodes might
have characteristics specified by attributes and values. In particular,
the virtual organization node probably needs a name reflecting the org.
name, however some names (e.g. for resources) may be inherited from the
Internet DNS domain names

Grid Directory Services

Grid Directory Services:

Information and Data Objects

• A variety of other information will probably require cataloguing and
global access, and the GDS should accommodate this in order to minimize
the number of long-lived servers that have to be managed. E.g.:

-- dataset metadata

-- dataset replica information

-- database registries

-- Grid system and state monitoring objects (could be a referral)

-- Grid entity certification/registration authorities (e.g. X.509
Certificate Authorities)

-- Grid Information Services object schema

Grid Directory Services: Operational Requirements

• Performance and Reliability

-- Queries, especially local queries, should be satisfied in times that
are comparable to other local queries, such as uncached DNS data for
local systems. E.g., seconds or fractions of seconds.

-- Local sites should not be dependent on remote servers to locate and
search local resources.

-- It should be possible to restrict searches to resources of a single,
local, administrative domain.

Grid Directory Services: Operational Requirements

• Site administrative domains may wish to restrict external access to
local information, and therefore will want control over a local, or set
of local, information servers.

-- This implies the need for servers intermediate between local
resources and the virtual org. root node that are under local control
for security, performance management, and reliability management.

-- (Note that in the Globus terminology that these intermediate
directory servers are called GIISs.)

Grid Directory Services: Operational Requirements

Grid Directory Services: Operational Requirements

Grid Directory Services: Operational Requirements

• Minimal manual management

Grid Directory Services: Operational Requirements

\* Security Aspects of Grids

\* Users are no longer listed in a single central database at a local
site, however positive identification to an entity that can provide
human accountability will still be required

\* Strong authentication to a globally unique identity

\* Grid cryptographic credentials should have a well understood
(published) relationship to the human subject

Security Aspects of Grids

\* There will be multiple stakeholders for the resources involved in
Grid

applications who will probably not have a uniform resource use policy:

Users may have to be authorized separately for every resource that in

incorporated into a Grid application system.

\* Strong, flexible, and easily used and managed policy based

authorization and access control (an R&D topic - see Akenti and GAA)

\* Grid services should not weaken security of local systems, and a

security compromise on one platform that is involved in a Grid
application

system should not propagate via Grid services to other platforms in the

system.

\- careful management of identity credentials

\- understanding what credential management clients are trusted (and
untrusted!)

\- require the use of authenticated control channels (GSI services)

between distributed application components

Security Aspects of Grids

\* Grid users will not have control over the security policy of remote

resources (e.g. computing platforms)

\- It may be necessary to \"rate\" systems on their security, and
provide

that rating as a system characteristic that may be used in choosing

resources from a candidate pool when constructing the resource base for

a distributed application.

## Requirements

## ATLAS-DataGrid Testbed

## ATLAS-iVDGL Testbed

## ATLAS-PPDG Testbed

# Tier 2 Regional Centers

## Tier 2 Facility Hardware 

### Physical Infrastructure

Supply building space, un-interruptable power supplies, HVAC, security,
fire suppression systems

Define needs for each of the categories listed: space required X-Y sq.
feet, UPS: number and size (VA rating), tones of air conditioning
required, type of physical security (card-key, locks, cabinets, etc.)

### Design Tier 2 Physical Infrastructure 

##  Tier 2 Facility Software 

# Tier 3 and Tier 4 Facilities

## Tier 3 (Institute) Facilities

## Tier 4 (Desktop) Systems

# Collaborative Services

ATLAS and LHC will not turn on for another 5 years. We expect many
changes in collaborative services using computer networks in that time,
so this is just a list of topics that can be addressed now or in the
immediate future. The task of tracking new technologies is one of the
most important.

## Web Services

Web was developed so that physicists could collaborate,

so not much to do. However, there is some:

\- Survey/database of ATLAS related web servers or sites

\- Organization of ATLAS web pages

\- ATLAS web search engine?

\- Catalogue of tools and documentation for web development

\- Policy on use of ATLAS logos or identification

> \- Policy on use of scripting (JavaScript can be dangerous) or other
> more advanced tools

## E-Mail

Establish policy and standardization with regard to attachments, HTML,
active content, and other new formats. Follow new developments.

## Document Distribution

Determine need for central document archive and distribution system,
including version control, access restrictions, indexing. Possible
models are Docushare, D0Notes, and/or the Los Alamos arXive. It may be
important to standardize on only one system, or it may be important to
support several systems for various purposes.

## News Distribution systems

Determine need for on-line news system for distributing TLAS
information. Determine if existing systems are adequate now and for
future. There are various modes in which news can be distributed and
will probably need to support several.

### Usenet news or similar (asynchronous distributed)

### Hypernews and the like (central repository)

### listservers and other mailing lists

## Software Management and Distribution

### Source code distribution for developers

### Distribution tools (eg rpm, PACman, AFS)

### Distribution of ATLAS software environment

### Documentation

## Web Lectures

### Archive

### Indexing

### Producing Web Lectures

### Software development

### New developments in this technology

##  Video Conferencing 

### Room sized systems (eg Access Grid)

### Desktop systems (eg VRVS)

### Vendor Systems (eg Polycom)

### Hybrid systems (eg phone bridge & VRVS w/o audio)

### New Developments

## Tele-immersion Environments

### Single site (asyncronous use)

### Interaction between Remote site (syncronous)

### Interoperability between different systems/software

### New Developments

## Whiteboard / Notepad

### Remote whiteboard collaboration,

Use of remote whiteboard devices either alone or with Access Grid

### Notepad device 

For remote conferencing between individual physicists, either in pairs
or small groups, from desktop to desktop

##  Instant Messaging

This new form of communication has become very popular. It may have
utility to remote collaboration by physicists. New variations on this
might turn out to be useful. The task is to determine whether instant
messaging technology is useful to ATLAS, and if so how.

##  New Collaborative Technologies

Need to keep up to date on new collaborative technologies. Track
developing technologies which may not be ready yet. Explore those that
might be ready, test them, and \"play\" with them.

##  Liason

Liason to other organizations interested in or working on collaborative
technologies.

# Network Systems

The advances in networking over the last 30 years have allowed us to
propose computing models such as the grid and hierarchical computing to
solve our computing needs. The increased importance of the network is
easily understood. Computers in the 1960s and 70s had minimal network
connections. Dial up lines running at BITS per second were the norm.
Computers accessed data and code locally from memory or disk. As the
ability to interconnect computers increased, first email and then the
World Wide Web became popular. Data and software could be remotely
accessed in much the same way as accessing it from local storage.
Permanent network connections became the norm. As persistent, reliable
interconnects reached speeds of 10+ megabits per second, people realized
the model of the computer could be extended to encompass networks of
computers, allowing massive leveraging of resources for specific tasks.
In this brave new world of the grid we must not forget that it is the
network which underpins the whole model. If the network fails to provide
the required bandwidth and services the grid model will fail to provide
a viable computing environment. In this new virtual computer the network
becomes the virtual "bus", interconnecting storage, memory and CPU.

In the sections below we will discuss the major network requirements,
services, research and costs to enable a robust grid environment and
viable computing infrastructure.

## Network Connectivity Requirements

This section outlines the general requirements for network capacity and
suggests ways in which these requirements may be estimated and
expressed. It also describes the major cost elements of providing the
network connectivity, with emphasis on the high-performance network
requirements of interconnecting Tier 1 and Tier 2 sites and providing
scientists across the US with access to these sites. It concludes with
comments on some possible next steps in planning to meet the networking
requirements of US ATLAS.

### Data Distribution Models

### Tier 1 Connectivity

### Tier 2 Connectivity

The MONARC report focused mainly on transatlantic links, and the links
between Tier 0 and Tier 1 regional centers. Here we begin the process of
the US network environment planning, with particular focus on Tier 2
sites. To extend our knowledge significantly beyond the scope of the
MONARC report is one purpose of Data Challenge activities planned for
US-International ATLAS.

We have added "DDO" to indicate Derived Data Objects, such as N-tuple
files, which are analyzed on the desktop, or perhaps over a high
performance storage area network located at a Tier 3 university. From
this table, which excludes data transfers related to Monte Carlo
simulations, we can infer the scale of the aggregate data rates into/out
of Tier 2 centers.

$$\begin{array}{r}
\text{Aggregate rate } = \text{ AOD loading }(\text{in}) + \text{ AOD archiving }(\text{out}) + \text{ DDO }(\text{out}) \\
\text{                        } = \left\lbrack \text{10 TB } \times \frac{4}{\text{month}} \right\rbrack \times 2 + \left( 0\text{.}\text{1 TB} \right) \times \frac{\text{40}}{\text{day}} \\
\text{                        } = \text{ 6}\text{.}\text{67 TB/day} \\
\text{                        } = \text{ 77 MB/sec} \\
\text{                        } = 0\text{.}\text{6 Gbs}
\end{array}$$

This estimate is quite uncertain, perhaps good to a factor of 2, but
should be considered on the conservative side. It should also be noted
that this aggregate rate spans different types of network service, the
majority of it occurring during business hours. It seems prudent that a
Tier 2 center should be expected to provide wide area network
connectivity of at least OC48, or 2.45 Gbs.

###  {#section-4 .unnumbered}

### Tier 3 and Desktop (Tier 4) Connectivity

### Uniform Requirements Definition

## Network Services

A common view of the network is the it provides a service analogous to
the phone company: it interconnects computers like the phone network
interconnects phones. In this view it is the network engineers job to
provide a "dialtone" to all users of the network. However, grid
computing and collaborative tools will require substantially more than
just a connection from the network. Grid computing will require
scheduling of network resources, just like it must schedule other
computing resources like disk, CPU and storage. Moreover the "network
resource" may not be simply bandwidth, but could include requirements on
packet jitter, delay and packet loss which can critically impact many
interactive or real-time applications. The ability to schedule
resources, once the ability to reserve them exists, requires security
and authentication services which can work between many different
security domains. The next few sections discuss some of these future
network services we will require.

###  {#section-5 .unnumbered}

### Security and Authentication Services

###  {#section-6 .unnumbered}

### QoS/\"Service Differentiation\"

The Grid Data Management Model put forward by ATLAS as well as

GriPhyN requires replica consistency across distributed databases and

distributed computation. The data traffic on the ATLAS grid can be

envisioned as of two different classes: large datasets being

transferred across centers at regular (or, predictable) periods for

updating or creating databases. This class of traffic requires a high

bandwidth but may not require low values of jitter or delay. In case

of a connection-oriented application protocol such as ftp over TCP/IP,

the packet loss can be tolerated to certain extent. On the other hand,

any computation being performed on the Grid (involving computing

resources and databases across multiple sites) will require not only a

high peak-rate, but also small values of loss, delay and

jitter. Therefore, it is proposed that the ATLAS project will utilize

the differentiated services framework as being implemented in

Abilene/Internet2 to distribute and analyze data.

The IETF working group on Diffserv has put forward the proposal of

Expedited Forwarding (EF) per-hop forwarding behavior that can

guarantee specified rate, delay and jitter per aggregate flow.

Internet2 has created the QBone Premium Service (QPS) as an equivalent

to providing EF service to data traffic across Abilene. In this

framework, a QPS reservation requests resources and is subsequently

granted a peak rate of EF traffic at a specified MTU and an explicit

bound on jitter. Each reservation is also characterized by a

diffserv-domain-to-diffserv-domain router and a specified time

interval (starttime, endtime). Therefore, a complete reservation has

the structure: (source, destination, route, starttime, endtime,

peakrate, MTU, jitter). Since majority of the traffic under US-ATLAS

will be routed on the Internet2, the following recommendations are

made:

1)  classify ATLAS data traffic into several classes, each with it\'s
    tolerable limits (i.e. minimum and maximum bounds) on peak rate,
    delay, jitter, packet loss, duration and time of day. This will
    depend on the type of center (i.e. Tier0, 1, 2 etc.) and the
    application associated with the data.

2)  work with Internet2 QoS Working Group and the network engineers to
    ensure that the DS domains within Internet2 support the ATLAS
    traffic classes. This is crucial since Internet2 will eventually
    carry a broad spectrum of traffic that varies both in its
    sensitivity to network performance as well as in relevance to
    research and education missions.

3)  perform a simulation of ATLAS traffic classes on a Gigabit Diffserv
    testbed such as at Michigan so that traffic parameters can be
    determined as per specific application needs.

4)  set up client systems at Tier 0, 1 and 2 centers with direct
    connections to Abilene/Internet2 (i.e. no or minimal traffic delay
    or loss in the stub network) and perform tests so that in-time
    calculation, capture and distribution of ATLAS data are possible
    within the participating centers.

5)  assign one staff member who can keep track of recent developments in
    Diffserv, RSVP and other recent protocols such as MPLS and
    lambda-switching; and suggest service differentiation schemes for
    ATLAS data grid.

### Bandwidth Brokering

A bandwidth broker (BB) manages resources across a network, and at the

edges of diffserv domains. BBs may span one or multiple domains

depending on the architecture of a particular network and it\'s peering

policies. A request for resource allocation and reservation from an

end-user may be granted by an edge router of a diffserv domain in

cooperation with a bandwidth broker. If the request includes network

resources outside the user\'s local domain, the bandwidth broker may

perform some form of admission control as per the service level

agreements with the adjacent autonomous domains. Some of the

functional requirements of a bandwidth broker may overlap with other

mechanisms proposed such as policy server, policy services manager

etc. In complex networks, bandwidth brokers, policy servers and

network management systems may be integrated into a single system. In

it\'s most basic form, a bandwidth broker collects and monitors the

state of QoS resources within its own domain as well as at the edges

of its peering domains. Across domain boundaries, the service level

agreements are often enforced on aggregate traffic and rarely on

individual flows. Therefore, a bandwidth broker may have to map an

incoming individual resource request to one of the aggregate traffic

classes being supported across a domain. In this model, the ingress

routers to a diffserv domain typically does policing and the egress

routers perform shaping so that the core of the network remains simple

and manageable.

The QBone Signaling Design team is working on the design and

architecture of the SIBBS (Simple Inter-domain Bandwidth Broker

Signalling) framework. The protocol description was completed in

March, 2001 and a prototype implementation is expected by September,

2001\. Besides SIBBS, a prototype BB has been implemented in Globus

called GARA ( General-purpose Architecture for Reservation and

Allocation). It can provide quality of service for different types of

resources, including netwroks, CPUs, batch job schedulers, disks, and

graphic pipelines; and include mechanisms for advanced as well as

real-time reservations in a Globus-enabled Grid. The CANARIE ANA

Bandwidth Broker is also another example. It was developed for the

Canadian educational network CA-NET by British Columbia Institute of

Technology. Finally, a prototype for cross-domain bandwidth broker is

being developed at the Real-Time Computing Laboratory (RTCL) at the

University of Michigan, targeting software routers such as Linux and

high-end Cisco routers.

The following suggestions are made with respect to Bandwidth Brokering

for ATLAS:

\(i\) study the different frameworks being proposed and choose a
framework

that will guarantee both intra-domain and inter-domain bandwidth and
other network parameters (delay, jitter, packet-loss).

\(ii\) work with the various groups to ensure that signalling protocols

and inter-BB messages are standardized so that end-to-end performance

guarantees are met across Tier0, 1 and 2 sites.

\(iii\) if possible, integrate BB as part of the Grid software so that

applications can take advantage of reservation of network resources

via an API (Application Programming Interface).

\(iv\) demonstrate a prototype BB on the ATLAS Grid testbed in

conjunction

with diffserv as outlined in section 9.2.2.

### Monitoring and Measuring Services

Performance measurement and monitor for network system is one of the
major issues in the Grid Computing. To improve network quality of
service, system administrators should evaluate how their systems are
working, and should operate their systems to meet the grid-computing
requirement, grid users\' requests, discover problems, and optimize the
performance. Here the Quality of Service for network consists of several
aspects: access delay, processing time, and data transfer bandwidth and
propagate delay, percentage of loss, and so on.

Many ways to monitor and measure network performance have been developed
so far, such as iperf, traceroute, tcpdump, pchar, pipechar and
tcptrace. However, performance measurement and monitoring are still
difficult because of the lack of effective tools to evaluate network
system usability for grid computing.

The goal of our work is to develop a performance evaluation tool for
network system quality of service based on existing tools. In our
approach, we measure network performance through mechanisms installed in
the grid nodes.

The purpose of this measurement tools are to understand what network
performance a grid application can utilize. These tools can reside on
the local grid node, and they can create a network performance log
database that can be used by grid resource managers, network diagnoses
experts.

In this plan, we will discuss the requirements of a monitoring tool for
grid network performance, propose a network monitoring paradigm, provide
a short term plan on the design and implementation of the network
performance monitoring tool.

**The requirements of a network performance monitoring tools**

Based on the network quality of service, a monitoring tool has to meet
the following requirements, but not limited by these requirements:

1.  The tool should be able to monitor and measure the performance in
    different levels, grid application level, grid level, and network
    level. The overall performance can interrelate with all these level.
    A good monitoring tool should monitor any factor affecting the
    overall performance. First, a tool is able to measure the throughput
    (bandwidth) and propagation delay at the level of grid application.
    If the underlying grid environment cannot guarantee 100% service
    (information could be lost during transfer), the monitoring tool is
    able to detect the percentage of loss. For example, a grid user
    copies a catalog from one site to another site, what is the
    bandwidth he/she can get. As we know, our performance monitoring
    tools aim to improve the overall performance for end users.
    Understand the performance they are experiencing is the first step.
    Secondly, a tool should monitor the performance of TCP/UDP stack.
    Most Grid software, such as GridFtp, GSI enabled ncFtp, run on top
    of TCP/UDP. By monitoring the performance of TCP/UDP, the tool can
    provide grid resource manager information on how to select a remote
    site and the path to that site. The tool can help system
    administration to discover the problem existing in TCP/UDP stack,
    and optimize the performance of the network. Finally, a tool should
    also monitor the performance of the datalinks. By measuring the
    datalink performance, the monitoring is able to provide information
    on data link, such as the latency of the data link, the actually
    bandwidth of the data link, and the percentage of loss. The tool can
    also measure the datalink usage. Network specialist to detect
    network path problem and optimize the network configuration will use
    this information.

2.  The tool should handle various kinds of datalinks. The Internet has
    been used on various datalinks such as Ethernet, FDDI, Token Ring,
    Integrated Services Digital Network (ISDN), and so on. Therefore,
    the measurement method should be independent of the datalinks.

3.  The monitoring tool can run in simulation mode and provide
    performance statistics on all levels of network. It can provide all
    kinds of simulations, for example, simulate many small file
    transferring, parallel data transferring, big file transfer.

4.  The measurement tool should be able to be applied to existing grid
    applications without any major modification. By designing
    user-friendly API (Application Program Interface), library modules,
    the application can get the running time performance factors.

5.  The measurement tool should be able to be plugged to running
    systems. It is difficult to measure all performance factors by
    running simulation. The real running system will have different
    performance behavior when the different applications run on it. It
    is not possible to simulate all of these applications. The best
    solution is to monitor the running system.

6.  The measurement tool should be able to be applied easily to any site
    which participants the performance monitoring. The tools should be
    independent on the platform of the site, the Operating system. It
    can be easily compiled and installed locally.

7.  Since the monitoring can be used by regular users, system
    administrators and network administrators. The interface should be
    user-friendly.

8.  The measure and monitoring tools should not consume too much system
    resources.

9.  The measurement tool should be interactive with Grid Resource
    Allocation Manager and Grid Meta-computing Directory Services (MDS)
    to provide the current network status.

10. Network performance monitoring is a part of performance monitor for
    grid facility. The measurement tool should be easily incorporated
    into a grid facility monitoring toolkit. Therefore, the
    network-monitoring tool should provide the network system status and
    information in a uniform, efficient format, i.e. the tool needs to
    produce readily interpretable data.

**The system paradigm for grid network monitor tool:**

The goal of the performance is to provide a suitable, efficient
framework to measure the network performance. The framework will monitor
the actually network performance for the different levels: Grid
application level, TCP/UDP level, datalink level. The framework will be
deployed in end system.

The monitor has three layers, Grid Observation Layer, TCP/UDP
Observation Layer and DataLink Observation Layer. Each Observation
listens on the traffic between two network entities (layers), and
calculate the performance of the traffic, put the result into database.
The visualization module reads the data from database; output the result
in graphical mode. The Monitoring Daemon can export the performance data
to the Grid Resource Manager.

Design and Implementation:

The existing network monitor tools are scattered in Internet. We do not
want to reinvent these tools. Our monitor tool can make use of these
tools. We need an efficient wrapper to pack all of these tools together.
For example, the datalink observation layer can be replaced by
*pathchar* or *pchar*. The TCP/UDP observation can use *tcpdump* and
*tcptrace* to monitor the network performance. We can use *truss* to
intercept the grid function calls and system calls. Grid Observation
Layer can analyze the intercepted system calls. The design and
implementation of *Netlogger* will be used in the Grid monitor tool to
monitor the end-to-end application, generate

*pathchar* is used to estimate per-hop bandwidth, propagation delay,
queue time and drop rate. It sends a series of packets with random
payload sizes over defined period of time to each hop in a path. The
reasoning is that it will provide a more accurate assessment of real
world network conditions. *pchar* is a tool to characterize the
bandwidth, latency, and loss of links along an end-to-end path through
the Internet. It is based on the algorithms of the pathchar.

Tcpdump is a tool used to echo packet information up to and including
payload content, to standard output or a file. The packet information is
gathered from the local network interfaces after it has been placed in
promiscuous mode.

Truss: The truss utility executes the specified command and produces a
trace of the system calls it performs, the signals it receives, and the
machine faults it incurs. Each line of the trace output reports either
the fault or signal name or the system call name with its arguments and
return value(s). System call arguments are displayed symbolically when
possible using defines from relevant system headers; for any path name
pointer argument, the pointed-to string is displayed.

NetLogger includes tools for generating precision event logs that can be
used to provide detailed end-to-end application and system level
monitoring, and tools for visualizing log data to view the state of the
distributed system in real time. NetLogger has proven to be invaluable
for diagnosing problems in networks and in distributed systems code.
This approach is novel in that it combines network, host, and
application-level monitoring, providing a complete view of the entire
system. NetLogger monitoring allows us to identify hardware and software
problems, and to react dynamically to changes in the system.

### Multicast (Possible uses for grid computing, concerns and viability)

Most network users are only vaguely aware of the different types of IP
connections available. Most probably think in terms of unicast when
considering how packets move about the internet. Actually there are
three primary types of IP transmission: unicast, broadcast and
multicast. Unicast is the most common: it transports a packet from a
specific IP address (source) to a specific IP destination. Broadcast
packets are transmitted to all network interfaces within a "broadcast
domain" (a region delimited by routers/layer 3 devices). Multicast is a
special way of transmitting from one source address to (potentially)
multiple destination network interfaces. It is ideal for supporting
applications like video and audio broadcasting, collaborative computing
and any application which needs to send to a group of receivers
simultaneously.

The primary benefit of multicast is a significant savings in bandwidth
when multiple destinations need to receive the same information. Imagine
a video server that provides a 1 Mbps video stream. If this is
transmitted via unicast, each client who wants to view the stream will
cause another 1 Mbps data flow to be added to the network. Ten clients
cause the server to emit 10 identical packets, one per client. With
multicast, the packet is sent once to a special multicast IP address. It
is the responsibility of the network to route this packet to all
destinations which "subscribe" to this address. The Internet Group
Management Protocol (IGMP) provides the means to automatically control
and limit the flow of multicast traffic thru the network.

US ATLAS needs to carefully evaluate the benefits and liabilities of
utilizing multicast to enable some of its needed applications. Thorough
analysis of the impact of multicast on the network and the overhead to
manage and support it is required. A recommended course of action:

-   Define use cases for multicast in a grid environment

-   Define use cases for collaborative tools

-   Research requirements for enabling multicast on a LAN. Provide an
    installation document focusing on common switch and router
    configurations.

-   Test implementation in a production environment. Define a set of
    measurables of network performance and compare
    multicast+configuration case with unicast and no configuration.

-   Evaluate overhead to supporting multicast. Identify problem areas
    and estimate amount of additional network management required to
    support

## End to End Performance

Distributed application performance problems traceable to poor TCP
performance has been identified as a major source of performance
degradation

in high-performance applications. The root causes of poor TCP
performance

are difficult to isolate and diagnose, and the effects of tuning efforts
on

TCP throughput are often difficult to gauge. This paper describes some
of

the sources of poor TCP performance, and a method to diagnose some of
these

problems based on a combination of existing performance tools and the
Web100

tuning package. An example of tuning an application developed for the

Visible Human Project to improve TCP performance by a factor of four
using

this approach is described.

#### Introduction

Distributed application performance problems traceable to poor TCP

performance has been identified as a major source of performance
degradation

in high-performance applications \[1\]. Appropriately provisioned
network

infrastructure is essential for providing support for high-performance

networking. However, lack of proper host tuning and unexpected levels of

packet loss can adversely affect actual network performance to the
extent

that it can nullify the benefits of investments made in improving the

network infrastructure to support high-performance networking. Most

operating systems are shipped with an overly conservative set of network

tuning parameters that can severely degrade aggregate TCP performance on

wide area networks.

Applications can greatly benefit from application and host tuning
efforts

targeted at improving aggregate network performance. Measurements of the

Edgewarp application \[2\] written for the Visible Human project at the

University of Michigan \[3\] has shown that the effects of poor host and

application tuning can seriously degrade bulk transport performance

necessary for delivering images by at least a factor of four.

System effects other than inadequate host tuning can also affect
end-to-end

performance. Software bugs in the implementation of TCP on host systems
can

contribute to poor TCP performance in very subtle ways \[4, 5\].

1.1 Actual TCP Bandwidth Delivered to the Application

If a host and application are properly tuned, effects outside the
control of

the host and application can adversely affect network performance.

Limitations on TCP bandwidth arise from the effects of packet loss and

packet round trip time on the network path between hosts. The TCP Slow
Start

and Congestion Control algorithms \[6\] probe the network path between
the

sender and receiver to both discover the maximum available transfer
capacity

of the network and at the same time minimize the effects of overloading
the

network and causing congestion.

Mathis \[7\] described the relationship between the upper bound of TCP

bandwidth , packet round trip time and packet loss by the equation

\(1\)

To achieve substantial network performance over a wide area network that
has

a relatively large , the required maximum packet loss rate must be very

low. The relationship derived by Mathis for the maximum packet loss rate

required to achieve a target bandwidth is defined by the relationship

\(2\)

For example, if the minimum link bandwidth between two hosts is OC-12
(622

Mbps), and the average round trip time is 20 msec, the maximum packet
loss

rate necessary to achieve 66% of the link speed (411 Mbps) is
approximately

0.00018%, which represents only 2 packets lost out of every 100000
packets.

Current loss rates on the commercial Internet backbone \[8\] are on the
order

of 0.1%, which puts a hard upper limit on the potential bandwidth
available

to an application.

Recent work \[28\] has demonstrated that packet losses come in bursts of

consecutive packet losses that may be due to the drop-tail queuing
mechanism

in routers \[29\]. As the implementation of the RED queuing mechanism
becomes

more widely deployed across the Internet infrastructure, this
characteristic

may change. Given the burst behavior of packet loss, obtaining
significantly

small packet loss rates may be very difficult. Looking at equation (1),
it

is apparent that increasing the MSS from the usual default value of 1500

bytes to the \"jumbo frame\" size of 9000 bytes can increase the upper
limit

on TCP bandwidth by a factor of six.

Recent experiences of one of the authors demonstrated that increasing
the

MSS by a factor of three from 1500 to 4470 bytes increased TCP
throughput by

a roughly equivalent factor.

##### \"Bad\" Application Network Behaviors

Application developers have learned to overcome poor TCP performance
with a

toolkit of \"bad\" (from the network administrator\'s perspective)
behaviors.

The first approach usually taken is to abandon the TCP transport service
and

to rely on UDP along with a transport layer written for the application.
In

this approach, the application simply transmits packets as fast as it
can.

If any packets are lost, the application either drops them (as in the
case

of multimedia applications), or performs packet retransmission on an

application level. This approach is considered \"bad\" for several
reasons. If

an application is injecting UDP packets into the network at a high rate,
the

network infrastructure has no way of signaling back to the application
that

the flow is congesting the network and affecting other users of the
network.

If the other users of the network are being \"good\" and using TCP for
their

connection, the UDP stream takes an unfair share of the available
network

bandwidth \[9, p. 246\].

The second approach is to open parallel TCP network sockets between

applications, and utilize software controlled striping of the data
across

the sockets, similar to disk striping \[10\]. This approach attempts to
take

more than the host\'s fair share of network bandwidth from other users
of the

network to deliver a higher aggregate network bandwidth to the end
hosts.

Other Sources of Poor Network Performance

Even when the end hosts are properly tuned, and the network packet loss
rate

is acceptable, other factors that may adversely affect application
network

performance come into play. Each of these factors should be considered
in

turn when diagnosing poor network performance, since a fault at a lower

layer will affect performance in all of the layers above it.

At the physical layer, network cables that are not within specification

limits can be a significant source of poor performance. A general rule
of

thumb is that Cat-5 cables are good for 10-BaseT, Cat-5 enhanced cables

(Cat-5e) are good for 100-BaseT, and Cat-6 cables are good for gigabit

Ethernet over copper. Network adapters are very good at getting around
bad

cables by decreasing their throughput or using data-link layer CRC

correction to compensate for a cable that is operating below
specification.

An additional source of problems is host network adapters that are

configured to operate at half-duplex mode rather than full-duplex mode.
If

both the network switch and the network adapters support full-duplex

transfers, both sides should be set to full duplex. If excessive losses
are

encountered in full-duplex mode, the cabling between the host and switch

should be tested or replaced.

At the data link layer, there are several potential sources of problems.

First, if the maximum transmission unit size (MTU) of the frame level

packets is set too low, TCP connections will suffer from poor
performance.

On 10 and 100 Mb/sec Ethernet, all adapter cards enforce a 1500 byte MTU

limit. On some Gigabit Ethernet cards, the MTU can be set to a \"jumbo
size\"

9000 byte frame. If we look back at equation (1), it\'s obvious that

increasing the MTU size (which is MSS + IP header) by a factor of 6 can

increase TCP bandwidth by a factor of 6! Unfortunately, most network

switches and routers have a hard 1500 byte MTU limit that cannot be
changed.

On the host side, another source of potential problems in the data link

layer is the number of CPU interrupts per second that are required to

service the network adapter. If a transfer is occurring at gigabit
Ethernet

speeds, with a limited 1500 byte MTU, the network adapter and CPU must

service over 83,000 packets per second. If the network adapter requires

service from the CPU for a small number of packets, the CPU will be

overwhelmed with servicing network adapter interrupts \[11\]. The device

driver must be configured to permit an appropriate degree of packet

coalescing to take advantage of the network adapter\'s packet buffer.

Finally, the size of the transmission queue in the operating system

(txqueuelen in Linux) can affect the packet loss rate on the host.
Finally,

the PCI slot where the network adapter card is placed can have an impact
on

performance. Some motherboards, such as the Intel L440GX+\[40\], have
dual PCI

busses, with specific PCI slots that are enhanced. If a host contains
RAID

adapters as well as network adapters, careful consideration of adapter
card

placement can have an impact on the aggregate performance of the
complete

system.

In the network layer, there are several potential sources of problems.

First, packet loss and round trip time affects TCP bandwidth as
described

above in equation 1. Second, there may be a network configuration error
that

routes traffic through an inadequate data link, or that adds unnecessary

additional hops in the path between the hosts. This problem can be

especially difficult to diagnose if any IP encapsulation (such as AAL5
for

IP over ATM) occurs on the network path, since IP based network tools
(such

as traceroute) do not have the ability to adequately penetrate an ATM
cloud

to diagnose ATM problems.

In the transport layer, mistuned host TCP options are a very common
source

of problems. Section 3 of this paper will describe some of these options
and

demonstrate how the can affect TCP performance.

Finally, the network I/O characteristics of the application can
dramatically

impact TCP performance. Application developers should consider

multithreading their applications to decouple network I/O from
computation.

Chapter 6 of Steven\'s text \[31\] is a good starting point for these
efforts.

At this point it is important to note that if a problem exists at a
lower

layer in the network, such as the physical layer, efforts directed at
tuning

high layer components to improve performance may not deliver the
expected

results. For example, if a physical link is improperly configured to
operate

at half duplex, attempts to increase performance by optimizing network

routes may yield little if any results. Thus, when diagnosing
application

network performance problems, it is important to make sure that tuning

opportunities at each layer are explored.

##### Web100

The remainder of this section will discuss experiences using Web100 for
host

and application TCP tuning. Web100 \[12\] has been used with great
success for

identifying and diagnosing the symptoms and causes of network
performance

problems and for immediately measuring the effects of performance
tuning. It

is hoped that the work described in this paper will be useful to
application

developers and system administrators for tuning their host systems and

improving network and application performance

#### Related Work

The suite of tools that are currently available for measurement and

diagnosis focus on specific characteristics of network performance. The

tools most frequently used include ping, traceroute, tcpdump, pchar, and

Iperf. Tools designed for network specialists include Treno, and TCP

testrig. This section will briefly describe each of these tools and how
they

are currently used for diagnosing network performance problems.

##### General Tools

The network measurement tools available to application developers and
system

administrators can measure physical data-link bandwidth, round trip
time,

loss rate, router buffer sizes at each hop in the network, and measure

end-to-end network bandwidth between hosts.

The UNIX ping utility is used to transmit and receive ICMP Echo packets
to a

destination host to determine if the host is reachable, to measure round

trip time (RTT), and to measure packet loss on the network path to the
host.

The RTT measurements made by ping can be used to estimate the \"pipe\"

capacity (capacity = BW \* RTT) of the network between two hosts. Since
the

test load put on the network by ping consists of small periodic ICMP

packets, the packet loss rate is not very useful for determining
available

TCP bandwidth using equation (1). The RTT measurement, however, is
useful

for determining the maximum packet loss rate necessary to support a
desired

bandwidth in equation (2).

Traceroute \[16\] is used to discover the IP network route between two
hosts

and the RTT to each hop in the network route. Traceroute is used to
diagnose

routing problems between hosts.

Tcpdump \[17\] is a packet capturing and display utility that displays
all

packets on a network segment connected to a network adapter that is

configured in \"promiscuous mode\'. Tcpdump is used to debug network
protocols

and to passively monitor the network traffic on a LAN segment.

Pchar\[13\] is a tool that measures bandwidth, round trip time, and
router

buffer space on every data link on the network path between two hosts.
Pchar

is used to diagnose and identify data link bottlenecks in the network
path

between two hosts. Pipechar \[14\] is another tool that can be used in

conjunction with Pchar to further examine network bottleneck

characteristics.

Iperf \[15\] is a tool that measures TCP and UDP transfer rates between
host

pairs. Iperf is used to estimate the maximum network bandwidth available
to

an application and to investigate the relationship between UDP packet

injection rate and packet loss on a network between two hosts.

There are many projects working on designing and developing user-level

network bandwidth prediction and management tools. These projects
include

Network Weather Service \[18\], NetLogger \[19\], and Gloperf \[20\]. A
complete

list of network measurement tools can be found at the NLANR website

http://ncne.nlanr.net/software/tools/.

##### Network Specialist\'s Tools

A small set of end-to-end performance measurement tools, such as Treno
\[21\],

TCP Testrig \[22\], are available, but the use of these tools requires
an

extensive knowledge of the characteristics of TCP and networks along
with

privileged access to network devices in the host operating system. To

address these problems, Web100 was developed by a team at Pittsburgh

Supercomputing Center to provide a window into the characteristics of a
TCP

connection for application developers and systems administrators.

Treno is a tool that performs a single stream transfer over a simulated
TCP

connection to diagnose TCP performance problems. TCP Testrig is a TCP
test

harness that is used in combination with tcptrace, xplot, tcpdump, and a
TCP

debugging flowchart \[23\] to aid specialists in characterizing and
diagnosing

TCP tuning problems.

Both Treno and TCP Testrig require a user to have an in-depth knowledge
of

the TCP protocol and network characteristics to realize maximum results.

##### Using Tools to Diagnose TCP Problems

An application developer or systems administrator use of a combination
of

these general and specialist\'s tools to diagnose and correct host and

application tuning problems, but there are several drawbacks to this

approach.

First, to make a fair estimate of the characteristics of the system
under

measurement, many measurements and data points must be collected, and

systematic sources of error (such as time of day) need to be taken into

account to eliminate artificial effects. Second, some of the tools
(pchar,

for example) require such a long time to run that the results of the

measurement may not accurately reflect the current state of the system
under

measurement. Third, some components of the network path (such as
switched

ATM clouds) are resistant to IP based measurement techniques. Finally, a

high degree of expertise in network and operating systems is required to

realize fruitful results from the use of these tools.

#### Web100

To provide an integrated performance measurement and diagnosis tool for

specialists, application developers and systems administrators, the
Web100

project \[12\] is developing software that will provide kernel level
access to

internal TCP protocol variables, settings, and performance
characteristics

for instantaneous feedback on TCP performance characteristics.

The current implementation of Web100 consists of two major components.
The first component is the set of Linux kernel modifications that export
TCP measurements, variables, and settings through the Linux \'/proc\'
interface. The second major component of Web100 is the graphical user
tool, Diagnostic Tool Builder (dtb), which provides an interface to the
Web100 TCP

instrumentation in the form of numerical displays, bar graphs, and pie

charts of the data values provided by Web100.

### Requirements

### Host-based Diagnostic Toolkits

### Network-based Diagnostic Toolkits

The end to end performance includes: the performance of both hosts on
the two ends to their wall jack, the performance of the wall jack to
wall jack. In this section, we are concerned with the performance of the
path from wall jack to wall jack. For a give path, it includes local
site network infrastructure, routers, firewalls, wide area network.
There should be measurement system that allows a person to query current
performance, to determine whether this performance is typical for this
path, and what is the theoretical performance of that path. In short,
the goal is to enable segmental end-to-end measurement, monitoring, and
analysis of a network infrastructure that carries IP and ICMP packages
in a systematic manner on a links which have local cable, fiber, switch,
router, firewall, network).

Generally, a network diagnostic infrastructure will have two parts,
monitoring system and analysis tools. The monitor tool can monitor any
segment in a network path. The analytical tools use the information
gathered from monitoring tools to isolate and identity problems in the
networks. The general analytical method to diagnose the location of a
problem is "divide and conquer". The success of a diagnostic tool relies
on the detailed accurate and flexible monitoring result. The diagnostic
toolkits should satisfy the following requirements:

1.  The measurement tools in the diagnostic toolkits should be
    standardized.

2.  The diagnostic tools should be distributed efficient efficiently
    along the network path. The way of distributing the toolkits will
    affect the accuracy of pinpointing where the problem is, it also
    affect the cost of diagnostic structure. The toolkits are able to be
    deployed in heterogeneous hosts.

3.  The monitoring machine should have uniform diagnostic interface so
    that a method can trigger monitoring and collect result independent
    to the type of the machine.

4.  A mechanism for adding new tests and monitoring service to the
    existing diagnostic system should be in a standardized, controllable
    manner. It is very hard to fix the range of monitoring and test due
    to the dynamics of the modern networks. Therefore, the diagnostic
    machine should be expandable and add new services in a systematic
    way.

5.  Security should be a key design consideration.

Implementing the diagnostic system. The implementation should include
the following activities:

-   Define a standardized set of measurement sets. These measurement
    sets can be implemented on a range of environment. The standard
    should also define the interface to the measurement tools: this
    includes how the control parameters, initial conditions can be input
    to the tools, how the result can be stored, retrieved.

-   Survey the existing monitoring and diagnostic tools. Determine
    whether there is already a core set of measurements and tools on
    which can be ported into our diagnostic system.

-   Build a diagnostic repository. We can save all examples of existing
    problem and solutions for them. We can expect that network problem
    can repeatedly happen in wide area network. The knowledge database
    can save the repeated effort for the same problem happened in
    different time and locations.

The main purpose of diagnostic tools is to find the problems existing in
the network path and solve them. In addition to this purpose, we need to
prevent them happening. Therefore, we need to identity and fix any
protocol-specific issues which cause the problem. We need to make use of
desirable protocol enhancements for better performance; We also identify
the desirable router and firewall enhancements; Another effort is to
exchange the performance diagnosis information with other research
groups who focus on the network performance, for example: Web100,
internet 2 and other US experiments

## Local Site Infrastructure Development

### Requirements

The ATLAS computing model requires high performance networking
infrastructure in place at all participating institutions. The Tier 0
thru Tier 2 sites will have the necessary infrastructure by design. The
major backbones (ESNet, Internet2, regional networks) will also provide
all necessary bandwidth and services based upon feedback and planning by
groups like ATLAS and others. What of the other institutions who are
participating in ATLAS?

Each institution is going to need to insure their local infrastructure
can provide the network bandwidth and services from their desktops to
the regional network gigapop.

### Hardware Infrastructure

### Liaison and Support

## Operations and Liaison

### Network Operations Centers 

### Liaison

#### CERN Liaison

#### Tier 1 Center Liaison

#### Tier 2 Centers Liaison

#### Internet2 Liaison

#### ESnet Liaison

#### International Networks Liaison

#### HENP Networking Forum Liaison

#### Grid Forum Networking Group Liaison

## Network R&D Activities

### Protocols

TCP/IP, UDP/IP, RTP, IPV6, DWDM, IP over λ, ...

### Network Performance and Prediction

### Network Security

#### General Security Requirements

\[General definition of security, levels of security.\]

Establish general network security requirements for ATLAS.

#### Policy and Standards

\- Overall Security Policy

\- Standards and Best Practices

\- Exceptions

#### Network Device Security (Routers and switches)

Determine requirements. Catalogue vulnerabilities

or known problems. (eg. hub vs switching, telnet sniffing)

Make recommendations .

#### Firewalls

Determine requirements, make recommendations for

standards or best practices. Interdomain coordination.

Deal with differing policies at different sites.

#### Current Events / Exploits / Advisories

Develop method for keeping up to date about new exploits or other
problems and distributing such info to the collaboration. Do we need
someone to read bugtraq all the time?

#### Web Security

Server problems,

Scripting Problems (eg JavaScript)

Server side includes, aout

New technologies

#### Encryption

Determine needs for encryption re E-mail, other applications, and at
transport level (see also 9.6.3.7). Promote the use of encrypted
protocols to replace telnet, either ssh or srp-telnet.

Encryption for E-mail (PGP vs SSL?)

#### Security Monitoring (passive)

Determine need for passive security monitoring.

Make recommendations for standards or best practices.

#### Security Testing (active)

Determine needs, make recommendations for standards or best practices.

#####  Host scanning

#####  Tiger Teams? 

Determine need for active penetration testing.

#  {#section-7 .unnumbered}

#### Software Tools 

Determine requirement for distribution of software tools for network
security. Distribute standards or best practices.

#### Software Updates

Implement methods for distributing software dates in a timely fashion.

#### Node Security 

Network is not secure if host nodes are not secure.

-   Unix host security

-   Win2K/NT host security

#### Authentication/Authorization 

Security domains (eg Kerberos) for distributed resources.

See also 9.2.2

#### Specific Protocol Security Issues

\- eg SNMP, telnetd, ftp, DHCP, SMTP, Ipsec, httpd

#### Liaison 

Each site should have a network security contact. This does not have to
be an expert, but it should besomeone with some knowledge of the issues.
There can be more than one (eg. a group might have someone, a department
might have someone else, and the university might have experts)

Perform liaison with other security organizations:

-   Campus or lab

-   Network operations

-   CERT, IEEE & related organizations

-   Law Enforcement

### Virtual Private Networks

### Technology Tracking and Evaluation

US ATLAS needs to monitor trends in networking technology. Significant
advances in such networking technologies as DWDM could transform the
role the network could play in the ATLAS computing model.\
Imagine what cheap terabit per second bandwidths could do for our
computing techniques. Of course, the most revolutionary and transforming
changes are usually unexpected. In anticipation of the unexpected, US
ATLAS needs to monitor networking technology as a matter of course. New
technologies need to be evaluated as they arise and their potential
impact upon the ATLAS computing model gauged.

### IETF Liaison

## Network Cost Evaluation

The Experiments at CERN rely on state-of-the-art computer networking to
enable ever large international collaborations. Collaborating of groups
from national laboratory and universities that are distributed in
different parts of the country create an immediate need for the new
technology network. The network solved the problem of transporting large
volume data among the collaborating sites, therefore it is critical in
enabling a collaboration to function at every stage of operation. For
example, large computer programs are developed to acquire, store, and
analyze large samples of data, and each of these processes will
typically involve collaborators at widely separated institutions. It
also became a driving force for even larger international collaboration.
ATLAS requires fully capable network connections no only between each of
the participating institutions and each of the experimental sites but
also among all of the participating institutions. The following are the
major networking requirements of the ALTAS experiment:

-   Contiune to upgrade and strengthen connectivity between CERN and BNL
    tier 1 center.

```{=html}
<!-- -->
```
-   Continue to upgrade and strengthen connectivity between major HEP
    labs and other sites

-   Continue to monitor network performance between HEP researchers in
    DOE labs and universities

-   Continue to assist in solving networking problem between HEP labs,
    universities.

-   Coordinate connectivity between Esnet, which is used by all DOE
    labs, and future domestic networks, such as Abilene/Internet2, to
    optimize the networking required by the university researcher to
    reach the major DOE national labs.

### International Networks

The large Hadron Collider (LHC) at CERN will open a new frontier in
particle physics due to its higher collision energy and luminosity
compared to existing accelerators. ATLAS is being constructed by 1850
collaborators in 150 institutes from 32 countries around the world. U.S.
groups, from 32 universites and national laboratories, are involved in
almost all of these components of the ATLAS detector. ATLAS data will be
divided into classes according to the degree of processing that has
taken place and the frequency of access that is expected during
analysis. Raw data estimates are based on a provisional average event
size of 2MB, though it is hope that can be reduced with initial
experience of reconstruction. Trigger rates for recorded events are
estimated to be 100 Hz during initial running in 2006, growing to 270
and 400 Hz by the end of 2007 and 2008 respectively. Thus raw data sets
(based on 10,000,000 seconds per year) will be about 2 Peta bytes for
the first year of data taking. The size of raw data will increase to 5.4
and 8 Peta bytes data for the following two years. Only small samples of
raw data will be transferred to the U.S. Tier 1 cernter. The data sets
to be transferred are the Even t summary data, analysis object data and
event tag metadata and smaller data sets. Based on estimation, the main
analysis data for transferring in the first three years will be 0.5, 1.4
and 2 peta bytes. The total data for transfer will be at least more than
5 times these numbers.

ATLAS will follow the hierarchical LHC computing model, with raw data
archived at the Tier 0 center at CERN. The U.S. Tier 1 center is at
Brookhaven National Laboratory. The U.S. Tier 1 center will be
complemented by five Tier 2 centers located at different areas in U.S.
International networking will be needed to move data from CERN to the
Tier 1 center at BNL. As discussed above, it is expected that at least
2.5 peta bytes data will be transferred at year 2006. The volume of data
to be transferred will grow with time at least by a factor of 3 when the
LHC reaches design luminosity. The aggregate data rates to international
sites are dominated by the Tier 0 to Tier 1 traffic, it will use the
service provided by Esnet. We listed the bandwidth requirement on the
international network path between CERN and BNL in the following table.
(Please see Report of the Transatlasntic Network Committee for details)

  ---------- ---------- ---------- ---------- ---------- ---------- ----------
  Link       2001       2002       2003       2004       2005       2006
  Between                                                           

  BNL-CERN   50         100        200        300        600        2500
  ---------- ---------- ---------- ---------- ---------- ---------- ----------

### Domestic Backbone Use

There are 32 institutions participating in US ATLAS, 29 universities and
3 national laboratories. The Tier-1 site is located at Brookhaven
National Lab, which is served by ESnet for its high-performance
networking needs. Tier-2 sites will be established at five locations.
Argonne and LBL are served by ESnet. 22 of the 29 U.S. ATLAS
universities are already connected to the Internet2/Abilene
high-performance backbone network and 2 more are in the process of
establishing connections. All of the universities are eligible for
Internet2 membership and Abilene connectivity.

In light of the excellent network connectivity available to project
participants, U.S. ATLAS should use existing high performance research
and education networks, Abilene and ESnet, to transfer data among Tier-1
and Tier-2 sites and to provide scientists access to data and computing
resources at these sites.

There are three cost elements to a university establishing the level of
network connectivity required to function as a Tier-2 site:

-   Local site network infrastructure (fiber, routers, network
    interfaces, etc.) to connect Tier-2 computational and data storage
    resources to a router or switch at the edge of the campus network.

-   Telecommunication service provider fees for a local loop connection
    from the edge of the campus network to the nearest gigaPOP.

-   Network connection fees paid to a network service provider (e.g.,
    UCAID for Abilene backbone services or ESnet for DOE Laboratories).

Fees to network service providers and telecommunication service
providers scale as a function of network capacity provided. Local
network infrastructure costs are a mixture of fixed costs (e.g., campus
fiber plant) and costs that vary, though not linearly, to increases in
network capacity (e.g., network interfaces).

Presently, most Internet2 universities connect to the Abilene backbone
at OC3 speed (155 megabits/second). The strategic direction for
Internet2 is to transition these users to higher speeds of OC12 (622
megabits/second) or OC48 (2.45 gigabits/second), both of which are
presently available.

### Tier 1 Cost Model

Following is a general model for calculating the cost of network
connectivity for a Tier-1 site

Table 2. Model inputs for Tier 1 Network Costs

+----------------------+-----------+-----------------+-----------------+
|                      | OC3       | OC12            | OC48            |
|                      |           |                 |                 |
|                      | 155 Mbs   | 622 Mbs         | (2.45 Gbs)      |
+----------------------+-----------+-----------------+-----------------+
| Local infrastructure |           |                 |                 |
+----------------------+-----------+-----------------+-----------------+
| Fiber/campus         | \(1\)     | \(1\)           | \(1\)           |
| backbone             |           |                 |                 |
+----------------------+-----------+-----------------+-----------------+
| Network interface    | \(2\)     | \(2\)           | \(2\)           |
+----------------------+-----------+-----------------+-----------------+
| Firewall             | \(3\)     | \(3\)           | \(3\)           |
+----------------------+-----------+-----------------+-----------------+
| Switch               | \(3\)     | \(3\)           | \(3\)           |
+----------------------+-----------+-----------------+-----------------+
| Router               | \(4\)     | \(4\)           | \(4\)           |
+----------------------+-----------+-----------------+-----------------+
| Network Connection   | (5a)      | (5b)            | (5c)            |
| Fee                  |           |                 |                 |
+----------------------+-----------+-----------------+-----------------+
| Performance          | \(6\)     | \(6\)           | \(6\)           |
| Monitoring & Tuning  |           |                 |                 |
+----------------------+-----------+-----------------+-----------------+
| Total                |           |                 |                 |
+----------------------+-----------+-----------------+-----------------+

1.  Tier 1 site needs to upgrade their campus network infrastructure.
    The minimum requirement for U.S. Atlas should be capable of
    operating at gigabit speed (and plans to upgrade to multi-gigabit
    speed

2.  Network interfaces for US-ATLAS computing equipment should be a
    relatively small expense, usually around \$1,000-2,000 per
    interface. There may be several such interfaces in a Tier-1
    configuration.

3.  Firewall and switch will be needed to set up local network
    configuration.

4.  Depending on existing local site backbone infrastructure, the Tier-1
    site may need to acquire a network router ( plan to have more than
    one routers which handle incoming traffic from CERN and outbound
    traffic Tier2 sites) that is dedicated to US-ATLAS data traffic.
    Expense might be in the range \$60,000-120,000.

5.  Annual connection fees for access to the Esnet were: a) OC3 =
    \$110,000; b) OC12 = \$270,000; and c) OC48 = \$430,000

6.  Internet end-to-end performance monitoring should be deployed in
    Tier 0, Tier 1 and Tier 2 centers. To keep pace with the increasing
    importance and extent of networking, future monitoring and tuning
    efforts will need to

-   Extend the deployment of monitoring (in both time and space)

-   Provide a scalable, manageable measurement infrastructure.

-   Address increased diversity in capabilities.

-   Meet new challenges imposed by security concerns.

-   Provide better visualization of results.

-   Provide automated detection and reporting of exceptions.

-   Provide automated tuning and self-recovery strategy in the event of
    poor performance.

### Tier 2 Cost Model

Following is a general model for calculating the cost of network
connectivity for a Tier-2 site. Although individual cost elements are
identified, each candidate Tier-2 site will require a site-specific
analysis and engineering study to determine the technical requirements
and cost of delivering end-to-end network connectivity at a specified
level of service.

Table 2. Model inputs for Tier 2 Network Costs

+--------------------+-------------+-----------------+-----------------+
|                    | OC3         | OC12            | OC48            |
|                    |             |                 |                 |
|                    | 155 Mbs     | 622 Mbs         | (2.45 Gbs)      |
+--------------------+-------------+-----------------+-----------------+
| Local              |             |                 |                 |
| infrastructure     |             |                 |                 |
+--------------------+-------------+-----------------+-----------------+
| Fiber/campus       | \(1\)       | \(1\)           | \(1\)           |
| backbone           |             |                 |                 |
+--------------------+-------------+-----------------+-----------------+
| Network interface  | \(2\)       | \(2\)           | \(2\)           |
+--------------------+-------------+-----------------+-----------------+
| Router             | \(3\)       | \(3\)           | \(3\)           |
+--------------------+-------------+-----------------+-----------------+
| Telecom service    | \(4\)       | \(4\)           | \(4\)           |
| provider           |             |                 |                 |
+--------------------+-------------+-----------------+-----------------+
| Network Connection | (5a)        | (5b)            | (5c)            |
| Fee                |             |                 |                 |
+--------------------+-------------+-----------------+-----------------+
| Total              |             |                 |                 |
+--------------------+-------------+-----------------+-----------------+

7.  All Internet2 member universities have committed to substantially
    upgrade their campus network infrastructure. US-ATLAS could consider
    requiring Tier-2 candidate universities to have in place an adequate
    campus backbone network and optical fiber plant capable of operating
    at gigabit speed (and plans to upgrade to multi-gigabit speed).
    Plans exist for upgrading DOE laboratories as demand requires.

8.  Network interfaces for US-ATLAS computing equipment should be a
    relatively small expense, usually around \$1,000-2,000 per
    interface. There may be several such interfaces in a Tier-2
    configuration.

9.  Depending on existing local site backbone infrastructure, the Tier-2
    site may need to acquire a network router that is dedicated to
    US-ATLAS data traffic. Expense might be in the range
    \$60,000-120,000.

10. Local loop charges between the campus and the nearest gigaPoP depend
    on local tariffs and distance. This is a highly variable figure that
    will need to be determined on a case-to-case basis. The final cost
    model should consider how to pro-rate a portion of this expense to
    US-ATLAS, since most universities will already have established a
    local loop connection to support their current Internet2/Abilene
    connectivity.

11. Annual connection fees presented at the Fall 2000 Internet2 meeting
    for access to the Abilene network were: a) OC3 = \$110,000; b) OC12
    = \$270,000; and c) OC48 = \$430,000. As with local loop charges,
    the annual connection fee should be apportioned between US-ATLAS and
    other institutional Abilene traffic.

These same cost elements and a similar cost model will likely apply to
establishing high-speed network connectivity to the US-ATLAS Tier-1 site
at Brookhaven National Laboratory.

We can estimate the network capacity that will become available to
universities through high-performance research and education networks
between now and 2006, based on the observation that the optical
bandwidth for commercial networks doubled over 9 times in the 18 year
period from 1979 to 1997. This rate of increase can be projected to
continue for at least the next 6 to 8 years, so that the capacity of
high-speed networks in 2006 might be around 4 times the speed of
today\'s OC48 networks, or approximately 10 gigabits/second.

### Next Steps in Network Planning

A detailed analysis of variables and factors outlined above needs to be
conducted, and dedicated staff resources are needed to carry out this
analysis. These staff may come from US-ATLAS institutions and/or from
representatives of the key network service providers (UCAID for Abilene,
DOE for ESnet). Regardless of who carries out the network analysis,
representatives of Abilene and ESnet should be involved in the process.
The analysis may also require access to existing networks, Abilene and
ESnet, for demonstration and empirical testing.

# APPENDIX A: Network Notes

Abilene, a project of the University Corporation for Advanced Internet
Development (UCAID) in partnership with Qwest Communications, Cisco
Systems, Nortel Networks and Indiana University, is an Internet2
backbone network providing nationwide high-performance networking
capabilities for over 150 Internet2 universities. For more information
on Abilene please see: http://www.ucaid.edu/abilene/.

ESnet provides a reliable communications infrastructure and leading-edge
network services that support the U.S. Department of Energy\'s missions.
The program emphasizes advanced network and distributed computing
capabilities needed for forefront scientific research and other
Department of Energy (DOE) programs, thus enhancing national
competitiveness and accelerating development of future generations of
communication and computing technologies. For more information, see:
http://www.es.net/.

The Global Research Network Operations Center (Global NOC) at Indiana
University manages the international network connections from advanced
research and education networks in the Asia/Pacific, Europe, Russia and
South America to the Science Technology and Research Transit Access
Point (STAR TAP) and the leading US high performance research and
education networks such as Abilene (the network that supports the
Internet2 project), the NSF's very high performance Backbone Network
System (vBNS) and the Department of Energy's ESNET. For more information
please see http://globalnoc.iu.edu/.

# APPENDIX B: Relevant MONARC Information

The following information is adapted from: "Models of Networked Analysis
at Regional Centers for LHC Experiments (MONARC)", Phase 2 Report,
CERN/LCB 2000-001.

Reconstruction of RAW data, at CERN (Tier 0):

These jobs create the ESD (Event Summary Data objects), the AOD and the
TAG datasets based on the information obtained from a complete
reconstruction of RAW data that has been already recorded. The newly
created ESD, AOD, and TAG are then distributed (by network transfers, or
other means) to the US ATLAS Tier 1 center at Brookhaven National
Laboratory (BNL). This is an International ATLAS Experiment Activity. It
is assumed that International ATLAS should be able to perform a full
reconstruction of the RAW data and distribution of the ESD, AOD and TAG
data, 2-4 times a year.

Re-definition of AOD and TAG data, at CERN:

This job re-defines the AOD and the TAG objects based on the information
contained in the ESD data. The new versions of the AOD and TAG objects
are then replicated to BNL by network transfers. This is an
International ATLAS Activity that is expected to take place with a
frequency of about once per month.

Selection of standard samples within Physics Analysis Groups:

This class of jobs performs a selection of a standard analysis group
sample, a subset of data that satisfies a set of cuts specific to an
analysis group. Event collections (subsets of the TAG database or the
AOD database with only the selected events, or just pointers to the
selected events) are created. Re-clustering of the objects in the
federated database might be included in this Analysis Group activity.

Generation (Monte Carlo) of "RAW" data set:

This job creates the RAW-like data to be compared with real data. These
jobs can be driven by a specific analysis channel (single signal) or by
the entire Collaboration (background or common signals). This is an
Analysis Group (performed at Tier 2 centers) or an ATLAS activity that
can take place at both CERN and at BNL.

**Reconstruction of "RAWmc" events to create ESDmc, AODmc and TAGmc:**

This job is very similar to the real data processing. Since RAWmc may be
created not only at the Tier 2 centers, the reconstruction may take
place at BNL or CERN, wherever the data have been created. The time
requirements of the reconstruction of these events are less stringent
than for the real RAW data.

Re-definition of the Monte Carlo AOD and TAG data.

Same as above. The difference may be in the need for the final analysis
to access the original simulated data (the "Monte Carlo truth") at the
level of the kinematics or the hits for the purpose of comparison.

Analysis of data sets to produce physical results.

These jobs start from data sets prepared for the respective analysis
groups, accessing Event Collections (subsets of TAG or AOD data-sets),
and follow associations (pointers to objects in the hierarchical data
model -- TAGAOD, AODESD, ESDRAW) for a fraction of all events.
Individual physicists, members of Analysis Groups submit these analysis
jobs. In some cases, co-ordination within the Analysis Group may become
necessary. Analysis jobs are examples of Individual Activities or Group
Activities (in the case of enforced co-ordination).

Analysis of data sets to produce private working selections.

This job is a pre-analysis activity, with a goal to isolate physical
signals and define cuts or algorithms (Derived Physics Data). These jobs
are submitted by individual physicists, and may access higher data
hierarchy following the associations, although (as test jobs) they
require perhaps a smaller number of events than Analysis jobs described
above. These jobs are examples of Individual Activities.

The main characteristics of the major analysis tasks, such as the
frequency with which the tasks will be performed, the number of tasks
run simultaneously, the CPU/event requirements, the I/O needs, the
needed time response et cetera, are summarised in Table 1.

Regional Centers and the Group Approach to the Analysis Process

The analysis process of experiments data follows a hierarchy:
Experiment-\>Analysis Groups-\>Individual Physicists. A typical Analysis
Group may have about 25 active physicists. Table 2 gives a summary of
the "Group Approach" to the Analysis Process.

Table B.1 Characteristics of the main analysis tasks (MONARC)

+--------+-----+--------+------+-------+-------+-------+-----+-------+
| :      | :   |        | :::  |       | :::   |       | :   |       |
| :: Def | ::  |        |  Def |       | Defin |       | ::  |       |
| initio | Def |        | init |       | ition |       | Def |       |
| n-Term | ini |        | ion- |       | -Term |       | ini |       |
| []{    | tio |        | Term |       | D     |       | tio |       |
| .mark} | n-T |        | R    |       | efine |       | n-T |       |
| :::    | erm |        | e-De |       | Group |       | erm |       |
|        | F   |        | fine |       | dat   |       | P   |       |
|        | ull |        | AOD  |       | asets |       | hys |       |
|        | re  |        | /TAG |       | :::   |       | ics |       |
|        | con |        | :::  |       |       |       | An  |       |
|        | str |        |      |       |       |       | aly |       |
|        | uct |        |      |       |       |       | sis |       |
|        | ion |        |      |       |       |       | Job |       |
|        | ::: |        |      |       |       |       | ::: |       |
+--------+-----+--------+------+-------+-------+-------+-----+-------+
|        | :   | :      | :::  | :::   | :::   | :::   | :   | :::   |
|        | ::  | :: Def |  Def | Defin | Defin | Defin | ::  | Defin |
|        | Def | initio | init | ition | ition | ition | Def | ition |
|        | ini | n-Term | ion- | -Term | -Term | -Term | ini | -Term |
|        | tio | range  | Term | Range | Value | Range | tio | Range |
|        | n-T | :::    | V    | :::   | used  | :::   | n-T | :::   |
|        | erm |        | alue |       | :::   |       | erm |       |
|        | Va  |        | used |       |       |       | Va  |       |
|        | lue |        | :::  |       |       |       | lue |       |
|        | u   |        |      |       |       |       | u   |       |
|        | sed |        |      |       |       |       | sed |       |
|        | ::: |        |      |       |       |       | ::: |       |
+--------+-----+--------+------+-------+-------+-------+-----+-------+
| :      | :   | :      | :::  | :::   | :::   | :::   | :   | :::   |
| :: Def | ::  | :: Def |  Def | Defin | Defin | Defin | ::  | Defin |
| initio | Def | initio | init | ition | ition | ition | Def | ition |
| n-Term | ini | n-Term | ion- | -Term | -Term | -Term | ini | -Term |
| Fre    | tio | 2-     | Term | 0     | 1/    | 0     | tio | 1-    |
| quency | n-T | 6/year | 1/m  | .5-4/ | month | .5-4/ | n-T | 8/day |
| :::    | erm | :::    | onth | month | :::   | month | erm | :::   |
|        | 2/y |        | :::  | :::   |       | :::   | 1/  |       |
|        | ear |        |      |       |       |       | day |       |
|        | ::: |        |      |       |       |       | ::: |       |
+--------+-----+--------+------+-------+-------+-------+-----+-------+
| :      | :   | 25     | :::  | :::   | :::   | :::   | :   | :::   |
| :: Def | ::  | 0-1000 |  Def | Defin | Defin | Defin | ::  | Defin |
| initio | Def |        | init | ition | ition | ition | Def | ition |
| n-Term | ini |        | ion- | -Term | -Term | -Term | ini | -Term |
| CPU    | tio |        | Term | 0.    | 25    | 10-50 | tio | 1-5   |
| /event | n-T |        | 0.25 | 1-0.5 | :::   | :::   | n-T | :::   |
| (SI    | erm |        | :::  | :::   |       |       | erm |       |
| 95\*s) | 250 |        |      |       |       |       | 2.5 |       |
| :::    | ::: |        |      |       |       |       | ::: |       |
+--------+-----+--------+------+-------+-------+-------+-----+-------+
| :      | :   | :      | :::  | :::   | :::   | :::   | :   | :::   |
| :: Def | ::  | :: Def |  Def | Defin | Defin | Defin | ::  | Defin |
| initio | Def | initio | init | ition | ition | ition | Def | ition |
| n-Term | ini | n-Term | ion- | -Term | -Term | -Term | ini | -Term |
| Input  | tio | RAW    | Term | ESD   | DB    | DB    | tio | DB    |
| data   | n-T | :::    | ESD  | :::   | query | query | n-T | query |
| :::    | erm |        | :::  |       | :::   | :::   | erm | :::   |
|        | RAW |        |      |       |       |       | DB  |       |
|        | ::: |        |      |       |       |       | qu  |       |
|        |     |        |      |       |       |       | ery |       |
|        |     |        |      |       |       |       | ::: |       |
+--------+-----+--------+------+-------+-------+-------+-----+-------+
| :      | :   | :      | :::  | :::   | :::   | :::   | :   | :::   |
| :: Def | ::  | :: Def |  Def | Defin | Defin | Defin | ::  | Defin |
| initio | Def | initio | init | ition | ition | ition | Def | ition |
| n-Term | ini | n-Term | ion- | -Term | -Term | -Term | ini | -Term |
| Input  | tio | 0.5-2  | Term | 0.0   | 0.1   | 0.0   | tio | 0.00  |
| size   | n-T | PB     | 0.1  | 2-0.5 | PB    | 2-0.5 | n-T | 1-1TB |
| :::    | erm | :::    | PB   | PB    | :::   | PB    | erm | (AOD) |
|        | 1   |        | :::  | :::   |       | :::   | 0   | :::   |
|        | PB  |        |      |       |       |       | .1- |       |
|        | ::: |        |      |       |       |       | 1TB |       |
|        |     |        |      |       |       |       | (A  |       |
|        |     |        |      |       |       |       | OD) |       |
|        |     |        |      |       |       |       | ::: |       |
+--------+-----+--------+------+-------+-------+-------+-----+-------+
| :      | :   | :      | :::  | :::   | :::   | :::   | :   | :::   |
| :: Def | ::  | :: Def |  Def | Defin | Defin | Defin | ::  | Defin |
| initio | Def | initio | init | ition | ition | ition | Def | ition |
| n-Term | ini | n-Term | ion- | -Term | -Term | -Term | ini | -Term |
| Input  | tio | TAP    | Term | DISK  | DISK  | DISK  | tio | DISK  |
| medium | n-T | E/DISK | DISK | :::   | :::   | :::   | n-T | :::   |
| :::    | erm | :::    | :::  |       |       |       | erm |       |
|        | D   |        |      |       |       |       | D   |       |
|        | ISK |        |      |       |       |       | ISK |       |
|        | ::: |        |      |       |       |       | ::: |       |
+--------+-----+--------+------+-------+-------+-------+-----+-------+
| :      | :   | :      | :::  | :::   | :::   | :::   | :   | :::   |
| :: Def | ::  | :: Def |  Def | Defin | Defin | Defin | ::  | Defin |
| initio | Def | initio | init | ition | ition | ition | Def | ition |
| n-Term | ini | n-Term | ion- | -Term | -Term | -Term | ini | -Term |
| Output | tio | ESD    | Term | AOD   | Colle | Colle | tio | Var   |
| data   | n-T | :::    | AOD  | :::   | ction | ction | n-T | iable |
| :::    | erm |        | :::  |       | :::   | :::   | erm | :::   |
|        | ESD |        |      |       |       |       | --  |       |
|        | ::: |        |      |       |       |       | ::: |       |
+--------+-----+--------+------+-------+-------+-------+-----+-------+
| :      | :   | :      | :::  | :::   | :::   | :::   | :   | :::   |
| :: Def | ::  | :: Def |  Def | Defin | Defin | Defin | ::  | Defin |
| initio | Def | initio | init | ition | ition | ition | Def | ition |
| n-Term | ini | n-Term | ion- | -Term | -Term | -Term | ini | -Term |
| Output | tio | 0.05-2 | Term | 10 TB | 0.    | 0.    | tio | Var   |
| size   | n-T | PB     | 10TB | :::   | 1-1TB | 1-1TB | n-T | iable |
| :::    | erm | :::    | (    |       | (AOD) | (AOD) | erm | :::   |
|        | 0.1 |        | aod) | :::   | :::   | :::   | --  |       |
|        | PB  |        | :::  | Defin |       |       | ::: |       |
|        | ::: |        |      | ition |       |       |     |       |
|        |     |        | :::  | -Term |       |       |     |       |
|        |     |        |  Def | (aod) |       |       |     |       |
|        |     |        | init | :::   |       |       |     |       |
|        |     |        | ion- |       |       |       |     |       |
|        |     |        | Term | :::   |       |       |     |       |
|        |     |        | 0    | Defin |       |       |     |       |
|        |     |        | .1TB | ition |       |       |     |       |
|        |     |        | :::  | -Term |       |       |     |       |
|        |     |        |      | 0.    |       |       |     |       |
|        |     |        | :::  | 1-1TB |       |       |     |       |
|        |     |        |  Def | :::   |       |       |     |       |
|        |     |        | init |       |       |       |     |       |
|        |     |        | ion- | :::   |       |       |     |       |
|        |     |        | Term | Defin |       |       |     |       |
|        |     |        | (    | ition |       |       |     |       |
|        |     |        | tag) | -Term |       |       |     |       |
|        |     |        | :::  | (tag) |       |       |     |       |
|        |     |        |      | :::   |       |       |     |       |
+--------+-----+--------+------+-------+-------+-------+-----+-------+
| :      | :   | :      | :::  | :::   | :::   | :::   | :   | :::   |
| :: Def | ::  | :: Def |  Def | Defin | Defin | Defin | ::  | Defin |
| initio | Def | initio | init | ition | ition | ition | Def | ition |
| n-Term | ini | n-Term | ion- | -Term | -Term | -Term | ini | -Term |
| Output | tio | DISK   | Term | DISK  | DISK  | DISK  | tio | DISK  |
| medium | n-T | :::    | DISK | :::   | :::   | :::   | n-T | :::   |
| :::    | erm |        | :::  |       |       |       | erm |       |
|        | D   |        |      |       |       |       | --  |       |
|        | ISK |        |      |       |       |       | ::: |       |
|        | ::: |        |      |       |       |       |     |       |
+--------+-----+--------+------+-------+-------+-------+-----+-------+
| :      | :   | :      | :::  | :::   | :::   | :::   | :   | :::   |
| :: Def | ::  | :: Def |  Def | Defin | Defin | Defin | ::  | Defin |
| initio | Def | initio | init | ition | ition | ition | Def | ition |
| n-Term | ini | n-Term | ion- | -Term | -Term | -Term | ini | -Term |
| Time   | tio | 2-6    | Term | 5-15  | 1 day | 0.5-3 | tio | 2-24  |
| re     | n-T | months | 10   | days  | :::   | days  | n-T | hours |
| sponse | erm | :::    | days | :::   |       | :::   | erm | :::   |
| (T)    | 4   |        | :::  |       |       |       | 12  |       |
| :::    | mon |        |      |       |       |       | ho  |       |
|        | ths |        |      |       |       |       | urs |       |
|        | ::: |        |      |       |       |       | ::: |       |
+--------+-----+--------+------+-------+-------+-------+-----+-------+
| :      | :   | :      | :::  | :::   | :::   | :::   | :   | :::   |
| :: Def | ::  | :: Def |  Def | Defin | Defin | Defin | ::  | Defin |
| initio | Def | initio | init | ition | ition | ition | Def | ition |
| n-Term | ini | n-Term | ion- | -Term | -Term | -Term | ini | -Term |
| Number | tio | 1      | Term | 1     | 1/    | 1/    | tio | 10    |
| of     | n-T | :::    | 1    | :::   | Group | Group | n-T | -100/ |
| jobs   | erm |        | :::  |       | :::   | :::   | erm | Group |
| in T   | 1   |        |      |       |       |       | 20/ | :::   |
| :::    | ::: |        |      |       |       |       | Gr  |       |
|        |     |        |      |       |       |       | oup |       |
|        |     |        |      |       |       |       | ::: |       |
+--------+-----+--------+------+-------+-------+-------+-----+-------+

Table B.2 Summary of the \"Group Approach\" to the Analysis Process
(MONARC)

+-----------------------------------+------------+--------------------+
| > **LHC Experiments**             | > Value    | > Range            |
|                                   | > USED     |                    |
+-----------------------------------+------------+--------------------+
| > Number of analysis groups (WG)  | > 20/      | > 10-25/experiment |
|                                   | experiment |                    |
+-----------------------------------+------------+--------------------+
| > Number of members per group     | > 25       | > 15-35            |
+-----------------------------------+------------+--------------------+
| > Number of Tier-1 Regional       | > 5/       | > 4-12/experiment  |
| > Centres (including CERN)        | experiment |                    |
+-----------------------------------+------------+--------------------+
| > Number of Analyses per Regional | > 4        | > 3-7              |
| > Centre                          |            |                    |
+-----------------------------------+------------+--------------------+
| > Active time of Members          | > 8        | > 2-14 Hour/Day    |
|                                   | > Hour/Day |                    |
+-----------------------------------+------------+--------------------+
| > Activity of Members             | > Single   | > More than one    |
|                                   | > regional | > regional centre  |
|                                   | > centre   |                    |
+-----------------------------------+------------+--------------------+

Table B.3 Model of Daily Activities of the Regional Centres (Tier 0, 1,
2) (MONARC)

+----------+----------+----------+----------+----------+-----------+
|          | :::      | :::      | :::      | :::      | :         |
|          |  Definit |  Definit |  Definit |  Definit | :: Defini |
|          | ion-Term | ion-Term | ion-Term | ion-Term | tion-Term |
|          | RAW      | ESD      | AOD      | TAG      | Monte     |
|          | :::      | :::      | :::      | :::      | Carlo     |
|          |          |          |          |          | :::       |
+----------+----------+----------+----------+----------+-----------+
| :::      |          |          |          |          |           |
|  Definit |          |          |          |          |           |
| ion-Term |          |          |          |          |           |
| Number   |          |          |          |          |           |
| of       |          |          |          |          |           |
| events   |          |          |          |          |           |
| in RAW,  |          |          |          |          |           |
| ESD,     |          |          |          |          |           |
| AOD, TAG |          |          |          |          |           |
| and      |          |          |          |          |           |
| Monte    |          |          |          |          |           |
| Carlo    |          |          |          |          |           |
| data     |          |          |          |          |           |
| types    |          |          |          |          |           |
| and      |          |          |          |          |           |
| their    |          |          |          |          |           |
| location |          |          |          |          |           |
| :::      |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| :::      | :::      | :::      | :::      | :::      | :         |
|  Definit |  Definit |  Definit |  Definit |  Definit | :: Defini |
| ion-Term | ion-Term | ion-Term | ion-Term | ion-Term | tion-Term |
| #events, | 1,000    | 1,000    | 1,000    | 1,000    | 10        |
| location | ,000,000 | ,000,000 | ,000,000 | ,000,000 | 0,000,000 |
| :::      | CERN     | each     | :::      | :::      | :::       |
|          | :::      | Tier1    |          |          |           |
|          |          | :locally | each RC: | each RC: | :         |
|          |          | :::      | locally  | locally  | :: Defini |
|          |          |          |          |          | tion-Term |
|          |          | each     |          |          | each      |
|          |          | Tier2:at |          |          | Tier      |
|          |          | Tier1    |          |          | 1:locally |
|          |          |          |          |          | :::       |
+----------+----------+----------+----------+----------+-----------+
| :::      |          |          |          |          |           |
|  Definit |          |          |          |          |           |
| ion-Term |          |          |          |          |           |
| Volume   |          |          |          |          |           |
| of       |          |          |          |          |           |
| re       |          |          |          |          |           |
| plicated |          |          |          |          |           |
| data     |          |          |          |          |           |
| (ftp);   |          |          |          |          |           |
| number   |          |          |          |          |           |
| of       |          |          |          |          |           |
| events   |          |          |          |          |           |
| and data |          |          |          |          |           |
| volume   |          |          |          |          |           |
| accessed |          |          |          |          |           |
| by       |          |          |          |          |           |
| analysis |          |          |          |          |           |
| ac       |          |          |          |          |           |
| tivities |          |          |          |          |           |
| :::      |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| :::      | :::      | :::      | :::      | :::      | :         |
|  Definit |  Definit |  Definit |  Definit |  Definit | :: Defini |
| ion-Term | ion-Term | ion-Term | ion-Term | ion-Term | tion-Term |
| Recons   | 6        | --       | --       | --       | 1,000,000 |
| truction | ,000,000 | :::      | :::      | :::      | :::       |
| input    | :::      |          |          |          |           |
| events   |          |          |          |          |           |
| :::      |          |          |          |          |           |
|          |          |          |          |          |           |
| :::      |          |          |          |          |           |
|  Definit |          |          |          |          |           |
| ion-Term |          |          |          |          |           |
| Accessed |          |          |          |          |           |
| per day  |          |          |          |          |           |
| :::      |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| :::      | :::      | :::      | :::      | :::      | :         |
|  Definit |  Definit |  Definit |  Definit |  Definit | :: Defini |
| ion-Term | ion-Term | ion-Term | ion-Term | ion-Term | tion-Term |
| Recons   | No       | Yes      | Yes      | Yes      | Yes       |
| truction | :::      | :::      | :::      | :::      | :::       |
| output   |          |          |          |          |           |
| :::      |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| :::      |          | :::      | 60 GB to | 600 MB   | :         |
|  Definit |          |  Definit | each     | to each  | :: Defini |
| ion-Term |          | ion-Term |          |          | tion-Term |
| FTP      |          | 0.6 TB   | :::      | :::      | 100 GB    |
| t        |          | to each  |  Definit |  Definit | from each |
| ransfers |          | :::      | ion-Term | ion-Term | **Tier1** |
| :::      |          |          | *        | **Tier1  | RC to     |
|          |          | :::      | *Tier1** | /Tier2** | CERN      |
| :::      |          |  Definit | and      | RC       | :::       |
|  Definit |          | ion-Term | *        | :::      |           |
| ion-Term |          | Tier1    | *Tier2** |          |           |
| (repl    |          | centre   | RC       |          |           |
| ication) |          | :::      | :::      |          |           |
| :::      |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| :::      | :::      | :::      | :::      | :::      |           |
|  Definit |  Definit |  Definit |  Definit |  Definit |           |
| ion-Term | ion-Term | ion-Term | ion-Term | ion-Term |           |
| De       | --       | 100      | --       | --       |           |
| finition | :::      | ,000,000 | :::      | :::      |           |
| of AOD / |          | ev       |          |          |           |
| input    |          | ents/day |          |          |           |
| :::      |          | :::      |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| :::      | :::      | :::      | :::      | :::      |           |
|  Definit |  Definit |  Definit |  Definit |  Definit |           |
| ion-Term | ion-Term | ion-Term | ion-Term | ion-Term |           |
| De       | --       | --       | Yes      | Yes      |           |
| finition | :::      | :::      | :::      | :::      |           |
| of AOD / |          |          |          |          |           |
| output   |          |          |          |          |           |
| :::      |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| :::      |          |          | 1 TB to  | :::      |           |
|  Definit |          |          | each     |  Definit |           |
| ion-Term |          |          | Tier1    | ion-Term |           |
| FTP      |          |          | and      | 10 GB to |           |
| t        |          |          | Tier2 RC | each     |           |
| ransfers |          |          |          | :::      |           |
| :::      |          |          |          |          |           |
|          |          |          |          | :::      |           |
| (repl    |          |          |          |  Definit |           |
| ication) |          |          |          | ion-Term |           |
|          |          |          |          | Tie      |           |
|          |          |          |          | r1/Tier2 |           |
|          |          |          |          | RC       |           |
|          |          |          |          | :::      |           |
+----------+----------+----------+----------+----------+-----------+
| :::      |          |          |          |          |           |
|  Definit |          |          |          |          |           |
| ion-Term |          |          |          |          |           |
| Number   |          |          |          |          |           |
| of       |          |          |          |          |           |
| events   |          |          |          |          |           |
| and data |          |          |          |          |           |
| volumes  |          |          |          |          |           |
| of       |          |          |          |          |           |
| d        |          |          |          |          |           |
| ifferent |          |          |          |          |           |
| type to  |          |          |          |          |           |
| be       |          |          |          |          |           |
| accessed |          |          |          |          |           |
| per day  |          |          |          |          |           |
| by       |          |          |          |          |           |
| d        |          |          |          |          |           |
| ifferent |          |          |          |          |           |
| analysis |          |          |          |          |           |
| ac       |          |          |          |          |           |
| tivities |          |          |          |          |           |
| :::      |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| :::      | :::      | :::      | :::      | :::      |           |
|  Definit |  Definit |  Definit |  Definit |  Definit |           |
| ion-Term | ion-Term | ion-Term | ion-Term | ion-Term |           |
| Physics  | 0.001%   | 0.1% of  | 10% of   | 100% of  |           |
| Group    | of       | :::      | :::      | :::      |           |
| :::      | :::      |          |          |          |           |
|          |          | :::      | :::      | 1,000    |           |
| :::      | :::      |  Definit |  Definit | ,000,000 |           |
|  Definit |  Definit | ion-Term | ion-Term | (per     |           |
| ion-Term | ion-Term | 1,000    | 1,000    | job)     |           |
| S        | 1,000    | ,000,000 | ,000,000 |          |           |
| election | ,000,000 | (per     | (per     | (100     |           |
| job      | (per     | job)     | job)     | GB/job)  |           |
| :::      | job)     | :::      | :::      |          |           |
|          | :::      |          |          |          |           |
| :::      |          | (0.1     | (1       |          |           |
|  Definit | (0.01    | TB/job)  | TB/job)  |          |           |
| ion-Term | TB/job)  |          |          |          |           |
| (data    |          |          |          |          |           |
| accessed |          |          |          |          |           |
| per      |          |          |          |          |           |
| single   |          |          |          |          |           |
| job); 20 |          |          |          |          |           |
| jobs     |          |          |          |          |           |
| running, |          |          |          |          |           |
| 1 per    |          |          |          |          |           |
| analysis |          |          |          |          |           |
| group.   |          |          |          |          |           |
| :::      |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| :::      | :::      | :::      | :::      | :::      |           |
|  Definit |  Definit |  Definit |  Definit |  Definit |           |
| ion-Term | ion-Term | ion-Term | ion-Term | ion-Term |           |
| Physics  | 0.01%of  | 1% of    | Follow   | Group    |           |
| Analysis | AOD data | AOD data | 100% of  | D        |           |
| (data    | :::      | :::      | the      | ata-set: |           |
| accessed |          |          | group    | :::      |           |
| per      | :::      | :::      | set (per |          |           |
| single   |  Definit |  Definit | job)     | 1-10 %   |           |
| job);    | ion-Term | ion-Term | :::      | of all   |           |
| 200 jobs | (per     | (per     |          | TAG      |           |
| running, | job)     | job)     | (on      |          |           |
| 10 jobs  | :::      | :::      | average  | :::      |           |
| per      |          |          |          |  Definit |           |
| analysis | (on      | (on      | :::      | ion-Term |           |
| group.   | average  | average  |  Definit | objects  |           |
| :::      |          |          | ion-Term | (per     |           |
|          | :::      | :::      | 0.45     | job)     |           |
|          |  Definit |  Definit | TB/job)  | :::      |           |
|          | ion-Term | ion-Term | :::      |          |           |
|          | 0.045    | 0.45     |          | (on      |           |
|          | TB/job)  | TB/job)  |          | average  |           |
|          | :::      | :::      |          |          |           |
|          |          |          |          | 4.5      |           |
|          |          |          |          | GB/job)  |           |
+----------+----------+----------+----------+----------+-----------+

[^1]: : "Models of Networked Analysis at Regional Centers for LHC
    Experiments (MONARC)", Phase 2 Report, CERN/LCB 2000-001.