**ITR: A Computing and Data Grid for the DØ Experiment**

**Ongoing Grid Research**

There are three major Grid efforts in the US, GriPhyN
\[www.griphyn.org\], iVDGL \[www.ivdgl.org\], and PPDG \[www.ppdg.net\].
These three groups are working together as a three prong approach to
meet the grid needs for US physics experiments through the Trillium
coordination. The **GriPhyN** (**Gri**d **Phy**sics **N**etwork)
collaboration is a team of experimental physicists and information
technology (IT) researchers who plan to implement the first
Petabyte-scale computational environments for data intensive science in
the 21st century. The GriPhyN effort includes massive computing
challenges in the fields of Astronomy, Gravitational Wave detection, and
HEP. The **iVDGL** (**i**nternational **V**irtual **D**ata **G**rid
Laboratory) is a global Data Grid that will serve forefront experiments
in physics and astronomy. Its computing, storage and networking
resources in the U.S., Europe, Asia and South America provide a unique
laboratory that will test and validate Grid technologies at
international and global scales. The **PPDG** (Particle Physics Data
Grid) collaboration was formed in 1999 to satisfy the need for Data Grid
services to enable the worldwide distributed computing model of current
and future high-energy and nuclear physics experiments. It has provided
an opportunity for early development of the Data Grid architecture as
well as evaluating some prototype Grid middleware. The DZero
Collaboration is a participant in the PPDG effort and thus closely
associated with its projects. Through these ties DZero also works
closely with complementary data grid initiatives in Europe and beyond:
GlobalGridForum\[www.gridforum.org\], European DataGrid
\[www.eu-datagrid.org\] and as part of the HENP InterGrid Coordination
Board (HICB) \[www.hicb.org\] and HICB Joint Technical Board.

# Current SAM-Grid Project Status

The SAM system was developed at Fermilab to accommodate the high volume
data management requirements for Run 2 Physics experiments. The system
is designed with a distributed architecture, using CORBA as its
underlying framework and is largely devoted to transparently delivering
files and managing large caches of data. Based on the success of SAM, a
larger architecture has been conceived that encompasses Grid level job
submission and management, and information services; this is called
SAM-Grid. This system expands the existing system to enable grid access
to additional computing resources within the D0 collaboration and
beyond.

The prototype version of the SAM-Grid software was released in early
October 2002, and demonstrated to the DZero experiment on Oct 10, 2002.
A refined and improved version was shown at SC2002 in Baltimore in
November of 2002. Features include:

-   Remote submission of via the SAM-Grid, Condor-G, and Globus layered
    system.

-   Grid job brokering based on the amount of data cached at the
    participating sites

-   Web-based monitoring of the grid system and of grid jobs. The
    monitoring is viewable at:
    <http://samadams.fnal.gov:8080/prototype/>

At each submission site, a user interface is provided which accepts jobs
written in a simple Job Description Language (JDL). A parser translates
the description into a Condor ClassAd which is delivered to the Condor
queuing server (Schedd). A Condor Collector gathers information from
each execution site's Grid sensor, or advertising entity, also through
ClassAd's. The information for submitted jobs, and available site
resources are matched in the Condor Negotiator to rank jobs for
submission. The Condor team provided the ability to use an external
module for the matching criteria, and this provides the ability to write
algorithms using known resource parameters to control the job
distribution for the system. The Condor negotiator sends the jobs
through the Condor Grid Manager to the GRAM server (gatekeeper) on the
gateway at the appropriate processing resource. Standard GSI mechanisms
are used to provide authentication for each user with grid certificates.
A kerberos to x509 translator is used to also enable the use of existing
Fermilab Kerberos principals.

The jobs that can be submitted include both "vanilla" and "SAM-enabled".
Vanilla refers to those jobs which do not take advantage of any SAM
provided data management services. SAM-enabled jobs include dataset
descriptions and the data handling facilities of the existing SAM
infrastructure provide files at the selected processing site for the job
to consume. The decision of where to send a processing job is currently
based on number of needed files for the project already cached at each
execution site. This algorithm will be extended and mature with
experience, but its flexibility is a major feature of the system. A
"sandboxing" mechanism was build that packages up a complex user job at
the submission site, and sends it with the job to the execution site.

(merge next 2 paragraphs and summarize list of deployment sites?)

For the SC2002 presentation both DZero and CDF submission, execution,
and monitoring sites were deployed. This was not really a special
demonstration that was prepared, but just a view of the system as it
existed a the time. Existing DZero SAM sites were selected and upgraded
with JIM software. The SAM-Grid job management was integrated with the
CDF Cluster Analysis Facility (CAF) software to provide a working
analysis system for CDF. CDF and DZero analysis and test jobs were
submitted and resulting histograms were collected from the grid to a
web-accessible area for display. For SC2002, there were eleven, and by
the end of 2002 there were thirteen sites enabled altogether. The DZero
sites included 1) Fermilab, 2) the University Texas, Arlington, TX, 3)
Michigan State University, East Lansing, MI, 4) The University of
Michigan, Ann Arbor, MI, 5)Imperial College, London, UK, and 6) GridKa
in Karlsruhe Germany. The CDF sites included 1)Fermilab, 2) Texas Tech
University, Lubbock, TX, 3) University of Toronto, Toronto, Canada,
4)Rutgers State University, NJ, 5) ScotGrid, Glasgow, UK, 6)Rutherford
Appelton Laboratory, Oxfordshire, UK, 7) Kyungpook National University,
Daegu, Republic of Korea.

A lot has been learned about the installation and operation of the
current software in the last quarter of 2002, but there remains
extensive work to be done before the system can be used for production
in the spring of 2003. The installation procedures are being improved
and automated so the system can be deployed to all of the existing sites
by personnel at each location, and not require extensive help form the
Fermilab team. Many problems related to various firewall configurations
and requirements have been confronted during the testbed deployment and
they are resolved and briefly documented. SAM has adapted well to
operating on various cluster configurations, including local disk on
each node, shared NFS disk, private and public networks. Additional
details about the system and installation can be found at
<http://www-d0.fnal.gov/computing/grid/SAMGridManual.htm>. After SC2002,
the main concentration has been on the evolution of the software version
1, and this resulted in design discussions on the manipulation of the
site xml-based configuration for most of the execution and monitoring
side server activities.

**Proposed Research**

(maybe get rid of this paragraph)

Grid computing is a promising area in the high-performance computing.
The challenges at the grid level include research areas such as
distributed computation, data storage, security, scheduling, and
resource optimization. Some of these problems can be studied in the
context of spatial and temporal dynamics of distributed systems.
Theoretically, these systems can be modeled by Neural Networks, Cellular
Automata, Coupled Maps Lattices, and L Systems (grammars).

The goal of the SAM-Grid effort has always been the use of standard grid
middleware and compliance with the emerging protocol standards to assure
reuse of our software and interoperability with other grid systems. As
the project moves ahead it will become increasingly important that it
adapts to new technologies and standards as they emerge. With the
experience gained in the existing project we propose to continue with
these ideas and adopt additional technologies to provide additional
performance, scalability, reliability and functionality for the system.
Areas included in our planned work are 1) More comprehensive Monitoring
and Information Service, 2) Careful attention to fault tolerance, 3)
innovative scheduling, 4) improved performance and security using NFS
version 4, and 5)additional security features including code signing and
needed improvements to security infrastructure, 6) Network performance
and Quality of Service, and 7) use of Web Services with emphasis on
interoperability with other grids. This effort is targeted at
functionality and success and therefore a large amount of work will be
devoted to testing and deployment to D0 collaborating sites. The focus
will be on major regional analysis and processing centers for experience
and feedback. Additional job submission sites will be enabled using the
existing infrastructure as a stepping stone to facilitate the project.

**Monitoring and Information Service** (The following is from Michigan
(Abhijit Bose))

Any large-scale wide-area distributed computing system such as the D0
Grid must be instrumented with monitoring and reporting agents so that a
set of performance metrics (e.g. average and peak times to transfer data
files among the computing sites, average number of jobs processed in the
system in a given period etc.) can be evaluated for the overall system.
The agents will also allow the system to detect potential problems, e.g.
hardware and software faults, and use this information to make
intelligent scheduling decisions for it's computing and storage nodes.
Therefore, grid monitoring and fault tolerance are considered two most
crucial aspects of any large-scale production-quality Grid computing
framework. (Another important aspect, grid- and node-level security
issues, is presented elsewhere in the ITR proposal.) . Both of these
areas are in early stages of development and therefore, present promise
of significant advancement by the D0 team, especially in the context of
an already-operational Grid environment. In the following, we present
requirements and our proposed activities in the context of the D0 Grid:

***Monitoring and Trace Collection:***

The basic design requirements of the D0/SAM Grid monitoring system,
expanded from \[1\] are as follows:

1)  Scalability: the monitoring architecture has to be scalable for
    deployment in a wide-area network involving hundreds and possibly
    thousands of computing and storage nodes.

2)  Reliability: the monitoring data should be correct during its
    intended time window and replicas should be placed and checked for
    consistency at periodic intervals.

3)  Performance: the monitoring architecture should pose minimum
    overhead on the participating sites as well as the network.

4)  Non-Intrusiveness: the monitoring protocol and messages should not
    interfere with the scheduled computation and data transfer
    operations.

5)  Flexibility: the architecture should be flexible and extensible
    enough so that new schemas and Grid standards can be incorporated
    without much integration efforts. This is an important requirement
    since the standards for Grid interoperability and monitoring are in
    very early stages of development and hence, are subject to change.
    The flexibility is also important for integrating monitoring tools
    and schemas that have already been developed.

6)  Security: the monitoring data should be made available based on a
    set of access control lists so that only intended users have access
    to such data. Unauthorized access to such data may result in launch
    of denial of service attacks or theft of resources in such a system.

There are two basic architectures for a grid monitoring and trace
collection system:

(i) Centrally-located server: in which client agents at the computing
    sites and network sensors periodically update appropriate databases
    in the central monitoring server. Consumers of such data such as
    performance evaluation agents, schedulers etc. register with the
    central server for access to the data over their respective
    administrative and functional domains. This model, although
    relatively simple to develop and manage, has a potential bottleneck
    as the number of sites in the Grid continues to grow.

(ii) Distributed servers: in which each administrative domain or a
     collection of administrative domains periodically update monitoring
     data with a local server and the servers themselves periodically
     exchange high-level information about the domains. This model
     scales well and is therefore, considered for our proposed
     architecture.

Figure 1 shows the basic architecture of D0 Grid monitoring and trace
collection system. Each of the participating sites registers with a
local monitoring server. The nearest server to a participating site can
be found from a central directory service. The traces from network
measurements, hardware and software monitors placed at the participating
site are stored at this server. The monitoring data is arranged in a
hierarchical fashion: local information about cpus, network status
constitute the lowest level of information about a computing site,
SAM-related information on data transfer rates, data file availability,
caches constitute higher-level information which can be used to make
global scheduling decisions. Such global or high-level information about
participating sites can be periodically replicated across all the
monitoring servers for reliability and performance using a distribution
protocol such as via *event pushes*. The Discovery and Monitoring Even
Description (DAMED) working group of the Global Grid Forum \[2\] has
described a set of basic attributes or schemas for the "top N" most
widely reported monitoring elements. While some of these attributes will
be relevant for the D0 Grid, there are several attributes unique to the
D0 project that will need to be developed. These relate to the
particular architecture of the SAM stations, database servers and
computing elements. It should be noted that every operational Grid will
have such unique monitoring attributes in addition to a standard set of
attributes proposed by the GGF. It is a research problem as to how to
provide a scalable and general framework for both the standard and
Grid-specific attributes to monitor.

(will add some more on schemas and will draw Fig 1)

# Fault-Tolerance: (this section is incomplete)

One of the operational requirements of the D0 Grid is to tolerate
partial failure. A partial failure can happen at a computing site, the
network connecting any two or more computing sites, at the central
database or replication catalog servers or at any point in the
distributed environment. Our proposed work in the fault-tolerance area
is to design the SAM-Grid system in such a way so that it can recover
from such partial failures without affecting its overall performance
over a specified level. In the following, we classify the various
failures that can happen in the context of the D0 Grid using definitions
in \[3\]:

1\. Crash failure: any of the compute or database servers crashes but
was working correctly until it halts.

2\. Omission failure: any SAM data transfer message (either request or
receive messages) is lost, or, data transfer is incomplete.

3\. Timing failure: a SAM station's or a compute cluster's response is
outside a specified time interval. This could be either due to network
congestion or server overload.

4\. Response failure: this happens typically when a server's response is
incorrect. In the context of D0 Grid, a likely scenario is during the
collection at the replication database from a computing site. (need to
verify)

5\. Other failure: scheduler failure, local compute-site specific
failure such as file server or NFS mount failures.

I will have to write more on (1) how we construct the Grid collective
communication strategies (improving on the existing SAM-Grid) to do
failure masking and replication with inputs from the monitoring system,
and (2) scheduler redundancy for SAM station servers.

**Job Scheduling** (The following stuff is from UTA (Jeff Lei and David
Levin))

(Clean up these sections)

There are several critical activities involved in making this Grid
operational. Work must be described as jobs to run, data sets needed,
time constraints, software versions required, and hardware constraints
(such as memory, scratch disk space, etc.) \[References will be added
later\]. Scheduling jobs will use the information described above and
knolwedge of available systems (known through monitoring) to allocate
jobs to clusters and individual/scheduler or machines. (add refs to
Condor/Globus, Pat McGuigons Thesis). Monitoring is a broad range of
activities that include dynamic monitoring the \"health\" of machines in
a cluster (machines running, available in network), the status of task
running on machines or clusters, semi-static information about versions
of the operating system, libraries, applications, memory avaialble, and
network monitoring which includes bandwidth, delay, and other network
QoS information.\
Monitoring of semi-static information will be done by discovery and
subsequent publishing of this information to a directory while dynamic,
fast-changing information will be pulled on-demand and cached where
appropriate.

(The following is from UTA (Farhad Kamangar))

Inherent to all distributed systems is their ability to be organized as
conceptual hierarchical structures. In the context of grid computing,
the conceptual hierarchical structuring allows the restructuring of
resources into groups of collaborative partners for achieving a
particular goal. It is desirable to be able to create appropriate
infrastructures and algorithms to dynamically assemble virtual pools of
resources (computers, storage devices, networks, etc.) to perform a
specific task. Such dynamically created structures must be able to
automatically handle member recruitment, role assignment, data storage,
job management, and scheduling.

Currently, the scheduling and restructuring of the resources are mostly
performed by centralized paradigms that make decisions based on global
information. The capabilities of such systems are determined at built
time and their ability to react to dynamics of the environment is
limited. I am interested in building large-scale cluster, grid, and
peer-to-peer computing systems where each node participates in the
scheduling and resource allocation based on available local information.
These systems are characterized by heterogeneous decentralized control
and have self-organization abilities that allow them to cope with the
extreme dynamism of their operating environment. Some of the important
characteristics of such systems include (Heylighen):

-   **Global order from local interactions:** Self-organized systems
    achieve a particular state by interaction of the elements on a local
    level. No global information is communicated between the local
    elements and individual nodes are not aware of the global state of
    the system.

-   **Distributed Control:** The control of the self-organized systems
    is distributed among the elements. No external agent is responsible
    for directing the system and no global information is communicated
    between the nodes.

-   **Robustness and Resilience:** Self-organized systems are inherently
    robust and fault-tolerant. Removing elements from such systems
    generally results in graceful degradation of the performance.

-   **Organizational closure, hierarchy and emergence:** Self-organized
    systems usually form a structured hierarchy with distinct
    boundaries. These structures are relatively stable (closed) and are
    not disturbed by external disturbances. Entities in higher levels
    exhibit properties that that do not exist in lower levels of
    hierarchy. These properties are called "emergent" and they can not
    be explained by examining the properties of the individual elements
    in the lower levels.

**Performance and Security Using NFSv4** (The following is from Michigan
(Charles Antonelli))

We propose the adoption of NFSv4, the new, emerging standard for secure,
reliable wide-area distributed file operations, as the basis for a grid
file system.

*Contributions of NFSv4*

Originally developed by Sun Microsystems in 1985, NFS was subsequently
turned over to the Internet Engineering Task Force (IETF) for
standardization and improvement . This has led to NFSv4 \[PSB00\], a
major new version of the protocol. NFSv4 resolves the limitations in
earlier versions that inhibited the use of NFS in wide-area networks.

There are three broad areas in which NFSv4 contributes to the long term
goals of this proposal. First, the NFSv4 protocol improves performance
while maintaining consistency via its *delegation* and *compound RPC*
features.. To improve consistent sharing, NFSv4 provides a mechanism by
which a client can be delegated control over a file; all client accesses
can be satisfied by a locally cached copy of the file, and consistent
access is assured while obviating the need to contact the server.
Especially in read-only environments, this improves performance
considerably. NFSv4 goes further by defining a compound RPC mechanism
that allows bundling multiple RPC requests in a single network transit.
For example, an "open" RPC might add a read request for the initial part
of a file.

These protocol features have the potential to improve the performance of
architectures such as the SAM-Grid \[BBG02\] for two reasons -- as SAM
data are read-only, delegation is especially efficient, and NFSv4 does
not require the transfer of an entire file before the application is
allowed to begin accessing the file's data.

Second, NFSv4 addresses security by requiring an Internet standard
extension to ONC (the NFS RPC layer) called RPCSEC_GSS. This marries the
Internet standard GSS API to ONC. The NFSv4 standard also requires two
mandatory GSS security flavors, one based on Kerberos V and the other on
an Internet standard public key protocol, LIPKEY, that makes few
infrastructure demands. In addition, NFSv4 authorizes access to files
and directories via access control lists, and supports data integrity
and privacy controls over RPC data.

These protocol features provide the immediate capability for limiting
grid users, via a rich set of access rights, to those disk resources for
which they are authorized, and for securing users' data in transit; the
latter especially in wide-area deployments. Furthermore, NFSv4 permits
the operation of a host system mostly and potentially entirely from
NFSv4-mounted filesystems, limiting damage from misbehaved applications
and thus reducing system administrator workload.

Third, NFSv4 is designed to operate efficiently in wide-area networks,
where large latencies and dropped packets are the norm, not the
exception. Compound RPC and delegation are especially effective in the
WAN environment. In addition, NFSv4 offers support for two vexing
aspects of naming: global naming of files and global naming of users.
For the former, NFSv4 defines a server-controlled mechanism that
dictates the placement of NFSv4 mounts in a client name space. For
global user names, NFSv4 defines names as arbitrary strings interpreted
according to client/server agreement. This will support Kerberos
user@realm style and X.509 distinguished names and is easily extensible.

This allows access to the filesystem beyond the conventional computing
cluster, enabling researchers at their desktops to access filesystem
data directly over the WAN. The full implication of the NFSv4 global
name space and wide area features on grid computing need to be more
fully explored and understood.

*CITI and NFSv4*

This proposal leverages considerable work performed at the Center for
Information Technology Integration (CITI) at the University of Michigan.
Since 1999, CITI students and technical staff have been engaged in the
development of an NFSv4 reference implementation in joint efforts with
industrial partners Sun and Network Appliance. CITI has participated in
a series of interoperability "bakeoffs" with Sun, Network Appliance, and
Hummingbird, fully achieving its goal of providing early, complete
implementations to flush out ambiguities in the specification and to
provide a path for the industrial implementations to follow. A Linux
2.4.18 version of NFSv4 that passes all Connectathon tests is available,
as is a pre-alpha OpenBSD version \[NFSv4\]. A full reimplementation for
Linux 2.5.x emphasizing performance is underway, with agreement reached
for insertion of our code into the Linux kernel. We expect this work, as
well as a back-port to 2.4.18, to be completed by summer 2003. The
results and expertise developed during this effort are highly
synergistic with the NFSv4 portion of this proposal, and we expect to
draw heavily upon them .

In addition, CITI has embarked on an ambitious effort to apply NFSv4
technologies at a grid-level scale. Supported by the ASCI Tri-Lab, we
are designing an NFSv4 front-end to a cluster-wide filesystem \[ASCI\].
NFSv4 clients running on cluster computer nodes access NFSv4 servers
running on I/O nodes and fronting GPFS backend filesystems. The intent
is to support scalable parallel I/O from GPFS through multiple data
streams back to the applications. A metadata server provides efficient
namespace and shared lock state management. There is considerable
synergy with this project as well.

**Security**

(need to fill in additional details, and/or consolidate with previous
section??)

In addition to using NFS v4 security features to build sandboxes to
limit access for clients on host systems, there are other security
issues to be explored. Signed code and other measures to allow trusted
access to compute and memory resources by grid clients. (REF Bor-Yuh
E.C., et al,"Trustless Grid Computing in ConCert", Grid Computing --
GRID 2002, p112-125.)

There are many issues related to Authentication and Authorization for
each Virtual Organization, and its individual members within the
security policies at each site. Currently, the SAM-Grid project uses
gridmap files to specify which user certificates are recognized at each
facility, and management of these files is problematic. The current
implementation of GSI and the use of gridmap files does not allow any
kind of access control that is tailored to individual users or
particular organizations and a much richer set of tools is required.
Work in this area is being done by other groups and we plan to leverage
these efforts. (REF GRID 2002, p169-179,199-206)

**Network Quality of Service**

We have estimated what our network needs, in terms of bandwidth, will be
based on event rates from the detector, and anticipated data access
patterns and processing scenarios. Network performance and monitoring is
an area whose resulting information will be fed back into the scheduling
algorithm of the request broker. Network Quality of Service (QoS) is an
area for which there is some expertise at the University of Michigan and
might be used to enhance the performance to some sites.

When problems arise in the network they can be very difficult to isolate
for a number of reasons: insufficient access to all the intervening
networks, lack of information on the current state of the various
network segments, lack of historical information representing the
operational baseline, and lack of information about the performance of
the host and applications involved. A critical part of our work in
ensuring the network can meet the requirements of DØ Grid computing
model, is collecting sufficient information about the state of the
network and determining how best to use this information to help
troubleshoot problems within the Grid network. We propose to build
"Monitors and Beacons": specially configured PCs that will be located at
critical points within the network (usually directly connected to
primary routers/layer 3 switches). The name "Monitors and Beacons"
refers to their dual role as both passive network monitors and active
network testers (beacons). These machines are designed to serve a number
of purposes: 1) provide local network segment monitoring (bandwidth,
losses, traffic analyis) 2) serve as an active network probe when
accessed via authenticated and authorized means (Ipef server/client, IP
reflector, etc), 3) router monitoring (syslog host), 4) end-to-end path
monitoring component, 5) multicast reflector/test-point, 6) QoS related
diagnostics and monitoring.

Timely analysis of DØ data requires the transfer of large amounts of
data across computer networks in a timely fashion. The internet as it is
presently constructed is not up to this task. Today's Internet provides
only a "best effort" data delivery service where all network traffic
receives the same priority, whether it contains time-sensitive medical
data or is just part of a popular music file. When network traffice
exceeds the capacity of some network to carry all of the traffic, the
network responds to the congestion by dropping some packets, which must
then be resent. Two approaches are being pursued to overcome the
limitations of "best effort" delivery. The first and so far trhe most
common approach is to build specialized portions of the network that are
capable of very high performance and which are open to relatively few
users. The other approach is to implement different levels of network
service that may be requested by an application, so that important or
time-sensitive traffic is given preference over other traffic, much like
first-class mail is given preference over parcel post. The general name
for such differentiated service is "Quality of Service" (QoS)...

**Interoperability and Web Services** (needs more work, Lee)

When the SAM project originated in 1997 many of the technologies needed
to construct grids were in their infancy. The architecture which was
used for SAM-Grid is largely compliant with the view of the grid as a
set of layered services \[Anatomy of the Grid paper\], but the emerging
field of web services promises to provide exciting new functionality and
interoperability \[physiology of the grid paper\]. SAM-Grid currently
employs two standards for communication among its distributed
components, 1) CORBA for the SAM Data Management component, and 2) the
Globus Toolkit version 2 (GT2) for the Job and Information Management.
As Globus transitions from the current GT2, to Open Grid Services
Architecture (OGSA) structured on web services to be distributed in
Globus Toolkit v3 (GT3) the SAM-Grid project will explore the features
and advantages it will provide. It is likely that the new architecture
will afford additional ease of use, scalability, reliability, and
certainly interoperability.

Web services are a set standards being defined within the W3C and other
standards bodies, and being used by the Globus team to provide the new
OGSA which defines Globus Toolkit v 3. This includes the use of web
services including SOAP (Simple Object Access Protocal), WSDL (Web
Services Description Language), and WS-Inspection to achieve the needed
resource advertisement, location, and , security for grid systems. There
is an effort within the Globus community to develop CoG's (Commodity
Grid Kits) which provide a bridge between existing languages or
frameworks, such as CORBA, and Grid services, these might be used as an
intermediary route to move the system quickly toward web services.
Adhering to these standards will provide the necessary interoperability
required for the SAM-Grid effort to move ahead and function with other
grids, and there is a straightforward roadmap of how to achieve this.

The SAM Data Handling system relies on CORBA as the framework for
communication among the various modules which comprise the system. CORBA
IDL interfaces are used to pass information through the distributed
servers, these include the user interface client, station servers,
middle tier database server, as well as a couple of miscellaneous other
serves that manage optimization and message logging. This portion of the
system is quite stable and has shown no evidence that it will not
continue to scale to meet the needs for some time. The system currently
employs a central naming service and requires that numerous ports be
opened through a site's firewall to allow for the CORBA, and Globus
communication to pass. Web services eliminates the need for a CORBA
naming service, also a single point of failure, the use of a distributed
naming service is reasonable, and quite straight forward. The system has
already employed remote naming services to allow resources on private
networks to be operated. By transitioning to web services the system
becomes more distributed, the interfaces are lighter weight, and
additional agility is afforded to function within the constraints of
firewalls. Moving this to a more distributed topology employing web
services will provide additional scalability, reliability, and
manageability.

The SAM-Grid system relies largely on Globus middleware for its resource
management and security services. This includes gateway resource
management using GRAM and GASS servers, information management using
MDS, grid security through GSI, and a component for file transfer,
GridFTP. The SAM system provides a centralized replica catalog managed
and stored in its central metadata repository in the form of an Oracle
database served to the system through a middle tier server. We plan to
transition this to a distributed Replica Location Service (RLS), similar
to the plans for distributing other parts of the system discussed above.
This will expose the data resources within SAM-Grid to other operational
grids, and facilitate the use of additional compute and storage
resources.

# Our vision is to ultimately work toward complete interoperability with other grids. This will provide many additional resources and prepare the system for the next generation of experiment needs. We plan throughout the project to test this interoperability with existing HEP experiments. The DZero experiment shares computing resources with many other experiments throughout the US, Europe, and Asia and will be forced to operate in an environment of grid standards. We will also explore interoperability with grids in other scientific areas, and the relationship to M-Grid and the Computer Science Departments at participating institutions will greatly facilitate this objective.

**Work Plan**

Institutional interest breakdown

-   Monitoring and Information Service, Fault tolerance, Scheduling --
    UTA, UM, FNAL

-   NFSv4 performance, NFSv4 sandbox, NFSv4 WAN, Network QoS, Code
    Signing -- UM, MSU

-   Web Services + interoperability -- FNAL, UM, UTA

-   Testing and Deployment -- MSU, UCR

Task list

The following is a list of proposed tasks as part of the ITR proposal
(these to be put with tasks below):

1.  develop a set of D0-specific attributes, the corresponding data
    models (i.e. hierarchical or sequential) and representative schemas
    based on LDAP and XML.

2.  develop a general monitor object model that can be a template for
    implementing the attributes.

3.  develop the distributed monitoring framework described earlier, with
    a high-level distribution protocol (i.e. exchange high-level data
    among the servers). (We can start with a simple basic push protocol
    but our design should be flexible enough to accommodate any future
    standard protocol being developed by the GGF. )

4.  develop the directory service for registering monitoring servers and
    sites that use them.

5.  install servers across a few sites to begin with, and test the
    monitoring system.

6.  add additional client site agents and perform scale-up studies.

Year 1

1.  Review the overall architecture of SAM-Grid, establish detailed plan
    for additional use of Web Services, modular component design, and
    minimizing DZero specific parts of existing software.

2.  Establish groundwork code sharing and documentation within the
    project.

3.  establish testing and deployment system for existing SAM-Grid at MSU

4.  Review existing scheduling and begin to test straightforward
    extensions

5.  Deploy shared disk SAM station using NVSv4 at MSU and begin
    extensive testing and performance measuring

6.  Begin deployment of NFSv4 at select Regional Analysis Centers,
    notably GridKa in Karlsruhe Germany.

7.  Begin using network beacons for network throughput and availability
    monitoring within DZero collaborating sites

8.  establish areas of potential interoperation outreach with other HEP
    grid efforts and potential interdisciplinary work on university
    campuses.

Year 2

1.  Monitoring infrastructure updates to provide information needed for
    scheduling

2.  begin multi parameter scheduling and fault tolerance work

3.  develop and distributed intercommunicating brokers and needed
    messaging

4.  signed (digital identity) code security project

5.  Begin use of NFSv4 security and ACL features to establish secure
    diskless systems at grid execution enabled sites.

6.  continue testing and deployment program

7.  Continue to upgrade existing system with Web services architecture

Year 3

1.  Continue work on scheduling and fault tolerance and Information
    collection

2.  Integration work among all systems

3.  Explore using NFSv4 WAN features within the context of grid file
    access

4.  Continue to testing and deployment program

Year 4

1.  Begin interoperability outreach effort

    1.  Other HEP experiments and European and Asian collaborators

    2.  Interdisceplinary university contacts

2.  provide fully operational systems