# ***SAM-Grid / LCG Interoperability System. Using the System in Production -- Final Report Andrew Baranovski, Adam Lyon, Gabriele Garzoglio***

[Table of content
1](#sam-grid-lcg-interoperability-system.-using-the-system-in-production-final-report-andrew-baranovski-adam-lyon-gabriele-garzoglio)

[Mission 2](#mission)

[Project statement 2](#__RefHeading___Toc132613755)

[About this document 2](#__RefHeading___Toc132613756)

[Required Functionality Additions to the System
3](#required-functionality-additions-to-the-system)

[Output bundling 3](#output-bundling)

[Scratch management 3](#scratch-management)

[Storage Selection 3](#storage-selection)

[Eliminate EDG UI Automatic Retrial
5](#eliminate-edg-ui-automatic-retrial)

[Additional scheduling system requirements from the SAM-Grid JDL
5](#__RefHeading___Toc132613762)

[Credential management 5](#__RefHeading___Toc132613763)

[Accounting 6](#accounting)

[System Expansion 6](#system-expansion)

[Cluster Additions 6](#__RefHeading___Toc132613766)

[Scalability Analysis 7](#__RefHeading___Toc132613767)

[Optional Deployment of new Integration Systems
8](#optional-deployment-of-new-integration-systems)

[Consistency of the system efficiency
8](#consistency-of-the-system-efficiency)

[Document efficiency of the system
8](#document-efficiency-of-the-system)

[Efficiency Stabilization 9](#efficiency-stabilization)

[Support 9](#support)

[Define Process to report user problems
10](#define-process-to-report-user-problems)

[Define Process to ask for support to LCG
10](#define-process-to-ask-for-support-to-lcg)

[Define Process to extend the system
10](#define-process-to-extend-the-system)

[Network Configuration 11](#network-configuration)

[Operations 11](#operations)

[Define Operational Model 11](#define-operational-model)

[Define Certification of LCG 11](#define-certification-of-lcg)

[Cluster Tagging 12](#cluster-tagging)

[Limitations 12](#limitations)

[Forwarding node/Resource broker 12](#forwarding-noderesource-broker)

[Storage element 12](#storage-element)

[Computing element 13](#computing-element)

[Conclusion 13](#conclusion)

[Current release cut 13](#current-release-cut)

[Future plans 13](#future-plans)

[Acknowledgements 13](#acknowledgements)

[Apendix 14](#apendix)

## **Mission**

Use the LCG Interoperability System consistently in production
operations with a consistent efficiency. Provide users with
documentation and support necessary to sustain system operations by D0
collaborators.

[]{#__RefHeading___Toc132613755 .anchor}***Project statement***

The SAM-Grid system is an integrated data, job, and information
management infrastructure. The system was designed to address the
distributed computing needs of the experiments of RunII at Fermilab. In
particular for the DZero experiment, SAM-Grid provides a user-friendly
environment to run typical production applications, such as montecarlo
generation and data reprocessing.

The goal of the SAM-Grid / LCG interoperability project is extending the
SAM-Grid system to use the LCG resources deployed by EGEE. The idea is
to handle the jobs within the SAM-Grid framework, retaining the
characteristics of user friendliness described above. In addition, the
deployment of the SAM-Grid can be reduced to a few \`\`forwarding\'\'
nodes, which act as interfaces to the LCG system. The actual deployment
to the cluster becomes responsibility of EGEE.

[]{#__RefHeading___Toc132613756 .anchor}***About this document***

The WBS of the project has been used to establish the core structure of
this report. For each item in the WBS we report status and work done.

## Required Functionality Additions to the System

We have added and improved several aspects of the system. These
improvements cover security, performance and operational features of the
project, features that enable efficient production operated by D0
collaborators.

### Output bundling

We have extended SAMGrid batch adapters to fetch and bundle
supplementary debug/log information to assist reporting instead of
troubleshooting of the LCG specific problems. The additional information
is especially useful for the end users that do not (should not) have
access to the forwarding nodes.

### Scratch management

During the test bed phase of the project we have established an
interface with the LCG scratch management system via the TMBDIR
environment variable. The TMBDIR is expected to point to an LCG WN
storage area that is guaranteed persisting for the job lifetime. The
TMBDIR area is used to dynamically deploy all the software and data
needed to run a user job. As we prepared the system for production, we
have improved the interface in the following ways: 1) assume and check
for a minimum space availability of 4Gb; 2) backup WN scratch space
cleanup and scratch location name uniqueness policies.

### Storage Selection

SAMGrid has been designed to support applications that by utilizing
1Ghz/day of computing consume and generate Gb's of physics data. The
data intensiveness of these applications incite careful thought of kinds
of data and data access patterns in order to define viable and
manageable storage deployment.

The primary application meant for the production on LCG is D0 MC and D0
MC merging. Generation phase of the MC requires job executable files,
min bias files, phase dataset files staged to the WN scratch space
before the start of the runtime. Furthermore, this data is frequently
reused by multiple MC requests that are "similar" in parameters of the
D0 runtime. Therefore we expect that most of the data traffic will be
generated by the MC applications repeatedly requesting access to the
same set of data from geographically (network-wise) scattered sites
participating in the LCG production.

A production ready system should implement means of mitigating temporary
resource failures to ensure consistent throughput. To achieve this, we
want to be able to reconfigure and/or enable critical resources involved
in the production process. We have identified storage as a critical
resource and worked to duplicate into production existing test bed
storage services. The storage selection is one of the subtasks addressed
on that course.

In conclusion, storage access and selection is an essential component of
the architecture and brings the project to the state where experiment
production can be run efficiently and consistently.

1.  Station Configuration in global LCG Environment\
    Station storage nodes consist of d0srv015.fnal.gov,
    d0srv065.fnal.gov,caterpillar.hep.man.ac.uk, dodge.hep.man.ac.uk and
    ccsvli02.in2p3.fr\
    Station routing is set to stage enstore files to d0srv015 and later
    re-transfer them to the randomly chosen nodes: caterpillar and
    ccsvli02. caterpillar and ccsvli02 are the SAMGRID/LCG SAM data
    input nodes.\
    At the same time, dodge.hep.man.ac.uk ccsvli02 are the SAM FSS
    buffer nodes. These nodes accept generated data and store the data
    intermediately for further processing(merging)

2.  SAMGrid Configuration in Global LCG Environment\
    SAMGrid/LCG configuration supports several application types: dzero
    reconstruction, dzero montecarlo and dzero merge. Each application
    is set to use its own storage configuration as below.\
    dzero reconstruction is tuned to stage physics data from d0src015
    while fetching binaries both from caterpillar and ccsvli02 (Fig. 2).
    In contrast, dzero montecarlo and dzero merge use caterpillar and
    ccsvli02 services for all their data input (Fig. 1). All
    applications use dodge.hep.man.ac.uk and ccsvli02.in2p3.fr as both
    buffer and durable location storages (Fig. 3).

### Eliminate EDG UI Automatic Retrial

Despite the original WBS, we have observed that grid job submission
retrials are extremely useful to mitigate deficiencies in the local
(batch system) deployment. A particular deficiency that was successfully
addressed by the retrials is the "black hole effect" . In summary,
"black hole effect" is a local scheduling policy that forces tasks to be
dispatched to the nodes that are empirically the fastest. Typically
these nodes do not respect requirements of the application at hand and
often cause premature termination of the job.

An increase in number of grid job submission retrials empirically
correlated with a chance of resubmission to the newly available
resources that are in compliance with application requirements. We have
anecdotic evidence that number of retrials set to 10 increases
submission efficiency from 30% to 70%.

[]{#__RefHeading___Toc132613762 .anchor}**Support additional scheduling
system requirements from the SAM-Grid JDL**

The initial phase of the integration project did not assume users
altering resource management policies pertaining to the LCG resource
broker. We found such feature inconvenient. It was often a necessity to
override static polices set on the forwarding node. A targeted site
validation / certification test is an example.

During the site validation phase, farmer or grid operator probes a site
by targeting LCG submission to a particular LCG resource. In order to do
so, he/she must be able to specify LCG resource selection requirements..
In a test bed, forwarding node configuration had to be changed. This
change was hard to coordinate and was inconvenient for a non experts.

In the SAMGrid JDL we have added lcg_requirement_string (and
osg_requirement_string) strings to by pass forwarding node LCG (OSG)
default resource selection parameters. The lcg_requirement_string and
osg_requirement_string allow a farmer to state internal LCG resource
selection policy that is the most convenient for the purpose at hand.

[]{#__RefHeading___Toc132613763 .anchor}**Credential management**

A fundamental aspect of the project is the forwarding node technology.
The core of the idea is to adapt software stacks of two grids via a
layer that understands mutual requirements and ensures proper job
propagation from SAMGrid to LCG. Availability of the fully delegated
user credentials at the LCG front-end (hence, on the forwarding node) is
one of the addressed requirements.

Fully delegated proxies are not generally generated by the forwarding
node gatekeeper. A test bed proof of principle resolution to that was to
ship original user credentials directly for the LCG front end use. The
solution has been recognized by many as insecure and had to be addressed
in order to commission a secure system to the physics community.

We have done R&D work to integrate MyProxy technology with SAMGrid
client and execution site software in order to mediate security
requirements between 2 grids. A cornerstone feature of the approach
relies on MyProxy service to grant access to fully privileged user proxy
anytime within the validity period of the original MyProxy-stored
credentials.

As a result, SAMGrid client and SAMGrid execution (forwarding node) site
now supports use of the MyProxy services. At the same time, we ensure
backwards compatibility with SAMGrid deployments not requiring the
solution. In addition, tools to install, configure and manage MyProxy
user credentials have also been developed.

The suggested default for the SAMGrid MyProxy service is
fermigrid4.fnal.gov -- maintained by FermiGrid deployment. FermiGrid
MyProxy deployment contact: Steeve Timm,
[mailto:timm@fnal.gov](mailto:timm@fnal.gov).

### Accounting

http://www-d0.fnal.gov/computing/grid/MC_LCG_Accounting.doc

## System Expansion

[]{#__RefHeading___Toc132613766 .anchor}**Cluster Additions**

The initial test bed consisted of two "friendly" LCG resources at
Clermont Ferrand and IN2P3. We have worked to pre-certify other LCG
resources that were willing to support the D0 VO. This work has
established a communication channel between site administrators and D0
VO representative and was effective mean to address LCG site
configuration, upgrade announcements, and network/firewall issues. To
date, our list of pre-certified computing elements consist of NIKHEF,
Lancaster, Prague, Manchester, Imperial College, Clermont Ferrand.

[]{#__RefHeading___Toc132613767 .anchor}**Scalability Analysis**

Our analysis of the project performance showed several areas that
limited productivity of the system. Below is the list of resources that
are used within SAMGrid job work flow and that are the largest
contributors to the limit:

3.  Forwarding node.\
    Forwarding node is responsible for accepting a job, preparing and
    serving job environment and polling job status. Each of these tasks
    requires certain fraction of computing charged against system idle.
    By architecture, the time to establish the job environment and job
    status polling are two major tasks that scale proportionally with
    request parallelism. Individually, it takes 2 second to retrieve
    individual jobs status and around 10 seconds to serve job
    environment to a WN.

4.  SAM storage deployment\
    Just like the forwarding node, SAM storage deployment scope is
    defined by the SAMGrid team. Storage is used to contain and serve
    executable binaries as well primary physics data. Storage is also
    used to store intermediate processing results and buffer output
    data. Storage resource usage depends on the type of the job as well
    as the job's cardinality. Basic characteristics of the SAMGrid
    storage deployed for SAMGrid/LCG are :

    1.  Space available: 500Gb per node ,

    2.  Number of nodes: 4

    3.  Download/upload rate: 1 Mb/s per job out of the European
        storage, 30Mb/s cumulative per node. 0.5 Mb/s per job out of the
        FNAL storage, 15MB/s cumulative per FNAL site.

5.  Fermilab storage system.\
    The SAMGrid / LCG production relies on DH services that are shared
    with other non LCG activities. In particular services that buffer
    and CRC check incoming data before storing data to tapes. We have
    observed (during refixing) that such services are prone to
    unpredictable spikes of load and consecutively need a bigger margin
    of the QOS than for exclusive access. This concern lead to
    deployment of addition FSS buffer machines to support offsite
    production (d0srv063 and d0srv065).

### Optional Deployment of new Integration Systems

1.  Forwarding node\
    Purpose of the forwarding node in production is to isolate software
    and configuration of the development environment and at the same
    time preserve the opportunity to change and enhance the project
    based on the input from running operations. The production node also
    helps to establish continuous software development cycle where newly
    added features can be certified by the experiment before their final
    use. Forwarding node in production is supported by Wuppertal team.We
    acknowledge Torsten Harenberg \<harenberg@physik.uni-wuppertal.de\>
    and <meder@physik.uni-wuppertal.de> for efforts to provide hardware,
    LCG deployment, and support for the new system.

2.  Station Node\
    We have extended our initial storage deployment using additional
    storage deployment at Manchester (caterpillar.hep.man.ac.uk,
    dodge.hep.man.ac.uk) and leveraged new storage space at IN2P3
    (ccsvli02.in2p3.fr). By applying the [storage
    selection](#storage-selection) work we've been able to leverage new
    hardware and increase the SAMGrid/LCG test bed productivity by
    factor of 2.

## Consistency of the system efficiency

### Document efficiency of the system

For simplicity, we recognized two kinds of inefficiencies. The
inefficiencies that bear a high cost on the resource usage and
inefficiencies that are low cost. All common issues related to LCG
middleware had a minimum impact on use of the SAMGrid services, thus
manifested as a low cost. To the contrary, issues stemming from SAMGrid
service failures were charging against overall system performance and
thus came at a high cost.

1.  Measure Efficiencies of the LCG infrastructure

    1.  Inefficiency due to LCG Resource Broker\
        Inefficiency due to LCG Resource Broker contributes little to a
        grand total of the system efficiency. However, few cases we
        observered required special handling and had no automatic
        recovery available for. Few examples:

        1.  edg-job-submit command may hang indefinitely.

        2.  Exhaustion of the V0 space at the RB with the grace period
            of 14 days.

    2.  Inefficiency due to Output Handling (Maradona Server)\
        Majority of failures due to Output Handling were observed at
        sites that had one or more nodes misconfigured. These nodes
        become "sinks" (see [black hole
        effec](#eliminate-edg-ui-automatic-retrial)t) that quickly fail
        the majority of jobs matched to the site. This inefficiency
        typically accounts for 70% of the submission failures.

    3.  Inefficiency due to Other Services\
        "Got an error while in the condor queue" is the typical failure
        that accounts for 5% present of the submission failures.

2.  Measure overall efficiency\
    While doing refixing, the SAMGrid/LCG production system has launched
    9600 jobs and stored 6800 files. This constitutes efficiency over
    70%.\
    Note: Because some of the jobs never reached SAMGrid bookkeeping
    code, this number does not fully reflect submission efficiency.

### Efficiency Stabilization

Efficiency stabilization has been achieved by diversifying use of
several storage and computing services. In order to achieve that, we've
deployed additional SAM storage nodes, enabled use of alternative LCG
resource broker, extended list of pre-certified sites, and deployed an
additional forwarding node in production.

## Support

https://lcg-registrar.cern.ch/

### Define Process to report user problems

Users are expected to do basic problem classification. The problem
classification is assisted by the web monitoring (samgrid.fnal.gov) and
job log files that list both SAMGrid and LCG specific details.

### Define Process to ask for support to LCG

At this stage of the project, LCG expert help through the
0_lcg_intgeration list has proved to be very efficient. As we move
production into more mature state and contribution of the LCG to D0 is
increased, better formalized methods of problem reporting will need to
be employed.

In contrast, reports related to SAM/SAMGrid issues are expected "SAM
issue tracker" <http://plone3.fnal.gov/SAMGrid> or via mail to
d0sam-admin@fnal.gov.

### Define Process to extend the system

Computing element, storage and forwarding node services are on the
critical path(are the bottlenecks) towards better productivity. Increase
in the cardinality of these services does have a measurable positive
impact on the system productivity.

Storage service in SAMGrid is a VO specific service and its deployment
is the responsibility of the VO. As a project, we've worked with D0 to
schedule deployment of the new hardware at Manchester and In2P3.

The SAMGrid/LCG forwarding node deployment is a joint effort shared
between core SAMGrid team and group of LCG experts from Wuppertal lead
by Torsten Harenberg. The SAMGrid team is responsible for deployment of
the SAMGrid services, LCG people provide LCG end installation and
support. At the moment, the project also relies on Wuppertal team to
provide hardware and hardware support.

New computing elements are added via "pre-certification" procedure. The
procedure is meant to ensure service usability with respect to SAMGrid
requirements. Pre-certification includes these steps:

1.  Publish IP range in order to open relevant firewalls at Wuppertal
    and In2p3 sites.

2.  Ensure availability of at least 4Gb of WN scratch space per batch
    slot.

3.  Define TMPDIR variable and configure LCG WM scratch management
    policy to ensure space allocation and cleanup.

### Network Configuration

Storage element at In2P3 is behind a firewall. Which is why, sites that
are part of the SAMGrid/LCG production must register their IP range with
In2P3 networking group via D0 liaison. In contrast, Manchester storage
elements are open for use to any site.

## Operations

### Define Operational Model 

The production effort on LCG via the SAMGrid is not fully automated. The
project expects D0 to contribute a group of people that are willing to
manually operate the infrastructure. The role of each member (farmer) of
the group is to oversee SAMGrid/LCG operations in a particular
geographic region /institution. The tentative workload of the farmer is:

1.  [Resource certification/
    re-certification](#define-certification-of-lcg)

2.  Production

3.  LCG issue reporting and followup

4.  SAMGrid issue reporting and follow.

The "by region" breakdown should help to spread support load and
diversify institutional contributions to D0 production.

### Define Certification of LCG

The quality of computing delivered by SAMGrid/LCG is monitored via the
certification procedure. The procedure relies on predefined set of basic
analysis/generation/reconstruction parameters to initialize SAMGrid
certification job. The outcome of such job is then analyzed by the D0 to
detect and qualify the divergence with respect to an acknowledged
"etalon". If the divergence is within acceptable limits, the
certification job is considered a success.

The fundamental presumption is that the certification success is a good
indicator of the quality of the generated data at a site even beyond the
parameters used to perform certification step itself.

These are the certification parameters

1.  Mc_run job release

2.  LCG computing element\
    LCG computing element is set via lcg_requirement_string and is
    reflected as a "facilityName" parameter in the file metadata.

3.  D0 run time version

### Cluster Tagging

The LCG farmers are encouraged to define lcg_requirement_string in a way
that supports relevant application certification requirements. An
application specific work load management tools (d0repro) may be used to
automate this process.

## Limitations

In this section we attempt to derive reasonable estimate of the system
productivity by analyzing following categories of the critical system
parameters.

### Forwarding node/Resource broker

One of the functions of the forwarding node is the grid job status
reporting. This task is accomplished by aggregating individual statuses
of all parallel sections that are part of the original SAMGrid/LCG grid
request.. Each status request is scheduled in a regular time intervals
and consumes about 2 seconds of CPU time. Assuming job status requests
are scheduled every hour (current setup), total number of concurrent LCG
jobs that can be supported by the forwarding node is 3600/2 = 1800 jobs.
Consecutively, 1800 is the maximum number of CPUs the infrastructure is
able to support at any given moment of time with hourly job status
updates.

### Storage element

Measured Fermi data import and export rates to LCG are around 17
Mbytes/s. Measured data import and export rates sustained by the storage
elements deployed close to the LCG sites are around 30 Mbytes/s.
Assuming each MC jobs requires 1000 Mb of pre-staged data, the
30Mbytes/s support up to 24\*3600\*30/1000 = 2600 MC jobs a day. At the
same time, the 17 Mbytes/s FNAL import rate corresponds to 20M events
per day of the physics output (assuming 74Kb per event).

### Computing element

5 pre-certified computing sites contribute around 500 "average" CPUs a
day, an equivalent of 4000 250 events jobs a day = 1 Million events per
day.

### Conclusion

We believe that MC production via SAMGrid/LCG infrastructure is neither
limited by the Fnal import/export IO factor nor by availability of CPUs.
It is however limited by the IO factor at the local storage deployments
near the LCG sites that cache frequently reused (such as binary and min
bias files) data. The gross production via LCG is estimated by 2600\*250
= 648000 events per day or **4.5 Million events per week**. This number
may be discounted by the efficiency factor of 70% that translates system
production into sustained **3 Million events per week.**

## Current release cut

TBD

## Future plans

Integrate LCG storage elements into SAMGrid/LCG infrastructure to enable
better caching of the frequently re-used data.

Better automation of the SAMGRid/LCG production

Improve SAMGrid/LCG monitoring and accounting.

## Acknowledgements

We want to thank Torsten Harenberg and David Meder-Marouelli for their
support of the LCG forwarding service and D0 VO administration, Jeff
Templon for his valuable input on LCG deployment issues, Tibor Kurca for
his persistence and patience that helped us all to bring the
infrastructure to the "certifiable" state, Sabah Salih of Manchester for
contributing to the expansion of SAMMGrid/LCG storage infrastructure,
Steve Timm for the Fermi grid MyProxy support, and at last but not the
least Gabrielle Garzoglio and Parag Mhashilkar for their efforts to
bring up the initial test bed.

## Apendix

SAM datahandling deployment for the SAMGrid/LCG

![](media/image1.wmf){width="5.997222222222222in"
height="4.134027777777778in"}Fig. 1 Flow of the MC input data

![](media/image2.wmf){width="5.99375in"
height="4.170833333333333in"}Fig. 2 Refix input data flow

![](media/image3.wmf){width="5.99375in" height="4.293055555555555in"}Fig
3. Reco and MC output data flow