SAMGrid Collaboration Effort Ideas

This document is supplied to the DØ CPB as a suggested guideline for
effective incorporation of collaboration effort into the SAMGrid
project, along lines which will advance the goals of interoperability on
multiple Grid systems (LCG, OSG, other...) and of a data handling system
with long-range maintainability and deployability strategies that match
the DØ experiment schedule and feature needs.

SAMGrid Architecture and Technology

SAMGrid is a system built around the sequential delivery of files which
are organized into flexible, user-defined datasets, based on
comprehensive descriptive metadata stored in a centrally maintained
Oracle database at Fermilab (Run II's Tier 0 center). The system stores
processing information in addition to file metadata. Its services are
primarily implemented in a set of servers written in Python,
communicating with each other via CORBA. It includes C++ and Python user
APIs and a test harness.

The Grid job submission tools are built on Condor and Globus tools in
the Virtual Data Toolkit.

SAMGrid Status

SAMGrid is in production with local job submission for central (i.e.
local to Fermilab) analysis and production activities, remote analysis
and production activities, production and user-generated file storage.
It is also in production with remote job submission for simulation
activities. Detailed status reports from February 04, and a more recent
update, are available at

<http://wwwserver2.fnal.gov/cfdocs/projectsdb/projdetail.cfm?ProjectID=122>
(scroll down to see the 4 talks from Feb 18)

[http://computing.fnal.gov/docdb/documents/0003/000314/001/SAM%20Project%20Status%20Jun%2004.pdf](http://computing.fnal.gov/docdb/documents/0003/000314/001/SAM Project Status Jun 04.pdf)
(updated project report from 6/16/04)

SAMGrid Plans

SAMGrid development projects for the next year, for D0, are shown on
slide 13 of the second hyperlink in the Status section above. SAM
deployment plans for D0 are shown on slide 6. The other slides in this
talk note the increasing commitments of the SAMGrid team to other
experiments.

SAMGrid Effort

SAMGrid development effort in three months or less is going to be down
by 3 FTE compared to 6 months ago, due to CD reassignments. At the same
time, as the recent project talk indicates, commitments to other
experiments are rising and the Grid deployments for D0 are entering an
integration/production phase, which requires large support commitments
plus a sufficient reserve of development effort to respond to problems
arising in that phase. At the same time, the project is expected by both
its stakeholders and the Computing Division to move rapidly in the
directions of interoperability with various Grid middleware
environments, technology changes driven by the need to consolidate
effort with other Grid projects in CD, and feature enhancements driven
less by experiment than by facility requirements (e.g., tape migration
activities, site security enhancements). Some of the effort reduction is
expected to be recovered in other reorganization moves, but we clearly
have a drastic short term shortfall as well as a long term ambitious
program whose loss would leave D0 in a very vulnerable position: with a
data handling system unable to respond to the changing Grid environment
and hence unable to supply the necessary flexibility in the quest for
resources that will grow in importance as Run II datasets multiply at
the same time Run II resource infusions decline. We could benefit
dramatically from experiment effort, IF it's efficiently and coherently
directed. Some successful examples to date of collaboration effort
include the GridPP collaboration with SAMGrid (Runjob interface to SAM,
SAM station testing and development of operational expertise that was
shared with the station community, JIM testing and development); the
contribution of strong shift coordinators from BNL (Yip) and Indiana
(Zieminska); integration testing of JIM deployment for simulation by
Langston U. (J. Snow); development effort for the MIS project (new but
making great strides this summer) from Prague (Vokac). Note I am not
including here the extensive effort for remote deployments in general,
but recent or ongoing efforts which have fed back directly to the
project goals.

Ideas for Collaboration Effort (not priority ordered)

-   Monitoring and Information Server development. Tasks --
    instrumenting SAM servers with appropriate monitoring events,
    reimplementing SAMTV to use the monitoring server as input,
    connecting the SAM MIS to MonALisa.\
    \[This project is well underway with Adam Lyon and Sinisa Veseli
    from the core team, Mariano Zimmler (summer student), Matt Leslie
    (CDF) and collaboration effort from the Prague group (Petr Vokac).
    There is room in this project to add effort to connect the MIS to
    the LCG monitoring software RGMA, and also to replace the summer
    student effort which we will lose in September for further system
    developments (connecting SAM TV to Enstore; writing monitoring
    applications for production activities like (re)processing or
    skimming, ...)\]

-   Interoperability & integration testing for SAMGrid job submission.
    Tasks -- (1) providing effort to enhance the test harness to include
    Grid job submission modes, using either the Fermilab SAMGrid testbed
    or offsite resources. \[UK Imperial post is already targeted for the
    testing portion of this -- would like to see the work undertaken in
    the test harness framework for more usability.\] (2) Working with
    the LHC metadata group to provide web-services based access modes to
    a SAM catalog. \[One of the European groups would be a logical
    candidate for this, although I am also hoping to interest US CMS
    effort in this project.\] (3) Working with the UK CDF GridPP effort
    to provide replacement dimensions code with enhanced functionality
    (parent-of, child-of, sibling-of; request-related dimensions) and
    better maintainability. \[ This is likely the hardest project on the
    list.\] (4) Demonstrating D0 job submission to Grid3+/OSG resources.
    \[Good project for a US group with Grid3 connections.\]

-   Further development of JIM tools. JIM development tasks which remain
    include implementing realistic brokering algorithms and helping
    deploy/debug those; revamping the interface and interactions with
    Runjob as it evolves; interfacing JIM to the MIS; doing integration
    testing for Reprocessing II (what Joel Snow did for MC production
    JIM deployment), testing JIM submission on central analysis
    systems... Also for UK Imperial post. The core team effort on this
    is being taken down from 2 FTE + UTA students to 0.5 FTE + UTA
    students. We will likely be effort limited here, so a contribution
    here will make a difference. \[The project to test JIM submission on
    central analysis systems could be something for a good postdoc as a
    learning experience.\]

-   Optimizer development. Some fairly simple optimizer improvements are
    needed to handle larger dataset sizes (mainly at this point for CDF
    actually), but no effort is available for optimizer development at
    all. Optimizer enhancements as a general project would be
    interesting from a Grid/computer science point of view, very
    separable as a project, and not critical path -- suitable for a
    group with strong computer science ties and/or a need for a long
    ramp-up time.

-   Offline shift coordinator. Daria has kindly agreed to continue her
    appointment, but it would be exploitative not to remember to keep
    this on the list for future reference. I'd just like to note that
    having good people like Kin and Daria in this job makes quite a
    difference to the shift performance and is very important to the
    project -- without good shifters, the core team gets very little
    development time.

-   Station SRM development. This project was scheduled on the basis of
    having 1.5 FTE from the core team available, and the reality is more
    like 0.5 FTE from the core team, due to loss of CDF effort and
    operational support for current station developments. This will slow
    the project significantly, and this is a project that enhances
    interoperability with the general direction of grid storage systems.
    Another development FTE would be a big help -- requires C++
    expertise and a willingness to work closely with the core developer
    and SRM developer at Fermilab. \[Best for someone who could spend
    time at Fermilab at least for a rampup period.\]

-   Working with the Fermilab Lambda Station project and SAMGrid to
    develop use of SAM resource management for networking. Requires
    access to StarLight with high speed lambda-separated networking
    (I've forgotten the acronym for this.).

-   General tools development. A number of fairly small but extremely
    useful projects (like a project audit tool, a dataset creation tool,
    ...) could now be quickly implemented given the new python API and a
    reasonable amount of user domain knowledge. \[Eminently suitable for
    a postdoc or graduate student with a fairly small amount of time and
    existing python skills, or somewhat more time and a commitment to
    acquiring python skills.\]