LHC Grid Computing Project

Common Use Cases for a

HEP Common Application Layer for Analysis

HEPCAL II

![](media/image1.png){width="3.7493055555555554in" height="0.44375in"}

  -------------------------- --------------- ---------------------------------
                             Document        HEPCAL II
                             identifier:     

                             Date:           7/7/2003

                             Authors:        **J.Beringer (CERN),\
                                             N.Brook (Bristol),\
                                             P.Buncic (CERN/IKF Frankfurt),\
                                             F.Carminati (CERN),\
                                             R.Cavanaugh (UF),\
                                             P.Cerello (INFN)\
                                             F.Donno (CERN/INFN),\
                                             D.Foster (CERN),\
                                             C.Grandi (INFN),\
                                             F.Harris (CERN/Oxford),\
                                             L.Perini (INFN),\
                                             A.Pfeiffer (CERN),\
                                             R.Pordes (FNAL),\
                                             D.Quarrie (LBNL),\
                                             A.Sciabà (CERN/INFN),\
                                             O.Smirnova (CERN/Lund),\
                                             J.Templon (NIKHEF),\
                                             A.Tsaregorodtsev (IN2P3)\
                                             C.Tull (LBNL)**

                             Editors:        **F.Carminati (CERN)**

                             Document        **DRAFT v0.1**
                             status:         
  -------------------------- --------------- ---------------------------------

  -----------------------------------------------------------------------
  [Abstract]{.underline}: This document identifies common use cases for
  LHC applications to use Grid services for data analysis.

  -----------------------------------------------------------------------

  ----------- ------------ -----------------------------------------------------
  **Issue**   **Date**     **Comment**

  0.0         12/05/2003   Output of HEPCAL week

  0.1         21/05/2003   First revision by F.Carminati for discussion within
                           GAG

  0.2         7/7/2003     Second revision for discussion
  ----------- ------------ -----------------------------------------------------

Content

1 Introduction [5](#introduction)

1.1 Things to add [5](#things-to-add)

2 The Analysis activity [6](#the-analysis-activity)

3 analysis scenarios [9](#analysis-scenarios)

3.1 Definitions [9](#definitions)

3.2 Scenarios [9](#scenarios)

3.3 metrics [10](#metrics)

4 ANALYSIS EXECUTION MODELS [11](#analysis-execution-models)

4.1 SUPPORT FOR QUERIES BY common layers
[11](#support-for-queries-by-common-layers)

4.2 Support for Analysis Job Execution by a Common Layer
[13](#support-for-analysis-job-execution-by-a-common-layer)

4.2.1 No special support by WMS [13](#no-special-support-by-wms)

4.2.2 Support at the Dataset Level [14](#support-at-the-dataset-level)

4.2.3 Support via job pipelines [14](#_Ref455137708)

4.2.4 Support for Data Access by A Common Layer (or mechanism)
[17](#support-for-data-access-by-a-common-layer-or-mechanism)

4.3 interactive vs batch grid activity
[18](#interactive-vs-batch-grid-activity)

5 system requirements [23](#system-requirements)

5.1 Provenance and Job Traceability
[23](#provenance-and-job-traceability)

5.2 INTERACTIVE ENVIRONMENT support
[24](#interactive-environment-support)

5.2.1 Log books and reports [24](#log-books-and-reports)

5.2.2 Persistent interactive environment
[25](#persistent-interactive-environment)

5.3 Deviant Flows and Errors [25](#deviant-flows-and-errors)

6 SERvice models [26](#service-models)

6.1 analysis software deployment [26](#analysis-software-deployment)

7 Collaborative work [27](#collaborative-work)

8 Recommendations [28](#recommendations)

9 Conclusions [29](#conclusions)

10 References and glossary [30](#references-and-glossary)

10.1 Applicable documents and reference documents
[30](#applicable-documents-and-reference-documents)

10.2 Terminology [30](#terminology)

11 Description of use cases [31](#description-of-use-cases)

# Introduction

Detailed knowledge of HEPCAL-prime is a prerequisite for reading this
document.

## Things to add

Things on which we should spend more time thinking

1.  Do we need private metadata attributes (individual/group) in the DMC
    \[the answer is yes, so put them in! But it should not be impossible
    to read other users metadata, include use case?\]

2.  Environment -- analysis, execution. More thinking of requirements
    for analysis... In particular we need to address Juerg's point about
    how the user should see the same environment from laptop level (on
    airplane even!) all the way to massive reconstruction level
    worldwide. The two obvious extremes are to make the grid visible at
    the command shell level (like slashgrid), or to make 'grid-on-a-box'
    where one could have a UI/CE/WN/SE all installed on your laptop.

3.  Analysis end to end -- hypothesis, formulation, analysis activities,
    validation, results...

4.  Requirements on application middleware and applications
    themselves... paratrooper paradigm

# Basic Concepts

Here we need to put enough background information so that the next
section is readable by mere mortals.

# The Analysis activity

We felt important to start this work with a definition of the analysis
activity. \"Physics data analysis\" refers to the iterative, exploratory
activity during the later stages of a physics experiment in which the
physicist attempts to transform via a set of algorithms reconstructed
event data into publishable scientific results. These results are
quantities derived from the experimental data that are accumulated in
histograms, tables or other statistical entities to provide evidence for
the subject of the investigation.

We start by defining the general flow of a data analysis activity, which
could be batch or interactive by starting with a very high level
scenario.

An experiment collaboration has reconstructed raw data and MonteCarlo
data generating a set of DSs that contain the events to be analysed.
This is done according to the experiment policy by the production
process. This process is explained in the HEPCAL document.

A member of an experiment collaboration wishes to become involved in
physics analysis activities. As a pre-requisite he needs to:

1.  Register as a user of the computing environment of experiment.

2.  Make sure that the analysis activity he intends to do is compatible
    with the resources (computational, network or storage) he is
    entitled to use. Physics analysis often happens within the context
    of an analysis working group, so one way for the user to gain
    resource access is for her to become a member of one or more working
    groups, or to create a new one.

After this, the analysis activity consists of some subset:

1.  Perform queries on the Metadata Catalogue(s) to determine which
    DS(s) meet their broad criteria (physics channels, processing runs,
    type of event data, etc.) Note that this set of input DSs may be
    already known from a previous query, and in this case this step is
    not performed every time. Current experience of the experiments
    indicates that it may be beneficial if the middleware could
    optionally cache the results of the query on behalf of the user (see
    \<interactive analysis section\>.

2.  Query the input set of DSs, selecting the event components of
    interest and the events of interest, using event-level metadata. The
    result of such a selection is a list of event component identifiers
    (experiment-dependent) that allow retrieving the relative event
    subset within the associated DSs.

3.  Optionally save for further use the results of 2 as defined in step
    5.

4.  Perform iterative analysis activity, looping over the event
    components selected by step 2 above. This activity may involve
    additional filtering (selecting a subset of the original event set
    from step 3), reprocessing (augmenting or replacing components of
    the selected events), generation of new calibration and statistical
    information. The analysis code is in general some combination of
    "official" experimental code and new user code being developed for
    end user analysis.

5.  Optionally save the results of 4 and publish them. The role the
    physicist is playing for this session will determine whether this
    publication is at the individual level, or is associated with a
    particular PWG. There are three modes of making the results
    persistent:

    a.  "Create TAG DS". The new DS consists only of references to event
        components, with no copying of the event data being performed.
        This is equivalent to 3 above. This is sometimes called "shallow
        copy", i.e. pointers are copied and not what they address.

    b.  "Make new reclustered DS" The selected subset of input event
        components is copied to a new DS so that they are available more
        efficiently to further processing. This is called "deep copy"
        because not the references but the information they point to is
        copied.

    c.  "Create new DS". If components of the events are modified (or
        new components are generated), these can only be saved to a new
        DS.

Jobs could be submitted for any internally consistent sequence of the
above steps and the whole procedure may loop over these steps. Input
files may not be known for step 4, e.g. recursive studies will need
files to be opened dynamically. However the user should be able to
evaluate in some way the cost of this step, to be able to allocate Grid
resources to the job.

When the analysis is complete, the user may wish to inspect the
provenance of the output for verification purposes.

Since the analysis is iterative, there may be lots of datasets left as
by-products of the intermediate steps. This may places extra
requirements dataset-deletion functionality. If an "analysis logbook" is
implemented, it will be useful to be able to mark a group of analysis
steps for deletion, which will delete in turn all the output datasets
associated with these steps.

## Peculiarities of the analysis activity

The process described above looks very much like a production use case,
but the following differences have been identified:

1.  Frequent use of "non-standard algorithms" and user- or PWG-code
    together with "official" experiment software release.

2.  Input DSs not necessarily known a priori.

3.  A high probability to have a very sparse data access pattern.

4.  Larger number of people submitting jobs concurrently and in an
    uncoordinated fashion. The consequence is a chaotic workload leading
    to more need to manage resource usage.

5.  A wide range of user expertise and familiarity with the system.

6.  A significant proportion of "interactive" jobs (see Sec. 5.3).

7.  In general strategy/metrics may differ. Ordered production tends to
    value total system throughput while response time is more important
    for end user analysis.

8.  Many more \"roles\" involved in analysis.

9.  Need for detailed provenance information. Overlooked in HEPCAL-I,
    chain of steps longer, and more chaotic activity.

10. Need for a resource estimator and for a way to limit resource
    consumption or elapsed time.

As explained above, the result of event selection / transformation
during the analysis activity may be new DSs. These DSs will be created
per node, physicist, physics channel, giving rise to a potentially large
number of small files that can be an issue for efficient access to the
data (of this more later in 4).

# analysis scenarios

We present now a more detailed description of the analysis activity via
a number of scenarios, before defining more precisely some use case. The
scenarios described in the following assume that the reconstruction of
the raw data ("RECO") is already done and has created ESD data with
their associated experiment-specific Metadata ("Experiment MetaData" or
"EMD") and possibly also an initial set of "Reconstruction TAGs".

The policies of the experiment will determine which information will be
stored in the provenance for the various stages of analysis scenarios.
It could for example be foreseen that every dataset that is going to be
replicated has this information available for everybody in the
experiment.

The connection between the various scenarios is done by using the
feedback (in the form of new and/or updated algorithms, condition
databases and selections) from the "later" stages in refining the
"earlier" stages of the scenarios.

## Definitions

In all of the discussion, we use the following definitions:

**Event**. The word "event" in general refers to the raw data associated
with a specific trigger, as well as all the entities derived from them.
In the case of real data the derived data products can be associated
with a single primary entity to be identified in the original trigger.
For simulation the situation is still the same in theory. In practice
due to the artefacts of the simulation technique, several primary
entities can be at the origin of the same event, and some of them can be
common to more than one event. However we may consider that, at a
certain level of detail, our definition is valid. We indicate with the
name of "event component" or "data objects" the raw data and any derived
quantity. In general the event components may also contain metadata,
describing e.g. the provenance of the data object (see sec. 6.1).

**Metadata.** "Metadata" refers to information such as:

1.  Information on event components, contained in a TAG collection, such
    as which subdetector or physics channel the component belongs to.

2.  Provenance information of datasets or data objects, describing the
    process of creating the object.

3.  Bookkeeping information on data objects (date created, size).

**Input dataset.** "Input datasets" (in the context of analysis)
generally refer to any pre-existing combination of ESD, RAW, EMD, AOD,
or TAGs from RECO (or re-RECO) needed by the job. Of course there may be
other required input datasets containing objects besides these (such as
calibration information). Additional input is needed like the versions
of the s/w and configuration used for the "PA". The provenance
information needs to be included; depending on the implementation this
information could be recorded in the DMC, in some system-provided
provenance archival service, or recorded in the EMD.

**Output dataset.** "Output datasets" refer to AOD, TAGs, new or updated
EMD information, histograms or statistical information produced by the
job. There may be multiple output streams (representing different kinds
of output information from the same job, e.g. a histogram stream plus
one or more AOD streams).

**Job.** The definition of job is given in the HEPCAL document to which
we refer.

## Scenarios

We present three scenarios from the one of the single user to the
"experiment-organised" analysis. We begin with the scenario "End-User
Analysis", since this scenario will be the one most frequently executed.
\[C.Tull gives us something on resources here\]

**End-User Analysis ("EUA")**

The user executes the analysis steps defined above in 3. She may apply a
private algorithm to the input data. The user may be using a private
dataset as input, and this dataset might not even be in the Grid DMS.
The interval between consecutive iterations of this activity probably
ranges from a few hours to a few days (or as frequently as resources
permit!) The analysed information may then be published back to the
experiment in the form of new selections/cuts/algorithms and be used in
a subsequent MGA or PA (see below). Alternatively the output might be a
new (grid or non-grid) DS.

**Group-Level Analysis ("MGA")**

A PWG gives to the Group Production Manager a set of input DSs (based on
a selection on the EMD, AOD or TAG from the reconstruction step). These
input DSs are selected by queries against EMD or TAGs from the
experiment, including the information from the (previous) PA\'s.
Depending on the policies of the experiment, the group might initiate
replication of the output data to other sites, or this might happen
automatically for "official" data products. We expect this scenario to
be executed about every month (per group). Usually all DS are on the
Grid and the results are also made available on the Grid.

This analysis is often a refinement of the preceding PA in preparation
for the following PA.

**Production Analysis ("PA")**

The various analysis groups of an experiment each suggest to the
Experiment Production Manager a set of input DSs (based on a selection
on the EMD or AOD/TAGs from the reconstruction step) and the algorithms
of a given version of software and configuration information. After the
experiment management endorses these algorithms and configuration data,
the Experiment Production Manager starts the "Production Analysis"
running the software on the selected (from the Reconstruction TAGs)
events from the input data sets. This program will produce a set of
output DSs and EMD, potentially creating multiple output streams that
will be published on the Grid (with "deep" or "shallow" copies) of the
relevant event components. Running the whole collection of "Physics
groups algorithms" is an experiment-wide centrally managed "production".
All DS are on the Grid and the results are also made available on the
Grid.

The three examples given before try to capture different aspects of what
is really a continuum. We are aware that the difference between the
"RECO" and "PA" as described may be mainly sociological and concerning
Quality Assurance and resource allocation. However in the context of
this document, going from "ESD" to "ESD-prime" for a subset of "RAW" is
defined *not* to be in the scope of a "PA".

## metrics

\[ This section needs considerable reworking, still looking for
brilliant ideas. \] It is expected to have about 10-15 physics analysis
groups in each experiment with probably 10-20 active people in each
extracting the data from the earlier scenarios above (PA or MGA). For
the later stages (scenario MGA or EUA) \[quote upper limits, refer to
ATLAS?\] the produced data may not necessarily be registered on the
Grid. In addition, it is expected to have about 30(?) people per
subdetector in each experiment (total of 300-500? per experiment)
accessing the data for detector studies and/or calibration purposes. So
a total of 400-600 people in each experiment are expected to do the
extraction of (possibly private) results. This number is representative;
depending on the stage of the experiment the profile might be quite
different.

# ANALYSIS EXECUTION MODELS

The scenarios described above for an analysis job can lead to a number
of execution models. These execution models in turn result in different
conclusions on what services and capabilities are expected from the
middleware layer and from the application layer. This is further
complicated by the situation where different communities have different
definitions of the boundary between "middleware" and "application" or
"experiment framework". In this section, we try to explore some of the
issues regarding the execution model.

The high-level statement of the problem derived from the above scenarios
is:

-   We have an algorithm that we would like to apply to a set of input
    data in a given environment.

-   These input data may be explicitly specified as a set of DS's or
    selected by the job itself. In most cases this selection is done via
    a query: the set of data to be processed is the set of data that
    matches the query.

-   The user provides an algorithm and a query to the workload
    management system.

-   "Something happens" and at the end the user receives the output of
    the algorithm.

Our task here is to identify ways in which "something happens", and to
explore how much of this "something" is handled by the middleware and
application layers, respectively. The discussions so far have identified
three areas in which it seems necessary to explore the division of
labour between the middleware and the application.

-   Query execution.

-   Workload distribution and execution.

-   Data access.

It is clear that we need all three to efficiently support the range of
analysis activities for the LHC experiments.

## SUPPORT FOR QUERIES BY common layers

As we said above, the input data may be the result of a query. The query
can invoke middleware services and experiment dependent services. As we
said in 3.1 it would be very useful to submit the query and get an
estimate of the cost for the data access. If this is too high the user
may change the query until its cost is acceptable or obtain permission
to use the resources required and then submit the job. We do not discuss
here the question that the query itself may have a cost in term of
resources. An English-language representation of a typical query might
look like this:

> All the data (or perhaps instead "at least 10,000 events" or
> "retrievable within 2 days") from first quarter 2007, taken in trigger
> configuration EFG, reconstructed with reco version 7.4 and calibration
> set 21.d3, for which p~T~\>150 GeV and a J/ψ was observed.

It will not be difficult to generate a query result that represents a
very large dataset. This needs attention. Users need to be encouraged to
think about the amount of resources that might be needed to process the
query results. The system may set some upper limit on the size of the
acceptable return; alternately this may be affected by user resource
allocation policies elsewhere in the system.

The most naïve and straightforward way to implement such a query is for
each experiment to have a gigantic table (like a classical DBMS) which
has a structure similar to the following one:

  ------ ------------ ---------- ------ ------------ -------- ------ ---------- ------ -----------
  evID   date         Trigconf   type   algversion   calibv   p~T~   Tag        LDN    Object
                                                                     particle          reference

  1231   31/12/2006   EFG        AOD    7.3          21.d2    120    J/ψ        A1     Ccdfe

  1232   01/01/2007   EFG        AOD    7.3          21.d2    121    K~L~       B1     C45ea

  1233   01/01/2007   EFA        AOD    7.4          21.d3    164    Bs         A2     E43fe

  1234   01/01/2007   EFG        ESD    7.4          21.d3    79     None       C4     Ae236

  1234   01/01/2007   EFG        AOD    7.4          21.d3    160    J/ψ        B3     910e3

  1235   01/01/2007   EFG        AOD    7.4          21.d3    141    Λ          Fred   F3e12
  ------ ------------ ---------- ------ ------------ -------- ------ ---------- ------ -----------

This example is vastly oversimplified. If such a table were to exist, it
would likely at least need many more columns, and even more likely would
need to have a richer structure. We hope it suffices for illustration
purposes.

The query described above will match only the next-to-last entry in this
table. The last two fields are needed to enable the user program (in
cooperation with the middleware) to access the matched event. The LDN is
needed to supply to an "open" call, and here we explicitly assume that
the middleware will provide a Gridified open such that we provide an LDN
to an open call, and get back a file handle or object from which we can
read the bytes just as if the file were on a disk local to the worker
node. The "object reference" is something the experiment software can
understand and use to find the selected event inside the dataset
referred to by the LDN. Examples might be a simple event number, a byte
offset into the dataset plus the size to read, or a sort of object
pointer -- this is completely up to the experiment. The execution model
for the job, at this level, would be:

1.  Apply the query to the event table.

2.  Take the list of object references so generated as input

3.  Generate job(s) that access these objects and run the algorithm on
    them

4.  Return the output.

Experiments are in general not enthusiastic about the use of such a
table. First the table would be very large. Such tables would typically
have on the order of 10^12^ entries per year. Furthermore the
experiments were not planning to construct such a single flat table
themselves, nor were they prepared to let a middleware layer provide and
manage such a table on their behalf.

There was some interest in various hybrid approaches. Part of the query
example above can be handled by facilities of the Dataset Metadata
Catalogue as described in HEPCAL. Columns two (date) through six
(calibration version) could be part of the dataset metadata, since
within a single dataset the events will almost certainly have the same
value for each of those columns. Therefore we decided it was interesting
to consider a "split" query -- one part that acts on a dataset-level
catalogue, and another that operates essentially at the
experiment-framework level. The split can be at an arbitrary level,
which is advantageous since the various experiments have different plans
for the Dataset Metadata Catalogue usage. There is no explicit
requirement on the ordering of events returned back from a query.

The experiments also were very interested in a hierarchical metadata
space or, more generally, a clustered metadata space. Such a hierarchy
can be an efficient way to cut down the search space compared to the
flat-table order of 10^12^ mentioned.

The experiments agreed that however the list of event objects is
returned, these event "identifiers" or "references" should be suitable
for handing to the experiment framework, which can essentially just
"open" them to get the bytes.

The conclusion is that the support for queries will be split between the
Dataset Metadata Catalogue (provided by the middleware layer) and the
experiment framework layer. The contents of the DMC will be experiment
specific, but the way in which these queries are executed will be the
same across all experiments. Typically this DMC-level query will result
in a list of LDNs. Since the complete query is a logical AND of the DMC
and the experiment dependent part, all DSs containing objects that match
the complete query will be in this list, but there may be more. We do
not require (since it appears impossible to guarantee) that all DSs
matching this middleware-level query will contain events matching the
complete query. The approach requires that a set of matching DSs will
not contain duplicate copies of events, or that the experiment software
will have to have some sort of guard against inclusion of duplicate
events in a single pass. In some of the scenarios below, this second
case is still not sufficient.

## Support for Analysis Job Execution by a Common Layer 

In HEPCAL, the middleware does not provide any special support for this
type of analysis job. At first glance this seems undesirable, since such
a job has the potential to access many datasets and consume a
considerable amount of computing resources. For David: here a problem is
being identified! If there is no special middleware support, the job may
not benefit from being run in the Grid environment, and analysis may
even take a step backward from pre-Grid days. We describe here several
scenarios for how the Workload Management System (WMS) -
experiment-independent middleware - might support execution of analysis
jobs as described above. \[replace DAG everywhere by workflow
representation\] \[add here publication on the fly of the software??\]

There are several options for running analysis jobs as have been
specified in the previous sections:

a)  \"User program does it all\": the user submits the job that is sent
    all the way to the WN where the execution starts. The query is
    executed from the worker node. Input data is accessed via standard
    mechanisms as described in HEPCAL -- relying on local access is not
    possible since the WMS does not know the list of LDNs to be accessed
    and hence cannot submit the job to a computing resource from which
    local copies are accessible.

b)  \"WMS queries DMC and then submits job": the user submits the job,
    the WMS performs the query to the DMC to optimise the CE selection,
    the job is sent all the way to WN with the experiment dependent
    query and the list of LDNs, job execution starts (see previous case)

The following items (c) and d)) are implemented making use of the DAG
mechanism. Experiment dependent tools that can be invoked by the WMS
(*plugins*) are required.

c)  \"WMS queries DMC, submits multi-jobs and merges output\": the user
    submits the job, and the WMS performs the DMC query as in b). The
    WMS generates several sub-jobs, one for each CE *close* to at least
    one of the input DSs. Each of the sub-jobs (experiment-level query
    plus algorithm) runs the algorithm on its local data. Some
    experiment-dependent merging of the sub-job outputs must take place
    at the end.

d)  "WMS queries DMC, performs multi-queries and merges input": the user
    submits the job, and the WMS performs the DMC query as in b). The
    WMS generates several sub-jobs as in c). Each of these sub-jobs
    selects the events matching the experiment-dependent part of the
    query and places these on its output. This output is merged at a
    final job in the pipeline, where it forms the input of the
    user-specified algorithm.

Below we will describe these cases in greater detail.

### No special support by WMS

The job description is prepared, which includes a specification of the
program to run, any required environment specification, and as input the
job receives the query parameters in some format that the specified
program can understand. The workload management system executes the job
on a site based on the job requirements (e.g. software environment or
available resources). Data access is not considered by the WMS since it
has no idea which data will be accessed, as they will be determined by
the query when the job is already in execution.

1.  The job starts, and if there is a "dataset level" piece of the
    query, the job can execute the "DS Metadata Access" use case of
    HEPCAL to get a list of LDNs

2.  The job accesses each of these DS's in turn. For each DS it applies
    the rest of the query as a filter to pick out only the selected
    event components. How the job does this (as a pipeline, or as an
    "if" statement in an event loop) is not considered -- we simply say,
    "the job does it". The job will make use of the middleware layer
    data access tools as described in HEPCAL (see sec. 5.2.4 for more
    discussion).

3.  The job returns the data by uploading it to the grid, or arranging
    for transfer back to the user.

### Support at the Dataset Level

In this scenario, we suppose that the WMS has the capability to execute
queries to the Dataset Metadata Catalog (as defined in HEPCAL) on behalf
of the user. As an example, in the EDG middleware at the moment we can
tell the WMS

> InputData = \"LF:stanlib.3\"

This tells the WMS that the job will require access to the DS with the
logical name "stanlib.3". We suppose in the current scenario that we can
replace this input data specification with a query. In the example
above, the query would be:

> Return all (or perhaps "at least N") datasets from first quarter 2007,
> taken in trigger configuration EFG, reconstructed with reco version
> 7.4 and calibration set 21.d3.

Of course we expect this query would have to be presented in some format
understood by the DMC instead of in plain English. The WMS would execute
this query and formally replace the query by the list of LDNs that match
it. At this point the job would execute as in HEPCAL:

1.  The WMS will optimize the selection of the computing resource
    according to the program selection, environment requirements,
    computing power availability, *as well as* optimization with respect
    to the list of LDNs generated by the query (as described in HEPCAL).

2.  The job is executed at the remote site, and receives as input the
    list of LDNs and the experiment-dependent part of the query.

3.  The job itself is responsible (as in the previous section) to select
    the objects from the supplied datasets, according to the
    experiment-dependent part of the query, and to apply the algorithm
    to these objects. Furthermore the job will have optimal access to
    the specified set of input data, but in most cases at least some of
    the DSs will need to be access remotely as in 5.2.1

4.  The job returns the data by uploading it to the grid, or arranging
    for transfer back to the user.[]{#_Ref455137708 .anchor}

### Support via job pipelines

A higher level of support can be provided if the WMS implements job
pipelines or DAGs as described in HEPCAL. Given this mechanism, a job
can supply its output as input to one or more subsequent jobs. We
suppose the existence of plug-in tools provided by the experiments (or
possibly end users) that would be used by the grid in these pipelines.
In the following, we call these tools: dump, query, input merge, output
merge, and select. It needs to be clear that these are tools provided by
the experiment.

**select** takes as input an experiment-dependent query and returns list
of object references

**dump** takes as input a list of object references, and returns as an
output stream the bytes of those objects, it is a "deep copy" tool.

**qdump** query+dump

**input merge** query and dump event components into an single input
file

**output merge** merge the output of several copies of the same job run
on different input DSs.

There are several ways in which such a construct could be used to
effectively implement an analysis job. Here we will list some.

#### Distributed Execution with no special analysis facility support

This mechanism is illustrated in the following figure.

![](media/image2.jpeg){width="3.921527777777778in"
height="4.560416666666667in"}

The flow for this is as follows:

1.  The first node in the pipeline is a "dataset metadata select" job.
    The input is the dataset part of the query. The job to be run is
    either a "DS Metadata Access" use case of HEPCAL, or it could be an
    experiment-dependent tool that knows how to access an experiment's
    private DS Metadata Catalogue. In either case, the output of this
    node is a list of matching LDNs.

2.  The next stage in the pipeline is a set of jobs, each one will have
    as input a subset of the LDNs generated by the first node. The input
    to each job is

    a.  The LDNs for the job. The WMS may optimize by executing a single
        job at a computing site with good access to physical copies of
        several matching datasets.

    b.  The experiment-dependent (event component level) part of the
        query.

    c.  The specification of the algorithm to run (executable,
        environment, input parameters and so on). This will be
        "inherited" from the original job submitted by the user.

3.  Each of these jobs can proceed as in the previous case, except that
    now the DS access has been optimized by the WMS.

4.  The output of each of these jobs is collected together at the last
    job of the pipeline, which runs an experiment-dependent or even
    user-specific tool that knows how to merge the results together into
    a single meaningful dataset.

5.  The job returns the data by uploading it to the Grid, or arranging
    for direct transfer back to the user (e.g. sandbox).

#### Distributed Data mining with experiment plugin support

The original HEPCAL team identified many problems with the concept of
merging partial output files. Another DAG construct can avoid this
problem if it turns out that merging of partial input files is simpler.
The disadvantage of this solution is that it can be very inefficient
(with regard to data movement) if the query selects a significant
fraction of all the events, or in other words if the "merged input file"
is nearly as large as the sum of the sizes of all the DSs matching the
DMC part of the query.

The mechanism is illustrated in the following figure.

![](media/image3.jpeg){width="3.8784722222222223in"
height="4.585416666666666in"}

The flow of work in this case is as follows.

1.  The first node in the pipeline is a "dataset metadata select" job.
    The input is the DS part of the query. The job to be run is either a
    "DS Metadata Access" use case of HEPCAL, or it could be a tool that
    knows how to access an experiment's private DS Metadata Catalogue.
    In either case, the output of this node is a list of matching LDNs.

2.  The next stage in the pipeline is a set of jobs, one for each group
    of DS matched by the first node. The input to each job is

    a.  The LDNs for this job. It might be several LDNs since the WMS
        can optimize by executing a single job at a computing element
        with good access to physical copies of several matching
        datasets.

    b.  The experiment-dependent part of the query.

    c.  An experiment-dependent tool called "qdump" loops over all
        events in a DS (or list of DSs) and applies a query to them. For
        matching objects, it dumps a copy of the object as output. The
        input of these jobs has been optimized by the WMS. The output of
        each of these jobs is a stream of bytes corresponding to the
        objects, contained in the input DS(s) for the job, which match
        the experiment-dependent part of the query[^1].

3.  The output of each of these jobs is collected at the following
    (single) node of the pipeline, which is an "input merge" job. It
    takes all the byte streams as input, and generates an output DS that
    has the same format as a normal DS for that experiment. "input
    merge" also must be provided by the experiment.

4.  This DS is passed to the last node of the pipeline, which runs the
    algorithm. It doesn't need the query as input, since the dataset it
    receives contains all the events matching the query, and none other.

This output is either registered via DS Upload or else returned to the
user. Note that we could stop at step 3 and register the output -- this
is essentially a "recluster" pipeline scenario.

#### Middleware Support for Dataset Queries

The previous two pipeline scenarios could be combined with the dataset
query mechanism described in section 5.2.2. This replaces the first node
in the pipeline (the metadata select) by a direct query initiated by the
WMS, but otherwise it is the same.

### Support for Data Access by A Common Layer (or mechanism)

The issues regarding DS access in support of analysis jobs are largely
addressed in HEPCAL, which assumed that the data management system would
transparently optimize data access on the user's behalf. HEPCAL
anticipated that at least the following options would be considered by
the DMS:

1.  access (possibly via remote protocol) to an existing physical copy
    of the DS;

2.  making a new replica to an SE -- because this SE has filesystems
    mountable from the chosen worker node, or perhaps it supports the
    protocol requested by the application -- and arranging for the user
    program to access this new one;

3.  making a local copy to temporary storage at the worker node where
    the job is running;

4.  if a virtual definition of the dataset exists, materializing the DS
    to either a suitable SE or local temporary storage at the node where
    the job will run.

The user will in general not be aware of this; her program will just
"open the DS". Subsequent reads on the returned handle will "get the
bytes".

For analysis jobs, this may not be enough. In many cases the dataset
access may be very sparse, meaning for example that a thousand-event
analysis may attempt to access one thousand separate datasets. The
previous sections discuss how the Workload Management System might
support such access.

The application itself can support such access via intelligent use of
the DMS. An example mechanism is an experiment-specific program (or
servlet) that dispenses events. This servlet would receive the results
of the query, e.g. as a list of (LDN, objref)-tuples as input, and as
output would dispense event data objects in a manner compatible with the
experiment software. In the background, we expect the event dispenser to
contact the Data Management System to discover access costs for the
various DSs, and then start accessing them in an advantageous order,
queuing the events for consumption as the bytes arrive. From the
application's viewpoint, the servlet object provides data on demand, and
the application need not be concerned with access optimization.

## interactive vs batch grid activity

We now propose a distinction between those tasks that are typically
viewed by end-users (physicists) as interactive versus those that are
considered batch. The utility of defining the distinction comes from
establishing a common vocabulary between physicists and middleware
developers, and from associating distinct user requirements on
middleware and services with the different categories.

The single, most obvious characteristic of an interactive job (relative
to a batch job) is the presence of a human interacting with the job
**during** execution. A batch job involves no significant human
interaction between job submission and completion verification. However,
the **amount** of human interaction with the job and feedback to the
human can vary considerably both quantitatively and qualitatively.

In Figure 1 we try to capture the gradual transition between batch an
interactive work.

![](media/image4.wmf)

Figure 1: Characterisation of batch and interactive jobs

The vertical axis we label as \"Influence\" and it denotes the amount of
control over job execution/job behaviour that the user exerts (N.B. The
user will typically [have]{.underline} more influence than she exerts,
but it is the upper-limit of influence which she may wish to exert which
defines the requirements on the system.).

We define 4 general categories of Influence (Regions A, B, C, D). They
are described from more batch-like to more interactive:

**Region A.** This region contains traditional Batch and Standard
Scheduled Jobs. These jobs are usually considered indisputably batch.
They are typically submitted to a batch queue or Grid job scheduler for
execution. The user can effectively ignore the job until it finishes.
These are also called "black box" according to the definition of
PPDG-CS11.

-   User Input:

    -   Only lifecycle commands (eg. start, stop, suspend, resume)

    -   Communicated to the job scheduler, not to the running
        application.

-   User Feedback:

    -   Only high-level job status and job health monitoring, once again
        via the job scheduler, not directly from the running
        application.

```{=html}
<!-- -->
```
-   Examples:

    -   All modern batch systems.

**Region B**. Jobs that are submitted to a batch queue or Grid job
scheduler, but with a communication channel available for minor
influence on job behaviour and some ability to examine intermediate
results before job completion. Fundamental flow of job (ie. algorithmic
code run, input parameters used) is fixed. These are also called
Real-Time Batch in the terminology of PPDG-CS11.

-   User Input:

    -   Uni-directional/asynchronous, pre-defined commands to the
        application that only change the behaviour of the job in
        predictable, and relatively minor ways (e.g. turn on/off
        monitoring, end execution gracefully, etc.).

-   User Feedback:

    -   Partial results (e.g. streamed, or incremental) available to
        user during job execution. (Can for instance be used as input to
        decisions for job termination.)

-   Examples:

    -   Many modern batch systems (allow examination of stdout/stderr).

    -   LSF/BOSS

    -   GMA-Instrumented Athena

    -   Job control via semaphore termination

**Region C.** Jobs that use a batch queue or Grid job scheduler to
schedule, place, and execute part of the job, but which have many
aspects and capabilities of a traditional interactive job, including
direct and strong control of program execution. These are also called
"Interactive Batch" (PPDG-CS11)

-   User Input:

    -   Bi-directional/synchronous control of the application at a level
        similar (identical?) to that of a traditional interactive job.

    -   Usage of the communication channel is optional and it can be
        opened and closed as desired.

    -   Default behaviour of job (e.g. when no new user input is being
        received) is defined. Similar to Region B/A if little or no
        influence is exerted.

    -   Ability to change control flow of job and algorithm, and
        input/control parameters.

    -   Restart of a job step does not necessarily require re-init of
        environment/application (e.g. re-open the DSs). This requirement
        is sometimes expressed as support for same level of
        \<Ctrl-C\>-like handling.

-   User Feedback:

    -   Can be through dedicated, distributed service (eg. distributed
        DB).

    -   If done manually, is typically chunked for efficiency.

-   Examples:

    -   DB I/O

    -   AliEn + ROOT session.

**Region D.** These corresponds to the common definition of fully
interactive jobs. They are also called Dedicated Interactive by
PPDG-CS11.

-   Jobs that meet most the layman idea of interactivity.

-   User Input:

    -   User has full control (at level defined by application) of
        application during execution.

    -   Provides a \<Ctrl-C\>-like interrupt facility.

    -   Use of the control channel is mandatory. Closing the control
        channel is tantamount to ending the application.

    -   Default behaviour of job is to wait for user input and display a
        prompt.

-   User Feedback:

    -   Immediate and finely grained.

    -   May be chunked for efficiency, but at a much smaller granularity
        than Region C.

-   Examples:

    -   LSRUN

    -   PROOF

    -   Distributed JAS

    -   Interactive applications such as PAW, ROOT, JAS, interactive
        event displays, etc.

    -   Most commercial desktop applications.

The horizontal axis of Figure 1 we label as \"Response Time". By
response time we mean the wall clock time elapsed from the issuing of a
command to the completion of the action that this command entails. This
applies to different situations such as job start-up time, request
execution latency, user feedback update period, and job completion time.

The horizontal axis can be divided into general regions (though we do
not do so on the figure), based largely on human time-scales:

-   \< 1 sec: Instantaneous. User\'s attention is continually focused
    upon the job.

-   \< 1 min: Fast. Time periods spent waiting for response or results
    is short enough that user will not start another task in the
    interim.

-   \< 1 hour: Slow. User will likely devote attention to another task
    while waiting for response/results, but will return to task in same
    working day.

-   \> 1 day: Glacial. User will likely release and forget. Will return
    to task after an extended period or only upon notification that task
    has completed.

To clarify different regions of the interactivity phase space, we draw
contours associated with different analysis scenarios. These scenarios
are intentionally vague and the contours are completely qualitative. We
have had extensive discussions on where various contours belong and
stress that the answer depends upon the perspective one is taking and/or
the point one is trying to illustrate.

The scenarios identified in FIGURE 3.1 are:

-   Interactive Event Display

    -   Demands instantaneous response to all commands/actions

-   Histogramming/Plotting/Browsing

    -   Human-time scale response to all commands/actions

    -   Can be orders of magnitude different than instantaneous

    -   Users are used to waiting a reasonable time period for feedback

-   Continuous Tuning/Optimisation

    -   Redoing calculation with different properties (e.g. control
        parameters, algorithm code, etc)

    -   Human-time scale processing time

-   Sporadic Tuning/Optimisation

    -   Redoing calculation w/ different properties (e.g. control
        parameters, algorithm, etc)

    -   Human needs to be prompted when input required or check-in
        occasionally

-   Reconstruction, other Production Jobs

    -   long (\> 24 hours), large scale jobs with no human interaction
        except monitoring

-   Useless

    -   This region is useless in any practical sense as the response
        latency of system incapacitates any human-time-scale
        interactivity

-   Irrelevant/Impractical

    -   This region is irrelevant to any Grid discussion, but is
        identified for completeness.

    -   For instance if completion time is much faster than time for
        interacting or if the sum of initialisation, startup, scheduling
        and so on is much larger than the execution time.

-   Client/Server or bi-modal

    -   Some tasks in which a user engages may span multiple domains.

    -   For instance an interactive client being fed by autonomous,
        batch-like servers.

We have focused primarily on the distinction from the users\'
perspective at this point. There are some aspects of the problem that we
have not addressed in detail but are corollaries of interactivity.

-   Is predictability inversely proportional to influence?

    -   Not in principle, but probably in practice. Batch jobs can
        specify sparse data sets as well as dense data sets, or can use
        navigational aspects of the event data model, or can use complex
        criteria for determining resources used. However, in practice
        most batch-like jobs will have a higher level of predictability
        than more interactive jobs.

    -   How interactive-job "influence" will work with typical
        middleware job optimisation and allocation strategies? Influence
        may change unpredictably the algorithm flow and the DSs
        accessed.

-   Does interactivity impose special requirements on software

    -   Portability/Distributability? No more than for any other kind of
        Grid activity. Portable, distributable control technologies like
        Java byte-code, Python scripts, CINT macros, etc. are already
        used by many HENP systems

-   Does interactivity impose special requirements on communication and
    security protocols?

    -   Probably yes. Both control and feedback imply communication one
        way or the other between a CE node and a user\'s desktop or
        laptop.

    -   We do not here try to address the technical issues, but are
        aware that they exist and are significant.

# system requirements

## Provenance and Job Traceability

Provenance at the event level is a requirement for data analysis.
Consider the following scenario. A rare physics channel is analysed and
results in a set of candidate events. After some years time, a sceptical
physicist revisits the analysis and, in so doing, wants to examine those
candidate events and how they were selected.

A very useful notion in provenance is that of a Directed Acyclic Graph
(DAG) where the nodes are "datasets" and the edges are "jobs" which
transform one data set into another. A common vocabulary was agreed upon
to describe the basic building blocks of provenance DAGs:

*Transformation*: A schema that defines a particular Grid executable or
function. A transformation is similar to a function definition with
formal parameters in a programming language. A transformation defines
the input and output types along with the name of the executable and the
required execution environment

*Derivation*: A particular invocation (or instantiation) of a
transformation. A derivation is similar to a function call with actual
parameters in a programming language. A derivation is called with actual
input parameters, input datasets, and output datasets.

The history of any particular dataset (be it a set of files, a set of
plots, etc), is then reduced to its derivation history which, if one
traces back to the original dataset, forms a DAG. As a result,
Provenance becomes useful at the *derivation* level and it was agreed
that, for *each* derivation, provenance metadata should be recorded for:

-   Transformation information

-   Executable name, and possibly shared libraries and so on

-   Instantiation parameters (input dataset names, output dataset names,
    etc)

-   Actual execution environment

-   Job log

Provenance metadata for the actual execution environment for the job
should be recorded. For each derivation, there are three decoupled
environment categories: the local Grid environment, the site/machine
environment, and the experiment software environment. It is not clear at
this moment how this will be done, but "extreme" solutions such as a
"dump of the RPM database" should be avoided, even if a rather complete
capture of the local Grid environment is probably necessary. Along that
line, the complete environment on the execution site may be described by
the following:

-   The software release version

-   Any patch versions applied

-   The test suite version last used to certify a site

-   and the date of the last LCG certification performed

The grid middleware should provide mechanisms to record these metadata
at job execution time.

Further, it is recommended that the middleware provide mechanisms for
gathering local site/machine information, including:

-   Site (CE) name

-   Machine hostname

-   Machine parameters (including OS, OS patches, Linux kernel version,
    machine memory, machine CPU, machine disk space, etc)

-   Job execution start and end date

Finally, the experiment software environment may consist of two
components: a common environment as well as a specific environment. We
did not yet address a possible common application environment. However,
recording the metadata which describes the experiment specific software
environment is the responsibility of the experiment and includes:

-   The version of the experiment software installed on the execution
    machine

-   The experiment test suite version last used to certify the site

-   The date of the last certification

Further, users must have the flexibility to modify the machine's
experiment software environment to suit individual analysis
applications, including, for example, replacing shared object libraries.
Responsibility for recording such perturbations away from the base
software environment may lie with the individual user however, in such a
case, there need to be mechanisms allowing the user to easily record
such information for provenance.

We are aware of the existence of several unresolved issues related to
recording provenance metadata. One such issue is related to automating
the gathering of provenance information corresponding to sub-job
creation. In addition, there may be complications related to tracking
provenance on machines that support multiple VOs. For example, different
experiments may require different versions of the same application
library to be installed on the same machine.

Finally, there is an intriguing possibility of branching and merging
provenance graphs creating new datasets in the process. Such a
possibility would be somewhat analogous to CVS, which is used as a
collaborative software development environment, and could be a powerful
mechanism for creating a collaborative analysis/dataset development
environment.

## INTERACTIVE ENVIRONMENT support

### Log books and reports

For the following discussion, we define a task as an activity that
produces a set of output data based on an algorithm and a (possibly
empty) set of input data. A typical example is a single (batch) job, but
a task may consist of several such jobs, may involve interactive
analysis sessions, and may include sub-tasks. On the other hand several
tasks can be included in the same job.

In the course of a given physics analysis, many such tasks will be
executed. Some of them will depend on the output of previously executed
tasks. The output of some tasks may partially or completely supersede
the output of previously executed tasks, e.g. when the algorithm used by
the task has been improved or the input data used by a previous task has
been revised. As physics analyses are often performed by groups of
people, tasks may depend on other tasks that have been executed by
different people. A typical physics analysis will involve a large number
of such tasks that one will have to keep track off.

We would therefore like to have an electronic bookkeeping tool (called
"logbook" in the following) with the following features:

-   A record should be kept of every task whose output is not
    immediately discarded as useless. This record should reference the
    input data, algorithms, scripts (or log files from interactive
    sessions), and all output data. The information stored should be
    sufficient to understand what happened during the task and to
    possibly repeat it. It should be possible to attach user comments
    and explanations to this record, including particular plots and
    intermediate results of interest.

-   The logbook should be tightly coupled with the WMS, so that the
    current status of a task can be queried, or a list of all pending
    tasks can be obtained. The recording of jobs submitted should be
    automatic and should not require the user to enter the job
    information manually.

-   The logbook should be able to provide provenance information at the
    task level, like e.g. for which tasks a given dataset was used as
    input.

-   The logbook should make it easy to repeat a set of tasks under
    different conditions. For example, one may want to rerun a set of
    tasks with a slightly changed algorithm.

-   The logbook should have some reporting capabilities, like the number
    of failed jobs, CPU time and other resources consumed, etc.

-   The logbook should be usable by individuals as well as by larger
    groups, with the possibility to import and export subsets of the
    information.

-   The logbook should be usable concurrently by several physicists
    working together on the same analysis, and each one of them should
    be able to add information and query the informatoin on tasks
    entered by everybody else.

### Persistent interactive environment

Here more work is needed. \[fca to do this\]

For each analysis session user should be able to assign a name (in
user's private namespace) to which he/she can subsequently refer in
order to

-   get additional information about analysis status, estimated time to
    completion,...

-   find and retrieve partial results of his/her analysis

-   re-establish complete analysis environment at later stage

Here is an example of pseudo-code that illustrates the analysis session:

// create a new analysis Object ( \<unique ID\>)

**Analysis\* analysis = new Analysis(\"MyAnalysis\");**

// set the program, which executes the Analysis Macro/Script

**analysis-\>Command(\"MyCommand.sh\",\"file:/home/user/test.C\");**

// submit query

**analysis-\>Query(\"200210/V3.08.Rev.04/00110/%galice.root?pt\>150.0\");**

// split the task in at most 10 subjobs

**analysis-\>Split(10);**

// submit subjobs

**analysis-\>Submit();**

// display job information

**analysis-\>GetInfo();**

// download partial/final results and merge them

**analysis-\>GetResults();**

## Deviant Flows and Errors

# SERvice models

## analysis software deployment

\[Introduce software installation on the fly?\]

# Collaborative work

# Recommendations

These are the recommendations of the RTAG

1.  Need to say something about pools of resources that can be linked to
    groups of users (group-level policies, roles, quotas).

2.  From Chapter 3 -- Current experience of the experiments indicates
    that it may be beneficial if the middleware could transparently save
    the results of the query on behalf of the user (see \<interactive
    analysis section\>. We need to think about and discuss what happens
    here, since in some situations this would be fine, but in some
    situations the user will want the up-to-date latest version of the
    query every time it is executed, and the full set of data matching
    the query may be changing rapidly. Example: how does you web browser
    check cached pages (every time, once per session, never). Probably
    need to be able to specify this on a per-query basis, with some
    reasonable user-settable default. Also -- do we want the cache to
    really be transparent (suppose the user didn't know!)?

3.  Garbage collection -- "Since the analysis is iterative, there may be
    lots of datasets left as by-products of the intermediate steps. This
    may places extra requirements dataset-deletion functionality. If an
    "analysis logbook" is implemented, it will be useful to be able to
    mark a group of analysis steps for deletion, which will delete in
    turn all the output datasets associated with these steps." Another
    possible way to handle it is to specify "expiration dates" for this
    type of dataset.

The question of output datasets is probably linked closely to the issue
of namespaces in the DMS. If we do not have namespaces, the requirements
on uniqueness and synchronization for the LDNs in the DMS are much
tougher.

# Conclusions

# References and glossary

## Applicable documents and reference documents

Applicable documents

  ------- ---------------------------------------------------------------
  R1      DataGrid-08-TEN-0201-1-11

          

          

          

          
  ------- ---------------------------------------------------------------

## Terminology

Glossary

  ------------------ ----------------------------------------------------
  Analysis Object    Event information containing the kinematics of the
  Data (AOD)         interaction reconstructed final state and additional
                     information about the event. The exact content of
                     the AOD is experiment dependent. AOD are the data
                     with which most of the physical analysis is done
                     with and are obtained from ESD.

  Deep Copy          

  File fragmentation 

  "history tree"     
  (provenance)       

  ; multiple streams ; multiple streams where an "event" may go to more
  where an "event"   than one stream (as "shallow" or "deep" copy)
  may go to more     
  than one stream    
  (as "shallow" or   
  "deep" copy)       
  ------------------ ----------------------------------------------------

# Description of use cases

## Use Case: Production ANAlysis (PA) {#use-case-production-analysis-pa .form}

+---------------+------------------------------------------------------+
| *             | *UC#**prodanalysis*                                  |
| *Identifier** |                                                      |
+---------------+------------------------------------------------------+
| **Goals in    | *Create AOD/TAG data from input for physics analysis |
| Context**     | groups*                                              |
+---------------+------------------------------------------------------+
| **Actors**    | *Experiment Production Manager*                      |
+---------------+------------------------------------------------------+
| **Triggers**  | *Need input for "individual" analysis*               |
+---------------+------------------------------------------------------+
| **Services    | *job submission* (from GridMiddleware (GMW))         |
| needed**      |                                                      |
|               | *access to exptMetaData to select* (GMW)             |
|               |                                                      |
|               | *access to calibration/conditions data (GMW)*        |
|               |                                                      |
|               | *access to input data based on the selection (GMW)*  |
|               |                                                      |
|               | *selection of s/w to be used for PA (from expt       |
|               | infrastructure) \[or installation on the flight?\]*  |
|               |                                                      |
|               | *storing and registration of output data (GMW)*      |
|               |                                                      |
|               | *storing and registration of updated exptMetaData    |
|               | (GMW)*                                               |
|               |                                                      |
|               | *trigger the creation of replicas of (part or all    |
|               | of) output data and updated exptMetaData (GMW)*      |
+---------------+------------------------------------------------------+
| **Specialised |                                                      |
| Use Cases**   |                                                      |
+---------------+------------------------------------------------------+
| **Pre         | *The experiment management has endorsed the input    |
| -conditions** | data and s/w for the analysis.*                      |
+---------------+------------------------------------------------------+
| **Post        | *"Ad-hoc groups" can analyse the output data from    |
| -conditions** | this step.*                                          |
|               |                                                      |
|               | *The next iteration of Production Analysis can be    |
|               | performed.*                                          |
+---------------+------------------------------------------------------+
| **Basic       | > *User specifies job information including*         |
| Flow**        | >                                                    |
|               | > *Selection criteria;*                              |
|               | >                                                    |
|               | > *Metadata DS (input);*                             |
|               | >                                                    |
|               | > *Information about s/w (library) and configuration |
|               | > versions*                                          |
|               | >                                                    |
|               | > *Output TAG and/or AOD DS (typical);*              |
|               | >                                                    |
|               | > *Program to be run;*                               |
|               | >                                                    |
|               | > *User submits job;*                                |
|               | >                                                    |
|               | > *Program is run;*                                  |
|               | >                                                    |
|               | > *Selection Criteria are used for a query on the    |
|               | > Metadata DS;*                                      |
|               | >                                                    |
|               | > *Event ID satisfying the selection criteria and    |
|               | > LDN of corresponding DSs are retrieved;*           |
|               | >                                                    |
|               | > *Input DSs are accessed;*                          |
|               | >                                                    |
|               | > *Events are read;*                                 |
|               | >                                                    |
|               | > *Algorithm (program) is applied to the events;*    |
|               | >                                                    |
|               | > *Output DS are uploaded;*                          |
|               | >                                                    |
|               | > *exptMetaData is updated;*                         |
|               | >                                                    |
|               | > *Report summarizing the output of the jobs is      |
|               | > prepared for the experiment (mgt) (eg. how many    |
|               | > evts to which stream, \...) extracting the         |
|               | > information from the application and GridMW*       |
+---------------+------------------------------------------------------+
| **Devious     |                                                      |
| Flow(s)**     |                                                      |
+---------------+------------------------------------------------------+
| **Importance  | *High importance. Frequency about a few times per    |
| and           | year.*                                               |
| Frequency**   |                                                      |
+---------------+------------------------------------------------------+
| **Additional  | *Input will probably be of the order 10\^9 events*   |
| R             |                                                      |
| equirements** | *Will have order of tens of output streams, each     |
|               | dealing with 10\^7 (??) events.*                     |
|               |                                                      |
|               | *Latency for data access: low ?*                     |
|               |                                                      |
|               | ***Latency for access to calibration/condition data: |
|               | low ??***                                            |
+---------------+------------------------------------------------------+
| **Example**   |                                                      |
+---------------+------------------------------------------------------+

## Use Case: Managed (Sub-)Group ANAlysis (MGA) {#use-case-managed-sub-group-analysis-mga .form}

+---------------+------------------------------------------------------+
| *             | *UC#**mgrpanalysis*                                  |
| *Identifier** |                                                      |
+---------------+------------------------------------------------------+
| **Goals in    | *Refine AOD/TAG data from a previous analysis step*  |
| Context**     |                                                      |
+---------------+------------------------------------------------------+
| **Actors**    | *Analysis-Group Production Manager*                  |
+---------------+------------------------------------------------------+
| **Triggers**  | *Need input for refined "individual" analysis*       |
+---------------+------------------------------------------------------+
| **Services    | *job submission* (from GridMiddleware (GMW))         |
| needed**      |                                                      |
|               | *access to exptMetaData and groupMetaData to select* |
|               | (GMW)                                                |
|               |                                                      |
|               | *access to calibration/conditions data (GMW)*        |
|               |                                                      |
|               | *access to input DataSets based on the selection     |
|               | (GMW)*                                               |
|               |                                                      |
|               | *selection of s/w to be used for MGA (group specific |
|               | info from expt infrastructure; e.g., from config mgt |
|               | tool)*                                               |
|               |                                                      |
|               | *storing and registration of output DataSet (GMW)*   |
|               |                                                      |
|               | *storing and registration of updated exptMetaData    |
|               | and groupMetaData (GMW)*                             |
|               |                                                      |
|               | *trigger the creation of replicas of (part or all    |
|               | of) output data and updated exptMetaData (GMW)*      |
+---------------+------------------------------------------------------+
| *             | ***Production Analysis***                            |
| *Specializing |                                                      |
| Use Cases**   |                                                      |
+---------------+------------------------------------------------------+
| **Pre         | *The physics group management has endorsed the input |
| -conditions** | data and s/w for the analysis.*                      |
+---------------+------------------------------------------------------+
| **Post        | *"Individuals" can analyse the output data from this |
| -conditions** | step.*                                               |
|               |                                                      |
|               | *The next iteration of Production Analysis can be    |
|               | performed.*                                          |
+---------------+------------------------------------------------------+
| **Basic       | > *User specifies job information including*         |
| Flow**        | >                                                    |
|               | > *Selection criteria;*                              |
|               | >                                                    |
|               | > *Metadata DS (input);*                             |
|               | >                                                    |
|               | > *Information about s/w (library) and configuration |
|               | > versions*                                          |
|               | >                                                    |
|               | > *Output AOD and/or TAG DS (typical);*              |
|               | >                                                    |
|               | > *Program to be run;*                               |
|               | >                                                    |
|               | > *User submits job;*                                |
|               | >                                                    |
|               | > *Program is run;*                                  |
|               | >                                                    |
|               | > *Selection Criteria are used for a query on the    |
|               | > Metadata DS;*                                      |
|               | >                                                    |
|               | > *Event ID satisfying the selection criteria and    |
|               | > LDN of corresponding DSs are retrieved;*           |
|               | >                                                    |
|               | > *Input DSs are accessed;*                          |
|               | >                                                    |
|               | > *Events are read;*                                 |
|               | >                                                    |
|               | > *Algorithm (program) is applied to the events;*    |
|               | >                                                    |
|               | > *Output DS are uploaded;*                          |
|               | >                                                    |
|               | > *exptMetaData is updated;*                         |
|               | >                                                    |
|               | > *Report summarizing the output of the jobs is      |
|               | > prepared for the group (eg. how many evts to which |
|               | > stream, \...) extracting the information from the  |
|               | > application and GridMW*                            |
+---------------+------------------------------------------------------+
| **Devious     |                                                      |
| Flow(s)**     |                                                      |
+---------------+------------------------------------------------------+
| **Importance  | *High importance. Frequency about a few times per    |
| and           | year.*                                               |
| Frequency**   |                                                      |
+---------------+------------------------------------------------------+
| **Additional  | *Input will be of the size of the output streams of  |
| R             | "PA" (and can consist of several of the output       |
| equirements** | streams from "PA"*                                   |
|               |                                                      |
|               | *Will have order of ten of output streams*           |
|               |                                                      |
|               | *Latency for data access: low ?*                     |
|               |                                                      |
|               | *Latency for access to calibration/condition data:   |
|               | low ??*                                              |
+---------------+------------------------------------------------------+
| **Example**   |                                                      |
+---------------+------------------------------------------------------+

## Use Case: End user ANAlysis (EUA) {#use-case-end-user-analysis-eua .form}

+---------------+------------------------------------------------------+
| *             | *UC#**enduseranalysis*                               |
| *Identifier** |                                                      |
+---------------+------------------------------------------------------+
| **Goals in    | *Find "the" physics signal*                          |
| Context**     |                                                      |
+---------------+------------------------------------------------------+
| **Actors**    | *End User*                                           |
+---------------+------------------------------------------------------+
| **Triggers**  | *Publish data andget the Nobel Prize :-)*            |
+---------------+------------------------------------------------------+
| **Services    | *job submission* (from GridMiddleware (GMW))         |
| needed**      |                                                      |
|               | *access to exptMetaData and groupMetaData to select* |
|               | (GMW)                                                |
|               |                                                      |
|               | *access to calibration/conditions data (GMW)*        |
|               |                                                      |
|               | *access to input DataSets based on the selection     |
|               | (GMW)*                                               |
+---------------+------------------------------------------------------+
| *             | ***Production Analysis***                            |
| *Specializing |                                                      |
| Use Cases**   |                                                      |
+---------------+------------------------------------------------------+
| **Pre         | *The user has defined a selection and an algorithm   |
| -conditions** | to extract the data*                                 |
+---------------+------------------------------------------------------+
| **Post        | *Local data analysis can be done*                    |
| -conditions** |                                                      |
+---------------+------------------------------------------------------+
| **Basic       | > *User specifies job information including*         |
| Flow**        | >                                                    |
|               | > *Selection criteria;*                              |
|               | >                                                    |
|               | > *Metadata DS (input);*                             |
|               | >                                                    |
|               | > *Output DataSet (optional);*                       |
|               | >                                                    |
|               | > *Program to be run;*                               |
|               | >                                                    |
|               | > *User submits job;*                                |
|               | >                                                    |
|               | > *Program is run;*                                  |
|               | >                                                    |
|               | > *Selection Criteria are used for a query on the    |
|               | > Metadata DS;*                                      |
|               | >                                                    |
|               | > *Event ID satisfying the selection criteria and    |
|               | > LDN of corresponding DSs are retrieved;*           |
|               | >                                                    |
|               | > *Input DSs are accessed;*                          |
|               | >                                                    |
|               | > *Events are read;*                                 |
|               | >                                                    |
|               | > *Algorithm (program) is applied to the events;*    |
|               | >                                                    |
|               | > *Output DS are uploaded to user\'s local store;*   |
+---------------+------------------------------------------------------+
| **Devious     |                                                      |
| Flow(s)**     |                                                      |
+---------------+------------------------------------------------------+
| **Importance  | *High importance. Frequency a few times per week.*   |
| and           |                                                      |
| Frequency**   |                                                      |
+---------------+------------------------------------------------------+
| **Additional  | *Input will be of the size of the output streams of  |
| R             | "PA" (and can consist of several of the output       |
| equirements** | streams from "PA"*                                   |
|               |                                                      |
|               | *Will produce (several) output files which will be   |
|               | copied to "local storage"*                           |
|               |                                                      |
|               | *Latency for data access: low ???*                   |
|               |                                                      |
|               | *Latency for access to calibration/condition data:   |
|               | low ??*                                              |
+---------------+------------------------------------------------------+
| **Example**   |                                                      |
+---------------+------------------------------------------------------+

##  {#section .form}

[^1]: Depending on how the experiment structures its metadata, it might
    be better to implement this in two stages: "query" and "dump". This
    scheme may be better in case the experiment keeps a metadata table
    separate from the dataset itself.