# Grid to Fabric Job Submission Interface in SAMGrid

A Theory with Applications

## What this is all about

First, a clarification of terminology is in order. "Fabric" is a fancy
word used to describe the collection of physical computing (as well as
storage etc) facilities that comprises "Grid". "Grid", well, is defined
elsewhere. "Grid to fabric interface" means a small collection of
services that are implemented on the wide boundary between the Grid and
Fabric levels. These services, inasmuch as job submission is concerned,
allow for a Grid job, scheduled at the site at hand by the Grid
scheduler, to be instantiated at the *execution site* as a collection of
*local* jobs (known to the local batch system), be submitted and
monitored, and, subsequently, for the *small output* (i.e. output files
such as logs that are not destined for an industrial-strength data
handling system such as SAM) to be propagated back to the Grid *client
site*. In general, SAMGrid provides several types of services on the
Grid-Fabric boundary; the present document focuses solely on the job
submission.

This document described what appeared to be the most challenging step in
the setup of a SAMGrid execution site. First, we present some ideas
behind what we have done. Then, we detail the actual installation and
configuration of the relevant pieces of the grid to fabric job
submission; other pieces are straightforward and configured by virtue of
the usual "upd install/ups tailor" commands for the packages comprising
the SAMGrid suite.

## What is the big deal? I could submit jobs before JIM!

Instantiation of a real Run II physics job (especially a multi-stage one
such as Monte-Carlo), differs dramatically from that of a "simple" job
consisting of pretty much a single binary performing a few KB
input/output, however computationally intensive (the kind that we
facetiously call "password crackers"). Hundreds, often thousands of
small files must be supplied with the job in a manner that is efficient
enough so as not to break local file transfer mechanisms. The fragile
jobs shouldn't interfere even when several of them are scheduled on a
single node. The number of local jobs running in parallel must be
determined so as to maximize the probability of job completion (within
the batch system imposed boundaries) yet not to have too many small jobs
producing too many small output files.

If you are a member of HEP community, then you are of course already
aware of the above (and other) application-imposed complexities. Few
people, however, realize how the Grid (even globally distributed)
computing is complicated by many "peculiarities" of individual sites. In
general, even if the site has been running real physics jobs locally
*for years*, it may not be straightforward to add this site to SAMGrid
(or any other serious Grid) for two principal reasons:

1)  There seem to exist an uncountable multitude of local configurations
    in terms of directory structures, shared file systems, conventions
    for naming standard output/diagnostic files, designated mechanisms
    of *intra-cluster* small file transfers, etc, which is not managed
    by any "standard" Grid (or not) software. For example, your local
    users may be accustomed to having the standard output/diagnostic
    deposited in their "home area" upon job completion. This doesn't
    quite work when the "grid" (i.e. foreign) user doesn't have
    NFS-shared home. You can create a local account for such "grid
    users" and you can make it even NFS-shared, but do you really want
    them to leave behind junk files? Compare yourself to a ranger in a
    state forest -- you want your campers to take their junk with them,
    and you want to provide, if necessary, trash bags and other tools to
    them.

2)  Job submission by a *machine* such as Grid middleware imposes
    considerably tougher requirements on the site's software services
    than job submission by a human scientist who has in her pocket the
    phone numbers of her friendly administrators. For example, all the
    batch systems we have used thus far managed to forget, at one point
    or another, what is actually running in them; whereas a human would
    simply shrug it off and re-issue the query command minutes later,
    the Grid machinery would have failed the job for no good reason. It
    is impossible for the local tools to guarantee 100% accuracy, but it
    becomes necessary to have wrappers that mitigate momentary lapses of
    memory and other transient glitches. As another example, if as much
    as a *single node* in the cluster is mis-configured to the point of
    being unusable, the batch system typically enjoys sending *all* the
    local jobs to the bad machine because it fails (completes) jobs too
    quickly and achieves "outstanding" turnaround; a human would easily
    modify his submission script to avoid this *black hole* effect but
    this is not straightforward to automate.

We urge you, our friendly site administrator, to understand these issues
and help us overcome them.

## Architectural perspective

Yes, there is even mini-architecture of the Grid-Fabric Interface job
submission service suite:

We now describe the roles of each block on the diagram.

### Batch system idealizers

These, as the name implies, "idealize" the batch systems to make their
interactions with the Grid machinery easier, by mitigating any
imperfections and adding any "missing" features. Mitigation includes:

-   retries in lookup commands for certain batch systems,

-   generation of easy to parse output (batch system commands return
    output that's usually too terse or too verbose),

-   dealing with confusing exit status from batch system commands.

Added features include:

-   grouping of jobs for all batch systems by an attribute such as
    generalized "project" (i.e. in SAMGrid rather than SAM sense),

-   local scratch management on the worker nodes, i.e. setup and cleanup
    of the scratch space before/after user job execution,

-   (optional) explicit preference/avoidance of nodes that are /are not
    well suited for the grid job(s) in question.

Scratch management deserves a special remark. Whereas in theory one can
argue whether this service belongs in the batch system or not, (PBS, for
example, knows nothing about it), we find it extremely useful and
*strongly recommend* that the batch jobs run in their separate scratch
directories, for reasons of performance and mutual isolation. All the
computing facilities we have seen have worker nodes with a few GB
scratch space which is seldom, if ever, used. For PBS, we provide a
"special" scratch setup script that wraps user job and creates/deletes a
subdirectory before/after the job in the location that must be
configured (see below).

There is one idealizer provided as a template for each batch system with
which we are familiar. Users are expected to look inside these and
modify them to accommodate the local configurations by e.g. setting the
full paths to correct values. These are located within the
sam_batch_adapter package.

### Batch system adapters

These adapters, located in the sam_batch_adapter package, were
originally intended for use by the "sam submit" command, which in turn
was intended to provide the correct interface to submit SAM analysis job
to a batch system, and performing actions such as starting/stopping of a
SAM project, see
<http://d0db.fnal.gov/sam_batch_adapter/sam_batch_adapter.html> . In the
expanded job management scheme, provided by the JIM and other SAMGrid
tools, this package serves as the configuration tool for the job
submission/lookup/kill commands, implemented in the aforementioned
idealizers, i.e. the adapters for jobs coming from the Grid must be
configured to use the idealizer appropriate for your local batch system.

The difference between "adapters" and "idealizers" is that the former
provide uniform interface to the batch system, whereas the latter
provide the scripts that actually correctly implement these interfaces.
For example, the adapter concept contains an interface to lookup a job
in the batch system, and an idealizer will actually perform the lookup,
handle some of the errors, and return a string that has a chance of
being parsed. In a broader sense, "adapters" include "idealizers".

### JIM Sandboxing

This service is provided within the jim_sandbox software package and
will soon be documented therein, so we merely provide a brief summary.
"Sandboxing" in SAMGrid refers to the ability to transfer and initialize
all the relevant input files for the user job, as well as correct
collection and return of "small output", thus avoiding the poorly
conceived and controversial "home area" concept for a Grid user on a
cluster which he/she does not own. It allows for *reproducibility* of
job results by providing independence from any pre-installed experiment
software. For input sandboxing, a staged *bootstrapping* process is used
whereby each subsequent stage uses results of the previous stage, the
last and most advanced stage being retrieval of a small input dataset
through the SAM data handling system.

### SAMGrid job managers

These implement the services of grid job instantiation at the execution
site, by means of mapping a logical grid job definition (with details
provided by e.g. SAM Monte-Carlo request system) to set of local jobs
submitted to the batch system. Our job managers come with the
jim_job_managers software package and are installed into the Globus job
manager area. When activated, they receive the job request via the
standard GRAM protocol and perform multiple creation, submission, lookup
and kill of the local jobs comprising the Grid job. In addition, they
allow for XMLDB-based monitoring of Grid jobs which is at the heart of
JIM Grid job monitoring.

### Putting it together

The SAMGrid job managers, when receiving the Grid job request for the
first time, calculate the number of the local jobs to be submitted and
then prepare these local jobs using the sandboxing mechanism. File name
stems for both standard output and diagnostics are chosen; local jobs
will use different files as unique suffixes are appended. The result of
the local job preparation is a self-extracting executable, i.e. an
executable containing other files needed for the bootstrapping process.
The job managers then consult the (properly configured) batch adapters
to learn which command to use for job submission, and execute the
command the appropriate number of times.

When the job is executing at the worker node, the sandbox bootstrapping
rolls out and eventually passes control to the job wrapper, provided by
the job managers from the head-node (i.e. local submission machine) for
each known job type (D0 and Monte-Carlo, file merging, general SAM
analysis, etc). These wrappers perform functions such as initialization
of the experiment's software "release tree" in the local scratch space
etc, and finally pass control to the job script (which is either
supplied by the Grid user as in the case with CDF MC or is standard for
the given job type, such as the mc_runjob-based script for D0 MC).

## OK, OK, let's get down to it

Once again, as the SAMGrid execution site installer, the very first step
you need to do is to configure local job submission. **Do not try to
proceed with JIM installation until after you have understood and
resolved all the issues in this document and are able to submit jobs as
described below.**

### Configure a batch system idealizer

Install (don't configure it yet) the sam_batch_adapter package from
KITS.

\~/\> setup sam_batch_adapter

\~/\> ls --l \$SAM_BATCH_ADAPTER_HANDLER_DIR

Choose the one that looks like sam_XXX_handler.sh where XXX is similar
to your batch system. **Copy** this file to a "JIM area", rename if you
like. Edit this file to ensure that all the paths are set correctly. If
you are using PBS, make sure that the full path to the
pbs_scratch_setup.sh script and the scratch disk path (on the **worker**
node) are correct.

Now go back to the "JIM area" or sub-area thereof from where the local
jobs will be submitted (this path will later be given to the
configurator script for the jim_sandbox package) and **test** the
idealizer. Be sure to do so as the same user as the one to which the
Grid jobs are mapped! We can't possibly list all the reasons why even
though it works for user "pete" or "josh" or "ann" it may not work for
user "samgrid" or whichever account you designate for Grid jobs! Execute
a command like this:

/big_disk/samgrid/jim/\> cp /bin/ls ./binary

/big_disk/samgrid/jim/\> ./sam_pbs_handler.sh job_submit \--project=test
--executable=\$PWD/binary --stdout=\$PWD/out --stderr=\$PWD/err
--arguments="-laF"

This should result, well, in a job submission. A second or so later,
try:

/big_disk/samgrid/jim/\> ./sam_pbs_handler.sh job_lookup \--project=test

This command **must** return information about the job you have just
submitted.

A few minutes later, when the job is finished, there **must** be a file
with the name "out" in this directory!!! If there was a problem, check
to see if the "err" file contains any message. If not, check if the
batch system has sent you an email (obviously, this is not good enough
for the Grid but is a means to troubleshoot local job submission). Most
often occurring problems are PBS-specific:

-   The batch system could not transfer the executable "./binary" to the
    execution node. It probably didn't' transfer "out" and "err",
    either. Debug the pbs_rcp command or, if you must use NFS, make sure
    that /big_disk/samgrid/jim is properly NFS-exported.

-   The scratch setup is mis-configured -- you should see something like
    "/scratch: directory not found" in the "err" file. Go ahead and
    create a directory with the same name on all the nodes in the
    cluster and put the full path to this directory in the
    pbs_scratch_setup.sh script.

If you see the file "out" and it contains (aside from any debug
messages) the listing of an almost blank directory on the worker node,
**congratulations!** You are almost done!

### Configure the batch adapter

Now tell the batch adapter that the way to handle your jobs is through
the above newly hacked script. Use the "sambatch" command, available
after setting up the sam_batch_adapter product. This command is
extremely intuitive and straightforward, with solid run-time
documentation (additional documentation is available at
<http://d0db.fnal.gov/sam_batch_adapter/sam_batch_adapter.html>). Be
sure to specify submit, lookup and kill commands (all implemented as the
corresponding actions in the handler script) to the batch adapter. Your
configuration should look similar to the following:

<sam@samgfarm2>:\~\>   sambatch display \--station=samgfarm\
Station: samgfarm\
  Default Adapter: grid\
  Available Adapters: \[\'grid\'\]\
    Adapter: grid\
      Available Commands: \[\'job kill command\', \'job lookup
command\', \'job submit command\'\]\
        Command: /bin/sh -c \'. /local/ups/etc/setups.sh;setup
fbsng;\${SAM_BATCH_ADAPTER_HANDLER_DIR}/sam_fbsng_handler.py job_kill
\--project=%\_\_USER_PROJECT\_\_ \--localJobId=%\_\_BATCH_JOB_ID\_\_\'\
          Type: job kill command\
          Known Outcomes:\
            Exit Status: 0\
            Outcome Description: Success\
            Exit Status: 1\
            Outcome Description: Failure\
        Command: /bin/sh -c \'. /local/ups/etc/setups.sh;setup
fbsng;\${SAM_BATCH_ADAPTER_HANDLER_DIR}/sam_fbsng_handler.py job_lookup
\--project=%\_\_USER_PROJECT\_\_ \--localJobId=%\_\_BATCH_JOB_ID\_\_\'\
          Type: job lookup command\
          Known Outcomes:\
            Exit Status: 0\
            Outcome Description: Success\
            Exit Status: 0\
            Expected Output: JobId=%\_\_BATCH_JOB_ID\_\_
Status=%\_\_BATCH_JOB_STATUS\_\_\
            Exit Status: 1\
            Outcome Description: Failure\
        Command: /bin/sh -c \'. /local/ups/etc/setups.sh;setup
fbsng;\${SAM_BATCH_ADAPTER_HANDLER_DIR}/sam_fbsng_handler.py job_submit
\--project=%\_\_USER_PROJECT\_\_ \--executable=%\_\_USER_SCRIPT\_\_
\--arguments=%\_\_USER_SCRIPT_ARGS\_\_
\--stdout=%\_\_USER_JOB_OUTPUT\_\_ \--stderr=%\_\_USER_JOB_ERROR\_\_\'\
          Type: job submit command\
          Known Outcomes:\
            Exit Status: 0\
            Outcome Description: Success\
            Exit Status: 0\
            Expected Output: %\_\_BATCH_JOB_ID\_\_\
            Exit Status: 1\
            Outcome Description: Failure

It is now necessary to execute the test script, found in the package:
\$SAM_BATCH_ADAPTER_DIR/etc/testBatchAdapters.py. This tests both the
current and the previous step of the installation. If your jobs execute
too fast or too slowly for this test script, feel free to hack it as
needed. If the script tells you that the job has executed and the
standard output was correctly received, **congratulations again**! You
can now return to the [main SAMGrid
manual](http://www-d0.fnal.gov/computing/grid/SAMGridManual.htm) to
complete your installation.

If this is not successful, try to debug the problems with your site
administrators. SAMGrid developers may be knowledgeable in many areas
but they are really not the best people to answer your questions related
to rcp/NFS, local account setup, batch system bugs, permissions on your
local directories etc ;-).

## Suggestions

Please send your suggestions (or) comments about this document to [Igor
Terekhov](mailto:terekhov@fnal.gov) and [Gabriele
Garzoglio](mailto:garzoglio@fnal.gov).

*Last updated on Monday, November 13, 2023.*