Future Production Paths Using Generic Job Types in Samgrid

# Introduction

This document is intended for users who want to enable running new DZero
applications using Samgrid as described in Generic WBS for DZero
Applications \[1\]. The document describes two generic job types
introduced in Samgrid that can be used to run new DZero applications.
Users can also run existing DZero applications like Monte Carlo (MC) or
Reconstruction (reco) with special uses cases using these generic job
types. The document also details characteristics that DZero application
should exhibit to classify under either of the two generic job types. It
also details other requirements that need to be satisfied to run a new
DZero application successfully on Samgrid.

# Who should use this document?

This document assumes that the reader --

-   has basic understanding of Computational Grids and Samgrid

-   is a valid member of DZero VO with at least one certificate/identity
    registered with DZero VOMRS/VOMS

-   wants to run new DZero applications through Samgrid or wants to run
    existing applications in a special way

-   has basic knowledge about DZero Data System, SAM, can create
    datasets in SAM

# Generic Job Types in Samgrid

Samgrid supports two generic job types' dzero_direct_io_type and
dzero_indirect_io_type. These job types can be used to run future DZero
applications using Samgrid. Generic job types differ based on how the
Samgrid jobs are split into the local jobs and based on how the local
jobs receive their input files. When the job starts at worker node,
Samgrid is responsible for fetching the job files, input files if any
and bootstrapping the job environment before the control is passed to
D0runjob. Except for merging jobs, D0runjob is responsible for
generating the SAM metadata for the output file and storing the file to
SAM based on the information available to D0runjob in its macro. This
section describes the two generic job types in detail.

## ***Generic job type* dzero_direct_io_type**

Any DZero application that is similar to reprocessing application can be
run via Samgrid using generic job type dzero_direct_io_type.

Job type *dzero_direct_io_type* exhibits following behavior --

1.  At the forwarding/head node, Samgrid job is split into number of
    OSG/Batch (local) jobs such that every local job processes exactly
    one input file.

2.  Samgrid job needs to start a SAM project for accounting purposes to
    fetch one input file from SAM per batch job.

A sample Samgrid JDF for a *dzero_reconstruction* job using
*dzero_direct_io_type* generic job is as shown below -

+-----------------------------------------------------------------------+
| job_type = dzero_direct_io_type                                       |
|                                                                       |
| dzero_application_name = dzero_reconstruction                         |
|                                                                       |
| sam_application_name = d0reco                                         |
|                                                                       |
| sam_application_family = reconstruction                               |
|                                                                       |
| sam_experiment = d0                                                   |
|                                                                       |
| sam_universe = prd                                                    |
|                                                                       |
| group = d0production                                                  |
|                                                                       |
| station_name = samgrid-osg-test                                       |
|                                                                       |
| input_dataset = parag-test-d0reco-dataset-6                           |
|                                                                       |
| check_consistency = true                                              |
|                                                                       |
| test_run = true                                                       |
|                                                                       |
| instances = 1                                                         |
|                                                                       |
| events_per_file = 5                                                   |
|                                                                       |
| d0_release_version = p17.09.07                                        |
|                                                                       |
| jobfiles_dataset = d0repro_jobfiles_p17.09.07_samgridV7-6             |
|                                                                       |
| grid_resource_requirement_string =                                    |
| fngp-osg.fnal.gov:2119/jobmanager-condor                              |
+-----------------------------------------------------------------------+

Figure 1: Samgrid JDF for dzero_reconstruction using
dzero_direct_io_type

New JDF parameters are supported by Samgrid to allow users to fully
express the properties of their application. These parameters are --

1.  ***dzero_application_name:***

> This parameter is the name of your new application. D0runjob should
> also know about your application with same name so that it can use
> appropriate plugins and run your application correctly. As explained
> later, this parameter is also needed in the configuration of product
> *jim_job_manager.* Example: *dzero_reconstruction, dzero_merge*, etc

2.  ***sam_application_name:***

> This parameter is the application name as registered with SAM. For
> example, in *ApplicationFamily(\'certification\', \'recocert\',
> \'p21.10.00\')*, the *sam_application_name* is *recocert*.

3.  ***sam_application_family:***

> This parameter is the application family as registered with SAM. For
> example, in *ApplicationFamily(\'certification\', \'recocert\',
> \'p21.10.00\')*, the *sam_application_family* is certification.

4.  ***runjob_directives:***

> Some DZero applications could be complex and the user may be required
> to provide additional macro parameters to D0runjob. These parameters
> need not have any significance in the context of Samgrid but are
> required by D0runjob to process your application. This additional
> information can be forwarded to D0runjob through the request system,
> but Samgrid also supports passing this information to the D0runjob via
> Samgrid JDF parameter *runjob_directives*.
>
> For example, to run your application, if D0runjob needs two parameters
> in its macro, say, *param1* and *param2* with values *value1* and
> *value2* respectively, you can add following line to the Samgrid JDF.

+-----------------------------------------------------------------------+
| job_type = dzero_direct_io_type                                       |
|                                                                       |
| \...                                                                  |
|                                                                       |
| runjob_directives = {param1 = value1}{param2 = value2}                |
+-----------------------------------------------------------------------+

Figure 2: runjob_directives in Samgrid JDF

> Opening and closing braces ({ })are required to pass multiple
> parameters to the D0runjob. Samgrid will forward these parameters to
> D0runjob by adding following information to the D0runjob macro --

+-----------------------------------------------------------------------+
| \[jdl\]                                                               |
|                                                                       |
| \...                                                                  |
|                                                                       |
| param1 = value1                                                       |
|                                                                       |
| param2 = value2                                                       |
+-----------------------------------------------------------------------+

Figure 3: runjob_directives in D0runjob macro

## ***Generic job type dzero_indirect_io_type***

Any DZero application that is similar to Monte Carlo application can be
run via Samgrid using generic job type *dzero_indirect_io_type*.

Job type dzero_indirect_io_type exhibits following behavior --

1.  If the parameter input_dataset is specified in Samgrid JDF, the
    Samgrid job is split into number of OSG/Batch (local) jobs such that
    every local job processes exactly one input file. User can specify
    number of events to process per file via parameter *events_per_file*
    in the Samgrid JDF. If the parameter input_dataset is not specified
    in Samgrid JDF, the Samgrid job is split into number of jobs based
    on number of events to generate/process per local job. This behavior
    is similar to MC jobs running from first stage.

2.  Samgrid job will not start a SAM project to fetch input file(s) from
    SAM per batch job.

Job type *dzero_indirect_io_type* supports new Samgrid JDL parameters
*dzero_application_name* and *runjob_directives* as described in Section
3.1.

A sample Samgrid JDF for a *dzero_monte_carlo* job using
*dzero_indirect_io_type* generic job is as shown below -

+-----------------------------------------------------------------------+
| job_type = dzero_indirect_io_type                                     |
|                                                                       |
| dzero_application_name = dzero_monte_carlo                            |
|                                                                       |
| station_name = samgrid-osg-test                                       |
|                                                                       |
| sam_experiment = d0                                                   |
|                                                                       |
| sam_universe = prd                                                    |
|                                                                       |
| group = d0production                                                  |
|                                                                       |
| \# Optional runjob_numevts and events_per_file                        |
|                                                                       |
| runjob_numevts = 18                                                   |
|                                                                       |
| events_per_file = 6                                                   |
|                                                                       |
| test_run = true                                                       |
|                                                                       |
| check_consistency = true                                              |
|                                                                       |
| instances = 1                                                         |
|                                                                       |
| \# PYTHIA REQUEST                                                     |
|                                                                       |
| runjob_requestid = 50196                                              |
|                                                                       |
| d0_release_version = p17.09.08                                        |
|                                                                       |
| jobfiles_dataset = d0mc_jobfiles_p17.09.08_samgridV7-7                |
|                                                                       |
| minbias_dataset = parag-Zerobias_p17_06_03MC_set1                     |
|                                                                       |
| grid_resource_requirement_string =                                    |
| fngp-osg.fnal.gov:2119/jobmanager-condor                              |
+-----------------------------------------------------------------------+

Figure 4: Samgrid JDF for dzero_reconstruction using
dzero_indirect_io_type

# Steps involved in running a new DZero Application on Samgrid

This section details the steps required to run DZero applications using
generic job types described in previous sections.

1.  Make your application part of a DZero portable distribution of
    executables.

    a.  Contact DZero members responsible for the executable, to help
        you with this.

    b.  Test your application code.

    c.  Your application should be able to run standalone on any node in
        a given cluster.

    d.  Your application should not depend on availability of UPS/UPD
        environment and SAM services on the machine it is run on.
        However, you may expect that SAM client commands are available
        in the PATH.

2.  Make sure your application can be run through D0runjob. This
    requires development on D0runjob plug-in. Talk to the D0runjob
    developers/maintainers to make this happen.

3.  Decide which of the generic job types described in the previous
    sections best fits your needs. Answer following questions to help
    you with your decision.

    a.  Does the metadata of the output file(s) need SAM process id or
        project id? In other words, does your application need to start
        SAM project to fetch input files? If so the input files will be
        fetched using "sam dh get lite" command and the status of
        consumed files will not be recorded.

    b.  Is your application similar to reconstruction programs? I.e.
        Your application does not depend on generating events when run
        but depends on processing an input file?

> If answer to above questions is

a.  **Yes:** select *dzero_direct_io_type*

b.  **No:** select *dzero_indirect_io_type*

```{=html}
<!-- -->
```
4.  Store the DZero executable and D0runjob in SAM. Official releases of
    D0runjob and DZero executable are usually available in SAM. Work
    with the D0runjob developers and DZero executable maintainers to
    achieve this.

5.  Create job files dataset in SAM containing DZero executable and
    D0runjob.

6.  In case of *dzero_direct_io_type*, request for a new application
    family to be declared in SAM.

7.  Create a SAM request if required by your application.

8.  Talk to the REX group and request them to add configuration
    parameters to *jim_job_manager* related to your application. This
    step is not required if you are using generic job type to run native
    Samgrid job types like *dzero_reconstruction* or *dzero_monte_carlo*
    in a special way. Configuration of *jim_job_manager* on the Samgrid
    head node and the OSG/LCG forwarding nodes should have configuration
    info as shown in Figure 5 and Figure 6. Replace
    *"dzero_application_name"* with the actual name you want to use for
    your application.

+-----------------------------------------------------------------------+
| \<*dzero_application_name*\>                                          |
|                                                                       |
| \<events number_per_output_file=\"xxx\" /\>                           |
|                                                                       |
| \<accelerator bootstrap_time=\"180\" interval=\"180\" /\>             |
|                                                                       |
| \</*dzero_application_name*\>                                         |
+-----------------------------------------------------------------------+

> Figure 5: jim_job_manager configuration tag for dzero_indirect_io_type
> generic job type

+-----------------------------------------------------------------------+
| \<*dzero_application_name*\>                                          |
|                                                                       |
| \<accelerator bootstrap_time=\"180\" interval=\"180\" /\>             |
|                                                                       |
| \</*dzero_application_name*\>                                         |
+-----------------------------------------------------------------------+

> Figure 6: jim_job_manager configuration tag for dzero_direct_io_type
> generic job type
>
> You can also ask REX group to configure data transfer queues and SAM
> storages similar to the native Samgrid applications.

9.  Create a Samgrid JDF for your application. Many JDF parameters are
    common. For more information on the parameters, refer to the Samgrid
    manual.

A sample Samgrid JDF for running *dzero_recocert* job on Samgrid OSG
test node using *dzero_direct_io_type* generic job is as shown below --

+-----------------------------------------------------------------------+
| job_type = dzero_direct_io_type                                       |
|                                                                       |
| dzero_application_name = dzero_recocert                               |
|                                                                       |
| sam_application_name = recocert                                       |
|                                                                       |
| sam_application_family = certification                                |
|                                                                       |
| sam_experiment = d0                                                   |
|                                                                       |
| sam_universe = prd                                                    |
|                                                                       |
| group = d0production                                                  |
|                                                                       |
| station_name = samgrid-osg-test                                       |
|                                                                       |
| input_dataset = parag-recocert-input-dataset-p17.05.01-1              |
|                                                                       |
| check_consistency = true                                              |
|                                                                       |
| **\# test_run = true should be used for testing jobs only**           |
|                                                                       |
| **\# For production run do not specify test_run**                     |
|                                                                       |
| test_run = true                                                       |
|                                                                       |
| instances = 1                                                         |
|                                                                       |
| d0_release_version = p17.05.01                                        |
|                                                                       |
| jobfiles_dataset = d0repro_jobfiles_p17.05.01_samgridV7-8             |
|                                                                       |
| grid_resource_requirement_string =                                    |
| fngp-osg.fnal.gov:2119/jobmanager-condor                              |
+-----------------------------------------------------------------------+

Figure 7: Samgrid JDF for dzero_recocert using generic job type

> A sample Samgrid JDF for running *dzero_monte_carlo* job starting from
> D0gstar stage on Samgrid OSG test node using *dzero_indirect_io_type*
> generic job is as shown in Figure 8. D0runjob knows details about the
> stages it needs to process through the *runjob_requestid* parameter.

+-----------------------------------------------------------------------+
| job_type = dzero_indirect_io_type                                     |
|                                                                       |
| dzero_application_name = dzero_monte_carlo                            |
|                                                                       |
| station_name = samgrid-osg-test                                       |
|                                                                       |
| sam_experiment = d0                                                   |
|                                                                       |
| sam_universe = prd                                                    |
|                                                                       |
| group = d0production                                                  |
|                                                                       |
| **\# test_run = true should be used for testing jobs only**           |
|                                                                       |
| **\# For production run do not specify test_run**                     |
|                                                                       |
| test_run = true                                                       |
|                                                                       |
| check_consistency = true                                              |
|                                                                       |
| instances = 1                                                         |
|                                                                       |
| runjob_requestid = xxxxx                                              |
|                                                                       |
| d0_release_version = p17.09.08                                        |
|                                                                       |
| jobfiles_dataset = d0mc_jobfiles_p17.09.08_samgridV7-7                |
|                                                                       |
| input_dataset = xxxxxx                                                |
|                                                                       |
| grid_resource_requirement_string =                                    |
| fngp-osg.fnal.gov:2119/jobmanager-condor                              |
+-----------------------------------------------------------------------+

Figure 8: Samgrid JDF for dzero_monte_carlo from D0gstar stage using
generic job type

10. If you want to run your Samgrid job on the OSG or LCG cluster, store
    your proxy/credentials in the MyProxy server (*myproxy.fnal.gov*)
    used by Samgrid.

11. Submit Samgrid job using the JDF file you created for your
    application by running following commands from d0mino0x --

+-----------------------------------------------------------------------+
| d0mino01\$ source /fnal/ups/etc/setups.sh                             |
|                                                                       |
| d0mino01\$ setup jim_client                                           |
|                                                                       |
| d0mino01\$ samg submit my.jdf                                         |
+-----------------------------------------------------------------------+

> Figure 9: Samgrid JDF for dzero_monte_carlo from D0gstar stage using
> generic job type

# Future Generic Job Types

This section lists possible Generic Job Types that could be implemented
in the future. Based on the limited input and use/test cases available
from the DZero Experiment at the time of implementation of the above
Generic job types, generic job types mentioned below were not considered
for development in Samgrid --

1.  *dzero_indirect_io_type* splits jobs based on one input file per
    batch job when the *input_datset* is specified in the Samgrid JDF. A
    variation to this generic job type would contain a single file
    *input_dataset*. The Samgrid job will be split into batch jobs based
    on the event ranges to be processed by local jobs. This type of job
    will be similar to phase datasets in Monte Carlo.

2.  One could potentially have generic job types for merging
    application. For historic reasons, merging was done in the context
    of Samgrid and not in the context of D0runjob. If this changes in
    future and D0runjob becomes responsible for merging files, having
    generic job type for merging is a viable option.

# References

> \[1\] Generic WBS for D0 Apps on Grid.
> (https://plone4.fnal.gov/P0/CD-OPMQA/project-management-activities/d0-grid-production-computing-initiative/subject-docs)