**Proposal for a D0 Distributed MC Generation Control System (D0MCGS)**

*Greg Graham*

*University of Maryland*

*Boaz Klima*

*Fermilab*

**INTRODUCTION**

The D0 experiment in Run II will exploit offsite computing resources to
generate large numbers of GEANT based Monte Carlo (MC) events using the
D0GSTAR detector simulation program. So far, we have five MC Production
Centers (MCPCs) up and running. These are at FNAL, Lyon, Amsterdam, UT
Arlington, and Prague. More MCPCs are being formed at Lancaster, Rio,
etc. There is a need for a unified control and monitoring system for
controlling the creation, distribution, and collection of statistics on
remotely running MC generation jobs. For Phase II of the MC Challenge
(MCC_2), we have been successful in generating more than 700 K events
over about 5 months. We have also achieved a peak capacity of about 5000
events per day *without a unified control system*. In order to extend
our capabilities in generating MC events, such a system is necessary.
For D0 Run II, we would like to introduce a Distributed MC Generation
Control System (D0MCGS) that will make better use of resources, improve
communication, require less human intervention, and scale with the
increasing amount of resources.

In MCC_2, MC generation software essentially designed for single users
(the D0 *mc_runjob* package) had been leveraged as a batch job
specification and production tool. Mc_runjob had been distributed to
remote sites and has been mainly successful in this capacity. Other
tasks, such as bookkeeping and job tracking, have been done by hand or
through the development of heterogeneous local solutions. The main
drawbacks of the current MCC_2 system include the following.

1\. The mc_runjob tool had to incorporate several local modifications

to accommodate different modes of running at the remote sites and

local batch configurations. This hampers the maintenance tasks of

mc_runjob since bug fixes and improvements have to be broadcast to

each remote site and incorporated locally.

2\. The task of collecting statistics from running many jobs at remote

sites is not addressed by mc_runjob and is currently done by hand

or by local scripting techniques. Many local improvements have been

made to aid in statistics collection, but there is no uniform

solution and the central statistics are in fact still compiled by

hand through the MCPCs local web pages and the SAM web page.

3\. Resource allocation, in the sense of assigning known resources to

specific MC generation tasks, is currently coordinated

through word of mouth and/or Email. There is no technical means

to aid central control of what gets generated and when. As the

number of remote sites increases, this task becomes more arduous.

4\. One area which has functioned well has been the system for storing
the

D0GSTAR output in SAM. Remote sites use ftp to place their output in

a centrally located disk cache for the import into SAM. Mc_runjob

has been able to create the SAM description files automatically to

allow this to happen. However, the task of bringing the output to

SAM is handled in a fashion that varies from site to site, and

is done by hand much of the time.

This proposal outlines the architecture of a Distributed MC Generation
Control System, D0MCGS, which addresses the shortcomings mentioned
above. In short, we would like the system to be less dependent on direct
human intervention. We would also like it to cater to the needs of top
D0 decision-makers in specifying what MC will be generated and with what
priority. Finally, we would like it to allow users to track job progress
through the system and access generation summaries. D0MCGS is a system
of clients and servers, which handle specification and execution of jobs
and collection of generation statistics in a unified way among the
remote MCPCs and the central site. The central site can be thought of as
a single resource client and each MCPC remote site can be thought of as
a resource server. This document will describe the proposed system in
more detail and will describe necessary upgrades to the successful
mc_runjob tool and new software components that must be written to
implement this system.

**REQUIREMENTS OF D0MCGS**

For D0MCGS in D0 Run II, to be implemented and tested in MCC_3, we
require that:

1\. Job specifications should be generated at the central site.

2\. There should be an easy to use interface for D0 decision-makers to

create job specifications at the central site.

3\. There should be a method to attach priorities to pending generation

jobs.

4\. Remote sites should provide resource information to the central
site,

so that jobs can be awarded on the basis of priority and available

resources.

5\. There should be an easy-to-use interface for D0 users and decision-

makers to check jobs' status. It should be possible to find the current

status of any job at any time from the central site.

6\. It should be possible to compile statistics at the central site on

overall generation and ultimate storage in SAM. The system should be

able to generate a report similar to what is currently displayed on the

MCC Web page (URL is http://www-d0.fnal.gov/\~ggraham/prod_mcc99_2.html)

7\. The system should be scalable from a system of a few PCs to a large
farm

PC system to a large SMP machine with its own batch system.

8\. The system should be packaged so that remote sites can run the
software

without the full D0 or Fermilab environments installed in order to

minimize interaction with remote system administrators.

9\. The software tools should be uniform and should decouple local

requirements as much as possible from the core software system.

10\. The implementation should not depend on the running of a GUI

interface to minimize human intervention

**IMPLEMENTATION**

Much of the control software for D0MCGS is envisioned to be written in
object oriented Python with CORBA (Fnorb 1.0) serving to provide the
backbone of communication between the central site and the remote sites.
It is also envisioned that for large file transfers, ftp, rcp, or scp
will be used. Several agents can be identified in the system. At the
central site, a job_specifier is needed to specify MC jobs, and a
job_mover is necessary to push the jobs onto remote queues. At the
remote sites, a job_executioner agent is needed to execute the jobs and
an output_delivery agent is needed to deliver the output back to the
central site. Each agent is responsible for logging its activity.
Finally, an agent is needed at the remote sites to collect information
from the central site and remote site log files and make it available to
the central site.

**JOB GENERATION**

A physics process, a centrally generated timestamp, random number seeds,
and other short parameters specify each job. The timestamp should
include the four digit year, a day of the year, and a five digit second
field. The random number seeds should be generated at the central site
by some method that is guaranteed to avoid duplication of random number
seeds from job to job. The job specifications are created from user
input at the central site, and this activity is also logged at the
central site.

![](media/image1.wmf)

The format of the job specification will be the mc_runjob marco script
format. The macro script is equivalent to a Python object and can
therefore be made accessible to remote sites using CORBA or can be
written to a file and transported using a file transfer method. It may
be advantageous to create the MC generated data at the central site as
this time. If the generator data file is created centrally, then
arrangements must be made to transfer the generator data file along with
the job specification object to the remote site using ftp, rcp, or scp.
This is advantageous if we decide that more than one remote site should
be allowed to process different sections of a given generator data file,
for example. Also, the generator data would be immediately stored in
SAM.

The implementation of this step is the job_specifier agent. This can be
leveraged from existing mc_runjob code with some minor modifications.
Modifications include the improved timestamp, random number generation,
and a new facility to allow for output of the job specification in
distributable format. The mc_runjob package GUI interface with script
reader can be used by the decision-makers for job specification. The GUI
will need improvements to allow for the creation of multiple jobs. Also,
this activity should be able to be decoupled from the GUI to allow for
command line processing. The job_specifier is not a daemon process and
exists only at the specific invocations of the user. The job_specifier
is shown in figure 1.

**DISTRIBUTING THE JOB TO REMOTE SITES**

Based on user assigned priority and remote site resource availability,
the job specifications are transferred to remote sites. It is envisioned
that the job specifications will either be pushed onto remote servers
using CORBA or transferred to remote staging areas using some file
transfer. The central site and the remote site receiving the job
specification also log this activity. A copy of the job specification
will be kept in a central repository while the job is being processed.
The central site will be responsible for determining to which remote
site to send a given job. The algorithm is not defined yet, but should
take into account priority and resource availability. It should also be
possible to earmark certain jobs to certain sites based on physics
process.

The implementation of this step is the job_mover. This agent can be
written in Python and should be able to communicate with remote servers
via CORBA. If any files (i.e. - generated files) need to be moved to
remote sites, this can be achieved via one of the above mentioned file
transfer methods. The job_mover does not need to be a daemon process,
but can be activated automatically at the end of the job specification
stage. This software does not exist and must be written from scratch.
The job_mover is shown in figures 1 and 2.

**RUNNING THE JOB**

The remote site will receive job specifications from the central site
via CORBA or file transfer. The remote site must therefore have a daemon
process waiting for incoming jobs from the central site. It will spawn a
new job processor for each incoming specification. In case of a lack of
resources, it will queue incoming jobs in a waiting area. This activity
will be logged at the remote site.

The implementation of this step is the job_executioner. The
job_executioner must be a daemon process. It will be written in Python
with CORBA communication

![](media/image2.wmf)

to the central site. The job executioner is responsible for the three
server

functions outlined above: running jobs requested by the central client,
returning resource usage and availability to the central site, and
tracking jobs in progress. While the first function above may be
achieved through file transfer, the last two need CORBA to efficiently
relay information back to the central site. The software for the
job_executioner does not exist and needs to be written from scratch.

The remote site will also be responsible for tracking jobs in progress.
This can simply be a periodic check on process IDs and exit status to be
kept in a circular log file (extending back for a few days only, for
example) and the results can be transmitted back to the central site for
instant job tracking. Finally, the remote site should provide local
resource information back to the central site on demand. The resource
information should convey at least the site capacity in peak number of
events per day and the current site usage.

The mc_runjob tool can be leveraged to run the actual job. It will
unpack the job specification, run the job, and package the output in a
way that is immediately importable by SAM via the SAM cache area. This
will involve some modification, and it also requires decoupling the core
functionality from the GUI so that it will run in batch mode. However,
as the back end of the job_executioner, it is here that local
modifications will become important in order to adapt to local
environments. Areas of mc_runjob that are known to require local
modifications will therefore be modularized and split off of the core
program.

On exit of the jobs, the executioner will be provided with a status
flag.

SUCCESS: The output is packaged in a form that can be immediately

deposited in the sam cache area and is transferred there.

This activity will be logged.

FAILURE: In case of failure, the job can be restarted a fixed number of

times before final failure is reported to the central site.

**RETURNING THE OUTPUT**

![](media/image3.wmf)On successful exit status, the output job should
report to the job executioner and package itself up for shipment back to
the central site. The output package should include the log files, the
output files, and the SAM description file. Another job_mover can be
configured to send the output package to the central site via one of the
above file transfer methods. This information is also logged. Local
copies of the output packages at the remote sites should be

deleted upon successful transfer. Thus the some mechanism for
determining success-of-transfer needs to be devised. The job output is
ultimately stored in SAM. Once job output is received at the central
site, the output package must be unpacked into the SAM cache import
area. This is illustrated in figure 3.

**MONITORING**

Log files at the remote sites and the central site should be queryable
from the central site. If the log entries are tagged by timestamp, we
can trace each job through the entire pipeline by collecting tagged
information from all of the log files. Extensive use will also be made
of the SAM database to provide the information about whether finished
jobs have actually been stored.

A monitor client process needs to be written for the central site, which
will be responsible for collecting statistics from the log files at the
central site and from the log files at the remote sites through the job
executioner. This monitor should be able to generate convenient reports
to the decision-makers on demand and provide job-tracking information
for the users. This software needs to be written from scratch. A WWW
interface will be designed to provide easy access the job tracking
information and summary results. This is shown in figure 4.

![](media/image4.wmf)

**CONCLUSIONS**

The system outlined above will provide the following functionality to
the decision-makers and the D0 community:

1\. Job specifications will be generated at the central site and

run at the remote sites. There will be an easy to use interface for

the decision-makers to create job specifications at the central site.

2\. Remote sites will provide resource information to the central site,

and jobs will be awarded on the basis of priority and available

resources.

3\. It will be possible to find the current status of any job at any

time from the central site. Furthermore, it will be possible to

compile statistics at the central site on job generation and

resource usage. These statistics will also track jobs through SAM.

4\. The system will be packaged so that remote sites can run the
software

without the full D0 environment installed and so that software tools

are uniform.

5\. The system will keep a tally of processed MC

events. It will be able to generate a report similar to what

is currently displayed on the MCC Web page.