## Problems Encountered

IGT ran CMS production by integrating CMS and Grid tools. Several
problems were identified and fixed during this run. These include
integration issues arising from non-grid CMS tools integrated with Grid
tools, bottlenecks arising form operating system limitations, bugs in
middleware and application software. Almost each component of this
production contributed to overall problem count.

### VDT Problems

The major components of IGT middleware comes from VDT (Virtual Data
Toolkit, http://www.lsc-group.phys.uwm.edu/vdt/) a project of
international Virtual Data Grid Laboratory (iVDGL,
[http://www.ivdgl.org](http://www.ivdgl.org/) ) and GriPhyN (
http://[www.griphn.org](http://www.griphn.org/)) project. The MOP
master/worker site requires installation of VDT Server and Client.

After installation of VDT 1.1.3 several problems were faced,

1.  Changed condor directory structure (as compare to VDT 1.1.1).

2.  Auto Condor configuration option not present anymore.

3.  Buggy FTSH binary.

4.  Incorrect permissions on libexec directory.

5.  Incorrect condor_jobmanager script.

Problems were conveyed to VDT developers and fixed. Installation was
verified to be working.

### RefDB

RefDB is the source of production assignment description; apparently
there is only one version of RefDB used for development and production.
It happened a couple of times that RefDB owner made some changes, or
someone else trying to update RefDB, corrupted the assignment
descriptions. We were unable to create new jobs from RefDB. This was
quickly reported and fixed. It is worth mentioning that we were able to
create new jobs even without RefDB, using the saved tracking
information.

### MCRunJob

1.  Bug in MCRunJob "job creation", after creating 1451 jobs (out of
    4000 in total), MCRunJob lost track and created rest of the jobs
    under the tracking directory holding job number 1451. MCRunJob
    tracking looks at the directory listing in the tracking area, before
    creating jobs, and large number of files in this directory, caused a
    system command to fail.

2.  Re-creation of jobs caused over-writing of existing tracking area
    and job-number count started from zero again. This caused
    over-writing of previously generated ntuple files at the end of job,
    which was also fixed, tweaking with MCRunJob tracking.

3.  Missing logs directory in the MCRunJob ntuple tracking area, created
    manually.

4.  In case when MOP master and worker sites were physically on same
    machine, jobs used files from Master area tracking instead of local
    to running job......this created problem of..???.

5.  There are some unwanted tracking option always present, which are
    valid only for non-grid based production, no error though!.

### MCRunJob and MOP Interfacing

Code level changes were required to interface MCRunJob with Grid tools
(namely MOP), especially for the reason that MCRunJob was initially
designed for Batch Farm job execution, and not on Grid.

### MCRunJob RUNTIME_AREA issues

MCRunJob runs job in a directory created for this purpose at job
submission time, called RUNTIME_AREA. In case of non-grid job execution
this area is generally created by Farm Batch system. MCRunJob then
create all data, log, temporary files and Objectivity federations in
this area. Later it is cleaned up by Farm Batch system to recover disk
space.

In grid environment using MOP, this area is created on (generally on NFS
mounted) shared disks, called stage-in area.

#### Objectivity Federations:

Objectivity doesn't allow using NFS/Shared disk areas to be used for
creating Federations/databases. So instead '/tmp' of farm worker node
was used to create Objectivity federations, by making changes to
MCRunJob. Quickly it was realized that on multiprocessor systems, more
than one processes are trying to create/access Objectivity federations
of same name in /tmp area. Lock contentions were avoided by creating
Objectivity federations in randomly generated directories in /tmp. This
does seem to solve the problem. But as the name of directories for
Federations were generated randomly, and then used by several processes
in the chain of production, there was no easy way to pass this name to
various processes at run time, and to the clean up processes in the end.
This additional task was achieved by making several code changes to
MCRunJob, saving name of generated directory names in some configuration
files, used by all processes, at run time.

#### FZ, Log and Temporary files:

It was observed that when a large number of processes (close to 80 or
more), run on a site, All jobs are staged-in on NFS mounted share area,
and all processes use this area to write log, temp, and data files.
While not fully understanding the impact, initially no changes were made
to MCRunJob in this regard. And later several crashes were faced. The
phenomenon was having misleading symptoms like batch system jobs were
failing because of NFS time outs and batch system was keep on submitting
jobs again and again as fresh jobs, increasing number of jobs to
thousands. The system load was rising and ultimately causing crash.
Initially this was observed at FermiLab. FermiLab was using newly
developed fbsng-globus jobmanager. This was quickly blamed for two three
crashes. Later investigations proved that using NFS mounted disk area
for I/O by a large number of processes caused the problem, and
fbsng-globus jobmanager was innocent. The situation was even worst as
MOP Master and MOP worker sites were the same physical machine using the
same NFS mounted disk. As it sounds, there were an over-whelming number
of Read/Writes operations going on same disk of same machine, causing
effects and after effects and bringing the system to its knees. After
FermiLab, some other sites also faced the same problem. This was a
lesson learned the hard way. And it couldn't be predicted without
running production at an over-whelming scale, like IGT.

MOP Worker and Master sites were physically separated at FermiLab. This
reduced the number of I/O to a certain extent. Several changes were made
to MCRunJob, the huge data file (FZ file) was then generated in some
area of the farm machine, instead of MCRunJob RUNTIME_AREA. This area
was called MOP_FZ_TEMP_AREA and defined as a MOP variable so that each
site could chose an appropriate directory for this purpose, available on
its farm nodes.

Unlike Objectivity federations, the FZ files have unique names, so
generation of more than on FZ file in same MOP_FZ_TEMP_AREA wasn't a
problem. But these FZ files were than need to be copied and or deleted
from this area to recover disk space after process finishes. Again there
was no specific way in MCRunJob to name a FZ file used by current chain
of production. The process generating the FZ file knows the name of this
file through some configuration files, while other processes simply did
"ls --1 \*.fz" to get the name of file from RUNTIME_AREA assuming that
there is only on FZ file generated in this area. The situation became
different when there were multiple FZ files MOP_FZ_TEMP_AREA. There was
no specific way to identify FZ file generated by earlier processes in
the chain, by later processes except by extracting it from the same
configuration file, using 'grep' and 'awk'.

MCRunJob does tie up several different step of chain of production, but
we will need a better way of handling their mutual integration and
integration with Grid based tools. Instead of running a chain of
production by calling each step in same script; a better architecture
will be required, as **proposed in section BLAH**. Each step of
production chain becomes an atomic step explicitly knowing its input and
output parameters. Making it much easier to do staging in and staging
out and parameter passing much easier.

#### More than one assignment:

MCRunJob, job submission using MOP depends on some hard-coded parameters
in its configuration files. This includes the tracking area where jobs
are created and then submitted. Running multiple assignments from same
Data Set ID mean maintaining two separate installation of MCRunJob.

### MOP

#### Site Files:

Problems with MOP are caused by required manual changes to its site
files. Each worker site is identified by a set of files, called site
files. Missing site file or error/typo in site file causes MOP jobs to
fail. The errors are misleading and only after closer inspection of log
files problems with site file(s) are identified.

#### Auto Restart/Resubmission:

Keeping in view the need of automatically restarting held jobs and
resubmission of failed jobs without manual interactions; MOP is using
auto restart features of Condor_G. A MOP job, failed to continue for any
reason is set in held state. Held jobs are then attempted to be
restarted by Condor_G for several times (05 times) and the submitted as
fresh jobs.

There have been several cases when actual jobs were still running on
remote site, while they lose contact with master site, becoming an
Orphan. Considering them lost jobs, MOP auto-restart mechanism resubmits
these jobs as Fresh jobs. The jobs are submitted in same runtime area,
already being used by Orphan job. So instead of a single job, two (newly
submitted and orphan) jobs are running in the same runtime area. In case
of higher number of orphan jobs, this affect was more obvious. Though
resubmitted jobs eventually finish successfully, but cause loss of CPU
hours.

#### Infinite Auto Restart/Resubmission:

There are some instances when jobs failed due to application code
problems, or some real reason like full disk partitions. With
auto-restart option, it becomes difficult to find out actual cause of
trouble, while failed jobs are restarted and resubmitted in an infinite
loop. The actual problem remain hidden unless jobs are peeked closely.

Restart/Resubmission is very helpful to minimize manual interaction with
the system and is a good Fault Tolerance, when many of the jobs could be
recovered automatically. But resubmission of jobs to same runtime area
is probably not a good idea. Future MOP developments are supposed to
address this issue.

### Condor_G

Condor_G running on the MOP Master site uses 'gahp_server' to handle its
communication with running processes on remote worker sites. The
gahp_server has trouble managing this communications at the scale we are
using it. Processes at MOP master site lose their contact with some of
running processes at worker sites. The lost process become Orphan
processes and in turn cause other troubles described earlier. To fault
tolerate, Condor_G (condor_gridmanager) process kills and restart all
gahp_server processes, whenever it find itself unable to communicate
with remote site(s). This problem becomes more severe when a large
number of jobs are submitted from MOP master site. The
condor_gridmanager restarts becomes too often to do any productive task,
eventually hampering production.

With over 400 CPUs available to IGT at later stages of production,
running two assignments to produce 1.5 Million events, we had to divide
production over two physically separate MOP master machines, to avoid
gahp_server problem. The Fault tolerance process of restarting
condor_gridmanager (Condor_G) didn't worked.

### Condor jobmanager

Most of the sites uses condor as job manager, its configuration seems to
be the hardest part of VDT installation. Many problems faced initially
by IGT were due to improper configuration of condor jobmanager.

A prior version of VDT had an interactive condor configuration option,
which was helpful to setup condor jobmanager, but was removed from the
latest versions. This make condor configuration even more difficult.

YP server crashed when a large number of jobs were submitted to a worker
site. This was avoided by maintaining the job queue at a lower number.

### Condor_q 

Condor_q is queue-monitoring tool widely used during production. It can
be used both at MOP master and worker sites. Using some command line
switches, useful information about status of running jobs can be viewed.
One can use '-dag' to display all DagMan processes and associated child
processes. With intentions to delete some DagMan processes having n o
children processes, several wrong DagMan processes were killed.
Eventually killing some useful running jobs. A bug in Condor_q was
displaying wrong information about various DagMan and respective child
processes. Bug was fixed later on.

### Application software issues

#### CMSIM and WriteAllDigi './/' issue: 

As mentioned in section XXXX, MOP_FZ_TEMP_AREA was introduced to write
FZ file into, to avoid NFS problems. At some of the sites, RAID was used
and also a smaller number of processes were running. So it was decided
that RUNTIME_AREA could still be used for FZ creation. Maintaining
consistency, it was not wise to apply MOP_FZ_TEMP_AREA change only to
few sites, while other sites continue to use old RUNTIME_AREA. Instead
MOP_FZ_TEMP_AREA was set to "./" (means current directory in Unix/Linux)
which in MOP environment would mean RUNTIME_AREA (the stage-in area
created by MOP). This information is put in parameter files, as the
legacy code used by some of the applications like CMSIM and WriteDigi
cannot read from environmental variables. The MOP_FZ_TEMP_AREA appeared
as ".//" in parameter files. This is a valid Unix/Linux directory path,
but the legacy code interpreted it as "comment" in the parameter file.
Looking for fz file in current directory on each file and then
Ultimately dumping core. Over 200 MB of core was dumped. The
stdout/stdin and stderr are managed by Globus called gass-cache. Gass
cache creates temporary directories in area specifically designated for
this purpose, when a job is created. All stdin/out/err is channeled to
this area from running program, and then to submission site.

Dumping core, over 200 MB, code was quitting with error code "0". So no
error being detected, and at the submission site it was never understood
that something is going wrong. Submission site, kept on submitting fresh
jobs, while current jobs were finishing apparently "normally" one after
the other without any indication of any problem at all. It was later
reported that gass-cache area is filling up and Globus is failing to
handle more jobs. As a solution gass-cache was either given more room,
or simply manually cleaned to make some space. Similar gass-cache
problems in past caused gass-cache clean up failure, so a biased opinion
pointed finger towards gass-cache problem. And experts in gass-cache
very invoked. Later it was identified that gass-cache had nothing to do
with the problem and actual reason is miss-interpretation of ".//" as
comments and strange behavior of application code. Instead of "./" as
MOP_FZ_TEMP_AREA "." Solved the problem.

The error would have been detected in the first place if failing code
would be returning some error code, and the integration of application
code in MCRunJob and MOP could have identified that there is something
wrong. If the previously part returns with error code "0", the later
part of jobs continues successfully assuming a successful completion.

It was also noted that a tool, which can look into running jobs, could
identify the errors at an earlier stage. This could be a script that can
monitor expected results and report error in case of some unexpected
behavior.

#### **WriteAllDigi** runs in a loop forever:

Few instances of WriteAllDigi code run in a loop forever. The
process(es) needs to be killed manually to save CPU time. The
WriteAllDigi happens to be last step in the production chain, so the
jobs get to it only after completing over 24 hours of cmsim FZ
production. Killing the job at this stage would mean losing over 24
hours of CPU time. The looping WriteAllDigi processes can only be
identified after monitoring job queue, and looking for jobs running over
several days, without restart.

In latest design of MOP, using atomic steps to run each step in
production chain, this type of problems will cause relatively lower
loss.

#### Long path names:

The Legacy code part of cmsim and other components were unable to open
files required for running the program,. The files were installed in
deep directory structures and program was unable to access them because
of longer Unix/Linux path names. Reinstallation of CMS software at
relatively less deeper directory structure resolved this problem.

### Globus-URL-Copy Limitations.

Attempting a large number of job submission to a site or number of site
from MOP master causes a large number of globus-url-copy processes to
run simultaneously which over-whelms file transfer daemons used by
globus-url-copy. This is being worked around by throttling job
submission at a time.

Globus-URL-Copy from a worker site to Master site fails, this is fault
tolerated by attempting a required g-u-c transfer a number of times
using Fault Tolerance Shell (FTSH), until the transfer is completed
successfully or given up with a complete failure.

### Experience from use of Job auto submission scripts

The decision to submit job(s) to a particular worker site is completely
human. To minimize effort few scripts to automate job submissions were
used. These scripts worked very well up to the extent that all sites
were behaving. In case of a failure at certain sites, these scripts
caused only more trouble. Continuous modifications were made to fine
tune automated job submissions to work around several possible
scenarios, which may appear, at a production worker.

#### G-U-C limitation

Initiating a submission means starting several g-u-c transfers from
master site. In cases where there were already many g-u-c processes
running, JAM scripts delayed submission until the number of g-u-c
processes at master site reaches to an acceptable level. Interesting is
the fact that there were some cases where then g-u-c number was
acceptable at submission time, but just after that some other transfers
were initiated by worker sites, which eventually killed FTP daemons.

#### Pre-emption and Orphan jobs:

Initially JAM scripts were monitoring number of free CPUs at a worker
site to submit more jobs. There were few cases when jobs were getting
pre-empted and then re-submitted by condor jobmanager at remote site,
due to configuration problem or because of the fact that these were
orphan jobs, which were getting submitted and then killed by condor
jobmanager, and still present in the queues (pretty horrible scenario!).
At moments JAM scripts found a number of free CPUs, submitted fresh jobs
to a worker site, and this continued until a real huge number of jobs
kept on submitting to worker site and brought it down for reasons like
YP bind problem or out of disk space.