# Installation Guide, IGT Operation and Shift Manual

**Installing the MOP master\
=========================**

The MOP master only runs at the master site (Fermilab). It does not need
to be installed anywhere else. It is, however, available for anonymous
cvs checkout for the interested. To check out the MOP master:

\% setenv CVSROOT :pserver:anonymous@cdcvs.fnal.gov:/cvs/cd_read_only\
% cvs login\
(Logging in to anonymous@cdcvs.fnal.gov)\
CVS password: anoncvs\
% cvs -Q checkout mop_master\
% cd mop_master\
% ./install\
Checking out impala\
Checking out mop_submitter\
%

To reiterate: mop_master is not needed at remote sites.

## Configuring the MOP Master User Account

Before doing anything on the MOP Master, you should source
mop_master/source.sh. This is most easily done by adding

source /path/to/mop_master/source.sh

to your .bash_profile. For maximum portability, the MOP Master should
probably use /bin/bash as its shell.

While you\'re chaning the .bash_profile, you may also want to add some
settings for Condor. It is recommended that you put the Condor user
tools into your path, wherever they are. Unless your condor_config file
is in \~condor/condor_config, you\'ll need to set the CONDOR_CONFIG
environment variable to point to the condor_config file.

PATH=\$PATH:\~/condorg/bin

declare -x CONDOR_CONFIG=\~/condorg/etc/condor_config

## 

## Setting Up a MOP Worker Site.

##  Overview MOP is a system for distributing CMS production jobs. There is a MOP master site where jobs are defined by the CMS production scripts. The mop_submitter then distributes those jobs to remote sites through CondorG and Globus. When the jobs are finished, the output is collected by GDMP. 

##        Preparing remote sites for MOP jobs             ===================================              Remote site overview:              In general, the remote site will need VDT installed on one or more machines and a Globus job             manager for the local batch system.                         Accounts:              MOP worker site should map cmsprod account from fnal.gov to one of its local user accounts.                         The contact string is:             \"/O=Grid/O=Globus/OU=fnal.gov/CN=CMS Production\"              Remote site summary:                         The following information needs to be conveyed to MOP Master site owner:              \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--             (1-1) stage-in job manager             (1-2) GLOBUS_LOCATION value.             (1-3) Shared directory for mop files if not home directory.              (2-1) run job manager.             (2-2) location of CMS DAR installation. NB: only the path needs to be provided. The DAR file(s)             themselves can be installed through MOP.              \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--              Example remote site values:              The first \"remote\" site is at Fermilab. Here are the Fermilab values:              (1-1) droidf.fnal.gov:/jobmanager             (1-2) /opt/globus/globus20             (1-3) /cms/work              (2-1) droidf.fnal.gov:/jobmanager-condor             (2-2) /data/dar              The site parameters are stored in the mop_submitter/site-info directory. Job manager and scratch             directory info is in the \*.site.\* files. The .vars files hold the following information:              Appendum 1:             ============              (1) In the \$GLOBUS_LOCATION/etc/globus-job-manager-condor.conf file, edit the two lines at the             bottom according to the comment above them, adding INTEL and LINUX as arguments, like so:              -condor-arch INTEL             -condor-os LINUX              and remove the two comment lines (appearing just before):              \# Edit the following two lines to complete             \# the configuration of your condor jobmanager              Then add another line to retain debugging info on errors:              -save-logfile on_errors              Finally, rename the file to globus-job-manager-condor-INTEL-LINUX.conf              (2) Let\'s use testulix.phys.ufl.edu as an example. In             \$GLOBUS_LOCATION/etc/jobmanager-condor, add \"-condor-os LINUX -condor-arch INTEL\" to             the end of the argument list, and change the existing -conf and -rdn arguments to refer to             globus-job-manager-condor-INTEL-LINUX.conf and             testulix.phys.ufl.edu/jobmanager-condor-INTEL-LINUX instead of the old names.              Finally, rename the file to jobmanager-condor-INTEL-LINUX.              We thereafter refer to that job manager as             testulix.phys.ufl.edu/jobmanager-condor-INTEL-LINUX instead of just             testulix.phys.ufl.edu/jobmanager-condor (when using globus-job-run, Condor-G,             etc).              At Florida, the condor job manager is then:              % cat jobmanager-condor-INTEL-LINUX             stderr_log,local_cred - /usr/local/globus/globus-2.0/libexec/globus-job-manager             globus-job-manager -conf             /usr/local/globus/globus-2.0/etc/globus-job-manager-condor-INTEL-LINUX.conf -type             condor -rdn testulix.phys.ufl.edu/condor-INTEL-LINUX -machine-type unknown             -publish-jobs -condor-os LINUX -condor-arch INTEL              and the condor job manager config file reads as:              % cat globus-job-manager-condor-INTEL-LINUX.conf             -home \"/usr/local/globus/globus-2.0\"             -e /usr/local/globus/globus-2.0/libexec             -globus-gatekeeper-host testulix.phys.ufl.edu             -globus-gatekeeper-port 2119             -globus-gatekeeper-subject \"/O=Grid/O=Globus/CN=testulix.phys.ufl.edu\"             -globus-host-cputype i686             -globus-host-manufacturer pc             -globus-host-osname Linux             -globus-host-osversion 2.2.14-5.0smp             -condor-arch INTEL             -condor-os LINUX             -save-logfile on_errors              Appendum 2:             ============              The default Condor installation is configured so that Condor will suspend all jobs on detection of             keyboard activity. The instructions on how to fix this are to modify \"PART 3\" of the             \$CONDOR_LOCATION/etc/condor_config file underneath where it says:              \#####################################################################             \## This where you choose the configuration that you would like to             \## use. It has no defaults so it must be defined. We start this             \## file off with the UWCS\_\* policy.             \######################################################################              The modifications are:              Original condor_config file:             START = \$(UWCS_START)             SUSPEND = \$(UWCS_SUSPEND)             CONTINUE = \$(UWCS_CONTINUE)             PREEMPT = \$(UWCS_PREEMPT)             KILL = \$(UWCS_KILL)              Modified condor_config file:             #START = \$(UWCS_START)             START = True             #SUSPEND = \$(UWCS_SUSPEND)             #CONTINUE = \$(UWCS_CONTINUE)             #PREEMPT = \$(UWCS_PREEMPT)             SUSPEND = False             CONTINUE = True             PREEMPT = False             #KILL = \$(UWCS_KILL)             KILL = \$(ActivityTimer) \> \$(MaxVacateTime)             #PREEMPTION_REQUIREMENTS = \$(UWCS_PREEMPTION_REQUIREMENTS)               PREEMPTION_REQUIREMENTS=False              Then, to enact the changes for the new Condor configuration, execute              % condor_reconfig node1 node2 node3 \...              where node1, node2, node3 \... are the different Condor compute machines (including the             Condor Master machine).

## IGT Operation Manual:

Debugging problems during a MOP based production over several sites is a
complex job and requires several different types of steps performed,
before the real nature of problem is diagnosed. The soultion sometimes
is trivial, but not obvious esaily found, or sometimes it is a complex
issue and will need expert attention.

Follwoing are the activities which seems to be very appropriate for
running a MOP based production.

**1. Verify health of MOP master:**\
    a. Do a condor_q and verify that you see a large number of jobs in
Running status.\
    b. do a \"**condor_q -l xxxx.x \| grep Out**\", and then open the
**run.out** file spitted by this command for condor job xxxx.x, verify
that job seems to be progressing into various stages, 
cmkin\--\>cmsim\--\> etc. Look for the last activity,most probably a
statement showing the latest event number being processed. Perform this
step on several jobs chosen randomly.\
    c. Perfom step c on cluster by cluster basis, by doing a
\"**condor_q -dag**\" and picking more then one job from a cluster.\
    d. Look for Held processes, few Held processes seems to be a
\"normal\" behaviour.\
    e.  Do a \'**uptime**\' time to time and verify that the load on MOP
master is not more than 3/4 at anytime.\
    f. Look for a number of gahp_server threads (use **pstree**), they
will go away, but if not kill the **condor_gridmanager** process.\
    g. Use ganglia/Monalisa monitoring tools, to see if system load and
other paramenets are looking healthy.\
    h. Do \"df -h\" and verify free disk space. There must be several
tens of GB free on /data and \"enough\" on other partitions.\
    i. Verify that various condor processes, including condor_master are
running.\
                     \<gallo.fnal.gov\>  ps -elf \| egrep condor\_\
                    040 S cmsprod   6509     1  1  60   0    -  1122
do_sel Nov13 ?        02:55:51 condor_master\
                    000 S cmsprod   6510  6509  0  60   0    -  1114
do_sel Nov13 ?        00:01:59 condor_collector -f\
                    000 S cmsprod   6511  6509  0  60   0    -  1061
do_sel Nov13 ?        00:02:02 condor_negotiator -f\
                    000 S cmsprod   6512  6509  0  60   0    -  1310
do_sel Nov13 ?        00:11:22 condor_startd -f\
                    000 S cmsprod   6513  6509  0  60   0    -  3095
do_sel Nov13 ?        00:23:57 condor_schedd -f\
    j. Look for produced ntuples in /data/out_ntup/RUN-II area, all
files, specially freshly arrived, should be of the order of similar
size.\
    k. Look into various log files, GridLog, condor logs etc. and look
into following  files if they exist,\
                    dprintf_failure.MASTER\
                    dprintf_failure.NEGOTIATOR\
                    dprintf_failure.SHADOW

**2. Verify health of MOP Worker site.**\
**   ** a. Should be able to \"globus-run-job \<site\> /bin/date\" to
all sites\
    b. should be able to verify condor_q/condor_status on all sites,
using globus-job-run.\
        globus-job-run grinulix.phys.ufl.edu /vdt/bin/condor_status\
        globus-job-run  t2cms0.sdsc.edu
/vdt/vdt-1.1.4/bin/condor_status\
        globus-job-run tier2.cacr.caltech.edu /vdt/bin/condor_q\
    c. Verify all sites have enough disk-spaces in respective
partitions.\
    d.\$MOP_DIR/mop_submitter/site-info, contain all site files, and
files ending with Igt are currently being used (for fermi its
velveeta.\*).\
        These file provides info. about each site, and one can reference
them for site verification.\
    e. Its sometime required to see if worker nodes are up and running
too, and they have enough disk space left (specially in /tmp area).\
    f. Look at changing size of .fz file(s) being produced, check back
after a period of more than an hour.\
    g. Look into various log files, GridLog, condor logs etc. and look
for failure files.

**3. Submitting new jobs:**\
    Follwoing are important points to remember:-\
        i. In absence of a scheduler, jobs are submitted manually.\
        ii. No need to keep a large number of jobs in waiting condition,
while there are only a certain number of CPUs avaialble,\
            and jobs take more than 24 hours, this gives  more liberty
and control and ability to diagnose problems, without losing jobs.

   So first look for free CPUs at various site.\
    Following return number of free CPUs at various sites,\
            globus-job-run grinulix.phys.ufl.edu /vdt/bin/condor_status
-total \| grep Total \| awk \'{print \$5}\'\
            globus-job-run tier2.cacr.caltech.edu /vdt/bin/condor_status
-total \| grep Total \| awk \'{print \$5}\'\
            globus-job-run  t2cms0.sdsc.edu
/vdt/vdt-1.1.4/bin/condor_status -total \| grep Total \| awk \'{print
\$5}\'

    FermiLab is using fbsng, so following will return number of jobs
already in queue, if this number is small or zero then its OK to add
more jobs.\
            globus-job-run velveeta /data/ANZAR/testfbs.sh\
   Otherwise login/ssh to velveeta and do a \'fbs queues\' to look at
the queue.

    The jobs are submitted while \'cwd\' is
/data/AZIZ/MOP/mop_master/impala_mcj/run-XX, where XX is a run number,
chooose the latest run number.\
    You should also source /data/AZIS/setup.csh to setup proper
environment for all activities.

    Note: Its always better to submit jobs in batches, not bigger than
15-20 at a time.

    RunJob.sh -m floridaIgt -v -j mcj eg02_BigJets 10\
    RunJob.sh -m caltechIgt -v -j mcj eg02_BigJets 10\
    RunJob.sh -m ucsdIgt -v -j mcj eg02_BigJets 10\
    RunJob.sh -m velveeta -v -j mcj eg02_BigJets 10

  Note: I have tried to use an auto submit script, that looks for free
CPUs every hour and submitts new jobs, but due to (known!) reasons its
not very wise to use it.

If **job submission fails**, with an error message like this,

> \<gallo.fnal.gov\> RunJob.sh -m velveeta -v -j mcj eg02_BigJets 20\
> localFunctions.sh resource file for US-MOP\
> TARBALL=true, installed in \_insert_MOP_REMOTE_DAR_ROOT\_.\
> No further action defined.\
> Selecting the list of scripts\
>       \...selecting from batch/created area.\
> **No jobs were selected.**

This will mean that MCRunJob tracking area is out of created batch jobs,
you will need to make some more jobs.\
    cd to /data/AZIZ/MOP/mop_master/impala_mcj/py_script\
    run \"./Linker.py script=eg02_BigJets_00002417.sav\"\
    This will make 200 new jobs, and you can resume job submission.

**4. Large number of Held Processes**\
    If there is a large number of Held processes, this needs more
attention.\
    1. Verify that Held processes belong to single site,\
            i. In this case do the steps specified in site health
verification.\
            ii. Trace down a couple of pocesses, upto the running jobs
at remote sites.\
            ii. If a site is completely down, its wise to just \"Hold\"
processes manually untill issue is resolved, or just remove them all.\
                holding the dagman running the batch is enough to hold a
set of jobs held. (**condor_q -dag**, **condor_hold, condor_rm**)\
     2. If processes are held for all sites (should never happen), this
obviously means problem at MOP master site.\
            i. Look for MOP master health status, track down running
jobs, condor processes, disk spaces etc\...

**5. Large number of Idle processes**\
    1. This could be normal.\
    2. Track clusters to sites, verify that either site also has a large
Idle queue, ( this is ok )  or all jobs at site are just Idle (not OK).\
    3. Master site do not have gahp_server, refere to master site
section.

**6. New submitted jobs do not move**\
1. Wait for 5-10 mins,\
2. Look for gahp_server processes\
        i. large number of gahp_server processes (wait \....if no
affect\.....kill condor_gridmanager process, and wiat 5-10 mins).\
        ii. No gahp_server process, most probably killed itself, wait a
while, system should start to move again.\
        iii. Worst case, submit some jobs manually to DagMan, try not to
use condor/other jobmanager at target site. Use the fork manager,. and
verify your\
              job goes through. I\
        iv. Sending condor_restart helps ? (worst case scenario!)\
        v. Restart condor_master (should be required only if city is on
fire!).\
3. Better to call experts, if noting helps.

**7. Jobs fail immediately after submission**\
A. Jobs fail in stage-in\
        i. Look into stage-in.out/err files for failing jobs,\
                a. is globus-url-copy working\
                b. no harm in trying a couple of copies manually\
                c. space left on site ?.\
                d. source file exits on MOP master (this happened when
something messes up with MCRunJob tracking area).\
B. Jobs held in stage-in\
        i. Verify globus-url-copy, site is reachable, site is healthy.\
C. Jobs fail in run stage\
      i. Look for run.out/run.err files (this is stdout/stderr of
running job).\
      ii. Look for \*out/err in the mop-tmp-bla-bla area for the failing
jobs.\
      iii. Might need to manually HOLD jobs as MOP master attempts
restart, over writting files, making it tough for debugging.

**8. Jobs fail in Run stage, and keeps on failing, while few of them are
still running (say 18 runs, and over 18 all fails)**\
            One/two machine is having problems !!!!\
            Which is draining the queue, so better identify this
machine, by looking for all jobs that failed at this machine.\
            Fix the problem, like disk-space etc.\
            Stop submitting more untill verified its OK.

**9. Orphan processes**\
    i. If you feel that number of jobs queued at MOP master is lesser
than the number of jobs running at a worker site, this might mean some
orphan\
        processes, taking the CPU.\
    ii. If possible, track them and kill them (only if you are damn
sure!).\
    iii Let them drain over time (collect .ntup files manually later).\
    iv. Avoid some Orphan processes by removing Held processes from MOP
master and killing, after tracking, the (once) related process at worker
site. (tough !!).

**10. Ever running processes**\
        i. If you notice jobs in the queue for a long time, 2-3 days or
more, this might be because of writeAllHits process caught in an
infinite loop.\
        ii. Track down the process to runnning job (uh!!). Kill the
writeAllHits process running for thousands of minutes.

**11. One of the stages fail, while its prior stage is successfuly
returned with MOP_RETURN_VAL=0**\
    i. Make sure all correct site files with correct entries are
present, especially for the failing stage.

**12. Number of Free CPUs at remote site keeps on revolving (example
20-10-0-20 )**\
    i. Most probably there are Orphan jobs\
    ii. Identify the jobs at Worker site\
    ii. Kill them.\
 

**Useful References:\
**http://home.fnal.gov/\~anzar/MOP_MASTER/mop.html\
http://www.cs.wisc.edu/\~adesmet/cms_mop_master.html\
http://grid.fnal.gov/test_beds/MOP_install.html