# How to get CAB Job Id from a SAM-Grid Job id

## Introduction

This document explains how to obtain the job id of a job running on an
OSG cluster (e.g. CAB) from a given SAM-Grid job id.

The process requires root access to the gatekeeper node of the OSG
cluster, in order to read /var/log/messages, AND access to a node with
the jim_client product installed as any user. This product is part of
the SAM-Grid release cut and is available through ups/upd.

## Summary of the commands

To get the list of OSG Job Ids for a given SAM-Grid job id
(*\<SAMGRID_JOB_ID\>*) (see Step 2):

> \# condor_q -pool osg-ress-1.fnal.gov -global --constraint\
> \'grid_jid==\"*\<SAMGRID_JOB_ID\>*\"\'

From the output of the previous command, select one OSG Job Id
(*\<OSG_JOB_ID\>*) from the first column. To get the process id of the
job manager running at the CAB gatekeeper (see Step 3):

> \# condor_q -pool osg-ress-1.fnal.gov -global *\<OSG_JOB_ID\>* -format
> \"%s\\n\"\
> GridJobId \| awk \'{print \$3}\' \| awk \'BEGIN {FS=\"/\"} {print
> \$4}\'

**IF** **you have root access** on the CAB gatekeeper node
(d0cabosg2.fnal.gov), log in as root. Given a job manager process id
(*\<JOB_MANAGER_PROCESS_ID\>*) from the previous command, the PBS job id
is (see Step 3):

\# grep *\<JOB_MANAGER_PROCESS_ID\>* /var/log/messages \| grep gridinfo

**IF you do NOT have root access** on the CAB gatekeeper node
(d0cabosg2.fnal.gov), log in as any unprivileged user. Given a job
manager process id (*\<JOB_MANAGER_PROCESS_ID\>*) from the previous
command, the PBS job id is (see Step 3):

> \# GK_PROC_ID=\`grep *\<JOB_MANAGER_PROCESS_ID\>*\
> usr/local/vdt/globus/var/globus-gatekeeper.log \| awk \'{print
> \$2}\'\` ;\
> grep \$GK_PROC_ID /usr/local/vdt/globus/var/accounting.log \|\
> grep GRAM_SCRIPT_JOB_ID

## Overview of the job workflow

Let me summarize the workflow for a job. For example
diesburg_d0mino01.fnal.gov_232809_28827

-   jim_client, e.g. on d0mino01, submit job to the condor_schedd @
    samgrid.fnal.gov

-   condor_schedd @ samgrid.fnal.gov talks to
    condor_negotiator/collector @ samgrid.fnal.gov (jim_broker) and
    selects the appropriate forwarding node. In this case
    samgfwd02.fnal.gov

-   condor_schedd @ samgrid.fnal.gov submit the job to globus_gatekeeper
    @ samgfwd02.fnal.gov

-   globus_gatekeeper @ samgfwd02.fnal.gov instanciate a job-manager,
    which splits the job in the appropriate amount of OSG jobs, and
    submits the to the condor_schedd @ samgfwd02.fnal.gov

-   condor_schedd @ samgfwd02.fnal.gov talks to
    condor_negotiator/collector @ osg-ress-1.fnal.gov (OSG ReSS) and
    select the appropriate OSG resource to submit the job\
    NOTE: condor_schedd @ samgfwd02.fnal.gov is registered with
    osg-ress-1.fnal.gov. You don\'t need to log in samgfwd02.fnal.gov to
    get condor_schedd info!

-   condor_schedd @ samgfwd02.fnal.gov talks to globus_gatekeeper @
    d0cabosg2.fnal.gov and submits the jobs to the local PBS (via the
    pbs job-manager)

## Step 1: Finding what forwarding node the job is submitted to

There are two ways: (1) by looking at the monitoring OR (2) using
command line interfaces.

NOTE: if you are looking for the OSG / Condor-G job id, you can skip
this step and get the forwarding node and the condor-g job in one time:
see Step 2.

### Finding the fwd node using web monitoring

Look at the list of SAM-Grid jobs for the samgrid.fnal.gov queuing
service:

http://samgrid.fnal.gov:8080/known_scheduler.php?schedulername=samgrid.fnal.gov

Find your SAM-Grid job id on the left-most column and find the name of
the forwarding node service at the corresponding column "Execution
Site". In our example, the SAM-Grid job id is
diesburg_d0mino01.fnal.gov_232809_28827 and the corresponding forwarding
node service is Samgrid-OSG-02.

Find the host name of the machine that runs the Samgrid-OSG-02
forwarding node service starting from the page of SAM-Grid resources

http://samgrid.fnal.gov:8080/list_of_resources.php

In this table, click the name of the "Station" (e.g. osg-ouhep)
corresponding to the "Site Name" Samgrid-OSG-02. This page shows all
classads corresponding to the osg-ouhep name. A classad is a list of
attribute/value pairs, used by the condor system to describe jobs,
resources, services, etc. Search the page for Samgrid-OSG-02 and look at
the value of gatekeeper_url in the same classad. In this case, the
gatekeeper_url is

\"samgfwd02.fnal.gov:2119/jobmanager-samgrid\". The machine that runs
the forwarding node service is samgfwd02.fnal.gov.

### Finding the fwd node using command line interfaces

Log in to any node where the jim_client product is installed; typically
any user has execute privileges to the necessary commands. One such node
is, for example, d0mino01.

Find to what forwarding node the job is submitted by looking at the
GridJobId attribute from the SAM-Grid job classad. A classad is a list
of attribute/value pairs that is used by the condor middleware to
describe characteristics of jobs, resources, services, etc.

The following command output the whole classad for the SAM-Grid job
diesburg_d0mino01.fnal.gov_232809_28827. To extract only the information
you need, you can 'pipe' the output of this command in 'grep GridJobId'.

\<d0mino01\> samg list jobs -lf diesburg_d0mino01.fnal.gov_232809_28827
\| grep GridJobId

GridJobId = \"gt2 samgfwd02.fnal.gov:2119/jobmanager-samgrid
https://samgfwd02.fnal.gov:38742/24826/1203571712/\"

Note that with jim_client v3_1_3 or later, you can form a more efficient
query obtaining only the GridJobId, with

samg list jobs -format \"%s\\\\n\" GridJobId
diesburg_d0mino01.fnal.gov_232809_28827

## Step 2: Finding all OSG / Condor-G jobs submitted to the fwd node

SAM-Grid jobs are unit of computations, typically consisting of complex
computational steps. These steps are best executed as multiple single
job instances. For example, a SAM-Grid montecarlo job typically
generates 50,000 simulated events; these events are best generated if
split into 200 single jobs, each producing 250 events. SAM-Grid jobs are
split into multiple OSG jobs at the forwarding node.

DZero uses the OSG Resource Selection System to submit jobs to OSG. In
other words, all its condor_schedd (condor-g scheduler) are registered
w/ the condor_collector at osg-ress-1.fnal.gov.

Find all running/idle/held jobs submitted to the forwarding node by
logging in to any machine with jim_client installed as any user. Execute
the command:

\<d0mino01\> condor_q -pool osg-ress-1.fnal.gov -global -constraint
\'grid_jid==\"diesburg_d0mino01.fnal.gov_232809_28827\"\'

\-- Schedd: \@samgfwd02.fnal.gov : \<131.225.217.228:57061\>

ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD

137587.0 samgrid 2/20 23:29 0+10:15:36 R 0 9.8 default_scratch_ma

137590.0 samgrid 2/20 23:29 0+10:15:35 R 0 9.8 default_scratch_ma

137598.0 samgrid 2/20 23:30 0+10:15:35 R 0 9.8 default_scratch_ma

137599.0 samgrid 2/20 23:31 0+10:15:35 R 0 9.8 default_scratch_ma

137600.0 samgrid 2/20 23:31 0+10:15:34 R 0 9.8 default_scratch_ma

137602.0 samgrid 2/20 23:31 0+10:15:34 R 0 9.8 default_scratch_ma

137607.0 samgrid 2/20 23:32 0+10:15:34 R 0 9.8 default_scratch_ma

137609.0 samgrid 2/20 23:32 0+10:15:34 R 0 9.8 default_scratch_ma

137611.0 samgrid 2/20 23:33 0+10:12:31 R 0 9.8 default_scratch_ma

137624.0 samgrid 2/20 23:35 0+06:57:07 H 0 9.8 default_scratch_ma

137626.0 samgrid 2/20 23:35 0+10:08:23 R 0 9.8 default_scratch_ma

137630.0 samgrid 2/20 23:35 0+10:08:22 R 0 9.8 default_scratch_ma

If you know the fwd node, you can make the search more efficient w/
(note -name)

\<d0mino01\> condor_q -pool osg-ress-1.fnal.gov -name
\@samgfwd02.fnal.gov -constraint
\'grid_jid==\"diesburg_d0mino01.fnal.gov_232809_28827\"\'

\-- Schedd: \@samgfwd02.fnal.gov : \<131.225.217.228:57061\>

ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD

137587.0 samgrid 2/20 23:29 0+10:19:01 R 0 9.8 default_scratch_ma

137590.0 samgrid 2/20 23:29 0+10:19:01 R 0 9.8 default_scratch_ma

137598.0 samgrid 2/20 23:30 0+10:19:00 R 0 9.8 default_scratch_ma

137599.0 samgrid 2/20 23:31 0+10:19:00 R 0 9.8 default_scratch_ma

137600.0 samgrid 2/20 23:31 0+10:18:59 R 0 9.8 default_scratch_ma

137602.0 samgrid 2/20 23:31 0+10:18:59 R 0 9.8 default_scratch_ma

137607.0 samgrid 2/20 23:32 0+10:18:59 R 0 9.8 default_scratch_ma

137609.0 samgrid 2/20 23:32 0+10:19:00 R 0 9.8 default_scratch_ma

137611.0 samgrid 2/20 23:33 0+10:15:56 R 0 9.8 default_scratch_ma

137624.0 samgrid 2/20 23:35 0+06:57:07 H 0 9.8 default_scratch_ma

137626.0 samgrid 2/20 23:35 0+10:11:49 R 0 9.8 default_scratch_ma

137630.0 samgrid 2/20 23:35 0+10:11:47 R 0 9.8 default_scratch_ma

## Step 3: Find the PBS job id for a given Condor-G job id.

Obtain the classad attribute "GridJobId" corresponding to a Condor-G job
id. We found the list of the Condor-G job id in Step 2 (see first field
of the output above in step 2). Note that the Condor-G GridJobId is
different from the SAM-Grid GridJobId that we sought in Step 1.

Note: In general, the GridJobId is the GRAM URL of the job manager
representing the job running at the remote cluster.

To obtain the GridJobId corresponding to a given Condor-G job id (e.g.
137630.0 i.e. the first field of the last row of the output above from
step 2), execute the following command from any machine with jim_client
installed

\<d0mino01\> condor_q -pool osg-ress-1.fnal.gov -global 137630.0 -format
\"%s\\n\" GridJobId

gt2 d0cabosg2.fnal.gov:2119/jobmanager-pbs
https://d0cabosg2.fnal.gov:32712/4307/1203572290/

Note: To see all information available about the job, use -l instead of
-format \"%s\...\.....

Extract the first number delimited by slashes (/), after the port in the
https URL. This is the job manager process id number. In the example
above, the number is 4307. In the examples below, the number used is
24286 (if you have root) or 6387 (if you do not have root).

**IF** **you have root access** on the gatekeeper node of the OSG
cluster, log in as root. For CAB, this is root@d0cabosg2.

Extract all lines with the number 24286 AND gridinfo from
/var/log/messages:

\[root@d0cabosg2 log\]# grep 24286 /var/log/messages \| grep gridinfo

Dec 18 14:59:31 d0cabosg2 gridinfo\[24286\]: JMA 2007/12/18 14:59:31

GATEKEEPER_JM_ID 2007-12-18.14:58:23.0000024091.0000000000 JM exiting

Dec 19 14:22:08 d0cabosg2 gridinfo\[24286\]: JMA 2007/12/19 14:22:08

GATEKEEPER_JM_ID 2007-12-19.14:20:47.0000023845.0000000000 for

/DC=org/DC=doegrids/OU=People/CN=Michael Diesburg 853856 on

131.225.219.207

Dec 19 14:22:08 d0cabosg2 gridinfo\[24286\]: JMA 2007/12/19 14:22:08

GATEKEEPER_JM_ID 2007-12-19.14:20:47.0000023845.0000000000 mapped to

dzeropro (42706, 1507)

Dec 19 14:22:08 d0cabosg2 gridinfo\[24286\]: JMA 2007/12/19 14:22:08

GATEKEEPER_JM_ID 2007-12-19.14:20:47.0000023845.0000000000 has

GRAM_SCRIPT_JOB_ID 455806.d0cabsrv2.fnal.gov manager type pbs

Dec 19 14:22:23 d0cabosg2 gridinfo\[24286\]: JMA 2007/12/19 14:22:23

GATEKEEPER_JM_ID 2007-12-19.14:20:47.0000023845.0000000000 JM exiting

The CAB pbs job id shows up as the GRAM_SCRIPT_JOB_ID in this log. In
this case 455806.d0cabsrv2.fnal.gov.

Note that this output shows also other information sometimes relevant,
such as the local username (dzerorepro) under which the job is running,
the user grid certificate subject
(/DC=org/DC=doegrids/OU=People/CN=Michael Diesburg 853856).

**IF you do NOT have root access** on the gatekeeper node of the OSG
cluster, log in as any unprivileged user. For CAB, the gatekeeper node
is d0cabosg2.

Given a GridJobId with job manager process id number 6387, get the
parent globus gatekeeper process id, which started the job manager, with
the following command:

\> grep 6387 /usr/local/vdt/globus/var/globus-gatekeeper.log

PID: 6270 \-- Notice: 0: Child 6387 started

In this case, process id is 6270.

Get the PBS job id with the following command

\> grep 6270 /usr/local/vdt/globus/var/accounting.log

JMA 2008/02/27 00:06:05 GATEKEEPER_JM_ID
2008-02-27.00:05:57.0000006270.0000000000 for
/DC=org/DC=doegrids/OU=People/CN=Michael Diesburg 853856 on
131.225.217.184

JMA 2008/02/27 00:06:05 GATEKEEPER_JM_ID
2008-02-27.00:05:57.0000006270.0000000000 mapped to dzeropro (42706,
1507)

JMA 2008/02/27 00:06:05 GATEKEEPER_JM_ID
2008-02-27.00:05:57.0000006270.0000000000 has GRAM_SCRIPT_JOB_ID
868177.d0cabsrv2.fnal.gov manager type pbs

JMA 2008/02/27 00:06:06 GATEKEEPER_JM_ID
2008-02-27.00:05:57.0000006270.0000000000 JM exiting

The PBS job id is next to the string GRAM_SCRIPT_JOB_ID in the output
above, in this case 868177.d0cabsrv2.fnal.gov