<!-- Title: 12:16:41  Fri Apr  4 2008 -->
<!-- Crewchief: Doug Benjamin -->
<!-- Op1: Marian Zvada -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>new file for glidewin/scondary schedd work</pre>


<!-- Date: Tue Apr 29 16:16:34 2008 -->
<pre>I blacklisted fcdfcaf362 (helpdesk ticket 114924)</pre>
<!-- Author: Krzysztof -->


<!-- Date: Fri May 9 16:54:23 2008 -->
<pre>"unblacklisted" fcdfcaf362</pre>
<!-- Author: klg -->


<!-- Date: Wed May 14 16:56:39 2008 -->
<pre>Blacklisted retired (per Stephan) nodes 
fncdf273 
fncdf274 
fncdf279 
fncdf292 
fncdf293 
fncdf297 
</pre>
<!-- Author: klg -->


<!-- Date: Tue Aug 5 14:15:35 2008 -->
<pre>[cdfcaf@fcdfhead2:~] $ date;condor_status -schedd 
Tue Aug  5 13:45:20 CDT 2008 
 
Name                 Machine    TotalRunningJobs TotalIdleJobs TotalHeldJobs 
 
fcdfhead2.fnal.gov   fcdfhead2.                0             0              0 
schedd_1@fcdfhead2.f fcdfhead2.              208             0             45 
schedd_2@fcdfhead2.f fcdfhead2.              543            51            205 
schedd_3@fcdfhead2.f fcdfhead2.              490            19            166 
schedd_4@fcdfhead2.f fcdfhead2.              356             1            161 
schedd_glide_fcdfosg fcdfhead2.                0             0              0 
schedd_glide_fcdfosg fcdfhead2.                0             0              0 
schedd_glide_fcdfosg fcdfhead2.                0             0              0 
schedd_glidecdf1@fcd fcdfhead2.                0             0              0 
schedd_glidecdf2@fcd fcdfhead2.                0             0              0 
schedd_glideins@fcdf fcdfhead2.                0             0              0 
 
                      TotalRunningJobs      TotalIdleJobs      TotalHeldJobs 
 
 
               Total              1597                 71                577 
 
 
 
 
[cdfcaf@fcdfhead2:~] $ for schedd_num in 1 2 3 4; do date; condor_q -name schedd_${schedd_num}@ -held | awk '{print $2}'| sort | uniq -c; done; 
Tue Aug  5 13:54:32 CDT 2008 
      3 
     18 canepa 
     26 casarsa 
      1 jobs; 
      1 njg 
      1 OWNER 
      1 Schedd: 
Tue Aug  5 13:54:32 CDT 2008 
      3 
     13 ag 
     38 casarsa 
      1 jobs; 
      1 OWNER 
    133 rshekhar 
      1 Schedd: 
      2 slaunwhj 
     11 taichi 
Tue Aug  5 13:54:33 CDT 2008 
      3 
     35 fedorko 
      3 hslee 
      1 jobs; 
     30 loakes 
      3 njg 
      1 OWNER 
     91 petar 
      1 Schedd: 
      2 slaunwhj 
Tue Aug  5 13:54:34 CDT 2008 
      3 
    135 canepa 
     10 casarsa 
      1 jobs; 
      1 OWNER 
      1 Schedd: 
 
 
[cdfcaf@fcdfhead2:~] $ for schedd_num in 1 2 3 4; do date; condor_q -name schedd_${schedd_num}@ -held | awk '{print $3}'| sort | uniq -c; done; 
Tue Aug  5 13:55:56 CDT 2008 
      3 
      1 0 
     18 7/26 
      1 7/29 
     26 8/2 
      1 HELD_SINCE 
      1 schedd_1@fcdfhead2.fnal.gov 
Tue Aug  5 13:55:57 CDT 2008 
      3 
      1 0 
    128 7/23 
     31 7/24 
     33 7/30 
      5 7/31 
      1 HELD_SINCE 
      1 schedd_2@fcdfhead2.fnal.gov 
Tue Aug  5 13:55:58 CDT 2008 
      3 
      1 0 
     28 7/22 
    128 7/23 
      2 7/24 
      3 7/25 
      3 7/31 
      1 HELD_SINCE 
      1 schedd_3@fcdfhead2.fnal.gov 
Tue Aug  5 13:56:05 CDT 2008 
      3 
      1 0 
    135 7/26 
     10 8/2 
      1 HELD_SINCE 
      1 schedd_4@fcdfhead2.fnal.gov 
 
 
glidein schedds :  
 
[cdfcaf@fcdfhead2:~] $ condor_q -global -constraint 'JobStatus==5' 
 
 
-- Schedd: schedd_glidecdf1@fcdfhead2.fnal.gov : &lt;131.225.240.106:40194&gt; 
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD 
2183502.0   condor          7/23 11:47   1+04:06:09 H  0   0.0  glidein_startup.sh 
2184223.0   condor          7/24 05:44   0+21:33:53 H  0   0.0  glidein_startup.sh 
2184433.0   condor          7/24 10:17   0+02:30:23 H  0   0.0  glidein_startup.sh 
2184511.0   condor          7/24 16:05   0+00:35:13 H  0   0.0  glidein_startup.sh 
 
 
-- Schedd: schedd_glidecdf2@fcdfhead2.fnal.gov : &lt;131.225.240.106:40198&gt; 
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD 
1071675.0   condor          7/23 22:03   0+17:05:29 H  0   0.0  glidein_startup.sh 
1072144.0   condor          7/24 12:07   0+00:30:42 H  0   0.0  glidein_startup.sh 
1072157.0   condor          7/24 12:08   0+00:25:36 H  0   0.0  glidein_startup.sh 
1072179.0   condor          7/24 12:09   0+01:15:43 H  0   0.0  glidein_startup.sh 
1072204.0   condor          7/24 12:10   0+00:25:42 H  0   0.0  glidein_startup.sh 
1072213.0   condor          7/24 12:10   0+00:31:36 H  0   0.0  glidein_startup.sh 
1072249.0   condor          7/24 12:12   0+00:35:42 H  0   0.0  glidein_startup.sh 
1072262.0   condor          7/24 12:12   0+00:36:08 H  0   0.0  glidein_startup.sh 
1072275.0   condor          7/24 12:13   0+00:30:36 H  0   0.0  glidein_startup.sh 
 
 
-- Schedd: schedd_glideins@fcdfhead2.fnal.gov : &lt;131.225.240.106:40191&gt; 
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD 
515860.0   condor          7/24 13:59   0+00:46:07 H  0   0.0  glidein_startup.sh 
 
 
[cdfcaf@fcdfhead2:~] $ date;condor_q -global -constraint 'JobStatus==5' 
Tue Aug  5 14:06:08 CDT 2008 
 
 
NAMCAF :  
 
[cdfcaf@fcdfhead5:~] $  date;condor_q -global -constraint 'JobStatus==5' 
Tue Aug  5 14:06:39 CDT 2008 
 
 
GroupCAF :  
 
[cdfcaf@fcdfhead3 ~]$ date; condor_status -schedd 
Tue Aug  5 14:09:32 CDT 2008 
 
Name                 Machine    TotalRunningJobs TotalIdleJobs TotalHeldJobs 
 
fcdfhead3.fnal.gov   fcdfhead3.                0             0              0 
schedd_100@fcdfhead3 fcdfhead3.               81           156              0 
schedd_101@fcdfhead3 fcdfhead3.              184             3              1 
schedd_102@fcdfhead3 fcdfhead3.              468            84              3 
schedd_103@fcdfhead3 fcdfhead3.             1076           154             75 
 
                      TotalRunningJobs      TotalIdleJobs      TotalHeldJobs 
 
 
               Total              1809                397                 79 
 
 
 
 
[cdfcaf@fcdfhead3 ~]$ date; condor_q -global -constraint 'JobStatus==5' 
Tue Aug  5 14:13:29 CDT 2008 
</pre>
<!-- Author: federica -->


<!-- Date: Thu Aug 7 17:00:09 2008 -->
<pre>[cdfcaf@fcdfhead2:~] $ date;for schedd_num in 1 2 3 4; do date; condor_q -name schedd_${schedd_num}@  -held | awk '{print $3}'| sort | uniq -c; done; 
Thu Aug  7 16:59:27 CDT 2008 
Thu Aug  7 16:59:27 CDT 2008 
      3 
      1 0 
     51 8/6 
      1 HELD_SINCE 
      1 schedd_1@fcdfhead2.fnal.gov 
Thu Aug  7 16:59:27 CDT 2008 
      3 
      1 0 
      1 HELD_SINCE 
      1 schedd_2@fcdfhead2.fnal.gov 
Thu Aug  7 16:59:28 CDT 2008 
      3 
      1 0 
    196 8/7 
      1 HELD_SINCE 
      1 schedd_3@fcdfhead2.fnal.gov 
Thu Aug  7 16:59:29 CDT 2008 
      3 
      1 0 
     80 8/6 
      1 HELD_SINCE 
      1 schedd_4@fcdfhead2.fnal.gov 
[cdfcaf@fcdfhead2:~] $ date;for schedd_num in 1 2 3 4; do date; condor_q -name schedd_${schedd_num}@  -held | awk '{print $2}'| sort | uniq -c; done; 
Thu Aug  7 16:59:39 CDT 2008 
Thu Aug  7 16:59:39 CDT 2008 
      3 
      1 jobs; 
     51 marcvog 
      1 OWNER 
      1 Schedd: 
Thu Aug  7 16:59:39 CDT 2008 
      3 
      1 jobs; 
      1 OWNER 
      1 Schedd: 
Thu Aug  7 16:59:39 CDT 2008 
      3 
      1 jobs; 
      1 OWNER 
      1 Schedd: 
    196 strolog 
Thu Aug  7 16:59:40 CDT 2008 
      3 
      1 jobs; 
     80 marcvog 
      1 OWNER 
      1 Schedd: 
[cdfcaf@fcdfhead2:~] $ 
 
 
removed</pre>
<!-- Author: federica -->


<!-- Date: Fri Oct 3 14:05:05 2008 -->
<pre> 
What I did, in a nutshell, with Krzysztof looking over my shoulder: 
 
CONFIG FILE 
~/condor/dist/etc/local_params.config  
WAS COPY OF 
~/condor/dist/etc/local_params.config.default 
IS NOW COPY OF 
~/condor/dist/etc/local_params.config_eons_long_queue_off_calib_prd_ntp_highprio_on 
 
 
CHECK THIS IS GOOD CONFIGURATION ON ONE NODE 
condor_reconfig fcdfcaf900 
condor_config_val -startd fcdfcaf900  CAF_DEFAULT_START 
condor_status fcdfcaf900 -long | grep -i start | more 
 
THEN ISSUED COMMAND FOR ALL NODES  
condor_reconfig -all 
 
THEN MONITOR TO MAKE SURE ITS DOING WHAT YOU EXPECT 
http://cdfcaf.fnal.gov/groupcaf/history/index.html 
</pre>
<!-- Author: Dennis Box -->


<!-- Date: Fri Oct 3 14:07:26 2008 -->
<pre>I forgot to add to the above elog entry that it was for groupcaf fcdfhead3 and it was done to drain the long queue for mondays power outage. </pre>
<!-- Author: Dennis Box -->


<!-- Date: Mon Oct 6 12:34:47 2008 -->
<pre>Stop of all queues on fcdfhead3: 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
 
% cd ~/condor/dist/etc/ 
% cp local_params.config_all_queues_off local_params.config 
 
THEN ISSUED COMMAND FOR ALL NODES   
% condor_reconfig -all 
 
Btw, we've developed scripts last year which do that, stop long, medium, short/all queues, and procude log file to check the status afterwards. You can find the scripts in here as the pointer for next downtimes: 
 
% ls ~/condor/sh/ 
[cdfcaf@fcdfhead3 ~]$ ls ~/condor/sh/set* 
/home/cdfcaf/condor/sh/set_condor_rc.sh 
/home/cdfcaf/condor/sh/set_held_all_queues.sh 
/home/cdfcaf/condor/sh/set_held_all_queues_test.sh 
/home/cdfcaf/condor/sh/set_held_longq.sh 
/home/cdfcaf/condor/sh/set_held_longq.sh~ 
/home/cdfcaf/condor/sh/set_held_longq_test.sh 
/home/cdfcaf/condor/sh/set_held_medium_longq_partial.sh 
/home/cdfcaf/condor/sh/set_held_medium_longq.sh 
/home/cdfcaf/condor/sh/set_held_medium_longq.sh~ 
/home/cdfcaf/condor/sh/set_held_medium_longq_test.sh 
/home/cdfcaf/condor/sh/set_release_all_queues.sh 
/home/cdfcaf/condor/sh/set_release_all_queues_test.sh 
/home/cdfcaf/condor/sh/set_release_fixedwntest_sam_group_jobs.sh 
/home/cdfcaf/condor/sh/set_release_fixedwntest_sam_group_jobs.sh~ 
/home/cdfcaf/condor/sh/set_release_sam_group_jobs.sh 
/home/cdfcaf/condor/sh/set_release_sam_group_jobs.sh~ 
 
THEN MONITOR TO MAKE SURE ITS DOING WHAT YOU EXPECT  
http://cdfcaf.fnal.gov/groupcaf/history/index.html</pre>
<!-- Author: Marian -->


<!-- Date: Mon Oct 27 15:44:30 2008 -->
<pre>removed ~65 held jobs from fermigrid 
and ~100 held jobs from groupcaf 
used method documented in twiki 
These jobs were held because the directory that they wanted to send log files to no longer existed.  Steve Timm suggested on a jira ticket that this may be due to the log file directory being full at the time. 
 
The twiki does not document that to do this on fermigrid you have to source /home/cdfcaf/grid/sh/glidein_grid.sh and to do this on groupcaf you have to be root.   
 
We should ask Igor if this needs to be done on glideinwms and if so what are the idiosyncracies</pre>
<!-- Author: dbox -->


<!-- Date: Thu Oct 30 11:41:02 2008 -->
<pre>Took Krzyzstof out of caf.cfg at his request on fcdfhead10. 
Should update all user lists on all caf.cfgs</pre>
<!-- Author: dbox -->


<!-- Date: Thu Oct 30 12:07:19 2008 -->
<pre>(last night, forgot to log then) 
 
Edited the crontab files on fcdfhead2,3,5,and 10 to check for held condor jobs and create a jira ticket if they found some.  Was almost immediately rewarded with 2 new jira tickets for held jobs on groupcaf and fermigrid.  Investigating.</pre>
<!-- Author: dbox -->


<!-- Date: Thu Oct 30 12:09:33 2008 -->
<pre>Edited caf.cfg.fcdfhead10.fnal.gov to 
1) add cdfopr proxy (Ray Culbertson) 
2) change stager proxy (genser to bellavan) 
 
restarted submitter and asked Roman to test, if it works I will do same to fermigrid</pre>
<!-- Author: dbox -->


<!-- Date: Thu Oct 30 14:41:55 2008 -->
<pre>cleared another ~100 held jobs from groupcaf 
and 4 from fermigrid 
 
HoldCodeReasons where 12 (missing directory in /export/CafCondor/cafIn/), 13 (user did a condor_rm ) and 14 (missing directories in /export/CafCondor/cafIn/) 
 
</pre>
<!-- Author: dbox -->