<!-- Title: 22:42:41  Wed Apr 25 2007 -->
<!-- Crewchief: Doug Benjamin -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Test to see if we can make an e-log entry</pre>


<!-- Date: Thu Apr 26 07:58:54 2007 -->
<pre>test of elog</pre>
<!-- Author: peter -->


<!-- Date: Thu Apr 26 09:32:30 2007 -->
<pre>I am installing pacman v 3.19 on fcdfhead2 and fcdfhead5 
in the ~cdfcaf/pacman-3.19. 
 
I am also fetching and installing the OSG client software stack 
 
by: 
 
cdfcaf@fcdfhead2:~/grid/VDT] $ cd ~cdfcaf/pacman-3.19 
[cdfcaf@fcdfhead2:~/pacman-3.19] $ cd 
[cdfcaf@fcdfhead2:~] $ cd ~cdfcaf/pacman-3.19 
[cdfcaf@fcdfhead2:~/pacman-3.19] $ source setup.sh 
[cdfcaf@fcdfhead2:~/pacman-3.19] $ cd ~cdfcaf/grid 
[cdfcaf@fcdfhead2:~/grid] $ mkdir VDT 
[cdfcaf@fcdfhead2:~/grid] $ cd VDT 
 
[cdfcaf@fcdfhead2:~/grid/VDT] $ pacman -get OSG:client 
 
 
I answered yes to all of the cache questions and license question 
I also chose local for the CA files 
 
after the installation finished as root I did the following commands. 
 
[root@fcdfhead2 VDT]# cd ~cdfcaf/grid/VDT 
[root@fcdfhead2 VDT]# source setup.sh 
[root@fcdfhead2 VDT]# vdt-register-service --name fetch-crl --enable 
[root@fcdfhead2 VDT]# vdt-register-service --name vdt-rotate-logs --enable 
[root@fcdfhead2 VDT]# vdt-control --on fetch-crl 
[root@fcdfhead2 VDT]# vdt-control --on vdt-rotate-logs 
[root@fcdfhead2 VDT]# vdt-control --list 
Service            | Type   | Desired State 
-------------------+--------+-------------- 
fetch-crl          | cron   | enable 
condor             | init   | do not enable 
vdt-rotate-logs    | cron   | enable 
globus-ws          | init   | do not enable 
 
Then as cdfcaf account  
 
[cdfcaf@fcdfhead2:~/grid/VDT] $ cd ~cdfcaf/grid/sh/ 
[cdfcaf@fcdfhead2:~/grid/sh] $ rm setup_grid.sh 
[cdfcaf@fcdfhead2:~/grid/sh] $ ln -s setup_grid.sh.VDT setup_grid.sh 
 
 
Important things to note... It takes a while to get the  
the software from OSG and VDT 
 
perl has to v5.8 or greater for VDT 1.6.1 
 
On fcdfhead5 I moved the older version of perl from  
/usr/local/bin/ to a subdirectory - /usr/local/bin/perl_moved_to_here/ 
 
</pre>
<!-- Author: Doug Benjamin -->


<!-- Date: Thu Apr 26 11:08:56 2007 -->
<pre>eLog entry test</pre>
<!-- Author: Aureliano Rama -->


<!-- Date: Mon Apr 30 12:11:39 2007 -->
<pre>entry e-Log test </pre>
<!-- Author: fede -->


<!-- Date: Wed May 2 17:09:45 2007 -->
<pre>I am increasing the log file sizes 
 
[cdfcaf@fcdfhead3 etc]$ diff '#condor_config#' condor_config 
467c467 
&lt; MAX_NEGOTIATOR_LOG    = 1000000000 
--- 
&gt; MAX_NEGOTIATOR_LOG    = 100000000 
479c479 
&lt; MAX_SCHEDD_LOG                = 100000000 
--- 
&gt; MAX_SCHEDD_LOG                = 1000000000 
482c482 
&lt; MAX_SHADOW_LOG                = 100000000 
--- 
&gt; MAX_SHADOW_LOG                = 1000000000 
485c485 
&lt; MAX_STARTD_LOG                = 100000000 
--- 
&gt; MAX_STARTD_LOG                = 1000000000 
</pre>
<!-- Author: Doug B -->


<!-- Date: Wed May 2 17:18:27 2007 -->
<pre>I used condor_reconfig to increase the log file size 
 
[cdfcaf@fcdfhead3 scripts]$ condor_reconfig -subsystem collector 
Sent "Reconfig" command to local collector 
[cdfcaf@fcdfhead3 scripts]$ condor_reconfig -subsystem schedd 
Sent "Reconfig" command to local schedd 
</pre>
<!-- Author: Doug -->


<!-- Date: Thu May 3 06:54:14 2007 -->
<pre>In order to get the schedd log to really accept larger size 
I had to use the following reconfig command. 
 
condor_reconfig -name fcdfhead3.fnal.gov -schedd 
 
 
Each log schedd log file now lasts 3 hrs. so We now have between 
3 and 6 hrs of log file time</pre>
<!-- Author: Doug -->


<!-- Date: Thu May 3 07:26:04 2007 -->
<pre>Used the following commands to go through that active user logs 
on fcdfhead3 (groupcaf) and get a list of restarted jobs: 
 
 find . -type d -maxdepth 1 -name '*submit*' -exec grep -i '001 (' {}/job.log \; | awk '{print $2 " " $3 " " $4 " " $9}' | sort -n -k 1 &gt; /tmp/current_running_jobs_`date +%Y_%m_%d_%H%M`.txt 
 
ls -ltr /tmp/current_running_jobs_*.txt 
 
~/find-restarts.py /tmp/current_running_jobs_2007_05_03_0721.txt  
</pre>
<!-- Author: Doug -->


<!-- Date: Thu May 3 07:36:46 2007 -->
<pre>Increased the schedd log size to 9000000000 to get almost 1 
day of information to help trap restarts</pre>
<!-- Author: Doug B -->


<!-- Date: Fri May 4 06:42:16 2007 -->
<pre>In an effort to reduce the load on the head node condor services due to logging. 
 
  I removed the D_SECURITY setting from NEGOTIATOR_DEBUG, SCHEDD_DEBUG, SHADOW_DEBUG, STARTD_DEBUG 
 
This was done - 3-May 13:27. I then issued a condor reconfig</pre>
<!-- Author: Doug B -->


<!-- Date: Fri May 4 09:29:49 2007 -->
<pre>due to unstability of the condor software (submission took couple of hours, userjobs were constantly restarted, etc.) i completely restarted condor and caf software on head3 yesterday evening. it seems to help for some time, but looking at the performance, there are again some problems</pre>
<!-- Author: peter bednar -->


<!-- Date: Fri May 4 12:35:12 2007 -->
<pre>the instructions on how to do a condor/caf restart are on the caf twiki. 
http://www-cdf.fnal.gov/htbin/twiki/bin/view/Main/CAFDCAFFAQ#The_Ultimate_GroupCAF_re_start_H</pre>
<!-- Author: peter -->


<!-- Date: Fri May 4 13:33:25 2007 -->
<pre>Changed  
 
Condor config file changes - 
 
[cdfcaf@fcdfhead3 etc]$ diff condor_config~ condor_config 
479c479 
&lt; MAX_SCHEDD_LOG                = 2000000000 
--- 
&gt; MAX_SCHEDD_LOG                = 1000000000 
647c647,649 
&lt; #START = $(START) && (CurMatches == 0) 
--- 
&gt; # 04-May-07 uncomment start expersion Federica Moscato, Doug B 
&gt; 
&gt; START = $(START) && (CurMatches == 0) 
1262a1265,1266 
&gt; # 04-May-07 Federica M. and Doug Benjamin uncomment changes 
&gt; 
1265,1268c1269,1273 
&lt; #WantAdRevaluate = TRUE 
&lt; #CurMatches = 0 
&lt; #STARTD_EXPRS = COLLECTOR_HOST_STRING, WantAdRevaluate, CurMatches 
&lt; #STARTD_AD_REEVAL_EXPR = target.UpdateSequenceNumber &gt; my.UpdateSequenceNumber 
--- 
&gt; WantAdRevaluate = TRUE 
&gt; CurMatches = 0 
&gt; STARTD_EXPRS = COLLECTOR_HOST_STRING, WantAdRevaluate, CurMatches 
&gt; STARTD_AD_REEVAL_EXPR = target.UpdateSequenceNumber &gt; my.UpdateSequenceNumber 
&gt; 
 
commented out the lines: 
##MASTER_ADDRESS_FILE = $(LOG)/.master_address 
 
#SCHEDD_ADDRESS_FILE	= $(LOG)/.schedd_address 
 
 
Stopped the schedd's by killing them with the  
kill -9 &lt;PID&gt;  
after trying the command file  
~/condor/sh/stop_condor_schedd.sh schedd_100 (etc) 
 
restarted the schedd's 
 
</pre>
<!-- Author: Doug B and Federica Moscato -->
<!-- Comment: Fri May  4 13;34;00 comment by...Doug B -->
<pre>This work was for group caf fcdfhead3</pre>


<!-- Date: Fri May 4 14:08:27 2007 -->
<pre>Add changes to fcdfhead2 condor_config file like previous 
e-log entry.</pre>
<!-- Author: Frederica and Doug -->
<!-- Comment: Fri May  4 14;09;38 comment by...db and fm -->
<pre>Changed condor_config.glexec file also</pre>


<!-- Date: Fri May 4 15:13:38 2007 -->
<pre>added following submission points to fermigrid: 
fcdfosg2_fcdfsrv001_glexec 
fcdfosg2_fcdfsrv002_glexec 
fcdfosg2_fcdfsrv003_glexec 
 
Tested submission points and they seem to work 
 
changed caf.cfg to include the new submission points. 
 
restarted glidekeeper. 
 
We followed the instructions here: 
http://www-cdf.fnal.gov/htbin/twiki/bin/view/Main/CAFDCAFFAQ 
 
used as an example to create the submission points the file 
on fcdfhead5: 
~/CafCondor/glidein/scripts/creates/fnal.txt 
 
The actual submission point creation is in the file on fcdfhead2: 
~/CafCondor/glidein/create/fnal_osg2.txt 
 
 ./create_entry.sh fcdfosg2_fcdfsrv001_glexec -g fcdfosg2.fnal.gov/jobmanager-condor -s OSG -d schedd_glidecdf2@fcdfhead2.fnal.gov -w http://fcdfhead2.fnal.gov:8000/condorcafstage/ -p fcdfsrv001.fnal.gov:3128 -f /home/cdfsoft -t FermiGridCDF -i FNAL 
 
./create_entry.sh fcdfosg2_fcdfsrv002_glexec -g fcdfosg2.fnal.gov/jobmanager-condor -s OSG -d schedd_glidecdf2@fcdfhead2.fnal.gov -w http://fcdfhead2.fnal.gov:8000/condorcafstage/ -p fcdfsrv002.fnal.gov:3128 -f /home/cdfsoft -t FermiGridCDF -i FNAL 
 
 
./create_entry.sh fcdfosg2_fcdfsrv003_glexec -g fcdfosg2.fnal.gov/jobmanager-condor -s OSG -d schedd_glidecdf2@fcdfhead2.fnal.gov -w http://fcdfhead2.fnal.gov:8000/condorcafstage/ -p fcdfsrv003.fnal.gov:3128 -f /home/cdfsoft -t FermiGridCDF -i FNAL 
</pre>
<!-- Author: Doug and Frederica -->


<!-- Date: Sun May 6 22:22:34 2007 -->
<pre>at 17:39 today killed all cdfcaf processes using  
kill -9 -1.  After discovering only 4 vms running. 
Started the condor daemons and the glidekeeper.  
Forgot to start the rest of the CafCondor middleware. 
 
Discovered mistake with the fcdfosg2 submission points. 
 
To remove a submission point and reconfigure it. I did the following commands: 
 
 rm -Rf /export/CafCondor/cafIn/glidein_fcdfosg2_fcdfsrv001_glexec 
 rm -Rf /export/CafCondor/stage/glidein_fcdfosg2_fcdfsrv001_glexec 
 rm -Rf /export/CafCondor/stage/glidein_fcdfosg2_fcdfsrv002_glexec 
  rm -Rf /export/CafCondor/cafIn/glidein_fcdfosg2_fcdfsrv002_glexec 
  rm -Rf /export/CafCondor/cafIn/glidein_fcdfosg2_fcdfsrv003_glexec 
  rm -Rf /export/CafCondor/stage/glidein_fcdfosg2_fcdfsrv003_glexec 
 
 
  ./create_entry.sh fcdfosg2_fcdfsrv001_glexec -g fcdfosg2.fnal.gov/jobmanager-condor -s Condor -d schedd_glidecdf2@fcdfhead2.fnal.gov -w http://fcdfhead2.fnal.gov:8000/condorcafstage/ -p fcdfsrv001.fnal.gov:3128 -f /home/cdfsoft -t FermiGridCDF -i FNAL  -C /home/cdfcaf/condor/dist 
  ./create_entry.sh fcdfosg2_fcdfsrv002_glexec -g fcdfosg2.fnal.gov/jobmanager-condor -s Condor -d schedd_glidecdf2@fcdfhead2.fnal.gov -w http://fcdfhead2.fnal.gov:8000/condorcafstage/ -p fcdfsrv002.fnal.gov:3128 -f /home/cdfsoft -t FermiGridCDF -i FNAL  -C /home/cdfcaf/condor/dist 
  ./create_entry.sh fcdfosg2_fcdfsrv003_glexec -g fcdfosg2.fnal.gov/jobmanager-condor -s Condor -d schedd_glidecdf2@fcdfhead2.fnal.gov -w http://fcdfhead2.fnal.gov:8000/condorcafstage/ -p fcdfsrv003.fnal.gov:3128 -f /home/cdfsoft -t FermiGridCDF -i FNAL  -C /home/cdfcaf/condor/dist 
 
Tests of submission points as outlined in the twiki: 
http://www-cdf.fnal.gov/htbin/twiki/bin/view/Main/CAFDCAFFAQ#Adding_a_new_glide_in_based_site 
 
indicates an error with the squid server. 
 
cat log/job_dbg.201523.err 
Failed to load file 'signature.sha1' from 'http://fcdfhead2.fnal.gov:8000/condorcafstage//glidein_fcdfosg2_fcdfsrv003_glexec' using proxy 'fcdfsrv003.fnal.gov:3128' 
 
I do not have root access so I can not investigate the problems with the squid servers (all three are complaining!!!???) 
</pre>
<!-- Author: Doug Benjamin -->


<!-- Date: Mon May 7 10:23:02 2007 -->
</br><b>Installation on test 012 </b></br> 
 
We have installed the head node on test012, and the workerer nodes on test013, test014 and test015.</br></br> 
<i>Environment preparation </i>  
</br> 
We ask to system administration to have the following configuration:   
 
<ul> 
 
/export/CafCondor mounts as  /export/CafCondor         </br> 
/export/condor_l  mounts as  /export/condor_local   </br> 
/export/cdfcaf    mounts as  /export/cdfcaf         </br> 
 
 
 
</ul> </br> 
 
Inside auto.home add the following lines 
 
<ul> 
 
cdfsoft         -ro,soft,intr,nosuid,rsize=32768,nfsvers=3      fcdfcode1:/export/cdfsoft 
cdfcaf          -rw,soft,intr,rsize=32768,wsize=32768,nfsvers=3 fcdftest012:/export/cdfcaf 
</ul> 
 
where the first supplies cdfsoft to the head node and  the worker nodes, and the second  mounts via nfs home directory of cdfcaf   </br> 
Create cdfcaf user account and change the UID, GID of this account 10155:9510 
<ul> 
usermod -u 10155 cdfcaf</br> 
groupmod -g 9510 cdfcaf  </br> 
usermod -g cdfcaf cdfcaf 
</ul> 
Change the ownership of this directories(CafCondor,condor_local,cdfcaf) to  cdfcaf:cdfcaf 
</br> 
We need to export the software on the worker nodes </br> 
Inside  
 
/home/cdfcaf/exports in test012 add </br> 
<ul> 
 
/export/cdfcaf  131.225.211.224(rw,no_root_squash) \</br> 
                131.225.211.225(rw,no_root_squash) \</br> 
                131.225.211.226(rw,no_root_squash) \</br> 
                131.225.211.223(rw,no_root_squash) 
 
</ul> 
 
</br> 
CVS client installed  on test012 </br> 
yum install cvs </br></br>		 
 
<i>head node installation</i></br> 
 
 
 
 
For the Condor installation on head node, we had followed the instructions on this link 
<a href='http://cdfcaf.fnal.gov/doc/cafcondorInstallation/node8.html'>condor on head node </a> 
</br> 
Condor software download from this site  
 
<a href='http://www.cs.wisc.edu/condor/downloads/'>condor 6.8.4 for SL 4.4 (Intel86)</a></br> 
Condor configuration files modified (<a href='http://cdfcaf.fnal.gov/doc/cafcondorInstallation/node9.html'>condor_config</a>, local_params,<a href='http://cdfcaf.fnal.gov/doc/cafcondorInstallation/node10.html'>fcdftest012.local</a>)</br> 
Condor installation on head node, following the steps at this link 
<a href='http://cdfcaf.fnal.gov/doc/cafcondorInstallation/node13.html'>condor_on_head_node</a>.</br> 
The instructions on how to do CafCondor installation are  on <a href='http://cdfcaf.fnal.gov/doc/cafcondorInstallation/node16.html'>Installing CafCondor </a></br> 
Change the /home/cdfcaf/CafCondor/submitter/SubmitModule.py like and the condor_config like  Fri May 4 13:33:25 eLog entry. </br> 
 
<u>note:</u> we had a problem of python library mismatching.It was fixed using, with root privileges,</br> 
<ul>yum  install compat-libstdc++-33.i386 </ul></br> 
 
 
</br></br> 
<i>worker node installation</i></br> 
The worker nodes installed are test013, test014, test015.</br></br> 
For all the machines used like a worker node: 
<p> 
 
 inside etc/auto.home </br> 
 <ul> 
cdfcaf          -rw,soft,intr,rsize=32768,wsize=32768,nfsvers=3 fcdftest012:/export/cdfcaf 
cdfsoft         -ro,soft,intr,nosuid,rsize=32768,nfsvers=3      fcdfcode1:/export/cdfsoft 
</ul> 
inside etc/mtab add the follow line</br> 
<ul> 
 
fcdftest012:/export/cdfcaf /home/cdfcaf nfs rw,soft,intr,rsize=32768,wsize=32768,nfsvers=3,addr=131.225.211.223 0 0 
 
</ul> 
 
 
 
</p> 
 
 
 
 
</br></br> 
 
 
 
<!-- 
<a href='http://cdfcaf.fnal.gov/doc/cafcondorInstallation/node13.html'>  
  head node installation </a> 
 
</br> 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
<a href='http://cdfcaf.fnal.gov/doc/cafcondorInstallation/cafcondorInstallation.html'> Condor CAF Installation Manual</a>-->
<!-- Author: Peter and Federica -->


<!-- Date: Mon May 7 10:46:34 2007 -->
<pre>Submitting a test debug job to each of the submission points 
 
here is the ClassAd 
 
head2&gt; cat job_dbg.ClassAd 
Universe = grid 
Grid_Resource = gt2 fcdfosg2.fnal.gov/jobmanager-condor 
 
 
Executable = glidein_startup.sh 
Arguments = dbg $(Cluster) schedd_glidecdf2@fcdfhead2.fnal.gov fcdfhead2.fnal.gov Condor http://fcdfhead2.fnal.gov:8000/condorcafstage//glidein_fcdfosg2_fcdfsrv003_glexec 1c1643a32387b4aa26d27a9251cd5a4231b3e758 fcdfsrv003.fnal.gov:3128 
 
Transfer_Executable = True 
transfer_Input_files = 
transfer_Output_files = 
WhenToTransferOutput = ON_EXIT 
 
Notification = Never 
 
+Owner = undefined 
+CAFGlideIn = "fcdfosg2_fcdfsrv003_glexec" 
 
Log = job.log 
Output = log/job_dbg.$(Cluster).out 
Error = log/job_dbg.$(Cluster).err 
 
# You can try to enable it, if your site supports it 
stream_output = False 
stream_error  = False 
 
Queue 
 
*********************** 
 
 
 
*********************** 
 
head2&gt;   source ~/condor/sh/condor_setup.sh; source ~/grid/sh/interactive_grid.sh; 
head2&gt; condor_submit -name `cat job.schedd` job_dbg.ClassAd; 
Submitting job(s). 
Logging submit event(s). 
1 job(s) submitted to cluster 644028. 
head2&gt; condor_reschedule -name `cat job.schedd`; 
Sent "Reschedule" command to schedd schedd_glidecdf1@fcdfhead2.fnal.gov 
head2&gt; cd ../glidein_fcdfosg1_fcdfsrv002_glexec/ 
head2&gt; condor_submit -name `cat job.schedd` job_dbg.ClassAd; 
Submitting job(s). 
Logging submit event(s). 
1 job(s) submitted to cluster 644029. 
head2&gt; condor_reschedule -name `cat job.schedd`; 
Sent "Reschedule" command to schedd schedd_glidecdf1@fcdfhead2.fnal.gov 
head2&gt; cd ../glidein_fcdfosg1_fcdfsrv003_glexec/ 
head2&gt; condor_submit -name `cat job.schedd` job_dbg.ClassAd; 
Submitting job(s). 
Logging submit event(s). 
1 job(s) submitted to cluster 644030. 
head2&gt; condor_reschedule -name `cat job.schedd`; 
Sent "Reschedule" command to schedd schedd_glidecdf1@fcdfhead2.fnal.gov 
head2&gt; cd ../glidein_fcdfosg2_fcdfsrv001_glexec/ 
head2&gt; condor_submit -name `cat job.schedd` job_dbg.ClassAd; 
Submitting job(s). 
Logging submit event(s). 
1 job(s) submitted to cluster 204104. 
head2&gt; condor_reschedule -name `cat job.schedd`; 
Sent "Reschedule" command to schedd schedd_glidecdf2@fcdfhead2.fnal.gov 
head2&gt; cd ../glidein_fcdfosg2_fcdfsrv002_glexec/ 
head2&gt; condor_submit -name `cat job.schedd` job_dbg.ClassAd; 
Submitting job(s). 
Logging submit event(s). 
1 job(s) submitted to cluster 204105. 
head2&gt; condor_reschedule -name `cat job.schedd`; 
Sent "Reschedule" command to schedd schedd_glidecdf2@fcdfhead2.fnal.gov 
head2&gt; cd ../glidein_fcdfosg2_fcdfsrv003_glexec/ 
head2&gt; condor_submit -name `cat job.schedd` job_dbg.ClassAd; 
Submitting job(s). 
Logging submit event(s). 
1 job(s) submitted to cluster 204106. 
head2&gt; condor_reschedule -name `cat job.schedd`; 
Sent "Reschedule" command to schedd schedd_glidecdf2@fcdfhead2.fnal.gov 
</pre>
<!-- Author: Doug B -->


<!-- Date: Mon May 7 10:48:01 2007 -->
<pre>Reconfigured the fcdfosg2 submission points to be: 
 
  ./create_entry.sh fcdfosg2_fcdfsrv001_glexec -g fcdfosg2.fnal.gov/jobmanager-condor -s OSG -d schedd_glidecdf2@fcdfhead2.fnal.gov -w http://fcdfhead2.fnal.gov/condorcafstage/ -p fcdfsrv001.fnal.gov:3128 -f /home/cdfsoft -t FermiGridCDF -i FNAL  -C /home/cdfcaf/condor/dist 
 
   ./create_entry.sh fcdfosg2_fcdfsrv002_glexec -g fcdfosg2.fnal.gov/jobmanager-condor -s OSG -d schedd_glidecdf2@fcdfhead2.fnal.gov -w http://fcdfhead2.fnal.gov/condorcafstage/ -p fcdfsrv002.fnal.gov:3128 -f /home/cdfsoft -t FermiGridCDF -i FNAL  -C /home/cdfcaf/condor/dist 
 
   ./create_entry.sh fcdfosg2_fcdfsrv003_glexec -g fcdfosg2.fnal.gov/jobmanager-condor -s OSG -d schedd_glidecdf2@fcdfhead2.fnal.gov -w http://fcdfhead2.fnal.gov/condorcafstage/ -p fcdfsrv003.fnal.gov:3128 -f /home/cdfsoft -t FermiGridCDF -i FNAL  -C /home/cdfcaf/condor/dist 
 
 
Some indications that there are problems with fcdfsrv003. Will investigate.</pre>
<!-- Author: Doug B -->


<!-- Date: Mon May 7 10:58:13 2007 -->
<pre>Attempted to restart squid server on fcdfsrv003. Found it down. 
 
Squid could not start due to missing directory. 
 
on fcdfsrv001  
[root@fcdfsrv001 root]# ls -l /export 
lrwxrwxrwx    1 root     root           16 Mar 14  2006 /export -&gt; /scratch_dcache/ 
 
[root@fcdfsrv001 root]# df -h 
Filesystem            Size  Used Avail Use% Mounted on 
/dev/hda2             9.7G  1.4G  7.8G  15% / 
/dev/hda1             494M   19M  450M   4% /boot 
/dev/hdb1              39G  242M   37G   1% /fbsng 
none                  1.5G     0  1.5G   0% /dev/shm 
/dev/hdb2             191G   36G  146G  20% /scratch_dcache 
/dev/hda3             4.9G   33M  4.6G   1% /tmp 
/dev/hda5             4.9G  265M  4.4G   6% /var 
 
While on fcdfsrv003 
 
Filesystem            Size  Used Avail Use% Mounted on 
/dev/hda2             9.7G  1.4G  7.9G  15% / 
/dev/hda1             494M   19M  450M   4% /boot 
none                  1.5G     0  1.5G   0% /dev/shm 
/dev/hda3             4.9G   33M  4.6G   1% /tmp 
/dev/hda5             4.9G  765M  3.9G  17% /var 
 
[root@fcdfsrv003 root]# ls -l /export 
lrwxrwxrwx    1 root     root           15 Mar 15  2006 /export -&gt; /scratch_dcache 
 
The /scratch_dcache partition will not mount.  I will log  
a help desk ticket 
 
 
 
</pre>
<!-- Author: Doug B -->


<!-- Date: Mon May 7 11:15:52 2007 -->
<pre>Right now fcdfsrv003 is missing the partition /scratch_dcache 
 
I am sending email to FEF</pre>
<!-- Author: Doug B -->


<!-- Date: Mon May 7 11:44:36 2007 -->
<pre>we have finally installed the testcaf. the condor/Caf is running, but we have problems adding users. the CafAdmin.py gives following error (with or without any 
paramters): 
 
*** glibc detected *** free(): invalid pointer: 0xb7ec98f0 *** 
./CafAdmin: line 10: 17136 Aborted ${CAF_ROOT}/admin/${PROGNAME}.py "$@" 
 
when it gets to the command krb5.get_login_principal().name() 
</pre>
<!-- Author: Peter and Federica -->


<!-- Date: Mon May 7 12:21:15 2007 -->
<pre>Given that the partition table was damaged on fcdfsrv003 /dev/hdb 
I created a new table with just one partition and mounted it. 
I then created the directories that the SQUID server needed and 
(owned by squid:squid). 
 
I issued the squid -z command to create the needed cache 
and issued the /etc/init.d/squid start command to restart 
the squid server.  
 
I will look at the logs later to check for success. 
 
</pre>
<!-- Author: Doug B -->


<!-- Date: Wed May 9 18:04:20 2007 -->
<pre>the partition holding secondary schedds directories on head3 (/export/condor_local) got full and schedds were crashing because they didn't have space for logs. i moved the schedd_100, schedd_101, schedd_102 and schedd_103 directories to /export/CafCondor/CafCondor_logs/ and made symlinks from the original location</pre>
<!-- Author: peter -->


<!-- Date: Wed May 9 19:07:14 2007 -->
<pre>max_submits - modified in the glidekeeper.py file to be 
100 instead of 20. Perhaps this will cause the fermigrid caf 
to fill with jobs. 
 
Also changed the CA_DIR in the caf.cfg 
; Where the CA signing files can be found 
; MUST set if you enable GSI 
; The default is good if you use UIPnP 
;CA_DIR=/home/cdfcaf/grid/UIPnP/files/etc/grid-security/certificates 
CA_DIR=/home/cdfcaf/grid/VDT/globus/TRUSTED_CA 
 
</pre>
<!-- Author: Doug B -->


<!-- Date: Wed May 9 19:45:51 2007 -->
<pre>TestCaf configuration 
 
On head node (test 010) 
 
 
-blue3.fnal.gov:/export/CafCondor/fcdftest010 mounts as  /home/cdfcaf 
 
-Inside auto.home  
 
  cdfcaf          -rw,intr,hard,async,tcp,wsize=32768,rsize=32768    blue3.fnal.gov:/export/CafCondor/fcdftest010 
 
- cp home/cdfcaf/.k5login  /blue/CafCondor/fcdftest010/. 
 
- created 2 new directories   
 
   /fbsng/CafCondor_test 
   /fbsng/condor_local_test 
 
 owned by cdfcaf:cdfcaf 
 
Inside /export made soft links:  
  CafCondor -&gt; /fbsng/CafCondor_test 
  condor_local -&gt; /fbsng/condor_local_test 
 
- copied from test 012 the CafCondor and condor directories 
 
- Inside /home/cdfcaf/condor made soft links: 
 
   dist -&gt; condor-6.8.4 
   sh -&gt; ../CafCondor/condor/scripts/ 
 
 
- modified  configuration file  caf.cfg 
 
- in /home/cdfcaf/CafCondor/icaf made a soft link  
  
   icaf.cfg -&gt; /home/cdfsoft/dist/releases/development/CafUtil/cafusers/icaf.cfg 
  
- copied the following directory  /export/CafCondor/tickets 
 
 
 
On the WN (test 013,014,015) 
 
- in etc/mtab :  
 
  blue3.fnal.gov:/export/CafCondor/fcdftest010 /home/cdfcaf nfs rw,soft,intr,rsize=32768,wsize=32768,nfsvers=3,addr=131.225.111.105 0 0 
 
-Inside etc/auto.home comment the line : 
 
   #cdfcaf         -rw,soft,intr,rsize=32768,wsize=32768,nfsvers=3 fcdftest012:/export/cdfcaf 
 
 
</pre>
<!-- Author: Peter and Federica -->


<!-- Date: Thu May 10 18:53:52 2007 -->
<pre>Hugh load on groupcaf has forced me to stop the  
submitter.</pre>
<!-- Author: Doug B -->


<!-- Date: Wed May 16 23:07:41 2007 -->
<pre>steps used to compile CafExe 
 
 1001  source ~cdfsoft/cdf2.shrc 
 1002  setup cdfsoft2 development 
 1003  setup sam_cpp_api v7_4_3_sl3 -q GCC-3.4.3 
 1005  cd ~/CafCondor/CafExe/ 
 1007  gmake clean 
 1008  gmake 
</pre>
<!-- Author: Doug B -->


<!-- Date: Sun May 20 16:11:35 2007 -->
<pre>I had 3 new submission point on nancaf  
 
 with the following scripts  
 
 
./create_entry.sh ucsd2_100 -g osg-gw-2.t2.ucsd.edu/jobmanager-condor -s OSG -d schedd_glidesdsc@fcdfhead5.fnal.gov -w http://fcdfhead5.fnal.gov:8000/condorcafstage -p clarens-1.local:3128 -b 131.225.211.232 -T 0 -f /code/osgcode/cdf/cdfsoft -t SDSC -k /export/CafCondor/stage/krb5.tgz -c /export/CafCondor/stage/compat_libs_ext.tgz 
 
 
 
with this response  
 
&gt;&gt;  
 
 Using default head node: fcdfhead5.fnal.gov 
Creating     : /export/CafCondor/cafIn/glidein_ucsd2_100 
Tar balls in : /export/CafCondor/stage/glidein_ucsd2_100 
Gatekeeper   : osg-gw-2.t2.ucsd.edu/jobmanager-condor 
RSL          : 
WN local dir : OSG 
Check 
/export/CafCondor/stage/glidein_ucsd2_100/glidein_config 
and 
/export/CafCondor/stage/glidein_ucsd2_100/local_params.config 
and configure as needed. 
 
 
./create_entry.sh ucsd2_101 -g osg-gw-2.t2.ucsd.edu/jobmanager-condor -s OSG -d schedd_glidesdsc@fcdfhead5.fnal.gov -w http://fcdfhead5.fnal.gov:8000/condorcafstage -p clarens-1.local:3128 -b 131.225.211.232 -T 0 -f /code/osgcode/cdf/cdfsoft -t SDSC -k /export/CafCondor/stage/krb5.tgz -c /export/CafCondor/stage/compat_libs_ext.tgz 
 
&gt;&gt; 
   Using default head node: fcdfhead5.fnal.gov 
Creating     : /export/CafCondor/cafIn/glidein_ucsd2_101 
Tar balls in : /export/CafCondor/stage/glidein_ucsd2_101 
Gatekeeper   : osg-gw-2.t2.ucsd.edu/jobmanager-condor 
RSL          : 
WN local dir : OSG 
Check 
/export/CafCondor/stage/glidein_ucsd2_101/glidein_config 
and 
/export/CafCondor/stage/glidein_ucsd2_101/local_params.config 
and configure as needed. 
 
Done 
 
 
./create_entry.sh ucsd2_102 -g osg-gw-2.t2.ucsd.edu/jobmanager-condor -s OSG -d schedd_glidesdsc@fcdfhead5.fnal.gov -w http://fcdfhead5.fnal.gov:8000/condorcafstage -p clarens-1.local:3128 -b 131.225.211.232 -T 0 -f /code/osgcode/cdf/cdfsoft -t SDSC -k /export/CafCondor/stage/krb5.tgz -c /export/CafCondor/stage/compat_libs_ext.tgz 
 
&gt;&gt; 
 
  
 Using default head node: fcdfhead5.fnal.gov 
Creating     : /export/CafCondor/cafIn/glidein_ucsd2_102 
Tar balls in : /export/CafCondor/stage/glidein_ucsd2_102 
Gatekeeper   : osg-gw-2.t2.ucsd.edu/jobmanager-condor 
RSL          : 
WN local dir : OSG 
Check 
/export/CafCondor/stage/glidein_ucsd2_102/glidein_config 
and 
/export/CafCondor/stage/glidein_ucsd2_102/local_params.config 
and configure as needed. 
 
Done 
 
 
 
add the new submission points inside caf.cfg 
 
 restart glidekeeper 
 
i tried to test the submission point and the output is  
 
  ERROR: Can't find address of schedd schedd_glidesdsc@fcdfhead5.fnal.gov   
 
i did  
 condor_q -name schedd_@ 
 Error: Collector has no record of schedd/submitter</pre>
<!-- Author: federica -->


<!-- Date: Tue May 22 11:49:14 2007 -->
<pre>For fix the namcaf : 
 
kill -1 -9  
 
after cd ~/condor/sh/ directory 
./start_fcdfhead5.sh  
 
 
I found a problem with the apache. 
 
Peter start the apache and glidekeeper. 
 
Namcaf is fixed. 
 
 </pre>
<!-- Author: Federica -->


<!-- Date: Tue May 22 11:50:54 2007 -->
<pre>I removed fcdfcaf480  
 
from ~cdfcaf/condor/dist/etc/rooster file</pre>
<!-- Author: Federica -->


<!-- Date: Thu May 24 19:46:49 2007 -->
<pre>moving collector and negotiator to different node: 
 
changes in condor_config on the main headnode: 
HOSTALLOW_NEGOTIATOR_SCHEDD = $(CONDOR_HOST), $(FLOCK_NEGOTIATOR_HOSTS), $(NEGOTIATOR_HOST) 
COLLECTOR_HOST = $(CONDOR_HOST),fcdfhead4.fnal.gov 
NEGOTIATOR_HOST = $(CONDOR_HOST),fcdfhead4.fnal.gov 
 
after changes, issue 
condor_reconfig -all 
 
the secondary headnode has to have its own condor copy (own config files). in headnode.local 
DAEMON_LIST   = MASTER, COLLECTOR, NEGOTIATOR 
 
start collector and negotiator on the secondary headnode. 
issue condor_reconfig -all on the primary headnode (should see reconfiguring also negotiator and collector of the secondary) 
 
then kill the collector/negotiator with their respective masters on primary headnode 
 
system keeps running using the secondary stuff.</pre>
<!-- Author: peter -->


<!-- Date: Thu May 24 21:42:18 2007 -->
<pre>found out, that i'm no longer able to submitt jobs after doing the previous steps</pre>
<!-- Author: pbednar -->


<!-- Date: Fri May 25 08:30:47 2007 -->
<pre>text used to try to debug help jobs 909970 (Starter/Shadow exception) 
911361 - missing initial dir 
 
cd /export/CafCondor/cafIn/history/jobs_331xxxx 
 
find . -type d -maxdepth 1 -name '*job*' -exec grep 911361 {}/job.dagman.dagman.out \; -exec echo {} \; 
 
</pre>
<!-- Author: Doug Benjamin -->
<!-- Comment: Fri May 25 09;08;17 comment by...Doug -->
<pre>in looking at the job.log file for  ./job_3316547

I see 
...
000 (911361.000.000) 05/24 18:40:09 Job submitted from host: &lt;131.225.211.116:35616&gt;
    DAG Node: Section_37
...
000 (911362.000.000) 05/24 18:40:09 Job submitted from host: &lt;131.225.211.116:35616&gt;
    DAG Node: Section_37

Thus the same section summitted twice right away

000 (909970.000.000) 05/24 13:33:16 Job submitted from host: &lt;131.225.211.116:60842&gt;
    DAG Node: Section_20
001 (909970.000.000) 05/24 18:40:21 Job executing on host: &lt;131.225.209.139:51431&gt;
...

but no entry of 909970 terminating



Also in job.dagman.dagman.log

5/24 18:27:10 ERROR: node Section_21: job ID in userlog submit event (911227.0) doesn't match ID reported earlier by submit command (911226.0)!  Trusting the userlog for now, but this is scary!
5/24 18:27:10 ERROR: node Section_22: job ID in userlog submit event (911237.0) doesn't match ID reported earlier by submit command (911235.0)!  Trusting the userlog for now, but this is scary!
5/24 18:27:10 ERROR: node Section_23: job ID in userlog submit event (911242.0) doesn't match ID reported earlier by submit command (911248.0)!  Trusting the userlog for now, but this is scary!
5/24 18:27:10 ERROR: node Section_23: job ID in userlog submit event (911248.0) doesn't match ID reported earlier by submit command (911242.0)!  Trusting the userlog for now, but this is scary!
5/24 18:27:10 ERROR: node Section_24: job ID in userlog submit event (911256.0) doesn't match ID reported earlier by submit command (911255.0)!  Trusting the userlog for now, but this is scary!
5/24 18:27:10 ERROR: node Section_25: job ID in userlog submit event (911269.0) doesn't match ID reported earlier by submit command (911262.0)!  Trusting the userlog for now, but this is scary!
5/24 18:27:10 ERROR: node Section_21: job ID in userlog submit event (911226.0) doesn't match ID reported earlier by submit command (911227.0)!  Trusting the userlog for now, but this is scary!
5/24 18:27:10 ERROR: node Section_21: job ID in userlog submit event (911227.0) doesn't match ID reported earlier by submit command (911226.0)!  Trusting the userlog for now, but this is scary!
5/24 18:27:10 ERROR: node Section_22: job ID in userlog submit event (911235.0) doesn't match ID reported earlier by submit command (911237.0)!  Trusting the userlog for now, but this is scary!
5/24 18:27:10 ERROR: node Section_22: job ID in userlog submit event (911237.0) doesn't match ID reported earlier by submit command (911235.0)!  Trusting the userlog for now, but this is scary!
5/24 18:27:10 ERROR: node Section_23: job ID in userlog submit event (911248.0) doesn't match ID reported earlier by submit command (911242.0)!  Trusting the userlog for now, but this is scary!
5/24 18:27:10 ERROR: node Section_24: job ID in userlog submit event (911255.0) doesn't match ID reported earlier by submit command (911256.0)!  Trusting the userlog for now, but this is scary!
5/24 18:27:10 ERROR: node Section_24: job ID in userlog submit event (911256.0) doesn't match ID reported earlier by submit command (911255.0)!  Trusting the userlog for now, but this is scary!
5/24 18:27:10 ERROR: node Section_25: job ID in userlog submit event (911262.0) doesn't match ID reported earlier by submit command (911269.0)!  Trusting the userlog for now, but this is scary!
5/24 18:27:10 ERROR: node Section_25: job ID in userlog submit event (911269.0) doesn't match ID reported earlier by submit command (911262.0)!  Trusting the userlog for now, but this is scary!
5/24 18:31:13 ERROR: node Section_26: job ID in userlog submit event (911289.0) doesn't match ID reported earlier by submit command (911288.0)!  Trusting the userlog for now, but this is scary!
5/24 18:31:13 ERROR: node Section_27: job ID in userlog submit event (911296.0) doesn't match ID reported earlier by submit command (911295.0)!  Trusting the userlog for now, but this is scary!
5/24 18:31:13 ERROR: node Section_28: job ID in userlog submit event (911303.0) doesn't match ID reported earlier by submit command (911302.0)!  Trusting the userlog for now, but this is scary!
5/24 18:31:13 ERROR: node Section_29: job ID in userlog submit event (911312.0) doesn't match ID reported earlier by submit command (911309.0)!  Trusting the userlog for now, but this is scary!
5/24 18:31:13 ERROR: node Section_30: job ID in userlog submit event (911316.0) doesn't match ID reported earlier by submit command (911318.0)!  Trusting the userlog for now, but this is scary!
5/24 18:31:13 ERROR: node Section_30: job ID in userlog submit event (911318.0) doesn't match ID reported earlier by submit command (911316.0)!  Trusting the userlog for now, but this is scary!
5/24 18:31:19 ERROR: node Section_26: job ID in userlog submit event (911288.0) doesn't match ID reported earlier by submit command (911289.0)!  Trusting the userlog for now, but this is scary!
5/24 18:31:19 ERROR: node Section_26: job ID in userlog submit event (911289.0) doesn't match ID reported earlier by submit command (911288.0)!  Trusting the userlog for now, but this is scary!
5/24 18:31:19 ERROR: node Section_27: job ID in userlog submit event (911295.0) doesn't match ID reported earlier by submit command (911296.0)!  Trusting the userlog for now, but this is scary!
5/24 18:31:19 ERROR: node Section_27: job ID in userlog submit event (911296.0) doesn't match ID reported earlier by submit command (911295.0)!  Trusting the userlog for now, but this is scary!
5/24 18:31:19 ERROR: node Section_28: job ID in userlog submit event (911302.0) doesn't match ID reported earlier by submit command (911303.0)!  Trusting the userlog for now, but this is scary!
5/24 18:31:19 ERROR: node Section_28: job ID in userlog submit event (911303.0) doesn't match ID reported earlier by submit command (911302.0)!  Trusting the userlog for now, but this is scary!
5/24 18:31:19 ERROR: node Section_29: job ID in userlog submit event (911309.0) doesn't match ID reported earlier by submit command (911312.0)!  Trusting the userlog for now, but this is scary!
5/24 18:31:19 ERROR: node Section_29: job ID in userlog submit event (911312.0) doesn't match ID reported earlier by submit command (911309.0)!  Trusting the userlog for now, but this is scary!
5/24 18:31:19 ERROR: node Section_30: job ID in userlog submit event (911318.0) doesn't match ID reported earlier by submit command (911316.0)!  Trusting the userlog for now, but this is scary!
5/24 18:33:13 ERROR: node Section_31: job ID in userlog submit event (911326.0) doesn't match ID reported earlier by submit command (911321.0)!  Trusting the userlog for now, but this is scary!
5/24 18:33:13 ERROR: node Section_32: job ID in userlog submit event (911332.0) doesn't match ID reported earlier by submit command (911327.0)!  Trusting the userlog for now, but this is scary!
5/24 18:33:13 ERROR: node Section_33: job ID in userlog submit event (911338.0) doesn't match ID reported earlier by submit command (911333.0)!  Trusting the userlog for now, but this is scary!
5/24 18:33:13 ERROR: node Section_34: job ID in userlog submit event (911342.0) doesn't match ID reported earlier by submit command (911337.0)!  Trusting the userlog for now, but this is scary!
5/24 18:33:52 From submit: Submitting job(s).
5/24 18:33:52 From submit: Logging submit event(s).
5/24 18:33:52 From submit: 1 job(s) submitted to cluster 911348.
5/24 18:33:53 ERROR: node Section_35: job ID in userlog submit event (911348.0) doesn't match ID reported earlier by submit command (911344.0)!  Trusting the userlog for now, but this is scary!
5/24 18:33:57 ERROR: node Section_31: job ID in userlog submit event (911321.0) doesn't match ID reported earlier by submit command (911326.0)!  Trusting the userlog for now, but this is scary!
5/24 18:33:57 ERROR: node Section_31: job ID in userlog submit event (911326.0) doesn't match ID reported earlier by submit command (911321.0)!  Trusting the userlog for now, but this is scary!
5/24 18:33:57 ERROR: node Section_32: job ID in userlog submit event (911327.0) doesn't match ID reported earlier by submit command (911332.0)!  Trusting the userlog for now, but this is scary!
5/24 18:33:57 ERROR: node Section_32: job ID in userlog submit event (911332.0) doesn't match ID reported earlier by submit command (911327.0)!  Trusting the userlog for now, but this is scary!
5/24 18:33:57 ERROR: node Section_33: job ID in userlog submit event (911333.0) doesn't match ID reported earlier by submit command (911338.0)!  Trusting the userlog for now, but this is scary!
5/24 18:33:57 ERROR: node Section_34: job ID in userlog submit event (911337.0) doesn't match ID reported earlier by submit command (911342.0)!  Trusting the userlog for now, but this is scary!
5/24 18:33:57 ERROR: node Section_33: job ID in userlog submit event (911338.0) doesn't match ID reported earlier by submit command (911333.0)!  Trusting the userlog for now, but this is scary!
5/24 18:33:57 ERROR: node Section_34: job ID in userlog submit event (911342.0) doesn't match ID reported earlier by submit command (911337.0)!  Trusting the userlog for now, but this is scary!
5/24 18:33:57 ERROR: node Section_35: job ID in userlog submit event (911344.0) doesn't match ID reported earlier by submit command (911348.0)!  Trusting the userlog for now, but this is scary!
5/24 18:33:57 ERROR: node Section_35: job ID in userlog submit event (911348.0) doesn't match ID reported earlier by submit command (911344.0)!  Trusting the userlog for now, but this is scary!
5/24 18:41:15 ERROR: node Section_36: job ID in userlog submit event (911358.0) doesn't match ID reported earlier by submit command (911359.0)!  Trusting the userlog for now, but this is scary!
5/24 18:41:15 ERROR: node Section_36: job ID in userlog submit event (911359.0) doesn't match ID reported earlier by submit command (911358.0)!  Trusting the userlog for now, but this is scary!
5/24 18:41:15 ERROR: node Section_37: job ID in userlog submit event (911361.0) doesn't match ID reported earlier by submit command (911362.0)!  Trusting the userlog for now, but this is scary!
5/24 18:41:15 ERROR: node Section_37: job ID in userlog submit event (911362.0) doesn't match ID reported earlier by submit command (911361.0)!  Trusting the userlog for now, but this is scary!
5/24 18:41:15 ERROR: node Section_38: job ID in userlog submit event (911364.0) doesn't match ID reported earlier by submit command (911365.0)!  Trusting the userlog for now, but this is scary!
5/24 18:41:15 ERROR: node Section_38: job ID in userlog submit event (911365.0) doesn't match ID reported earlier by submit command (911364.0)!  Trusting the userlog for now, but this is scary!
5/24 18:41:15 ERROR: node Section_39: job ID in userlog submit event (911368.0) doesn't match ID reported earlier by submit command (911369.0)!  Trusting the userlog for now, but this is scary!
5/24 18:41:15 ERROR: node Section_39: job ID in userlog submit event (911369.0) doesn't match ID reported earlier by submit command (911368.0)!  Trusting the userlog for now, but this is scary!
5/24 18:41:15 ERROR: node Section_40: job ID in userlog submit event (911373.0) doesn't match ID reported earlier by submit command (911375.0)!  Trusting the userlog for now, but this is scary!
5/24 18:41:15 ERROR: node Section_40: job ID in userlog submit event (911375.0) doesn't match ID reported earlier by submit command (911373.0)!  Trusting the userlog for now, but this is scary!
5/24 18:41:15 ERROR: node Section_36: job ID in userlog submit event (911359.0) doesn't match ID reported earlier by submit command (911358.0)!  Trusting the userlog for now, but this is scary!
5/24 18:41:15 ERROR: node Section_37: job ID in userlog submit event (911362.0) doesn't match ID reported earlier by submit command (911361.0)!  Trusting the userlog for now, but this is scary!
5/24 18:41:15 ERROR: node Section_38: job ID in userlog submit event (911365.0) doesn't match ID reported earlier by submit command (911364.0)!  Trusting the userlog for now, but this is scary!
5/24 18:41:15 ERROR: node Section_39: job ID in userlog submit event (911369.0) doesn't match ID reported earlier by submit command (911368.0)!  Trusting the userlog for now, but this is scary!
5/24 18:41:15 ERROR: node Section_40: job ID in userlog submit event (911375.0) doesn't match ID reported earlier by submit command (911373.0)!  Trusting the userlog for now, but this is scary!
</pre>


<!-- Date: Fri May 25 09:17:29 2007 -->
<pre>this command in the directory about job_xxxxx dir's 
lists all of the jobs with double start of dag 
 find . -type d -maxdepth 1 -name '*job*' -exec grep -q 'match ID reported earlier' {}/job.dagman.dagman.out \; -exec echo {} \; 
 
[cdfcaf@fcdfhead3 jobs_330xxxx]$ grep 'ID reported earlier' ./job_3301041/job.dagman.dagman.out  
5/3 16:14:55 ERROR: node Section_1: job ID in userlog submit event (3301760.0) doesn't match ID reported earlier by submit command (3301759.0)!  Trusting the userlog for now, but this is scary! 
5/3 16:14:55 ERROR: node Section_2: job ID in userlog submit event (3301761.0) doesn't match ID reported earlier by submit command (3301762.0)!  Trusting the userlog for now, but this is scary! 
5/3 16:14:55 ERROR: node Section_2: job ID in userlog submit event (3301762.0) doesn't match ID reported earlier by submit command (3301761.0)!  Trusting the userlog for now, but this is scary! 
5/3 16:14:55 ERROR: node Section_3: job ID in userlog submit event (3301764.0) doesn't match ID reported earlier by submit command (3301763.0)!  Trusting the userlog for now, but this is scary! 
5/3 16:14:55 ERROR: node Section_4: job ID in userlog submit event (3301766.0) doesn't match ID reported earlier by submit command (3301765.0)!  Trusting the userlog for now, but this is scary! 
5/3 16:14:55 ERROR: node Section_5: job ID in userlog submit event (3301768.0) doesn't match ID reported earlier by submit command (3301767.0)!  Trusting the userlog for now, but this is scary! 
5/3 16:14:55 ERROR: node Section_1: job ID in userlog submit event (3301759.0) doesn't match ID reported earlier by submit command (3301760.0)!  Trusting the userlog for now, but this is scary! 
5/3 16:14:55 ERROR: node Section_1: job ID in userlog submit event (3301760.0) doesn't match ID reported earlier by submit command (3301759.0)!  Trusting the userlog for now, but this is scary! 
5/3 16:14:55 ERROR: node Section_2: job ID in userlog submit event (3301762.0) doesn't match ID reported earlier by submit command (3301761.0)!  Trusting the userlog for now, but this is scary! 
5/3 16:14:55 ERROR: node Section_3: job ID in userlog submit event (3301763.0) doesn't match ID reported earlier by submit command (3301764.0)!  Trusting the userlog for now, but this is scary! 
</pre>
<!-- Author: Doug -->


<!-- Date: Fri Jun 1 13:15:28 2007 -->
<pre>modify the configuration file for the list of the caf on cafmon  
for dcafmon 
 
in  
 
/home/cafmon/dcafmon/config/config.xml  
add the following farms 
 
 &lt; farm name="KORCAF" type="condorcaf" &gt; 
          &lt; head node="cluster46.knu.ac.kr"xml_port="8999" &gt; 
          &lt; web node="cluster46.knu.ac.kr" html_base="/condorcaf" cgi_base="/cgi-bin/condorcaf"/ &gt; 
  &lt; /farm &gt; 
 
 
  &lt; farm name="BCNCAF" type="condorcaf" &gt; 
     &lt; head node="cdf-bcnhead.pic.org.es"xml_port="8999"/ &gt; 
         &lt; web  node="cdf-bcnhead.pic.org.es" html_base="/condorcaf" cgi_base="/cgi-bin/condorcaf"/ &gt; 
 &lt; /farm &gt; 
 
 
Created a backup of old configuration  
 
in  
 
/home/cafmon/dcafmon/config/config.xml.06012007 
 
 
Now the output of CafAnalyze list is : 
 
 
&lt;fcdflnx4.fnal.gov&gt; ./CafAnalyze list 
 
ASCAF:http://ascaf.sinica.edu.tw/cgi-bin/condorcaf 
BCNCAF:http://cdf-bcnhead.pic.org.es/cgi-bin/condorcaf 
CAF:http://cdfcaf.fnal.gov/cgi-bin/groupcaf 
CNAFCAF:http://webserver.infn.it/cgi-bin/cdf/condorcaf 
FermiGrid:http://cdfcaf.fnal.gov/cgi-bin/glidecaf 
JPCAF:http://head1.tsukuba.jp.hep.net:8081/cgi-bin/condorcaf 
KORCAF:http://cluster46.knu.ac.kr/cgi-bin/condorcaf 
LyonCAF:http://webserver.infn.it/cgi-bin/cdf/lyoncaf 
MITCAF:http://cdfsrv2.mit.edu/cgi-bin/condorcaf 
RUTCAF:http://hexfm1.rutgers.edu/cgi-bin/condorcaf 
TORCAF:http://bigmac-www.physics.utoronto.ca/cgi-bin/condorcaf 
 
&lt;fcdflnx4.fnal.gov&gt; 
 
 
</pre>
<!-- Author: Federica -->


<!-- Date: Fri Jun 1 15:03:55 2007 -->
<pre>After the shutdown of the computer systems at FCC yesterday (31-May-07), forgot to restart the gcb servers 
 
my notes on restarting the gcb servers - 
 
ssh cdfcaf@ 
 
cd condor/sh 
./start_condor.sh 
-bash-2.05b$ ps ax | grep condor 
11306 ?        S      0:00 condor_master 
11307 ?        S      0:00 /home/cdfcaf/condor/dist/sbin/condor_procd -A /fbsng/condor_local/condor.lock/procd_pipe -P 11306 7455399 -C 10155 
11309 ?        S      0:00 /home/cdfcaf/condor/dist/libexec/gcb_relay_server -i 131.225.211.232 -p 32842 
11883 ?        S      0:00 /home/cdfcaf/condor/dist/libexec/gcb_relay_server -i 131.225.211.232 -p 32957 
12327 ?        S      0:00 /home/cdfcaf/condor/dist/libexec/gcb_relay_server -i 131.225.211.232 -p 33086 
12511 ?        S      0:00 /home/cdfcaf/condor/dist/libexec/gcb_relay_server -i 131.225.211.232 -p 33220 
12629 ?        S      0:00 /home/cdfcaf/condor/dist/libexec/gcb_relay_server -i 131.225.211.232 -p 33339 
12722 ?        S      0:00 /home/cdfcaf/condor/dist/libexec/gcb_relay_server -i 131.225.211.232 -p 33473 
12726 ?        S      0:00 /home/cdfcaf/condor/dist/libexec/gcb_relay_server -i 131.225.211.232 -p 33593 
13034 pts/0    S      0:00 grep condor 
</pre>
<!-- Author: Doug -->
<!-- Comment: Fri Jun  1 15;04;51 comment by...Doug B -->
<pre>The current gcb servers are:
fcdfsrv102
fcdfsrv103
fcdfsrv104</pre>


<!-- Date: Fri Jun 1 15:06:50 2007 -->
<pre>Moving the xml_monitor outside test010  
 
 
1- 
 
inside the fstab file of the machine when we want move the xml_monitor  
add the /export/CafCondor via nfs  
 
fcdftest010:/fbsng/CafCondor_test  /export/CafCondor nfs  rw,hard,intr,rsize=32768,wsize=32768,nfsvers=3 0 0 
 
 
2-  create a new directory where put inside the xml_monitor code  
    with the startup  
 
 
[cdfcaf@fcdftest013:~/test] $  ls 
total 8 
-rwxr-xr-x    1 cdfcaf   cdfcaf       2171 May 23 21:52 startup 
drwxr-xr-x    3 cdfcaf   cdfcaf       4096 May 25 16:26 xml_monitoring 
 
 
 
3- 
modify the startup  to do the startup of only the xml_monitor 
 
 
 
#!/bin/sh 
 
# 
# usage: 
#   startup &lt;progname&gt; &lt;dirname&gt; &lt;command&gt; 
# 
 
echo "$1/$2 $3" 
 
#CAF_ROOT=`pwd` 
CAF_ROOT=~cdfcaf/CafCondor 
cd $CAF_ROOT 
 
echo $CAF_ROOT 
 
PIDDIR=`cd $CAF_ROOT/config;python getPidDir.py` 
CAFPYTHON=`cd $CAF_ROOT/config;python getCAFPython.py` 
DIRNAME=$1 
PROGNAME=$2 
 
echo $CAFPYTHON 
need_ca_dir=0 
 
if [ "$DIRNAME" == "xml_monitoring" ]; then 
  enable_grid_monitoring=`cd $CAF_ROOT/config;python getEGM.py` 
  enable_grid_submit=`cd $CAF_ROOT/config;python getEGS.py` 
  if [ "$enable_grid_monitoring" == "1" -o "$enable_grid_submit" == "1" ]; then 
    need_ca_dir=1 
  fi 
fi 
 
 
if [ "$need_ca_dir" == "1" ]; then 
    ca_dir=`cd $CAF_ROOT/config;python getCADir.py` 
    if [ "$ca_dir" == "NONE" ]; then 
      echo "You need to configure CA_DIR" 
      echo "See caf.cfg.fnal" 
      exit 1 
    fi 
    export X509_CERT_DIR=$ca_dir 
echo "X509_CERT_DIR=$ca_dir" 
fi 
 
 
 
################################################### 
## here put the right dir !!!!!! ################# 
################################################## 
 
cd /home/cdfcaf/test/$DIRNAME 
export LD_LIBRARY_PATH=/usr/krb5/lib:${LD_LIBRARY_PATH} 
export PYTHONPATH=${CAF_ROOT}/config:/home/cdfcaf/test/xml_monitoring:${PYTHONPATH} 
echo $PYTHONPATH 
 
 
############# end !!!!!!!!! ################### 
 
 
 
 
 
if [ "$3" == "start" -o "$3" == "stop" -o "$3" == "restart" -o "$3" == "check" -o "$3" == "status" ] ; then 
 
    case "$3" in 
 
    "start") 
       $CAFPYTHON $PROGNAME.py & 
    ;; 
 
    "stop" | "restart") 
 
       PIDFILE=$PIDDIR/$PROGNAME.pid 
       if [ -f $PIDFILE ]; then 
           PID=`cat $PIDFILE` 
           if [ ! -z "$PID" ]; then 
               kill $PID 
               rm $PIDFILE 
           fi 
       fi 
 
       if [ "$3" == "restart" ]; then 
           sleep 5 
           $CAFPYTHON $PROGNAME.py & 
       fi 
    ;; 
 
    "check" | "status") 
 
    
       PIDFILE=$PIDDIR/$PROGNAME.pid 
       if [ -f $PIDFILE ]; then 
          PID=`cat $PIDFILE` 
          if [ ! -z "$PID" ]; then 
              CNT=`ps $PID|wc -l` 
              if [ $CNT -ne 2 ]; then 
                echo $PROGNAME died 
              else 
                echo $PROGNAME is running under pid: $PID 
              fi 
          else 
              echo $PROGNAME is not running 
          fi 
       else 
          echo $PROGNAME is not running 
       fi 
    ;; 
 
    esac 
else 
    echo "usage: $0 $1 $2 {start|stop|restart|check|status}" 
 
    exit 1 
fi 
 
exit 0 
 
 
 
4 - inside the startup on the head node remove  
   the code for the xml_monitor  
    note : TODO ! 
 
 
5 - the directory contains this following files : 
 
 
[cdfcaf@fcdftest013:~/test/xml_monitoring] $ ls 
total 316 
-rw-r--r--    1 cdfcaf   cdfcaf        509 May 23 21:35 archfile.py 
-rw-r--r--    1 cdfcaf   cdfcaf       4960 May 23 21:35 CAFConfig.py 
-rw-r--r--    1 cdfcaf   cdfcaf      28034 May 23 21:36 CafInfo.py 
-rw-r--r--    1 cdfcaf   cdfcaf      23551 May 23 21:35 CAFLogParser.py 
-rw-r--r--    1 cdfcaf   cdfcaf      11425 May 23 21:35 CODMonitor.py 
-rw-r--r--    1 cdfcaf   cdfcaf       9481 May 23 21:35 CommModule.py 
-rw-r--r--    1 cdfcaf   cdfcaf      21355 May 25 13:39 CondorMonitor.py 
-rw-r--r--    1 cdfcaf   cdfcaf       5270 May 23 21:36 conf.py 
drwxr-xr-x    2 cdfcaf   cdfcaf       2048 May 23 21:35 CVS 
-rw-r--r--    1 cdfcaf   cdfcaf      21460 May 23 21:35 DAGManParser.py 
-rw-r--r--    1 cdfcaf   cdfcaf       9355 May 25 16:26 DiskModule_old.py 
-rw-r--r--    1 cdfcaf   cdfcaf      10098 May 25 16:26 DiskModule.py 
-rw-r--r--    1 cdfcaf   cdfcaf      21078 May 23 21:35 JobManager.py 
-rw-r--r--    1 cdfcaf   cdfcaf      44335 May 23 21:35 JobMonitor.py 
-rw-r--r--    1 cdfcaf   cdfcaf       5866 May 23 21:35 TelnetModule.py 
-rw-r--r--    1 cdfcaf   cdfcaf      29112 May 23 21:35 xml_format.py 
-rwxr-xr-x    1 cdfcaf   cdfcaf         58 May 23 21:47 xml_monitor 
-rw-r--r--    1 cdfcaf   cdfcaf      30770 May 25 12:17 xml_monitor.py 
 
 
6 - change the code   
 
CondorMonitor.py  
 
 cmd = condor_bin+'/condor_userprio -l -pool &lt; name_of_pool &gt; 
 
note : only if collector or negotiator are outside head node (old case of test010) 
 
 
DiskModule.py  
 
 
class CachedDiskIO: 
    def __init__(self,filename,cachename,default_fields): 
        self.filename = filename 
        self.cachename = cachename 
        self.fields = default_fields 
 
    def readlines(self): 
        lines = [] 
        try: 
             
            fd = open(self.filename,"r")` 
            lines = fd.readlines() 
        finally: 
            fd.close() 
        return lines       
            #fd = open(self.cachename,"r") 
            #fcntl.flock(fd.fileno(),fcntl.LOCK_SH) 
            #try: 
             #   cache_stats = os.fstat(fd.fileno()) 
              #  cache_data = cPickle.load(fd) 
               # cache_last_size =  cache_data["last_size"] 
                #cache_last_check =  cache_data["last_check"] 
              #  cache_fields =  cache_data["fields"] 
            #finally: 
             #   fcntl.flock(fd.fileno(),fcntl.LOCK_UN) 
              #  fd.close() 
            #cache_ok = 1 
        #except: # cache does not exist yet 
         #   cache_ok = 0 
 
        #data_stats = os.stat(self.filename) 
 
        #if cache_ok: 
         #   if ((data_stats[stat.ST_MTIME]&gt;cache_stats[stat.ST_MTIME]) or 
          #      (data_stats[stat.ST_MTIME]&gt;cache_last_check) or 
           #     (data_stats[stat.ST_SIZE]!=cache_last_size)): 
            #    cache_ok = 0 
 
      #  if cache_ok: 
            #cache valid, use cached data 
       #     self.fields = cache_fields 
        #    self.last_size = cache_last_size 
         #   self.last_check = cache_last_check 
            #lines = [] 
        #else: 
            # cache not valid, reread data 
          #  fd = open(self.filename,"r") 
 
           # try: 
            #   fcntl.flock(fd.fileno(),fcntl.LOCK_SH) 
             #  try: 
              #     data_stats = os.fstat(fd.fileno()) # reread stats to have the same values as readlines 
                  # lines = fd.readlines() 
               #finally: 
             #      fcntl.flock(fd.fileno(),fcntl.LOCK_UN) 
               #   fd.close() 
           # except IOError,e: 
            #     try: 
                   #data_stats=fd.fileno() 
             #      data_stats=fd 
              #   finally: 
               #    fd.close()     
                        
           # self.last_size = data_stats[stat.ST_SIZE] 
            #self.last_check = data_stats[stat.ST_MTIME] 
            # leave fields to the default values 
 
       # return lines 
  
 
 
6 - change the configuration file 
 
on the head node : 
 
  /home/cdfcaf/CafCondor/www/scripts/monitor.cfg	 
 
 
# 
# COndorCAF monitor configuration 
# 
 
xmlmon_node = fcdftest013 
xmlmon_port = 9999 
 
 
 
 
todo : add inside caf.cfg a new variabile for the xml_monitor  
 
 
7 - inside title.py change  
 
base_wpath="/igorcaf/" 
 
 
  
</pre>
<!-- Author: Federica -->


<!-- Date: Fri Jun 1 16:59:00 2007 -->
<pre>move xml_mon fro head3 to head 4  
 
i did the same steps did on test . 
 
I list here the diff. 
 
1- 
 
  on head4 the file system is  
 
[cdfcaf@fcdfhead4 cdfcaf]$ df 
Filesystem           1K-blocks      Used Available Use% Mounted on 
/dev/sda1              4032092   3453220    374048  91% / 
/dev/sda5              8064272     34904   7619716   1% /tmp 
/dev/sda6              4032092    145604   3681664   4% /var 
/dev/sda3             10080520    434728   9133724   5% /misc 
/dev/sda2             10080520   5927168   3641284  62% /export 
none                   4099720         0   4099720   0% /dev/shm 
df: `/export/CafCondor/stage': No such file or directory 
/dev/sdb1             10231104   3187512   7043592  32% /export/condor_local 
fcdfhead3:/export/CafCondor 
                      61282128  42935584  18346544  71% /export/CafCondor 
fcdfhead1:/export/home/cdfcaf 
                     141106144  91084832  42853504  69% /home/cdfcaf 
 
 
2- i created a dir on /home/cdfcaf/xml_head3 
 
3- inside the script for the start of xml_mon inside this dir  
   change the following line  
 
 
 
cd /home/cdfcaf/xml_head3/$DIRNAME 
export LD_LIBRARY_PATH=/usr/krb5/lib:${LD_LIBRARY_PATH} 
export PYTHONPATH=${CAF_ROOT}/config:/home/cdfcaf/xml_head3/xml_monitoring:${PYTHONPATH} 
echo $PYTHONPATH 
 
4. inside caf.cfg  
 
 
  XML_PORT = 9999 
 
(NOTE : todo -&gt; add a new parameter inside the caf.cfg for the node of xml_mon) 
 
5. change the node on monotor.cfg in head4  
  
</pre>
<!-- Author: Federica -->


<!-- Date: Mon Jun 4 18:39:45 2007 -->
<pre>update CafCondor/monitor/DAGManParser.py 
                         DiskModule.py 
                         JobMonitor.py 
                         xml_monitor.py 
                /mailer/ mailer.py 
                /www/scripts/updateCompleted.py 
on head2,head3, head5. 
 
Restart cafcondor. 
 </pre>
<!-- Author: Federica -->
<!-- Comment: Mon Jun  4 19;15;08 comment by...Federica -->
<pre>reactivate updateCompleted.py for head3 
</pre>


<!-- Date: Tue Jun 5 08:41:54 2007 -->
<pre>This is a task list for the Week of 29-May-07 
Hi, 
 
 We have a few tasks that we should do this week. (These are in no particular order - just what 
I remember) 
  
 1)  modify monitor.py to use less memory 
 2)  modify xml_monitor.py to use less memory 
 3)  restrict the number of running monitor processes to 1 for monitor.py and xml_monitor.py 
 4)  Save changes in CVS 
 5)  update monitoring changes on fcdfhead2 and fcdfhead5 
 6)  move xml_monitoring off of fcdfhead3 to another node 
 7)  restart caf code on fcdftest010 
 8)  finish upgrading fcdftest010 to Condor 6.9.2 
 9)  finish installing fermigrid submission point to fcdftest010 
 10) finish testing SAM-CAF interface 
 11)  On Thursday or before (if done w/ 10) deploy SAM-CAF interface 
 12) test new glexec with roles 
 13)   -- Removed due to poor taste --- (Self censureship) 
 14) Hope that we don't have more crisis this week - though we have at least one good one a 
       week. 
 
   Who want to do what?  I started 8-10 so I should finish it. 
 
 Doug 
 
</pre>
<!-- Author: Doug -->


<!-- Date: Tue Jun 5 08:43:25 2007 -->
<pre>Plan for monitoring code changes made by Igor Sfiligoi over weekend 
 
Hi, 
 
 I am in Wilson Hall this morning working on my CV. So I will not be coming to the CDF trailers until 
the afternoon. Over the weekend you should have gotten e-mail from the CVS system as Igor checked 
in new code.  Would one of you please install this code on fcdftest010. Run some test jobs to make 
sure things work. Assuming that these tests have been sucessful then it can be ployed in the head 
nodes. 
 
   thanks, 
 
  Doug 
</pre>
<!-- Author: Doug B -->


<!-- Date: Wed Jun 6 12:43:48 2007 -->
<pre>Namcaf headnode fcdfhead5 
Portion of Collector log around where the collector 
core dump with code 11 
 
Enact = "NO" 
Subsystem = "COLLECTOR" 
ParentUniqueID = "fcdfhead5:21378:1181139833" 
ServerPid = 27115 
SessionDuration = "8640000" 
6/6 11:53:56 DC_AUTHENTICATE: the_policy: 
MyType = "" 
TargetType = "" 
Authentication = "YES" 
Encryption = "YES" 
Integrity = "YES" 
AuthMethodsList = "GSI" 
AuthMethods = "GSI" 
CryptoMethods = "3DES,BLOWFISH" 
SessionDuration = "8640000" 
Enact = "YES" 
6/6 11:53:56 DC_AUTHENTICATE: generating 3DES key for session fcdfhead5:27115:1181148836:335... 
6/6 11:53:57 SECMAN: Sending following response ClassAd: 
MyType = "" 
TargetType = "" 
Authentication = "YES" 
Encryption = "YES" 
Integrity = "YES" 
AuthMethodsList = "GSI" 
AuthMethods = "GSI" 
CryptoMethods = "3DES,BLOWFISH" 
SessionDuration = "8640000" 
Enact = "YES" 
RemoteVersion = "$CondorVersion: 6.9.2 Apr  9 2007 $" 
6/6 11:53:57 SECMAN: new session, doing initial authentication. 
6/6 11:53:57 DC_AUTHENTICATE: authenticating RIGHT NOW. 
6/6 11:53:57 AUTHENTICATE: in authenticate( addr == NULL, methods == 'GSI') 
6/6 11:53:57 AUTHENTICATE: can still try these methods: GSI 
6/6 11:53:57 HANDSHAKE: in handshake(my_methods = 'GSI') 
6/6 11:53:57 HANDSHAKE: handshake() - i am the server 
6/6 11:53:57 condor_read(): Socket closed when trying to read 5 bytes from &lt;137.110.141.34:41260&gt; 
6/6 11:53:57 IO: EOF reading packet header 
6/6 11:53:57 AUTHENTICATE: handshake failed! 
6/6 11:53:57 AUTHENTICATE: auth_status == 0 (?!?) 
6/6 11:53:57 Authentication was a FAILURE. 
6/6 11:54:17 passwd_cache::cache_uid(): getpwnam("condor") failed: Inappropriate ioctl for device 
 
6/6 11:54:17 passwd_cache::cache_uid(): getpwnam("condor") failed: Inappropriate ioctl for device 
 
6/6 11:54:17 KEYCACHE: created: 0x8a85570 
</pre>
<!-- Author: Doug B -->


<!-- Date: Mon Jul 2 03:38:19 2007 -->
<pre>comands used to check the job.log files to look for restarts 
 
 1004   find . -type d -maxdepth 1 -name '*submit*' -exec grep -i '001 (' {}/job.log \; | awk '{print $2 " 
" $3 " " $4 " " $9}' | sort -n -k 1 &gt; /tmp/current_running_jobs_`date +%Y_%m_%d_%H%M`.txt  
 1005   find . -type d -maxdepth 1 -name '*submit*' -exec grep -i '001 (' {}/job.log \; | awk '{print $2 " " $3 " " $4 " " $9}' | sort -n -k 1 &gt; /tmp/current_running_jobs_`date +%Y_%m_%d_%H%M`.txt  
 
 
head2&gt;  find history/jobs_8xxxx -type d -maxdepth 1 -name 'job_*' -exec grep -i '001 (' {}/job.log \; | awk '{print $2 " " $3 " " $4 " " $9}' | sort -n -k 1 &gt; /tmp/old_jobs_8xxxx_jobs_`date +%Y_%m_%d_%H%M`.txt  
head2&gt;  find history/jobs_9xxxx -type d -maxdepth 1 -name 'job_*' -exec grep -i '001 (' {}/job.log \; | awk '{print $2 " " $3 " "$4 " " $9}' | sort -n -k 1 &gt; /tmp/old_jobs_9xxxx_jobs_`date +%Y_%m_%d_%H%M`.txt  
 
found these errors: 
 
grep: history/jobs_9xxxx/job_90186/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_90439/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_91950/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_91804/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_91799/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_92609/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_92608/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_92607/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_92645/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_92853/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_92930/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_93101/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_93172/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_93586/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_93625/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_94282/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_94584/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_94587/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_95124/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_95408/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_95418/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_95417/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_95428/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_95469/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_95477/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_95956/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_95955/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_95953/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_96049/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_96052/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_96084/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_96144/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_96157/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_96156/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_96263/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_96302/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_96473/job.log: No such file or directory 
grep: history/jobs_9xxxx/job_96471/job.log: No such file or directory 
 
</pre>
<!-- Author: Doug B -->


<!-- Date: Mon Jul 2 04:28:57 2007 -->
<pre>Investigating the problem of connection failures leading to job 
restarts 
 
Went through the history logs using this command: 
 
head2&gt;  find history/jobs_9xxxx -type d -maxdepth 1 -name 'job_*' -exec grep -i '022 (' -A 3 {}/job.log \; &gt; /tmp/old_jobs_9xxxx_connection_failures_`date +%Y_%m_%d_%H%M`.txt  
 
head2&gt; for idate in 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 ; do echo -n "June ${idate} 2007 disconnects "; grep '022 (' /tmp/old_jobs_9xxxx_connection_failures_2007_07_02_0340.txt | grep '06\/'${idate} | wc -l ; done; 
June 01 2007 disconnects      95 
June 02 2007 disconnects      27 
June 03 2007 disconnects      40 
June 04 2007 disconnects      30 
June 05 2007 disconnects     182 
June 06 2007 disconnects      57 
June 07 2007 disconnects     144 
June 08 2007 disconnects     147 
June 09 2007 disconnects       3 
June 10 2007 disconnects       3 
June 11 2007 disconnects     197 
June 12 2007 disconnects     846 
June 13 2007 disconnects     482 
June 14 2007 disconnects     112 
June 15 2007 disconnects    1117 
June 16 2007 disconnects      99 
June 17 2007 disconnects      51 
June 18 2007 disconnects      82 
June 19 2007 disconnects    1663 
June 20 2007 disconnects    1412 
June 21 2007 disconnects    2667 
June 22 2007 disconnects     955 
June 23 2007 disconnects       6 
June 24 2007 disconnects    1825 
June 25 2007 disconnects    1526 
June 26 2007 disconnects    2198 
June 27 2007 disconnects    5246 
June 28 2007 disconnects    1506 
June 29 2007 disconnects     371 
June 30 2007 disconnects     153 
 
 
Typical disconnect message - 
head2&gt; tail  /tmp/old_jobs_9xxxx_connection_failures_2007_07_02_0340.txt 
... 
-- 
022 (264667.000.000) 06/30 23:26:11 Job disconnected, attempting to reconnect 
    Socket between submit and execute hosts closed unexpectedly 
    Trying to reconnect to vm2@16762@fcdfcaf1156.fnal.gov &lt;131.225.210.65:55824&gt; 
... 
022 (267353.000.000) 06/29 10:50:50 Job disconnected, attempting to reconnect 
    Socket between submit and execute hosts closed unexpectedly 
    Trying to reconnect to vm2@14256@fcdfcaf1171.fnal.gov &lt;131.225.210.80:43731&gt; 
... 
</pre>
<!-- Author: Doug B -->


<!-- Date: Tue Jul 10 16:29:24 2007 -->
<pre>command to get list of Condor job starts that includings 
the Dagman id and the condor id 
(on any head node) 
 
cd /export/CafCondor/cafIn 
 
 find . -type d -maxdepth 1 -follow -name 'job_*' -exec grep -H -i '001 (' {}/job.log \; | sort -k 1 | sort -n -k 2 
 
</pre>
<!-- Author: Doug  -->
<!-- Comment: Tue Jul 10 16;43;06 comment by...Doug B -->
<pre>Here is one example where there was an AUTH_ERROR at same time
as a restart
cdfcaf@fcdfhead3 condor_local]$ grep 'AUTH_ERROR: KDC reply did not match expectations' schedd_102/log/ShadowLog | grep 1017365
7/10 14:44:01 (1017365.0) (21162): AUTH_ERROR: KDC reply did not match expectations

[cdfcaf@fcdfhead3 cafIn]$ grep 1017365 /tmp/current_running_jobs_2007_07_10.txt
./job_3346285/job.log:001 (1017365.000.000) 07/10 14:03:57 Job executing on host: &lt;131.225.208.14:55990&gt;
./job_3346285/job.log:001 (1017365.000.000) 07/10 14:44:39 Job executing on host: &lt;131.225.208.113:32775&gt;



************ Here is an example where there was none ********
[cdfcaf@fcdfhead3 condor_local]$ grep 'AUTH_ERROR: KDC reply did not match expectations' schedd_102/log/ShadowLog | grep 1017677
7/10 16:21:24 (1017677.0) (16369): AUTH_ERROR: KDC reply did not match expectations
7/10 16:21:28 (1017677.0) (16369): AUTH_ERROR: KDC reply did not match expectations

[cdfcaf@fcdfhead3 cafIn]$  grep 1017677 /tmp/current_running_jobs_2007_07_10.txt
./job_3346392/job.log:001 (1017677.000.000) 07/10 16:28:48 Job executing on host: &lt;131.225.209.87:48170&gt;

</pre>


<!-- Date: Wed Jul 11 15:10:50 2007 -->
<pre>here are the current Fermigrid submission points: 
 
GLIDEIN_PORTALS=fcdfosg1_fcdfsrv001_glexec_new,fcdfosg1_fcdfsrv002_glexec_new,fcdfosg1_fcdfsrv003_glexec_new,fcdfosg2_fcdfsrv001_glexec_SL44_new,fcdfosg2_fcdfsrv002_glexec_SL44_new,fcdfosg2_fcdfsrv003_glexec_SL44_new</pre>
<!-- Author: Doug B -->


<!-- Date: Wed Jul 11 15:36:25 2007 -->
<pre>In the group_params.config file on fcdfhead2 
change this setting: 
 
GROUP_QUOTA_DYNAMIC_group_highprio = 0.99 
 
to 
 
GROUP_QUOTA_DYNAMIC_group_highprio = 0.40 
 
and 
 
GROUP_QUOTA_DYNAMIC_group_sam = 0.00 
 
to  
 
GROUP_QUOTA_DYNAMIC_group_sam = 0.05 
 
Note the Dynamic quota can not be more that 0.5 or the quota 
will be higher than the available slots. 
 
</pre>
<!-- Author: Doug B -->


<!-- Date: Wed Jul 11 23:56:49 2007 -->
<pre>[cdfcaf@fncdfsrv0 sh]$ ./check_alive.sh | grep 'not found' | awk '{ print $1}' | while read node ; do ping -c 1 ${node} | grep -B 1 Unreachable ;  done; 
PING fncdf212.fnal.gov (131.225.239.212) 56(84) bytes of data. 
From fncdfsrv0.fnal.gov (131.225.239.58) icmp_seq=0 Destination Host Unreachable 
PING fncdf223.fnal.gov (131.225.239.223) 56(84) bytes of data. 
From fncdfsrv0.fnal.gov (131.225.239.58) icmp_seq=0 Destination Host Unreachable 
ping: unknown host fncdf224 
PING fncdf246.fnal.gov (131.225.239.246) 56(84) bytes of data. 
From fncdfsrv0.fnal.gov (131.225.239.58) icmp_seq=0 Destination Host Unreachable 
PING fncdf263.fnal.gov (131.225.239.14) 56(84) bytes of data. 
From fncdfsrv0.fnal.gov (131.225.239.58) icmp_seq=0 Destination Host Unreachable 
PING fncdf268.fnal.gov (131.225.239.19) 56(84) bytes of data. 
From fncdfsrv0.fnal.gov (131.225.239.58) icmp_seq=0 Destination Host Unreachable 
PING fncdf273.fnal.gov (131.225.239.24) 56(84) bytes of data. 
From fncdfsrv0.fnal.gov (131.225.239.58) icmp_seq=0 Destination Host Unreachable 
PING fncdf279.fnal.gov (131.225.239.30) 56(84) bytes of data. 
From fncdfsrv0.fnal.gov (131.225.239.58) icmp_seq=0 Destination Host Unreachable 
PING fncdf286.fnal.gov (131.225.239.37) 56(84) bytes of data. 
From fncdfsrv0.fnal.gov (131.225.239.58) icmp_seq=0 Destination Host Unreachable 
PING fncdf290.fnal.gov (131.225.239.41) 56(84) bytes of data. 
From fncdfsrv0.fnal.gov (131.225.239.58) icmp_seq=0 Destination Host Unreachable 
PING fncdf293.fnal.gov (131.225.239.44) 56(84) bytes of data. 
From fncdfsrv0.fnal.gov (131.225.239.58) icmp_seq=0 Destination Host Unreachable 
PING fncdf304.fnal.gov (131.225.239.55) 56(84) bytes of data. 
From fncdfsrv0.fnal.gov (131.225.239.58) icmp_seq=0 Destination Host Unreachable 
PING fncdf307.fnal.gov (131.225.239.114) 56(84) bytes of data. 
From fncdfsrv0.fnal.gov (131.225.239.58) icmp_seq=0 Destination Host Unreachable 
PING fncdf309.fnal.gov (131.225.239.116) 56(84) bytes of data. 
From fncdfsrv0.fnal.gov (131.225.239.58) icmp_seq=0 Destination Host Unreachable 
PING fncdf336.fnal.gov (131.225.239.103) 56(84) bytes of data. 
From fncdfsrv0.fnal.gov (131.225.239.58) icmp_seq=0 Destination Host Unreachable 
</pre>
<!-- Author: Doug B -->


<!-- Date: Thu Jul 12 08:01:40 2007 -->
<pre>Changes made to Fermigrid (fcdfhead2) head node to increase 
the number of running jobs.  Will need to figure out how to limit the number of jobs to fcdfosg1 and fcdfosg2 (different for each) 
 
[cdfcaf@fcdfhead2:~] $ condor_config_val -verbose GRIDMANAGER_MAX_PENDING_SUBMITS_PER_RESOURCE 
GRIDMANAGER_MAX_PENDING_SUBMITS_PER_RESOURCE: 20 
  Defined in '/home/cdfcaf/condor/dist/etc/fcdfhead2.local', line 211. 
 
[cdfcaf@fcdfhead2:~] $ condor_config_val -verbose GRIDMANAGER_MAX_SUBMITTED_JOBS_PER_RESOURCE 
GRIDMANAGER_MAX_SUBMITTED_JOBS_PER_RESOURCE: 2000 
  Defined in '/home/cdfcaf/condor/dist/etc/fcdfhead2.local', line 209. 
 
and 
 
MAX_JOBS_RUNNING: 3000 
  Defined in '/home/cdfcaf/condor/dist/etc/fcdfhead2.local', line 29. 
 
</pre>
<!-- Author: Doug B -->


<!-- Date: Thu Jul 12 11:43:06 2007 -->
<pre>To restart the xml-monitor  when it is locate outside the headnode. 
 
now only the xml_monitor of fcdfhead2 is outside (on fcdftest012) 
 
on fcdftest012 there is  
 
/home/cdfcaf/xml_head2 
 
go inside  
 
[cdfcaf@fcdftest012:~/xml_head2/xml_monitoring] $ ./xml_monitor {start|stop|restart} 
 
</pre>
<!-- Author: federica -->


<!-- Date: Thu Jul 12 15:52:17 2007 -->
<pre>TODO : check problem of the user rgp230. 
I have account on fcdflnx4 not in fcdflnx8. 
 
so it is not possible to add like caf user</pre>
<!-- Author: federica -->


<!-- Date: Thu Jul 12 16:17:46 2007 -->
<pre>In order to solve problem for the very old running jobs : 
 
1- kill all held jobs (condor_rm *) 
2- put inside at the mailer.todo all the jobs that are finish 5/25 
3-investigate on the priority  */-20 on the webmonitor 
4-check the number of slots for all the group on group caf</pre>
<!-- Author: federica -->


<!-- Date: Mon Jul 16 14:01:43 2007 -->
<pre>Changed the highprio quota on fermigrid from 0.40 to 
0.20 and reconfigured condor</pre>
<!-- Author: Doug B -->


<!-- Date: Mon Jul 16 15:05:24 2007 -->
<pre>While trying to create new schedd's for the fcdfosg2 gatekeeper, I 
got an error about creating the directories. So I changed the ownership of /export/condor_local from root:root to cdfcaf:root 
 
[cdfcaf@fcdfhead2:~/condor/sh] $ ls -l /export 
total 12 
drwxr-xr-x   25 cdfcaf   root         4096 Mar 14 11:31 backup 
drwxr-xr-x    8 cdfcaf   root          109 Mar 21  2006 CafCondor 
drwxr-xr-x   25 root     root         4096 Dec 26  2006 condor_local 
drwxr-xr-x    3 cdfcaf   root         4096 Jun 30  2005 glideins 
[cdfcaf@fcdfhead2:~/condor/sh] $ ksu 
Authenticated benjamin@FNAL.GOV 
Account root: authorization for benjamin@FNAL.GOV successful 
Changing uid to root (0) 
[root@fcdfhead2 sh]# chown cdfcaf /export/condor_local 
[root@fcdfhead2 sh]# exit 
exit 
[cdfcaf@fcdfhead2:~/condor/sh] $ ls -l /export 
total 12 
drwxr-xr-x   25 cdfcaf   root         4096 Mar 14 11:31 backup 
drwxr-xr-x    8 cdfcaf   root          109 Mar 21  2006 CafCondor 
drwxr-xr-x   25 cdfcaf   root         4096 Dec 26  2006 condor_local 
drwxr-xr-x    3 cdfcaf   root         4096 Jun 30  2005 glideins 
 
Which now worked. 
 
I created three new schedd's 
./init_condor_schedd.sh schedd_glide_fcdfosg2_1 
./init_condor_schedd.sh schedd_glide_fcdfosg2_2 
./init_condor_schedd.sh schedd_glide_fcdfosg2_3 
 
I started the schedd's 
 
./start_condor_schedd.sh schedd_glide_fcdfosg2_1 
./start_condor_schedd.sh schedd_glide_fcdfosg2_2 
./start_condor_schedd.sh schedd_glide_fcdfosg2_3 
 
Then created the entry points : 
cd /home/cdfcaf/CafCondor/glidein/scripts 
 
 ./create_entry.sh fcdfosg2_glide_1_fcdfsrv001_glexec -g fcdfosg2.fnal.gov/jobmanager-condor -s Condor -d schedd_glide_fcdfosg2_1@fcdfhead2.fnal.gov -p fcdfsrv001.fnal.gov:3128 -C /home/cdfcaf/condor/dist -t FermiGridCDF -f /home/cdfsoft -i FNAL 
 
./create_entry.sh fcdfosg2_glide_2_fcdfsrv002_glexec -g fcdfosg2.fnal.gov/jobmanager-condor -s Condor -d schedd_glide_fcdfosg2_2@fcdfhead2.fnal.gov -p fcdfsrv002.fnal.gov:3128 -C /home/cdfcaf/condor/dist -t FermiGridCDF -f /home/cdfsoft -i FNAL 
 
./create_entry.sh fcdfosg2_glide_2_fcdfsrv003_glexec -g fcdfosg2.fnal.gov/jobmanager-condor -s Condor -d schedd_glide_fcdfosg2_3@fcdfhead2.fnal.gov -p fcdfsrv003.fnal.gov:3128 -C /home/cdfcaf/condor/dist -t FermiGridCDF -f /home/cdfsoft -i FNAL 
 
I will test the newly created entry points later. 
</pre>
<!-- Author: Doug B -->
<!-- Comment: Mon Jul 16 15;10;51 comment by...Doug B -->
<pre>Had to create a new entry point: fcdfosg2_glide_3_fcdfsrv003_glexec
Due to a typo in the first one 
created the following files which included the create entry string:
[cdfcaf@fcdfhead2:~/condor/sh] $ cat /export/CafCondor/cafIn/glidein_fcdfosg2_glide_1_fcdfsrv001_glexec/create_entry.txt
./create_entry.sh fcdfosg2_glide_1_fcdfsrv001_glexec -g fcdfosg2.fnal.gov/jobmanager-condor -s Condor -d schedd_glide_fcdfosg2_1@fcdfhead2.fnal.gov -p fcdfsrv001.fnal.gov:3128 -C /home/cdfcaf/condor/dist -t FermiGridCDF -f /home/cdfsoft -i FNAL
[cdfcaf@fcdfhead2:~/condor/sh] $ cat /export/CafCondor/cafIn/glidein_fcdfosg2_glide_2_fcdfsrv002_glexec/create_entry.txt
./create_entry.sh fcdfosg2_glide_2_fcdfsrv002_glexec -g fcdfosg2.fnal.gov/jobmanager-condor -s Condor -d schedd_glide_fcdfosg2_2@fcdfhead2.fnal.gov -p fcdfsrv002.fnal.gov:3128 -C /home/cdfcaf/condor/dist -t FermiGridCDF -f /home/cdfsoft -i FNAL
[cdfcaf@fcdfhead2:~/condor/sh] $ cat /export/CafCondor/cafIn/glidein_fcdfosg2_glide_3_fcdfsrv003_glexec/create_entry.txt
./create_entry.sh fcdfosg2_glide_3_fcdfsrv003_glexec -g fcdfosg2.fnal.gov/jobmanager-condor -s Condor -d schedd_glide_fcdfosg2_3@fcdfhead2.fnal.gov -p fcdfsrv003.fnal.gov:3128 -C /home/cdfcaf/condor/dist -t FermiGridCDF -f /home/cdfsoft -i FNAL
</pre>


<!-- Date: Tue Jul 17 12:30:39 2007 -->
<pre>SQUID server commands to help restart it: 
 
log into fcdfsrv001, fcdfsrv002 or fcdfsrv003 as root 
 
[root@fcdfsrv002 root]# /etc/init.d/squid status 
squid dead but subsys locked 
squid: ERROR: No running copy 
[root@fcdfsrv002 root]# ls -l /var/lock/subsys/squid 
-rw-r--r--    1 root     root            0 Jul 17 11:02 /var/lock/subsys/squid 
[root@fcdfsrv002 root]# rm /var/lock/subsys/squid 
rm: remove regular empty file `/var/lock/subsys/squid'? y 
[root@fcdfsrv002 root]# /etc/init.d/squid status 
squid is stopped 
squid: ERROR: No running copy 
 
The squid config file is : 
/etc/squid/squid.conf 
 
This error message usually means that the squid.pid file is missing. Since the PID file is normally present when squid is running, the absence of the PID file usually means Squid is not running. If you accidentally delete the PID file, Squid will continue running, and you won't be able to send it any signals. 
 
If you accidentally removed the PID file, there are two ways to get it back. 
 
One is to run ps and find the Squid process id. You'll probably see two processes, like this: 
 
bender-wessels % ps ax | grep squid 
83617  ??  Ss     0:00.00 squid -s 
83619  ??  S      0:00.48 (squid) -s (squid) 
 
You want the second process id, 83619 in this case. Create the PID file and put the process id number there. For example: 
 
echo 83619 &gt; /var/run/squid.pid 
 
The second is to use the above technique to find the Squid process id. Send the process a HUP signal, which is the same as squid -kreconfigure: 
 
kill -HUP 83619 
 
The reconfigure process creates a new PID file automatically. 
 
Noticed that the /var/ file system used by the squid server for its logs is full on fcdfsrv002 and fcdfsrv003 
</pre>
<!-- Author: Doug -->
<!-- Comment: Tue Jul 17 14;11;05 comment by...Doug B -->
<pre>in order to get the squid server to work on fcdfsrv002 properly.
I moved the log files from /var/log/squid to /fbsng/squid.
This is a 40 GB partition. The configuration file 

/etc/squid/squid.conf   

was changed according.</pre>

<!-- Comment: Tue Jul 17 15;38;55 comment by...Doug  -->
<pre>I reconfigured the squid server on fcdfsrv001 to direct its log
files to /fbsng/squid/*</pre>

<!-- Comment: Tue Jul 17 16;09;19 comment by...Doug -->
<pre>Jason Allen reminded me that when moving the log files to a new location, I should also modify the file: /etc/logrotate.d/squid

So - /etc/logrotate.d/squid on fcdfsrv001 fcdfsrv002 and fcdfsrv003 have been modified to be:

/fbsng/squid/access.log {
    weekly
    rotate 5
    copytruncate
    compress
    notifempty
    missingok
}
/fbsng/squid/cache.log {
    weekly
    rotate 5
    copytruncate
    compress
    notifempty
    missingok
}

/fbsng/squid/store.log {
    weekly
    rotate 5
    copytruncate
    compress
    notifempty
    missingok
# This script asks squid to rotate its logs on its own.
# Restarting squid is a long process and it is not worth
# doing it just to rotate logs
    postrotate
      /usr/sbin/squid -k rotate
    endscript
}
</pre>

<!-- Comment: Wed Jul 18 08;33;58 comment by...Doug  -->
<pre>
-bash-3.00$ /opt/condor/bin/condor_q -name fcdfosg2.fnal.gov 938591.0


-- Schedd: fcdfosg2.fnal.gov : &lt;131.225.208.218:32787&gt;
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
938591.0   cdf             7/17 15:34   0+16:25:57 R  0   322.3 data dbg 6 schedd_
</pre>


<!-- Date: Mon Jul 23 09:38:26 2007 -->
<pre>In order to have the monitoring pickup the new submission points automatically. 
Need to restart the monitoring after the submission points have been created. 
 
For fermigrid this means - 
monitor and state_monitor on fcdfhead2 
xml_monitor on fcdftest012 
 
When I did this and updated the status and history on the monitoring node fcdfmon5 by hand (executed the corresponding crontab entries) and waited a little while the new submission points appeared. 
 
</pre>
<!-- Author: Doug  B -->


<!-- Date: Mon Jul 23 09:50:21 2007 -->
<pre>Have noticed that we have idle and waiting jobs in Fermigrid when we should be running with more sections. Our glidein's are dying too quickly.    
The ~/CafCondor/glidein/glidekeeper/glidekeeper.py file had these 
three setting (internally): 
 
[cdfcaf@fcdfhead2:~/CafCondor/glidein/glidekeeper] $ diff glidekeeper.py~ glidekeeper.py 
249,251c249,251 
&lt;     max_submits = 200 
&lt;     max_removes = 10 
&lt;     max_idle = 100 
 
!!!!!!!!  New settings 23-July-07 dpb    
 
&gt;     max_submits = 1000 
&gt;     max_removes = 50 
&gt;     max_idle = 300 
 
 
Have wait and see if there are more glidein's now.</pre>
<!-- Author: Doug B -->
<!-- Comment: Mon Jul 23 14;38;12 comment by...Doug B -->
<pre>reduced the max_idle value to 20

and restarted the glidekeeper</pre>
<!-- Date: Mon Jul 23 10:33:30 2007 -->
<pre>cleaning up held entries on fcdfhead2 
 
JID - 97920  Section_65  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_necula_medium_234284_24218: No such file or directory 
JID - 97930  Section_137  Reason - Error from starter on vm2@838@fcdfcaf1071.fnal.gov: STARTER at 131.225.212.220 failed to send file(s) to &lt;131.225.240.106:54401&gt;; SHADOW at 131 
.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_necula_medium_234303_25830/x509cc_necula: (errno 2) No such file or directory 
JID - 97930  Section_88  Reason - Error from starter on vm2@22899@fcdfcaf1083.fnal.gov: STARTER at 131.225.212.232 failed to send file(s) to &lt;131.225.240.106:49920&gt;; SHADOW at 13 
1.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_necula_medium_234303_25830/section_88.out: (errno 2) No such file or directory 
JID - 97930  Section_187 JID - 98703  Section_5  Reason - Error from starter on vm2@872@fcdfcaf1002.fnal.gov: STARTER at 131.225.212.151 failed to send file(s) to &lt;131.225.240.10 
6:36962&gt;; SHADOW at 131.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_johnpaul_medium_234784_13421/section_5.out: (errno 2) No such file or directory 
JID - 98624  Section_14  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_21  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_24  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_15  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_3  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_11  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_16  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_25  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_17  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_2  Reason - Error from starter on vm2@22091@fcdfcaf1224.fnal.gov: STARTER at 131.225.213.140 failed to send file(s) to &lt;131.225.240.106:47645&gt;; SHADOW at 131 
.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5672/section_2.out: (errno 2) No such file or directory 
JID - 98624  Section_22  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_10  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_13  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_23  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_9  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_20  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_12  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_4  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_6  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_18  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_19  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_5  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_1  Reason - Error from starter on vm2@9660@fcdfcaf1374.fnal.gov: STARTER at 131.225.210.122 failed to send file(s) to &lt;131.225.240.106:47468&gt;; SHADOW at 131. 
225.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5672/section_1.IOlog: (errno 2) No such file or directory 
JID - 98624  Section_7  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98624  Section_8  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5672: No such file or directory 
JID - 98621  Section_8  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_19  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_3  Reason - Error from starter on vm2@28661@fcdfcaf1344.fnal.gov: STARTER failed to receive file(s) from &lt;131.225.240.106:56415&gt;; SHADOW at 131.225.240.106 f 
ailed to send file(s) to &lt;131.225.213.100:36290&gt;: error reading from /export/CafCondor/cafIn/submit_schsu_long_234691_5473/stage/iomap.txt: (errno 2) No such file or directory 
JID - 98621  Section_20  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_13  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_25  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_21  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_23  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_2  Reason - Error from starter on vm2@932@fcdfcaf1345.fnal.gov: STARTER at 131.225.213.101 failed to send file(s) to &lt;131.225.240.106:47830&gt;; SHADOW at 131.2 
25.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234691_5473/section_2.err: (errno 2) No such file or directory 
JID - 98621  Section_11  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_16  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_24  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_18  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_17  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_10  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_1  Reason - Error from starter on vm2@16155@fcdfcaf1271.fnal.gov: STARTER at 131.225.213.26 failed to send file(s) to &lt;131.225.240.106:47409&gt;; SHADOW at 131. 
225.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234691_5473/section_1.procs: (errno 2) No such file or directory 
JID - 98621  Section_14  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_4  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_22  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_12  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_7  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_15  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_9  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_5  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98621  Section_6  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5473: No such file or directory 
JID - 98622  Section_9  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_5  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_10  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_15  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_14  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_18  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_3  Reason - Error from starter on vm2@18798@fcdfcaf1097.fnal.gov: STARTER at 131.225.210.6 failed to send file(s) to &lt;131.225.240.106:47426&gt;; SHADOW at 131.2 
25.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234691_5610/section_3.err: (errno 2) No such file or directory 
JID - 98622  Section_2  Reason - Error from starter on vm2@30826@fcdfcaf1027.fnal.gov: STARTER at 131.225.212.176 failed to send file(s) to &lt;131.225.240.106:47838&gt;; SHADOW at 131 
.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234691_5610/section_2.out: (errno 2) No such file or directory 
JID - 98622  Section_8  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_11  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_23  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_4  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_19  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_25  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_17  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_16  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_12  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_13  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_7  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_1  Reason - Error from starter on vm2@9394@fcdfcaf1202.fnal.gov: STARTER at 131.225.210.111 failed to send file(s) to &lt;131.225.240.106:43802&gt;; SHADOW at 131. 
225.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234691_5610/section_1.procs: (errno 2) No such file or directory 
JID - 98622  Section_21  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_24  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_20  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_22  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98622  Section_6  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234691_5610: No such file or directory 
JID - 98623  Section_6  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_9  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_4  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_1  Reason - Error from starter on vm2@16253@fcdfcaf1292.fnal.gov: STARTER at 131.225.213.47 failed to send file(s) to &lt;131.225.240.106:43702&gt;; SHADOW at 131. 
225.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5645/section_1.out: (errno 2) No such file or directory 
JID - 98623  Section_22  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_24  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_7  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_17  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_23  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_20  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_11  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_13  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_16  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_15  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_2  Reason - Error from starter on vm2@17196@fcdfcaf1432.fnal.gov: STARTER at 131.225.210.180 failed to send file(s) to &lt;131.225.240.106:47519&gt;; SHADOW at 131 
.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5645/section_2.err: (errno 2) No such file or directory 
JID - 98623  Section_14  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_10  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_25  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_8  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_21  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_18  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_12  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_5  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_19  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5645: No such file or directory 
JID - 98623  Section_3  Reason - Error from starter on vm2@7931@fcdfcaf1201.fnal.gov: STARTER at 131.225.210.110 failed to send file(s) to &lt;131.225.240.106:47536&gt;; SHADOW at 131. 
225.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5645/CafExe_3.log: (errno 2) No such file or directory 
JID - 97730  Section_3  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_18  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_20  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_19  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_9  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_17  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_4  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_7  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_10  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_16  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_12  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_2  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_15  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_5  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_6  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_8  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_14  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_13  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 97730  Section_11  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_tatsuya_medium_234051_15849: No such file or directory 
JID - 98626  Section_16  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_5  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_18  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_21  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_15  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_13  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_17  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_4  Reason - Error from starter on vm2@25445@fcdfcaf1203.fnal.gov: STARTER at 131.225.210.112 failed to send file(s) to &lt;131.225.240.106:56894&gt;; SHADOW at 131 
.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5834/section_4.procs: (errno 2) No such file or directory 
JID - 98626  Section_7  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_11  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_20  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_14  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_22  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_9  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_8  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_24  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_12  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_25  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_2  Reason - Error from starter on vm2@5623@fcdfcaf1108.fnal.gov: STARTER at 131.225.210.17 failed to send file(s) to &lt;131.225.240.106:47522&gt;; SHADOW at 131.2 
25.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5834/section_2.out: (errno 2) No such file or directory 
JID - 98626  Section_10  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_6  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_3  Reason - Error from starter on vm2@27695@fcdfcaf1252.fnal.gov: STARTER at 131.225.213.7 failed to send file(s) to &lt;131.225.240.106:47629&gt;; SHADOW at 131.2 
25.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5834/section_3.err: (errno 2) No such file or directory 
JID - 98626  Section_19  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_23  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5834: No such file or directory 
JID - 98626  Section_1  Reason - Error from starter on vm2@16434@fcdfcaf1037.fnal.gov: STARTER at 131.225.212.186 failed to send file(s) to &lt;131.225.240.106:43799&gt;; SHADOW at 131 
.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5834/section_1.out: (errno 2) No such file or directory 
JID - 97872  Section_110  Reason - Error from starter on vm2@10649@fcdfcaf1265.fnal.gov: STARTER at 131.225.213.20 failed to send file(s) to &lt;131.225.240.106:46583&gt;; SHADOW at 13 
1.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_necula_medium_234233_32605/section_110.procs: (errno 2) No such file or directory 
JID - 97873  Section_151  Reason - Error from starter on vm2@9366@fcdfcaf1242.fnal.gov: STARTER at 131.225.213.158 failed to send file(s) to &lt;131.225.240.106:38665&gt;; SHADOW at 13 
1.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_necula_medium_234236_23117/section_151.err: (errno 2) No such file or directory 
JID - 97875  Section_181  Reason - Error from starter on vm2@14042@fcdfcaf1362.fnal.gov: STARTER at 131.225.213.118 failed to send file(s) to &lt;131.225.240.106:51210&gt;; SHADOW at 1 
31.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_necula_medium_234240_29250/section_181.err: (errno 2) No such file or directory 
JID - 97876  Section_73  Reason - Error from starter on vm2@11319@fcdfcaf1382.fnal.gov: STARTER at 131.225.210.130 failed to send file(s) to &lt;131.225.240.106:60888&gt;; SHADOW at 13 
1.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_necula_medium_234244_2820/section_73.IOlog: (errno 2) No such file or directory 
JID - 97876  Section_138  Reason - Error from starter on vm2@28046@fcdfcaf1091.fnal.gov: STARTER at 131.225.212.240 failed to send file(s) to &lt;131.225.240.106:39708&gt;; SHADOW at 1 
31.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_necula_medium_234244_2820/section_138.out: (errno 2) No such file or directory 
JID - 97876  Section_91  Reason - Error from starter on vm2@26258@fcdfcaf1206.fnal.gov: STARTER at 131.225.210.115 failed to send file(s) to &lt;131.225.240.106:35328&gt;; SHADOW at 13 
1.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_necula_medium_234244_2820/section_91.procs: (errno 2) No such file or directory 
JID - 97878  Section_136  Reason - Error from starter on vm2@12774@fcdfcaf980.fnal.gov: STARTER at 131.225.212.129 failed to send file(s) to &lt;131.225.240.106:47475&gt;; SHADOW at 13 
1.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_necula_medium_234248_16856/x509cc_necula: (errno 2) No such file or directory 
JID - 98629  Section_20  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_24  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_8  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_7  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_23  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_25  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_11  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_6  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_22  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_17  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_21  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_3  Reason - Error from starter on vm2@17222@fcdfcaf1501.fnal.gov: STARTER at 131.225.210.250 failed to send file(s) to &lt;131.225.240.106:56636&gt;; SHADOW at 131 
.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5898/section_3.err: (errno 2) No such file or directory 
JID - 98629  Section_2  Reason - Error from starter on vm2@17436@fcdfcaf1194.fnal.gov: STARTER at 131.225.210.103 failed to send file(s) to &lt;131.225.240.106:47647&gt;; SHADOW at 131 
.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5898/CafExe_2.log: (errno 2) No such file or directory 
JID - 98629  Section_18  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_12  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_14  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_15  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_1  Reason - Error from starter on vm2@10272@fcdfcaf1402.fnal.gov: STARTER at 131.225.210.150 failed to send file(s) to &lt;131.225.240.106:56323&gt;; SHADOW at 131 
.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5898/CafExe_1.log: (errno 2) No such file or directory 
JID - 98629  Section_9  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_10  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_4  Reason - Error from starter on vm2@17086@fcdfcaf1501.fnal.gov: STARTER at 131.225.210.250 failed to send file(s) to &lt;131.225.240.106:56784&gt;; SHADOW at 131 
.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5898/CafExe_4.log: (errno 2) No such file or directory 
JID - 98629  Section_13  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_19  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_16  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5898: No such file or directory 
JID - 98629  Section_5  Reason - Error from starter on vm2@32118@fcdfcaf1403.fnal.gov: STARTER at 131.225.210.151 failed to send file(s) to &lt;131.225.240.106:56986&gt;; SHADOW at 131 
.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5898/section_5.err: (errno 2) No such file or directory 
JID - 97882  Section_77  Reason - Error from starter on vm2@14341@fcdfcaf1425.fnal.gov: STARTER at 131.225.210.173 failed to send file(s) to &lt;131.225.240.106:47384&gt;; SHADOW at 13 
1.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_necula_medium_234251_23148/section_77.err: (errno 2) No such file or directory 
JID - 97884  Section_101  Reason - Error from starter on vm2@22380@fcdfcaf1128.fnal.gov: STARTER at 131.225.210.37 failed to send file(s) to &lt;131.225.240.106:51738&gt;; SHADOW at 13 
1.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_necula_medium_234255_30158/section_101.procs: (errno 2) No such file or directory 
JID - 98659  Section_1  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_johnpaul_medium_234704_31759: No such file or directory 
JID - 98659  Section_4  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_johnpaul_medium_234704_31759: No such file or directory 
JID - 98659  Section_3  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_johnpaul_medium_234704_31759: No such file or directory 
JID - 98659  Section_7  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_johnpaul_medium_234704_31759: No such file or directory 
JID - 98659  Section_5  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_johnpaul_medium_234704_31759: No such file or directory 
JID - 98659  Section_6  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_johnpaul_medium_234704_31759: No such file or directory 
JID - 98659  Section_2  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_johnpaul_medium_234704_31759: No such file or directory 
JID - 98625  Section_25  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_9  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_20  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_15  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_18  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_8  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_13  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_3  Reason - Error from starter on vm2@7547@fcdfcaf984.fnal.gov: STARTER at 131.225.212.133 failed to send file(s) to &lt;131.225.240.106:47477&gt;; SHADOW at 131.2 
25.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5709/section_3.out: (errno 2) No such file or directory 
JID - 98625  Section_12  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_2  Reason - Error from starter on vm2@20498@fcdfcaf1216.fnal.gov: STARTER at 131.225.213.132 failed to send file(s) to &lt;131.225.240.106:47806&gt;; SHADOW at 131 
.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5709/section_2.out: (errno 2) No such file or directory 
JID - 98625  Section_6  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_14  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_19  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_23  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_11  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_7  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_17  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_5  Reason - Error from starter on vm2@9842@fcdfcaf1287.fnal.gov: STARTER at 131.225.213.42 failed to send file(s) to &lt;131.225.240.106:57015&gt;; SHADOW at 131.2 
25.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5709/section_5.err: (errno 2) No such file or directory 
JID - 98625  Section_22  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_24  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_1  Reason - Error from starter on vm2@17837@fcdfcaf1273.fnal.gov: STARTER at 131.225.213.28 failed to send file(s) to &lt;131.225.240.106:43820&gt;; SHADOW at 131. 
225.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5709/section_1.err: (errno 2) No such file or directory 
JID - 98625  Section_4  Reason - Error from starter on vm2@9939@fcdfcaf1113.fnal.gov: STARTER at 131.225.210.22 failed to send file(s) to &lt;131.225.240.106:47469&gt;; SHADOW at 131.2 
25.240.106 failed to write to file /export/CafCondor/cafIn/submit_schsu_long_234692_5709/section_4.err: (errno 2) No such file or directory 
JID - 98625  Section_10  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_21  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 98625  Section_16  Reason - Cannot access initial working directory /export/CafCondor/cafIn/submit_schsu_long_234692_5709: No such file or directory 
JID - 97901  Section_100  Reason - Error from starter on vm2@13499@fcdfcaf1216.fnal.gov: STARTER at 131.225.213.132 failed to send file(s) to &lt;131.225.240.106:43271&gt;; SHADOW at 1 
31.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_necula_medium_234265_12993/section_100.out: (errno 2) No such file or directory 
JID - 97913  Section_111  Reason - Error from starter on vm2@3443@fcdfcaf1067.fnal.gov: STARTER at 131.225.212.216 failed to send file(s) to &lt;131.225.240.106:47021&gt;; SHADOW at 13 
1.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_necula_medium_234276_5661/section_111.out: (errno 2) No such file or directory 
JID - 97913  Section_126  Reason - Error from starter on vm2@30507@fcdfcaf1413.fnal.gov: STARTER at 131.225.210.161 failed to send file(s) to &lt;131.225.240.106:39465&gt;; SHADOW at 1 
31.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_necula_medium_234276_5661/section_126.IOlog: (errno 2) No such file or directory 
JID - 97916  Section_60  Reason - Error from starter on vm2@31960@fcdfcaf1272.fnal.gov: STARTER at 131.225.213.27 failed to send file(s) to &lt;131.225.240.106:49210&gt;; SHADOW at 131 
.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_necula_medium_234280_18758/section_60.IOlog: (errno 2) No such file or directory 
JID - 97916  Section_98  Reason - Error from starter on vm2@25453@fcdfcaf1059.fnal.gov: STARTER at 131.225.212.208 failed to send file(s) to &lt;131.225.240.106:51619&gt;; SHADOW at 13 
1.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_necula_medium_234280_18758/x509cc_necula: (errno 2) No such file or directory 
JID - 97916  Section_135  Reason - Error from starter on vm2@16861@fcdfcaf1246.fnal.gov: STARTER at 131.225.213.162 failed to send file(s) to &lt;131.225.240.106:54459&gt;; SHADOW at 1 
31.225.240.106 failed to write to file /export/CafCondor/cafIn/submit_necula_medium_234280_18758/section_135.err: (errno 2) No such file or directory 
 
Need to execute these commands to remove them 
 
source ~/condor/sh/condor_setup.sh 
source ~/grid/sh/interactive_grid.sh 
 
          condor_rm  -name schedd_2@fcdfhead2.fnal.gov -constraint 'DAGManJobId==97920' 
          condor_rm  -name schedd_2@fcdfhead2.fnal.gov -constraint 'DAGManJobId==97930' 
          condor_rm  -name schedd_1@fcdfhead2.fnal.gov -constraint 'DAGManJobId==98703' 
          condor_rm  -name schedd_4@fcdfhead2.fnal.gov -constraint 'DAGManJobId==98624' 
          condor_rm  -name schedd_4@fcdfhead2.fnal.gov -constraint 'DAGManJobId==98621' 
          condor_rm  -name schedd_4@fcdfhead2.fnal.gov -constraint 'DAGManJobId==98622' 
          condor_rm  -name schedd_4@fcdfhead2.fnal.gov -constraint 'DAGManJobId==98623' 
          condor_rm  -name schedd_2@fcdfhead2.fnal.gov -constraint 'DAGManJobId==97730' 
          condor_rm  -name schedd_4@fcdfhead2.fnal.gov -constraint 'DAGManJobId==98626' 
          condor_rm  -name schedd_2@fcdfhead2.fnal.gov -constraint 'DAGManJobId==97872' 
          condor_rm  -name schedd_2@fcdfhead2.fnal.gov -constraint 'DAGManJobId==97873' 
          condor_rm  -name schedd_2@fcdfhead2.fnal.gov -constraint 'DAGManJobId==97875' 
          condor_rm  -name schedd_2@fcdfhead2.fnal.gov -constraint 'DAGManJobId==97876' 
          condor_rm  -name schedd_2@fcdfhead2.fnal.gov -constraint 'DAGManJobId==97878' 
          condor_rm  -name schedd_4@fcdfhead2.fnal.gov -constraint 'DAGManJobId==98629' 
          condor_rm  -name schedd_2@fcdfhead2.fnal.gov -constraint 'DAGManJobId==97882' 
          condor_rm  -name schedd_2@fcdfhead2.fnal.gov -constraint 'DAGManJobId==97884' 
          condor_rm  -name schedd_1@fcdfhead2.fnal.gov -constraint 'DAGManJobId==98659' 
          condor_rm  -name schedd_4@fcdfhead2.fnal.gov -constraint 'DAGManJobId==98625' 
          condor_rm  -name schedd_2@fcdfhead2.fnal.gov -constraint 'DAGManJobId==97901' 
          condor_rm  -name schedd_2@fcdfhead2.fnal.gov -constraint 'DAGManJobId==97913' 
          condor_rm  -name schedd_2@fcdfhead2.fnal.gov -constraint 'DAGManJobId==97916' 
</pre>
<!-- Author: Doug B -->



<!-- Date: Mon Jul 23 19:23:26 2007 -->
<pre>Monitoring of production farm (finished on Thursday/Friday last week). 
http://fncdfsrv0.fnal.gov/condorcaf/ 
 
Problem: Necessary update of condor monitor due to high consume of memory. Monitoring not running, script version mismatch between cgi web server scripts and scripts for data collecting (cafmon stuff). 
Action: synchronize all versions of scripts regarding to monitoring of GroupCAF and CondorCAF (prod. farm), that means, between fcdfmon5 and fncdfsrv0. 
 
NOTE: Notice, that there were two types of file manipulation. As user cafmon and as user cdfcaf. (In some cases also root :-), but shouldn't be needed in the future) 
 
 
1) Backuped current stuff on fncdfsrv0 
 
@fncdfsrv0: 
[cafmon@fncdfsrv0 www]$ cd /home/cafmon/CafCondor/www 
[cafmon@fncdfsrv0 www]$ mv cgi/ cgi-backup/ 
[cafmon@fncdfsrv0 www]$ mv scripts/ scripts-backup/ 
 
 
2) Created tar of cgi and script folder from fcdfmon5 and copied into fncdfsrv0 
 
@fcdfmon5: 
[cafmon@fcdfmon5 ~]$ cd /var/www/cgi-bin/groupcaf/ 
[cafmon@fcdfmon5 groupcaf]$ tar cvzf /tmp/www_cgi_scripts_mon5.tar.gz . 
[cafmon@fcdfmon5 groupcaf]$ cd /home/cafmon/CafCondor/www/ 
[cafmon@fcdfmon5 www]$ tar cvzf /tmp/www_scripts_mon5.tar.gz scripts/ 
[cafmon@fcdfmon5 www]$ cd 
[cafmon@fcdfmon5 ~]$ scp /tmp/www_cgi_scripts_mon5.tar.gz cafmon@fncdfsrv0:/home/cafmon/CafCondor/www/. 
[cafmon@fcdfmon5 ~]$ scp /tmp/www_scripts_mon5.tar.gz cafmon@fncdfsrv0:/home/cafmon/CafCondor/www/. 
 
@fncdfsrv0: 
[cafmon@fncdfsrv0 cafmon]$ cd /home/cafmon/CafCondor/www 
[cafmon@fncdfsrv0 cafmon]$ mkdir cgi 
[cafmon@fncdfsrv0 cafmon]$ cd cgi 
[cafmon@fncdfsrv0 cgi]$ cp ../www_cgi_scripts_mon5.tar.gz . 
[cafmon@fncdfsrv0 cgi]$ tar xvzf www_cgi_scripts_mon5.tar.gz 
[cafmon@fncdfsrv0 cgi]$ cp ../cgi-backup/config/* config/ 
[cafmon@fncdfsrv0 cgi]$ pwd 
/home/cafmon/CafCondor/www/cgi 
[cafmon@fncdfsrv0 cgi]$ cd .. 
[cafmon@fncdfsrv0 www]$ tar xvzf www_scripts_mon5.tar.gz 
[cafmon@fncdfsrv0 www]$ cp scripts-backup/monitor.cfg* scripts/ 
 
Little fix, because of wrong reference to history/status web link: 
[cafmon@fncdfsrv0 www]$ cp cgi-backup/history/title.py cgi/history/. 
[cafmon@fncdfsrv0 www]$ cp cgi-backup/status/title.py cgi/status/. 
 
 
3) Condor monitor re-install: 
(logged in as cdfcaf user) 
@fncdfsrv0: 
[cdfcaf@fncdfsrv0 cdfcaf]$ cd CafCondor 
[cdfcaf@fncdfsrv0 CafCondor]$ tar cvzf ~/CafCondor/condor-monitor-fncdfsrv0.tar.gz monitor/ 
[cdfcaf@fncdfsrv0 CafCondor]$ cd monitor 
[cdfcaf@fncdfsrv0 CafCondor]$ ./startup monitor xml_monitor stop 
[cdfcaf@fncdfsrv0 CafCondor]$ ./startup monitor state_monitor stop 
[cdfcaf@fncdfsrv0 CafCondor]$ ./startup monitor monitor stop 
[cdfcaf@fncdfsrv0 monitor]$ rm *.py* 
[cdfcaf@fncdfsrv0 monitor]$ export CVSROOT=:pserver:anonymous@cdfcvs.fnal.gov:/cdf/code/cdfcvs/run2 
[cdfcaf@fncdfsrv0 monitor]$ cvs update 
[cdfcaf@fncdfsrv0 monitor]$ cd .. 
[cdfcaf@fncdfsrv0 CafCondor]$ ./restartmon 
 
Finally, works!</pre>
<!-- Author: Marian -->


<!-- Date: Mon Jul 23 19:51:18 2007 -->
<pre>Performed on Friday 20, 2007 
 
Added ~150 slots to Alignment group on FermiGrid 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
@fcdfhead2: GlideCAF 
[cdfcaf@fcdfhead2:~] $ cd condor/dist/etc 
[cdfcaf@fcdfhead2:~/condor/dist/etc] $ vim group_params.config 
GROUP_QUOTA_DYNAMIC_group_alignment = 0.00 
-&gt; GROUP_QUOTA_DYNAMIC_group_alignment = 0.04 
:wq! 
[cdfcaf@fcdfhead2:~/condor/dist/etc] $ cd /export/CafCondor/acctgroups/ 
[cdfcaf@fcdfhead2:/export/CafCondor/acctgroups] $ vim acctgroups.cfg 
[alignment] 
max_parallel_jobs = 150 
:wq! 
[cdfcaf@fcdfhead2:/export/CafCondor/acctgroups] $ cd 
[cdfcaf@fcdfhead2:~] $ cd CafCondor 
[cdfcaf@fcdfhead2:~/CafCondor] $ source ~/condor/sh/condor_setup.sh 
[cdfcaf@fcdfhead2:~/CafCondor] $ ./startup acctgroups acctgroups restart 
[cdfcaf@fcdfhead2:~/CafCondor] $ condor_reconfig -all 
[cdfcaf@fcdfhead2:~/CafCondor] $ ./startup monitor monitor restart 
[cdfcaf@fcdfhead2:~/CafCondor] $ ./startup monitor state_monitor restart 
[cdfcaf@fcdfhead2:~/CafCondor] $ ./startup monitor xml_monitor restart 
 
 
Change of slots for groups alignment, prd, sam on GroupCAF 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
@fcdfhead3: GroupCAF 
[cdfcaf@fcdfhead3 cdfcaf]$ cd condor/dist/etc 
[cdfcaf@fcdfhead3 etc]$ vim group_params.config 
GROUP_QUOTA_group_alignment = 40 
-&gt; GROUP_QUOTA_group_alignment = 100 
GROUP_QUOTA_group_prd  = 440 
-&gt; GROUP_QUOTA_group_prd  = 575 
GROUP_QUOTA_group_sam = 150 
-&gt; GROUP_QUOTA_group_sam = 75 
:wq! 
[cdfcaf@fcdfhead3 etc]$ cd /export/CafCondor/acctgroups/ 
[cdfcaf@fcdfhead3 acctgroups]$ vim acctgroups.cfg 
-&gt; changes for every group as above 
[cdfcaf@fcdfhead3 acctgroups]$ cd 
[cdfcaf@fcdfhead3 cdfcaf]$ cd CafCondor 
[cdfcaf@fcdfhead3 CafCondor]$ source ~/condor/sh/condor_setup.sh 
[cdfcaf@fcdfhead3 CafCondor]$ ./startup acctgroups acctgroups restart 
[cdfcaf@fcdfhead3 CafCondor]$ condor_reconfig -all 
[cdfcaf@fcdfhead3 CafCondor]$ ./startup monitor monitor restart 
[cdfcaf@fcdfhead3 CafCondor]$ ./startup monitor state_monitor restart 
[cdfcaf@fcdfhead3 CafCondor]$ ./startup monitor xml_monitor restart 
</pre>
<!-- Author: Marian -->


<!-- Date: Tue Jul 24 10:14:10 2007 -->
<pre>Problem: Some jobs stalled in the queue, but not assigned to the WN. 
Action: removing of them  
 
[cdfcaf@fncdfsrv0 cdfcaf]$ condor_q -name fncdfsrv0.fnal.gov -running 
-- Schedd: fncdfsrv0.fnal.gov : &lt;131.225.239.58:38298&gt; 
 ID      OWNER            SUBMITTED     RUN_TIME HOST(S) 
535545.0   cdfprd          7/13 14:19   0+00:21:23 [????????????????] 
535548.0   cdfprd          7/13 14:19   0+00:21:42 [????????????????] 
535549.0   cdfprd          7/13 14:20   0+00:22:06 [????????????????] 
535550.0   cdfprd          7/13 14:20   0+00:22:57 [????????????????] 
535553.0   cdfprd          7/13 14:20   0+00:21:25 [????????????????] 
535556.0   cdfprd          7/13 14:21   0+00:22:02 [????????????????] 
535560.0   cdfprd          7/13 14:22   0+00:22:07 [????????????????] 
535561.0   cdfprd          7/13 14:22   0+00:22:19 [????????????????] 
535562.0   cdfprd          7/13 14:22   0+00:21:29 [????????????????] 
535569.0   cdfprd          7/13 14:23   0+00:21:32 [????????????????] 
535572.0   cdfprd          7/13 14:24   0+00:21:21 [????????????????] 
535587.0   cdfprd          7/13 14:27   0+00:20:16 [????????????????] 
535589.0   cdfprd          7/13 14:27   0+00:21:13 [????????????????] 
535593.0   cdfprd          7/13 14:28   0+00:20:20 [????????????????] 
 
[cdfcaf@fncdfsrv0 cdfcaf]$ for i in `condor_q -name fncdfsrv0.fnal.gov -running | grep '?????' | awk '{print $1}'`; do condor_rm $i; done; 
Permission denied to remove job 535545.0 
...etc... 
 
[cdfcaf@fncdfsrv0 cdfcaf]$ ksu 
Authenticated zvada@FNAL.GOV 
Account root: authorization for zvada@FNAL.GOV successful 
Changing uid to root (0) 
[root@fncdfsrv0 cdfcaf]# for i in `condor_q -name fncdfsrv0.fnal.gov -running | grep '?????' | awk '{print $1}'`; do condor_rm $i; done; 
Job 535545.0 marked for removal 
Job 535548.0 marked for removal 
Job 535549.0 marked for removal 
Job 535550.0 marked for removal 
Job 535553.0 marked for removal 
Job 535556.0 marked for removal 
Job 535560.0 marked for removal 
Job 535561.0 marked for removal 
Job 535562.0 marked for removal 
Job 535569.0 marked for removal 
Job 535572.0 marked for removal 
Job 535587.0 marked for removal 
AUTHENTICATE:1002:Failure performing handshake 
No result found for job 535589.0 
Job 535593.0 marked for removal 
[root@fncdfsrv0 cdfcaf]# exit 
 
[cdfcaf@fncdfsrv0 cdfcaf]$ condor_q -name fncdfsrv0.fnal.gov -running                                         
-- Schedd: fncdfsrv0.fnal.gov : &lt;131.225.239.58:38298&gt; 
 ID      OWNER            SUBMITTED     RUN_TIME HOST(S) 
[cdfcaf@fncdfsrv0 cdfcaf]$</pre>
<!-- Author: Marian -->


<!-- Date: Sun Jul 29 21:59:39 2007 -->
<pre>Hi, 
 
   I noticed that the proxy information is slightly different between fcdfhead2, fcdfhead5 and fcdftest010. 
It most likely makes little difference.   
 
    Doug 
 
*************************   fcdfhead2 (FERMIGRID) 
 
cdfcaf@fcdfhead2:~/CafCondor/config] $ voms-proxy-info -file /export/CafCondor/tickets/x509_service_proxy 
WARNING: Unable to verify signature! Server certificate possibly not installed. 
Error: Cannot find certificate of AC issuer for vo cdf 
subject   : /DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN=cdf/CN=Renzo Borgatti/USERID=borgatti/CN=proxy 
issuer    : /DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN=cdf/CN=Renzo Borgatti/USERID=borgatti 
identity  : /DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN=cdf/CN=Renzo Borgatti/USERID=borgatti 
type      : proxy 
strength  : 1024 bits 
path      : /export/CafCondor/tickets/x509_service_proxy 
timeleft  : 103:45:20 
 
 
************************ fcdfhead5  (NamCaf) 
 
[cdfcaf@fcdfhead5:~] $ voms-proxy-info -file /export/CafCondor/tickets/x509_service_proxy 
WARNING: Unable to verify signature! Server certificate possibly not installed. 
Error: Cannot find certificate of AC issuer for vo cdf 
subject   : /DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN=cdf/CN=Doug Benjamin/USERID=benjamin/CN=proxy 
issuer    : /DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN=cdf/CN=Doug Benjamin/USERID=benjamin 
identity  : /DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN=cdf/CN=Doug Benjamin/USERID=benjamin 
type      : proxy 
strength  : 1024 bits 
path      : /export/CafCondor/tickets/x509_service_proxy 
timeleft  : 149:28:18 
 
 
********************* fcdftest010 (IgorCaf -  test CAF) 
-bash-2.05b$ hostname 
fcdftest010.fnal.gov 
-bash-2.05b$ voms-proxy-info -file /export/CafCondor/tickets/x509_service_proxy 
subject   : /DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN=cdf/CN=Doug Benjamin/USERID=benjamin/CN=proxy 
issuer    : /DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN=cdf/CN=Doug Benjamin/USERID=benjamin 
identity  : /DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN=cdf/CN=Doug Benjamin/USERID=benjamin 
type      : proxy 
strength  : 1024 bits 
path      : /export/CafCondor/tickets/x509_service_proxy 
timeleft  : 148:27:06 
 
</pre>
<!-- Author: Doug B -->


<!-- Date: Tue Jul 31 05:02:22 2007 -->
<pre>In order to restart  monitor I followed this steps : 
 
1- restart nfs in fcdftest010 like root : 
 
[cdfcaf@fcdftest010:/export] $ ksu 
Authenticated fmoscato@FNAL.GOV 
Account root: authorization for fmoscato@FNAL.GOV successful 
Changing uid to root (0) 
[root@fcdftest010 export]# /sbin/service nfs start 
Starting NFS services:                                     [  OK  ] 
Starting NFS quotas:                                       [  OK  ] 
Starting NFS daemon:                                       [  OK  ] 
Starting NFS mountd:                                       [  OK  ] 
 
2- I did a backup of the old fcdftest013 /export/CafCondor . You can find in 
/tmp/bkup.310707 
 
3-reboot fcdftest013 
 
4- restart xml_monitor on fcdftest013. 
 
 
Now the fcdftest013 df output is 
 
Filesystem           1K-blocks      Used Available Use% Mounted on 
/dev/hda2             10080520   1383644   8184808  15% / 
/dev/hda1               505605     18884    460617   4% /boot 
/dev/hdb1             40313964    282124  37983956   1% /fbsng 
none                   1541112         0   1541112   0% /dev/shm 
/dev/hdb2            200048616     32828 189853876   1% /scratch_dcache 
/dev/hda3              5036316     36312   4744172   1% /tmp 
/dev/hda5              5036284    266688   4513764   6% /var 
fcdftest010:/fbsng/CafCondor_test 
                      40313968  15084848  23181240  40% /export/CafCondor 
blue3.fnal.gov:/export/CafCondor/fcdftest010 
                     2147483648  85209024 2062274624   4% /home/cdfcaf 
 
 
This is the updateStatus (cafmon5) log : 
 
Tue Jul 31 04:38:11 2007: CondorCAF updateStatus: Contacting server fcdftest013.fnal.gov:9999 
Tue Jul 31 04:38:11 2007: CondorCAF updateStatus: Got data 
Tue Jul 31 04:38:11 2007: CondorCAF updateStatus: Data merged 
Tue Jul 31 04:38:11 2007: CondorCAF updateStatus: Summary ready 
Tue Jul 31 04:38:12 2007: CondorCAF updateStatus: Ready to write files 
Tue Jul 31 04:38:12 2007: CondorCAF updateStatus: Files written 
Tue Jul 31 04:38:14 2007: CondorCAF updateStatus: RRDs updated 
Tue Jul 31 04:40:11 2007: CondorCAF updateStatus: Reading cached data 
Tue Jul 31 04:40:11 2007: CondorCAF updateStatus: Cached data read 
Tue Jul 31 04:40:11 2007: CondorCAF updateStatus: Found 4 sections to ignore 
Tue Jul 31 04:40:11 2007: CondorCAF updateStatus: Reading cached data5 
Tue Jul 31 04:40:11 2007: CondorCAF updateStatus: Cached data5 read 
Tue Jul 31 04:40:11 2007: CondorCAF updateStatus: Contacting server fcdftest013.fnal.gov:9999 
 
 
So you can see the normal lifecycle of the code. 
And also the updateCompleted log  seems good : 
 
Tue Jul 31 04:14:03 2007: CondorCAF updateCompleted: Loading cache data from /home/cafmon/igorcaf/data/completed 
Tue Jul 31 04:14:04 2007: CondorCAF updateCompleted: Getting current list of completed jobs from fcdftest013.fnal.gov:9999 
Tue Jul 31 04:14:04 2007: CondorCAF updateCompleted: Updating cache (cache: 1016, data: 1118) 
Tue Jul 31 04:14:13 2007: CondorCAF updateCompleted: Found new job: 1521 
Tue Jul 31 04:14:16 2007: CondorCAF updateCompleted: Found new job: 1520 
Tue Jul 31 04:14:16 2007: CondorCAF updateCompleted: Found new job: 1519 
Tue Jul 31 04:14:16 2007: CondorCAF updateCompleted: Added 3 jobs, failed 0, ignored 99 
</pre>
<!-- Author: federica -->


<!-- Date: Thu Aug 2 00:09:26 2007 -->
<pre>Here is something for a best practices section of write up. 
 
I recreated entry points with the same name on the test CAF. 
Prior to recreating the entry points, I modified the condor_config code in the web_base area. Once I created the entry point the 
signature file contents changed as expected.  The glidein would not start because of the error: 
 
************ At the end of the *.out file ************* 
------- =================== --------------- 
Started in /grid/home/cdf/gram_scratch_RiutyvYJzd 
Running in /local/stage1/condor/execute/dir_14321/glide_p14345 
signature.sha1: FAILED 
 
******** In the *.err file ********************** 
-bash-2.05b$ cat /export/CafCondor/cafIn/glidein_fcdfosgt1_srv001_glexec/log/job_dbg.60933.err 
sha1sum: WARNING: 1 of 1 computed checksum did NOT match 
Corrupted signature file! 
 
I realized that because we use a squid, I had to flush the squid of the old files. 
 
logging into fcdfsrv001 as root: 
 
[root@fcdfsrv001 root]# squidclient -m PURGE http://fcdftest010.fnal.gov/condorcafstage//glidein_fcdfosgt1_fcdfsrv001_glexec/signature.sha1 
HTTP/1.0 200 OK 
Server: squid/2.5.STABLE3 
Mime-Version: 1.0 
Date: Thu, 02 Aug 2007 04:17:43 GMT 
Content-Length: 0 
X-Cache: MISS from fcdfsrv001.fnal.gov 
Proxy-Connection: close 
 
[root@fcdfsrv001 root]# squidclient -m PURGE http://fcdftest010.fnal.gov/condorcafstage//glidein_fcdfosgt1_fcdfsrv001_glexec/condor_config 
HTTP/1.0 200 OK 
Server: squid/2.5.STABLE3 
Mime-Version: 1.0 
Date: Thu, 02 Aug 2007 04:18:00 GMT 
Content-Length: 0 
X-Cache: MISS from fcdfsrv001.fnal.gov 
Proxy-Connection: close 
 
 
squidclient -m PURGE http://.....  purges the file 
from the squid cache. 
 
Once the squid cache's were purged the proper files got to the 
glidein's and the system ran correctly. 
 
I have reused a submission point before, but had not changed the code used to recreate it. 
 
 
 
</pre>
<!-- Author: Doug -->


<!-- Date: Sun Aug 5 22:46:21 2007 -->
<pre>Created roster file for FermiGrid. 
Official nodes in CDF grid cluster 1--&gt; fcdfcaf972-1091 
                  CDF grid cluster 2--&gt; fcdfcaf1092-1501 
 
[cdfcaf@fcdfhead2:~/CafCondor/condor/scripts] $ vim create_roster.sh 
 
#!/bin/bash 
 
var=972 
LIMIT=1502 
 
touch /tmp/roster 
 
while [ "$var" -lt "$LIMIT" ] 
do 
  echo "fcdfcaf"$var".fnal.gov" &gt;&gt; /tmp/roster 
  var=`expr $var + 1` 
done 
 
exit 0 
 
:wq 
 
[cdfcaf@fcdfhead2:~/CafCondor/condor/scripts] $ mv /tmp/roster /home/cdfcaf/condor/dist/etc/roster 
</pre>
<!-- Author: Marian -->


<!-- Date: Mon Aug 6 11:25:06 2007 -->
<pre>Hi, 
 
  Because fcdfhead3 has the lovely omaws process running (a "nice" feature from Dell) 
 
I had to do the following steps to restart the submitter. 
 
1) log into fcdfhead3 as cdfcaf 
2) ksu 
 
[root@fcdfhead3 CafCondor]# netstat -anp -tcp | grep 8000 
tcp        0      0 0.0.0.0:8000                0.0.0.0:*                   LISTEN      4544/omaws32 
tcp        0      0 131.225.211.116:8000        131.225.232.190:32816       TIME_WAIT   - 
tcp        0      0 0.0.0.0:8000                0.0.0.0:*                   LISTEN      4544/omaws32 
tcp        0      0 131.225.211.116:8000        131.225.232.190:32816       TIME_WAIT   - 
tcp        0      0 0.0.0.0:8000                0.0.0.0:*                   LISTEN      4544/omaws32 
tcp        0      0 0.0.0.0:8000                0.0.0.0:*                   LISTEN      4544/omaws32 
 
&lt;ctrl-C&gt; 
 
[root@fcdfhead3 CafCondor]# kill -9 4544 
[root@fcdfhead3 CafCondor]# exit 
exit 
[cdfcaf@fcdfhead3 CafCondor]$ ./startup submitter submitter start 
submitter/submitter start 
[cdfcaf@fcdfhead3 CafCondor]$ CommModule.py:5: RuntimeWarning: Python C API version mismatch for module krb5: This Python has API version 1011, module krb5 has version 1007. 
  import krb5 
SUBMITTER PORT: 8000 
PID: 14507 
 
&lt;return&gt; 
 
[cdfcaf@fcdfhead3 CafCondor]$ ./startup submitter submitter status 
submitter/submitter status 
submitter is running under pid: 14508 
</pre>
<!-- Author: Doug B -->


<!-- Date: Mon Aug 6 13:51:21 2007 -->
<pre>To start the squid servers on each of the nodes 
fcdfsrv001, fcdfsrv002 and fcdfsrv003 
 
/etc/init.d/squid start 
 
I also turned the squid service on for boot levels 345 
for each node.</pre>
<!-- Author: Doug B -->


<!-- Date: Tue Aug 7 20:12:35 2007 -->
<pre>Hi, 
 
 Here are all of the steps needed to fermigrid somewhat running today. 
1) restart all of squid servers.  -  changing the chkconfig setttings to start squid servers automatically 
2)  Modifying the condor startup script to start the extra schedd's for fcdfosg2 submission point (should have been done 
   before - Needs to go into CVS) 
3) starting nfs on fcdfhead2 (chaning chkconfig settings to start nfs automatically) 
4) modifying the automouter on fcdftest012 to mount /export/CafCondor from fcdfhead2 
5) restarting xml monitoring from fcdftest012 
6) giving up and doing a kill -9 -1 fromcdfcaf account on fcdfhead2, 
   a) restarting condor 
   b) then restarting the CAF code. 
 
Fermigrid seems to be working.  At least condor_status gives more than 4 slots on fcdfhead2 and 
shows the running jobs on the glidein nodes. 
 
condor_status -master shows all of the slots on the various worker nodes. 
 
[cdfcaf@fcdfhead2:~/CafCondor/glidein] $ condor_status -total 
 
                    Total Owner Claimed Unclaimed Matched Preempting Backfill 
 
        INTEL/LINUX   334   167     167         0       0          0        0 
       X86_64/LINUX   320   160     160         0       0          0        0 
 
              Total   654   327     327         0       0          0        0 
 
So the numbers are not too wildly off.</pre>
<!-- Author: Doug B -->


<!-- Date: Tue Aug 7 21:10:41 2007 -->
<pre>Hi, 
 
after cvs update tricks and some hard copies I've got monitoring running (details -see in --addon). 
 
Here are some comments what I figured out and maybe helpful for further investigation. 
 
*1) This is last normal monitoring cycle which passed on Mon Aug  06:* 
Mon Aug  6 00:09:01 2007: CondorCAF updateStatus: Reading cached data 
Mon Aug  6 00:09:04 2007: CondorCAF updateStatus: Cached data read 
Mon Aug  6 00:09:04 2007: CondorCAF updateStatus: Found 15191 sections to ignore 
Mon Aug  6 00:09:04 2007: CondorCAF updateStatus: Reading cached data5 
Mon Aug  6 00:09:06 2007: CondorCAF updateStatus: Cached data5 read 
Mon Aug  6 00:09:06 2007: CondorCAF updateStatus: Contacting server fcdfhead3.fnal.gov:8999 
Mon Aug  6 00:09:17 2007: CondorCAF updateStatus: Got data 
Mon Aug  6 00:09:17 2007: CondorCAF updateStatus: Data merged 
Mon Aug  6 00:09:19 2007: CondorCAF updateStatus: Summary ready 
Mon Aug  6 00:09:19 2007: CondorCAF updateStatus: Ready to write files 
Mon Aug  6 00:09:39 2007: CondorCAF updateStatus: Files written 
Mon Aug  6 00:10:00 2007: CondorCAF updateStatus: RRDs updated 
 
*2) Then happened following:* 
Mon Aug  6 00:12:01 2007: CondorCAF updateStatus: Reading cached data 
Mon Aug  6 00:12:04 2007: CondorCAF updateStatus: Cached data read 
Mon Aug  6 00:12:04 2007: CondorCAF updateStatus: Found 15200 sections to ignore 
Mon Aug  6 00:12:04 2007: CondorCAF updateStatus: Reading cached data5 
Mon Aug  6 00:12:07 2007: CondorCAF updateStatus: Cached data5 read 
Mon Aug  6 00:12:07 2007: CondorCAF updateStatus: Contacting server fcdfhead3.fnal.gov:8999 
Traceback (most recent call last): 
 File "updateStatus.py", line 228, in ? 
   raw_data = XMLMonRead.GetStatus(xmlmon_node,xmlmon_port,ignore_sections) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/XMLMonRead.py", line 8, in GetStatus 
   data=getData(node,port,ignore_sections) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/XMLMonRead.py", line 39, in getData 
   raise Exception, "XMLMon connection unexpectedly terminated" 
Exception: XMLMon connection unexpectedly terminated 
 
*3) After this, next cron call of monitoring cycle was normal again.* 
 
*4) And at 6:48am happened following:* 
Mon Aug  6 06:48:02 2007: CondorCAF updateStatus: Reading cached data 
Mon Aug  6 06:48:04 2007: CondorCAF updateStatus: Cached data read 
Mon Aug  6 06:48:04 2007: CondorCAF updateStatus: Found 15220 sections to ignore 
Mon Aug  6 06:48:04 2007: CondorCAF updateStatus: Reading cached data5 
Mon Aug  6 06:48:07 2007: CondorCAF updateStatus: Cached data5 read 
Mon Aug  6 06:48:07 2007: CondorCAF updateStatus: Contacting server fcdfhead3.fnal.gov:8999 
Traceback (most recent call last): 
 File "updateStatus.py", line 228, in ? 
   raw_data = XMLMonRead.GetStatus(xmlmon_node,xmlmon_port,ignore_sections) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/XMLMonRead.py", line 8, in GetStatus 
   data=getData(node,port,ignore_sections) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/XMLMonRead.py", line 31, in getData 
   fd.connect((node,port)) 
 File "&lt;string&gt;", line 1, in connect 
socket.error: (111, 'Connection refused') 
 
*5) ...and this continuously repeated until 08:15am:* 
Mon Aug  6 08:15:12 2007: CondorCAF updateStatus: Contacting server fcdfhead3.fnal.gov:8999 
Traceback (most recent call last): 
 File "updateStatus.py", line 228, in ? 
   raw_data = XMLMonRead.GetStatus(xmlmon_node,xmlmon_port,ignore_sections) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/XMLMonRead.py", line 8, in GetStatus 
   data=getData(node,port,ignore_sections) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/XMLMonRead.py", line 31, in getData 
   fd.connect((node,port)) 
 File "&lt;string&gt;", line 1, in connect 
socket.error: (111, 'Connection refused') 
* 
6) ...in the next cron call it disappeared and we'got different error message:* 
Mon Aug  6 08:18:01 2007: CondorCAF updateStatus: Reading cached data 
Mon Aug  6 08:18:04 2007: CondorCAF updateStatus: Cached data read 
Mon Aug  6 08:18:04 2007: CondorCAF updateStatus: Found 15220 sections to ignore 
Mon Aug  6 08:18:04 2007: CondorCAF updateStatus: Reading cached data5 
Mon Aug  6 08:18:07 2007: CondorCAF updateStatus: Cached data5 read 
Mon Aug  6 08:18:07 2007: CondorCAF updateStatus: Contacting server fcdfhead3.fnal.gov:8999 
Mon Aug  6 08:18:36 2007: CondorCAF updateStatus: Got data 
Mon Aug  6 08:18:36 2007: CondorCAF updateStatus: Data merged 
Mon Aug  6 08:18:37 2007: CondorCAF updateStatus: Summary ready 
Mon Aug  6 08:18:37 2007: CondorCAF updateStatus: Ready to write files 
Mon Aug  6 08:18:54 2007: CondorCAF updateStatus: Files written 
Traceback (most recent call last): 
 File "updateStatus.py", line 369, in ? 
   RRDUpdate.updateAll(base_dir+"/rrd",update_time,summary_arr,raw_data) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/RRDUpdate.py", line 209, in updateAll 
   updateDHModes(base_dir,update_time,summary_data) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/RRDUpdate.py", line 100, in updateDHM 
odes 
   return updateMulti(list) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/RRDUpdate.py", line 12, in updateMult 
i 
   RRDWriter.writeCreateMulti(sublist) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/RRDWriter.py", line 149, in writeCrea 
teMulti 
   createNewMulti(createlist) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/RRDWriter.py", line 57, in createNewM 
ulti 
   raise Exception, string.join(tempErr,"\n") 
Exception: ERROR: can't parse argument '/home/cafmon/groupcaf/rrd/dhmode_SAM.rrd' 
 
*7) ...this error continued until 17:09 and then combination with XMLMonRead.py error:* 
Mon Aug  6 17:09:01 2007: CondorCAF updateStatus: Reading cached data 
Mon Aug  6 17:09:02 2007: CondorCAF updateStatus: Cached data read 
Mon Aug  6 17:09:02 2007: CondorCAF updateStatus: Found 11916 sections to ignore 
Mon Aug  6 17:09:02 2007: CondorCAF updateStatus: Reading cached data5 
Mon Aug  6 17:09:04 2007: CondorCAF updateStatus: Cached data5 read 
Mon Aug  6 17:09:04 2007: CondorCAF updateStatus: Contacting server fcdfhead3.fnal.gov:8999 
Traceback (most recent call last): 
 File "updateStatus.py", line 228, in ? 
   raw_data = XMLMonRead.GetStatus(xmlmon_node,xmlmon_port,ignore_sections) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/XMLMonRead.py", line 8, in GetStatus 
   data=getData(node,port,ignore_sections) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/XMLMonRead.py", line 39, in getData 
   raise Exception, "XMLMon connection unexpectedly terminated" 
Exception: XMLMon connection unexpectedly terminated 
Mon Aug  6 17:12:01 2007: CondorCAF updateStatus: Checking if /proc/26465 is still running 
Mon Aug  6 17:12:01 2007: CondorCAF updateStatus: Reading cached data 
Mon Aug  6 17:12:03 2007: CondorCAF updateStatus: Cached data read 
Mon Aug  6 17:12:03 2007: CondorCAF updateStatus: Found 11916 sections to ignore 
Mon Aug  6 17:12:03 2007: CondorCAF updateStatus: Reading cached data5 
Mon Aug  6 17:12:05 2007: CondorCAF updateStatus: Cached data5 read 
Mon Aug  6 17:12:05 2007: CondorCAF updateStatus: Contacting server fcdfhead3.fnal.gov:8999 
Mon Aug  6 17:13:26 2007: CondorCAF updateStatus: Got data 
Mon Aug  6 17:13:26 2007: CondorCAF updateStatus: Data merged 
Mon Aug  6 17:13:29 2007: CondorCAF updateStatus: Summary ready 
Mon Aug  6 17:13:29 2007: CondorCAF updateStatus: Ready to write files 
Mon Aug  6 17:13:43 2007: CondorCAF updateStatus: Files written 
Traceback (most recent call last): 
 File "updateStatus.py", line 369, in ? 
   RRDUpdate.updateAll(base_dir+"/rrd",update_time,summary_arr,raw_data) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/RRDUpdate.py", line 209, in updateAll 
   updateDHModes(base_dir,update_time,summary_data) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/RRDUpdate.py", line 100, in updateDHM 
odes 
   return updateMulti(list) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/RRDUpdate.py", line 12, in updateMult 
i 
   RRDWriter.writeCreateMulti(sublist) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/RRDWriter.py", line 149, in writeCrea 
teMulti 
   createNewMulti(createlist) 
 File "/home/cafmon/condor_condorc/CafCondor/www/scripts/RRDWriter.py", line 57, in createNewM 
ulti 
   raise Exception, string.join(tempErr,"\n") 
Exception: ERROR: can't parse argument '/home/cafmon/groupcaf/rrd/dhmode_SAM.rrd' 
 
 
*Summary: 
~~~~~~~~ 
*I believe, that whole this error cycle *from '*Mon Aug  6 08:18:01 2007' *until* 'Tue Aug  7 13:48:01 2007' caused that plots where decreasing at http://cdfcaf.fnal.gov/groupcaf/history/user.html. 
 
Now, after some update tricks and restarts, error of RRD update cycle is fixed and running NORMAL looking at /tmp/cron_groupcaf_updateStatus.log @fcdfmon5. 
 
But, increasing of plots doesn't start yet and there is bunch of jobs running: 
[cdfcaf@fcdfhead3 CafCondor]$ condor_q -global -constraint "JobStatus==2" | wc -l 
  2619 
 
I also made restart of monitor, state_monitor, xml_monitor on head3 but didn't help. 
 
---addon regarding to mentioned cvs update tricks--- 
CVS update "tricks" on fcdfmon5: 
(Note, that purging and cvs update scripts didn't help to fix life cycle error in log of RRD updateStatus, therefore I took CVS folder from glidecaf and then made hard copy of some files - see below) 
 
[cafmon@fcdfmon5 ~]$ cd /home/cafmon/CafCondor/www 
[cafmon@fcdfmon5 www]$ pwd 
/home/cafmon/CafCondor/www 
[cafmon@fcdfmon5 www]$ tar cvzf scripts-backup.tar.gz scripts/ 
[cafmon@fcdfmon5 www]$ cd scripts 
[cafmon@fcdfmon5 scripts]$ rm *.py*; rm 2; rm monitor.cfg.fnal; rm monitor.cfg.sdsc 
[cafmon@fcdfmon5 scripts]$ rm -rf CVS 
 
Note: keep the monitor.cfg file in scripts folder 
 
[cafmon@fcdfmon5 scripts]$ cp -R /home/cafmon/glidecaf/scripts/CVS/ . 
[cafmon@fcdfmon5 scripts]$ export CVSROOT=:pserver:anonymous@cdfcvs.fnal.gov:/cdf/code/cdfcvs/run2 
[cafmon@fcdfmon5 scripts]$ cvs update 
[cafmon@fcdfmon5 scripts]$ cd ../ 
[cafmon@fcdfmon5 www]$ pwd 
/home/cafmon/CafCondor/www 
[cafmon@fcdfmon5 www]$ mkdir scripts-unpacked 
[cafmon@fcdfmon5 www]$ cp scripts-backup.tar.gz scripts-unpacked/ 
[cafmon@fcdfmon5 www]$ cd scripts-unpacked/ 
[cafmon@fcdfmon5 scripts-unpacked]$ tar xvzf scripts-backup.tar.gz 
[cafmon@fcdfmon5 scripts-unpacked]$ ls 
scripts  scripts-backup.tar.gz 
[cafmon@fcdfmon5 scripts-unpacked]$ cd ../scripts 
[cafmon@fcdfmon5 scripts]$ pwd 
/home/cafmon/CafCondor/www/scripts 
[cafmon@fcdfmon5 scripts]$ cp ../scripts-unpacked/scripts/updateStatus.py . 
[cafmon@fcdfmon5 scripts]$ cp ../scripts-unpacked/scripts/XMLMonRead.py . 
[cafmon@fcdfmon5 scripts]$ cp ../scripts-unpacked/scripts/RRDUpdate.py . 
 
Now, life cycle update of RRDs works pretty nice (no errors), but plots don't show real running sections. We've realizes, that also CafMon doesn't work and this has nothing to do with web monitor. Finaly we deleted condor monitor on head3 as following: 
 
[cdfcaf@fcdfhead3 cdfcaf]$ cd CafCondor 
[cdfcaf@fcdfhead3 CafCondor]$ ./startup monitor monitor stop 
[cdfcaf@fcdfhead3 CafCondor]$ ./startup monitor state_monitor stop 
[cdfcaf@fcdfhead3 CafCondor]$ ./startup monitor xml_monitor stop 
[cdfcaf@fcdfhead3 CafCondor]$ mv monitor oldmonitor 
[cdfcaf@fcdfhead3 CafCondor]$ mkdir monitor 
[cdfcaf@fcdfhead3 CafCondor]$ cp -R oldmonitor/CVS monitor/. 
[cdfcaf@fcdfhead3 CafCondor]$ cd monitor 
[cdfcaf@fcdfhead3 monitor]$ cvs update 
... 
[cdfcaf@fcdfhead3 monitor]$ cd ../ 
[cdfcaf@fcdfhead3 CafCondor]$ ./startup monitor monitor start 
[cdfcaf@fcdfhead3 CafCondor]$ ./startup monitor state_monitor start 
[cdfcaf@fcdfhead3 CafCondor]$ ./startup monitor xml_monitor start 
 
And at the end, mailer: 
[cdfcaf@fcdfhead3 CafCondor]$ ./startup mailer mailer start 
 
After all of these steps, web monitoring and CafMon finally started to produce correct output status/history. 
 
</pre>
<!-- Author: Marian -->


<!-- Date: Tue Aug 7 22:32:23 2007 -->
<pre>Matt and I discovered that the condor processes would not start on NamCaf because the 
partition holding the logs was completely full. 
 
   When Condor will not start any daemon and write nothing into the log file then look 
for a full partition. Matt also noticed in one log that the logging stopped mid word. 
 
    We removed many old core files and Matt moved the Procd log files away. We also 
reduced the maximum size of the Procd log. (Procd - needed for glexec). 
 
  This afternoon I started the system. Put the jobs are not running. The monitoring claims 
things are running but they are not.  I guess that the monitor code should be upgraded to 
match the same version as the rest of the CAF's. 
 
</pre>
<!-- Author: Doug B -->


<!-- Date: Mon Aug 20 09:44:11 2007 -->
<pre>squid server fcdfsrv001 ran out of file descriptors 
 
2007/08/12 04:03:39| logfileRotate: /fbsng/squid/access.log 
2007/08/14 09:23:59| urlParse: Illegal character in hostname 'catsubmit_stn.tgz' 
2007/08/16 15:18:41| WARNING! Your cache is running out of filedescriptors 
2007/08/16 15:18:57| WARNING! Your cache is running out of filedescriptors 
2007/08/16 15:19:13| WARNING! Your cache is running out of filedescriptors 
2007/08/16 15:19:29| WARNING! Your cache is running out of filedescriptors 
 
I restarted it.</pre>
<!-- Author: Doug B -->


<!-- Date: Mon Aug 20 09:52:18 2007 -->
<pre>fcdfsrv003 squid is dead.  
 
Steps used to restart squid 
 
[root@fcdfsrv003 root]# /etc/init.d/squid status 
squid dead but pid file exists 
squid: ERROR: Could not send signal 0 to process 17231: (3) No such process 
[root@fcdfsrv003 root]# rm /var/run/squid.pid 
rm: remove regular file `/var/run/squid.pid'? y 
[root@fcdfsrv003 root]# /etc/init.d/squid status 
squid dead but subsys locked 
squid: ERROR: No running copy 
[root@fcdfsrv003 root]# ls -l /var/lock/subsys/squid 
-rw-r--r--    1 root     root            0 Aug  6 13:45 /var/lock/subsys/squid 
[root@fcdfsrv003 root]# rm /var/lock/subsys/squid 
rm: remove regular empty file `/var/lock/subsys/squid'? y 
[root@fcdfsrv003 root]# /etc/init.d/squid status 
squid is stopped 
squid: ERROR: No running copy 
[root@fcdfsrv003 root]# /etc/init.d/squid start 
Starting squid: .                                          [  OK  ] 
</pre>
<!-- Author: Doug B -->


<!-- Date: Sat Aug 25 15:30:49 2007 -->
<pre>Steps used to remove old jobs that condor thinks are running. 
condor_q -name fcdfhead3.fnal.gov  | grep '5/' &gt; /tmp/oldjobs_to_remove.txt 
 
for line in `cat /tmp/oldjobs_to_remove.txt | awk '{print $1}'`; do echo "condor_rm -name fcdfhead3.fnal.gov ${line}"; condor_rm -name fcdfhead3.fnal.gov ${line} ; done; 
 
After those commands.  The mailer took over and move the job logs. 
(The users should have gotten emai).</pre>
<!-- Author: Doug B -->


<!-- Date: Tue Aug 28 12:14:29 2007 -->
<pre>Dear all, 
 
  I have made the following changes to reduce the on fcdfosg2 from the CDF side of the 
equation. 
 
in condor_config 
 
   Condor variable - 
 
   JOB_START_COUNT = 4  (it was 8) 
   (Note JOB_START_DELAY = 2) 
 
in fcdfhead2.local 
 
  Condor variables - 
 
GRIDMANAGER_MAX_PENDING_SUBMITS_PER_RESOURCE=10 (was 25) 
GRIDMANAGER_MAX_JOBMANAGERS_PER_RESOURCE = 10   (was 25) 
 
 In side of glidekeeper: 
    submit_glideins routine - increase the sleep time to 1.5 seconds (was 0.5) 
 
  
  I have restarted the glidekeeper, I have reconfigured -all condor (after sourcing condor_setup.sh and interactive_grid.sh). 
I also killed the condor_gridmanagers to ensure that they got the changes required. 
 
   Cheers, 
 
  Doug 
</pre>
<!-- Author: Doug B -->


<!-- Date: Tue Aug 28 12:21:37 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=caf&action=view&page=-12&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=caf&action=view&page=-12  width='300' align='top'></a></td><td>CPU useage from ganaglia - fcdfhead2 after killing 
collector/negotiator (11:40) and condor_gridmanagers(12:00)</td></tr></table>
<!-- Author: Doug Benjamin -->


<!-- Date: Tue Aug 28 12:22:55 2007 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=caf&action=view&page=-10&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=caf&action=view&page=-10' width='300'></a></center>
<!-- Author: Doug B -->
<!-- Comment: Tue Aug 28 12;24;09 comment by...Doug -->
<pre>Memory useage fcdfhead2. Collector/Negotiator killed 11:40
condor_gridmanager killed at ~12:00
</pre>


<!-- Date: Tue Aug 28 16:37:27 2007 -->
<pre>In order to have the right timestamp for the production script  
I followed this steps: 
 
1-code (fncdfsrv0 & groupcaf): 
  CafCondor/www/cgi/xml/summary.py : 
  I added a new internal_attribute inside "job_info" folder  
  with the timestamp loaded to current.summary.pk 
  CafCondor/www/cgi/xml/running_list.py && running_job.py: 
  changed the timestamp -&gt; now for running_job is the short.pk   timestamp while for the running_list is current.summary.pk timestamp 
 
2- I changed the cron on fncdfsrv0: 
   
  
*/5 * * * * cd /home/cafmon/condorcaf/scripts; /usr/bin/python2 updateStatus.py &gt;&gt; /tmp/cron_condorcaf_updateStatus.log 
# run at same frequency, but the second should start a little bit later 
*/8 * * * * cd /home/cafmon/condorcaf/scripts; sleep 15; /usr/bin/python2 updateCompleted.py &gt;&gt; /tmp/cron_condorcaf_updateCompleted.log  
  </pre>
<!-- Author: federica -->


<!-- Date: Thu Sep 6 09:44:31 2007 -->
<pre>reconfigured the squid servers fcdfsrv001-fcdfsrv003 to 
listen to the snmp_port 
 
root@fcdfsrv001 root]# grep snmp_port /etc/squid/squid.conf 
#  TAG: snmp_port 
# snmp_port 0 
  snmp_port 3401 
[root@fcdfsrv001 root]# netstat -plun | grep -E '3401|squid' 
udp        0      0 0.0.0.0:33448           0.0.0.0:*                           16723/(squid) 
udp        0      0 0.0.0.0:3130            0.0.0.0:*                           16723/(squid) 
udp        0      0 0.0.0.0:3401            0.0.0.0:*                           16723/(squid) 
</pre>
<!-- Author: Doug B -->
<!-- Comment: Thu Sep  6 09;52;10 comment by...Doug B -->
<pre>Link to proof that the squid servers are working.
http://cdfdbfrontier4.fnal.gov:8888/indexspec</pre>


<!-- Date: Fri Sep 7 18:05:20 2007 -->
<pre>I installed new version of pacman on fcdftest010. v3.20 
as cdfcaf 
$cd ~ 
$wget http://vdt.cs.wisc.edu/software/pacman/3.20/pacman-3.20.tar.gz 
$tar xzvf pacman-3.20.tar.gz 
 
$ cd ~/pacman-3.20 
$ source setup.sh 
$ cd /scratch_dcache/VDT 
$ pacman -get http://vdt.cs.wisc.edu/vdt_180_cache:VDT-Client 
[yes to all questions] 
 
vdt-control --list 
(lists what was installed) 
 
As root: 
[root@fcdftest010 VDT]# vdt-register-service --enable --name fetch-crl 
vdt-register-service: updated cron service 'fetch-crl' 
vdt-register-service: desired state = enable 
vdt-register-service: cron time     = '10 3 * * *' 
vdt-register-service: cron command  = '/scratch_dcache/VDT/fetch-crl/share/doc/fetch-crl-2.6.2/fetch-crl.cron' 
[root@fcdftest010 VDT]# vdt-control --on fetch-crl 
enabling cron service fetch-crl... ok 
[root@fcdftest010 VDT]# vdt-register-service --enable --name vdt-rotate-logs 
vdt-register-service: updated cron service 'vdt-rotate-logs' 
vdt-register-service: desired state = enable 
vdt-register-service: cron time     = '0 0 * * *' 
vdt-register-service: cron command  = '/scratch_dcache/VDT/vdt/bin/vdt-rotate-logs' 
[root@fcdftest010 VDT]# vdt-control --on vdt-rotate-logs 
enabling cron service vdt-rotate-logs... ok 
</pre>
<!-- Author: Doug B -->


<!-- Date: Mon Sep 10 10:30:54 2007 -->
<pre>Performed on 5th of september: 
 
I've cleaned up schedd's from held jobs (all were dated from 30th of August and earlier): 
 
[cdfcaf@fcdfhead3 cdfcaf]$ condor_status -schedd 
Name                 TotalRunningJobs TotalIdleJobs TotalHeldJobs 
fcdfhead3.fnal.gov       0               0              0 
schedd_100@fcdfhead3     261            20             64 
schedd_101@fcdfhead3     542            65            118 
schedd_102@fcdfhead3     278           148              6 
schedd_103@fcdfhead3     1038           29             24 
 
[cdfcaf@fcdfhead3 cdfcaf]$ ksu 
 
---cleaning schedd_100 
[root@fcdfhead3 cdfcaf]$ condor_q -name schedd_100@ | grep H &gt; /tmp/heldjobs_schedd100_to_remove.txt 
[root@fcdfhead3 cdfcaf]# for line in `cat /tmp/heldjobs_schedd100_to_remove.txt | awk '{print $1}'`; do echo "condor_rm -name schedd_100@ ${line}"; condor_rm -name schedd_100@ ${line}; done; 
condor_rm -name schedd_100@ 997672.0 
Job 997672.0 marked for removal 
condor_rm -name schedd_100@ 1017665.0 
Job 1017665.0 marked for removal 
... 
 
---cleaning schedd_101 
[root@fcdfhead3 cdfcaf]# condor_q -name schedd_101@ | grep H &gt; /tmp/heldjobs_schedd101_to_remove.txt 
[root@fcdfhead3 cdfcaf]# for line in `cat /tmp/heldjobs_schedd101_to_remove.txt | awk '{print $1}'`; do echo "condor_rm -name schedd_101@ ${line}"; condor_rm -name schedd_101@ ${line}; done; 
condor_rm -name schedd_101@ 1342115.0 
Job 1342115.0 marked for removal 
... 
 
---cleaning schedd_102 
[root@fcdfhead3 cdfcaf]# condor_q -name schedd_102@ | grep H &gt; /tmp/heldjobs_schedd102_to_remove.txt 
[root@fcdfhead3 cdfcaf]# for line in `cat /tmp/heldjobs_schedd102_to_remove.txt | awk '{print $1}'`; do echo "condor_rm -name schedd_102@ ${line}"; condor_rm -name schedd_102@ ${line}; done; 
... 
 
---cleaning schedd_103 
[root@fcdfhead3 cdfcaf]# condor_q -name schedd_103@ | grep H &gt; /tmp/heldjobs_schedd103_to_remove.txt 
[root@fcdfhead3 cdfcaf]# for line in `cat /tmp/heldjobs_schedd103_to_remove.txt | awk '{print $1}'`; do echo "condor_rm -name schedd_103@ ${line}"; condor_rm -name schedd_103@ ${line}; done; 
... 
 
Result: 
[root@fcdfhead3 cdfcaf]# condor_status -schedd 
Name                 TotalRunningJobs TotalIdleJobs TotalHeldJobs 
fcdfhead3.fnal.gov       0               0              0 
schedd_100@fcdfhead3     462            14              0 
schedd_101@fcdfhead3     421             0              0 
schedd_102@fcdfhead3     434            42              0 
schedd_103@fcdfhead3     847           211              0</pre>
<!-- Author: Marian Z -->


<!-- Date: Tue Sep 11 16:10:03 2007 -->
<pre>Hi, 
 
due to problems of missing 'libtcl8.3.so' library after system upgrade, I put listed nodes below temporary to blacklist. 
 
[cdfcaf@fcdfhead3 cdfcaf]$ cd condor/dist/etc/ 
[cdfcaf@fcdfhead3 etc]$ cat fcdfcaf404.local 
... 
# Are standard jobs allowed to run? 
CAF_DEFAULT_START     = 0 
# Are test jobs allowed to run? 
CAF_TEST_START        = 0 
# Are production jobs allowed to run? 
CAF_PRODUCTION_START  = 0 
 
[cdfcaf@fcdfhead3 etc]$ cp fcdfcaf404.local fcdfcaf339.local 
[cdfcaf@fcdfhead3 etc]$ cp fcdfcaf404.local fcdfcaf371.local 
[cdfcaf@fcdfhead3 etc]$ cp fcdfcaf404.local fcdfcaf850.local 
[cdfcaf@fcdfhead3 etc]$ source ~/condor/sh/condor_setup.sh 
[cdfcaf@fcdfhead3 etc]$ ls *.local 
fcdfcaf339.local  fcdfcaf371.local  fcdfcaf404.local  fcdfcaf850.local  fcdfhead3.local 
[cdfcaf@fcdfhead3 etc]$ condor_reconfig -all 
 
Altogether, four nodes are in blacklist for now. fcdfcaf404 has bad hard drive. 
</pre>
<!-- Author: Marian Z -->


<!-- Date: Thu Sep 13 15:56:40 2007 -->
<pre>Steve Timm reminded me that when I have problems with the OSG:Client install to check for the python-devel package. 
yum install python-devel</pre>
<!-- Author: Doug B -->


<!-- Date: Mon Sep 17 16:13:35 2007 -->
<pre>Monitor for the new glidetestcaf 
 
I put the monitor on fcdfmon5. The folder is fcdffheadt1. 
For mount the monitor i followed this instruction  
 
http://cdfcaf.fnal.gov/doc/cafcondorInstallation/node25.html 
 
the base_name of the new test caf is glidetestcaf. 
i changed the following files: 
-fcdfheadt1/CafCondor/www/html/analyze/(index.html  menu.html  title.html) 
-fcdfheadt1/CafCondor/www/html/analyze(index.html  menu.html  title.html) 
-fcdfheadt1/CafCondor/www/html(index.html  user.html) 
-fcdfheadt1/CafCondor/www/cgi/history (title.py) 
-fcdfheadt1/CafCondor/www/cgi/status (title.py) 
 
the url is http://cdfcaf.fnal.gov/glidetestcaf/ 
 
 
</pre>
<!-- Author: federica -->


<!-- Date: Tue Sep 18 15:19:17 2007 -->
<pre>examples of getting service certificate for various processes on fcdfheadt1. 
 
See  file ~cdfcaf/grid/VDT/cert-scripts/cdf-service-account-request.sh 
 
as cdfcaf  
 
source ~/grid/VDT/setup.sh 
 
bash-3.00$ ./bin/cert-retrieve -serialnum 18408 -label fcdfheadt1-stager 
checking script version, V2-4,  This one is OK. Latest version is V2-5. 
 using CA doegrids 
Using URL https://pki1.doegrids.org/displayBySerial?op=displayBySerial&serialNumber=18408 
Checking that the usercert and /home/cdfcaf/.globus/fcdfheadt1-cdfoprkey.pem match 
writing RSA key 
/home/cdfcaf/.globus/usercert.pem and /home/cdfcaf/.globus/userkey.pem now contain your Globus credential 
 
************************ Become root *************** 
-bash-3.00$ ksu 
Authenticated benjamin@FNAL.GOV 
Account root: authorization for benjamin@FNAL.GOV successful 
Changing uid to root (0) 
mkdir /etc/grid-security/stager 
 
[root@fcdfheadt1 cert-scripts]# ~/move_certs.sh stager 
`/home/cdfcaf/.globus/usercert.pem' -&gt; `/etc/grid-security/stager/stagercert.pem' 
removed `/home/cdfcaf/.globus/usercert.pem' 
`/home/cdfcaf/.globus/userkey.pem' -&gt; `/etc/grid-security/stager/stagerkey.pem' 
removed `/home/cdfcaf/.globus/userkey.pem' 
mode of `/etc/grid-security/stager/stagerkey.pem' changed to 0400 (r--------) 
mode of `/etc/grid-security/stager/stagercert.pem' changed to 0444 (r--r--r--) 
[root@fcdfheadt1 cert-scripts]# cat ~/move_certs.sh 
#!/bin/bash 
 
service=$1 
 
 
mv -v /home/cdfcaf/.globus/usercert.pem /etc/grid-security/${service}/${service}cert.pem 
mv -v /home/cdfcaf/.globus/userkey.pem /etc/grid-security/${service}/${service}key.pem 
chmod -v 400 /etc/grid-security/${service}/${service}key.pem 
chmod -v 444 /etc/grid-security/${service}/${service}cert.pem 
 
 
Care must be taken because typo's can require a new certificate to be applied for. 
 </pre>
<!-- Author: Doug B -->


<!-- Date: Wed Sep 19 12:46:42 2007 -->
<pre>Hi *, 
 
finally, today, I've set up with support of helpdesk/run2sys worker nodes with SLF4.4 [fcdfcaf{339,371,850}.fnal.gov] to TestCAF environment [fcdftest010.fnal.gov]. 
 
Steps performed: 
1) WNs blacklisted in GroupCAF 
 
2) Turned off automount of auto.home area on WNs (/home/cdfcaf, /home/cdfsoft) 
 
3) Symlink of missing library created: 
[cdfcaf@fcdfcaf850:~] $ ls -al /usr/lib/libtcl8.* 
lrwxrwxrwx  1 root root     12 Sep 10 14:56 /usr/lib/libtcl8.3.so -&gt; libtcl8.4.so 
-rwxr-xr-x  1 root root 682752 Feb 17  2005 /usr/lib/libtcl8.4.so 
[cdfcaf@fcdfcaf850:~] $ 
 
4) Static mounts of /home/cdfcaf and /home/cdfsoft: 
blue3.fnal.gov:/export/CafCondor/fcdftest010    /home/cdfcaf 
fcdfcode1:/export/cdfsoft       /home/cdfsoft 
 
5) Changes on fcdftest010: 
[cdfcaf@fcdftest010:~] $ cat ~/condor/dist/etc/roster 
fcdftest013 
fcdftest014 
fcdfcaf339 
fcdfcaf371 
fcdfcaf850 
 
6) Restart condor client on WNs: 
[cdfcaf@fcdftest010:~/condor/sh] $ /usr/krb5/bin/rsh -l root fcdfcaf850 killall -9 condor_master 
This rsh session is using DES encryption for all data transmissions. 
condor_master: no process killed 
[cdfcaf@fcdftest010:~/condor/sh] $ /usr/krb5/bin/rsh -l root fcdfcaf850 killall -9 condor_startd 
This rsh session is using DES encryption for all data transmissions. 
condor_startd: no process killed 
[cdfcaf@fcdftest010:~/condor/sh] $ /usr/krb5/bin/rsh -l root fcdfcaf850 "export CONDOR_CONFIG=/home/cdfcaf/condor/dist/etc/condor_config; /home/cdfcaf/condor/sh/start_condor.sh" 
This rsh session is using DES encryption for all data transmissions. 
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel) 
(done for 339,371 as well) 
 
7) Condor reconfig: 
[cdfcaf@fcdftest010:~/condor/sh] $ condor_reconfig -all 
[cdfcaf@fcdftest010:~/condor/sh] $ condor_status -state -total 
                     Total Owner Unclaimed Claimed Preempting Matched 
 
                Busy     4     0         0       4          0       0 
                Idle    26     5        21       0          0       0 
               Total    30     5        21       4          0       0 
 
Nodes are in the TestCAF pool and ready for testing of regular jobs. "Sleep" jobs are finishing successfully. 
</pre>
<!-- Author: Marian -->


<!-- Date: Thu Sep 20 10:52:43 2007 -->
<pre>note : the monitor/CommonModule.py and submitter/CommonModule.py  
are different !! 
 
</pre>
<!-- Author: Federica -->


<!-- Date: Thu Sep 20 11:16:02 2007 -->
<pre>Example how to setup VDT packages that we need for the glidecaf 
can be found here: 
 
http://www-cdf.fnal.gov/htbin/twiki/bin/view/Main/GlideCAFInstallation#VdtUI 
 
</pre>
<!-- Author: Doug -->


<!-- Date: Thu Sep 20 12:20:12 2007 -->
<pre> ./bin/cert-request -host fcdftest015.fnal.gov -ou s -phone 6308408432 -agree -affiliation fnal -vo CDF -email benjamin@fnal.gov -name "Doug Benjamin" -label fcdftest015-host -reason "Host certificate request for CDF test cluster node" 
 
--- result --- 
 
Your Certificate Request has been successfully submitted 
Your Certificate Request id: 27823 
 
        You will receive a notification email from the CA when your certificate 
        has been issued. Please disregard the instructions to download your 
        certificate though a web browser and use the cert-retrieve script instead. 
 
-bash-2.05b$ pwd 
/home/cdfcaf/grid/VDT/cert-scripts 
-bash-2.05b$ whoami 
cdfcaf 
 
</pre>
<!-- Author: Doug -->


<!-- Date: Wed Sep 26 09:36:15 2007 -->
<pre>Noticed today that on fcdfhead5 the directory 
 /home/cdfcaf/grid/VDT/glite/etc was missing and inside it 
The file -  
  
/home/cdfcaf/grid/VDT/glite/etc/vomses 
 
and had to remove 23 from line suggested by Steve Timm</pre>
<!-- Author: Doug -->


<!-- Date: Wed Sep 26 09:37:23 2007 -->
<pre>Error found on fcdfheadt1 (SL 4.4 with python 2.3.4) 
 
bash-3.00$ ./CafAdmin refreshPrincipal all 
/home/cdfcaf/CafCondor/admin/CafAdmin.py:11: RuntimeWarning: Python C API version mismatch for module krb5: This Python has API version 1012, module krb5 has version 1010. 
  import krb5 
refreshPrincipal 
['all'] 
Traceback (most recent call last): 
  File "/home/cdfcaf/CafCondor/admin/CafAdmin.py", line 941, in ? 
    sys.exit(main()) 
  File "/home/cdfcaf/CafCondor/admin/CafAdmin.py", line 938, in main 
    globals()[sys.argv[1]+"Impl"](opts,args) 
  File "/home/cdfcaf/CafCondor/admin/CafAdmin.py", line 303, in refreshPrincipalImpl 
    createtime= time.mktime(time.strptime(date,"%m/%d/%y %H:%M:%S")) 
  File "/usr/lib/python2.3/_strptime.py", line 427, in strptime 
    raise ValueError("unconverted data remains: %s" % 
ValueError: unconverted data remains: 
 
Of Course it must be fixed. 
</pre>
<!-- Author: Doug -->


<!-- Date: Sat Sep 29 13:02:40 2007 -->
<pre>adding user/ group apache to new machine - 
as root - 
 /usr/sbin/groupadd -g 48 -r apache 
 /usr/sbin/useradd -d /var/www  -m -g apache -c Apache -r -u 48 -s /sbin/nologin apache 
 
 
Only node fcdfhead5 has tux installed and runnig. 
/etc/rc.d/init.d/ and now fcdfheadt1 
</pre>
<!-- Author: Doug B -->


<!-- Date: Mon Oct 1 01:03:11 2007 -->
<pre>tar Pzcvpfh /scratch_dcache/tars/fcdftest010-for-fcdftest014.tgz /home/cdfcaf/CafCondor/glidein/scripts/ /home/cdfcaf/grid/sh/ /home/cdfcaf/grid/VDT/ /home/cdfcaf/condor/dist/etc/ /export/CafCondor/tickets/</pre>
<!-- Author: Doug B -->
<!-- Comment: Mon Oct  1 02;29;42 comment by...Doug  -->
<pre>tar Pzcvpfh /scratch_dcache/tars/fcdftest010-for-fcdftest014.tgz /home/cdfcaf/CafCondor/ /home/cdfcaf/grid/VDT/ /home/cdfcaf/condor/dist/etc/ /export/CafCondor/tickets/

After creating the tar file :

cd /
tar xzvf /scratch_dcache/tars/fcdftest010-for-fcdftest014.tgz

cd /home/cdfcaf/condor
cp -v dist/etc/* condor-6.9.4/etc
rm -Rf dist
ln -s /home/cdfcaf/condor/condor-6.9.4 dist
ln -s /home/cdfcaf/CafCondor/condor/scripts/ sh
cd /home/cdfcaf/grid
ln -s /home/cdfcaf/CafCondor/condor/scripts/ sh
cd /home/cdfcaf/grid/VDT
rm -Rf TRUSTED_CA/
ln -s /home/cdfcaf/grid/VDT/globus/share/certificates TRUSTED_CA

now - editing fcdftest014.local

CONDOR_HOST             = fcdftest014.fnal.gov

DAEMON_LIST   = MASTER, COLLECTOR

afterwards -
mkdir /export/condor_local/ViewHist

cd /home/cdfcaf/condor/sh
./start_condor_head.sh collector
./start_condor_head.sh negotiator

Condor works on fcdftest014.

log into fcdftest010 as cdfcaf
edit condor_config to have condor_host point to fcdftest014.fnal.gov
only have master 


</pre>


<!-- Date: Fri Oct 5 22:42:19 2007 -->
<pre>could not start the submitter on port 8000 
so as root looked for program using this port. 
[root@fcdfhead3 CafCondor]#  /bin/netstat -l --numeric-ports -p | grep 8000 
tcp        0      0 *:8000                      *:*                         LISTEN      3780/omaws32         
 
To stop omaws32 program: 
[root@fcdfhead3 CafCondor]# /sbin/chkconfig omawsd off 
[root@fcdfhead3 CafCondor]# /sbin/service omawsd stop 
Shutting down Secure Port Server:                          [  OK  ] 
[root@fcdfhead3 CafCondor]#  /bin/netstat -l --numeric-ports -p | grep 8000 
 
Now the port is free. log into cdfcaf account and restart submitter</pre>
<!-- Author: Doug  -->


<!-- Date: Thu Oct 11 01:14:02 2007 -->
<pre>create cdfcaf:cdfcaf user/group on fcdfhead7 
[root@fcdfhead6 ~]# /usr/sbin/groupadd -g 9510 -r cdfcaf 
[root@fcdfhead6 ~]# grep cdfcaf /etc/group 
cdfcaf:x:9510: 
[root@fcdfhead6 ~]# /usr/sbin/useradd -d /home/cdfcaf -m -g cdfcaf -C "CAF CAF user" -r -u 10155 -s /bin/bash cdfcaf 
/usr/sbin/useradd: invalid option -- C 
usage: useradd  [-u uid [-o]] [-g group] [-G group,...]  
                [-d home] [-s shell] [-c comment] [-m [-k template]] 
                [-f inactive] [-e expire ] [-p passwd] [-M] [-n] [-r] [-l] name 
       useradd  -D [-g group] [-b base] [-s shell] 
                [-f inactive] [-e expire ] 
[root@fcdfhead6 ~]# /usr/sbin/useradd -d /home/cdfcaf -m -g cdfcaf -c "CAF CAF user" -r -u 10155 -s /bin/bash cdfcaf 
[root@fcdfhead6 ~]# grep cdfcaf /etc/passwd 
cdfcaf:x:10155:9510:CAF CAF user:/home/cdfcaf:/bin/bash 
 
- create the directories /export/CafCondor 
/home/cdfcaf /export/condor_local and set the ownership to cdfcaf:cdfcaf. 
- copy the cdfcaf .k5login file from fcdfhead2 
</pre>
<!-- Author: Doug -->
<!-- Comment: Thu Oct 11 01;25;56 comment by...Doug -->
<pre>make sure that /etc/shadow has this entry

cdfcaf:*:13059:0:99999:7:::


Our you will not be able to log in.</pre>


<!-- Date: Thu Oct 18 01:38:38 2007 -->
<pre>Adding 3 nodes from fcdfhead3 to fcdfheadt1 cluster for testing of Gen7 timing purpose: 
 
on fcdfhead3 (in ~/condor/dist/etc/): 
[cdfcaf@fcdfhead3 scripts]$ cp fcdfcaf371.local fcdfcaf370.local 
[cdfcaf@fcdfhead3 scripts]$ cp fcdfcaf371.local fcdfcaf849.local 
[cdfcaf@fcdfhead3 scripts]$ cp fcdfcaf371.local fcdfcaf848.local 
[cdfcaf@fcdfhead3 scripts]$ source ~/condor/sh/condor_setup.sh 
[cdfcaf@fcdfhead3 scripts]$ condor_reconfig fcdfcaf370 -subsystem startd 
Sent "Reconfig" command to startd fcdfcaf370.fnal.gov 
[cdfcaf@fcdfhead3 scripts]$ condor_reconfig fcdfcaf849 -subsystem startd 
Sent "Reconfig" command to startd fcdfcaf849.fnal.gov 
[cdfcaf@fcdfhead3 scripts]$ condor_reconfig fcdfcaf848 -subsystem startd 
Sent "Reconfig" command to startd fcdfcaf848.fnal.gov 
[cdfcaf@fcdfhead3 scripts]$ 
 
on fcdfheadt1: 
1) added fcdfcaf{371,849,848} in roster 
[cdfcaf@fcdfheadt1:~/condor/dist/etc] $ cat roster 
fcdfcaf339 
fcdfcaf371 
fcdfcaf370 
fcdfcaf850 
fcdfcaf849 
fcdfcaf848 
 
2) make local config for every node, I made cp fcdfcaf339.local fcdfcaf370.local etc... 
[cdfcaf@fcdfheadt1:~/condor/dist/etc] $ ls *.local 
-rw-r--r--  1 cdfcaf cdfcaf  543 Oct 17 22:58 fcdfcaf339.local 
-rw-r--r--  1 cdfcaf cdfcaf  543 Oct 17 23:27 fcdfcaf370.local 
-rw-r--r--  1 cdfcaf cdfcaf  543 Oct 17 22:57 fcdfcaf371.local 
-rw-r--r--  1 cdfcaf cdfcaf  543 Oct 17 23:27 fcdfcaf848.local 
-rw-r--r--  1 cdfcaf cdfcaf  543 Oct 17 23:27 fcdfcaf849.local 
-rw-r--r--  1 cdfcaf cdfcaf  543 Oct 17 22:56 fcdfcaf850.local 
-rw-r--r--  1 cdfcaf cdfcaf 8331 Oct  1 21:25 fcdfheadt1.local 
 
3) Allow nfs export for them: 
[cdfcaf@fcdfheadt1:~/condor/dist/etc] $ ksu 
Authenticated zvada@FNAL.GOV 
Account root: authorization for zvada@FNAL.GOV successful 
Changing uid to root (0) 
[root@fcdfheadt1 etc]# vi /etc/exports 
/home/cdfcaf    131.225.209.19(rw,no_root_squash) \ 
                131.225.209.50(rw,no_root_squash) \ 
                131.225.209.51(rw,no_root_squash) \ 
                131.225.211.248(rw,no_root_squash) \ 
                131.225.211.247(rw,no_root_squash) \ 
                131.225.211.246(rw,no_root_squash) 
 
[root@fcdfheadt1 etc]# /etc/init.d/nfs reload 
 
On every node set up following: 
1) /etc/fstab/: 
fcdfheadt1:/home/cdfcaf   /home/cdfcaf    nfs     rw,hard,intr,rsize=32768,wsize=32768,nfsvers=3 0 0 
fcdfcode1:/export/cdfsoft       /home/cdfsoft   nfs     rw,hard,intr,rsize=32768,wsize=32768,nfsvers=3 0 0 
 
2) autofs MUST be OFF 
[root@fcdfcaf370 root]# chkconfig --list | grep autofs 
autofs          0:off   1:off   2:on    3:on    4:on    5:on    6:off 
 
[root@fcdfcaf370 root]# chkconfig --level 2345 autofs off 
[root@fcdfcaf370 root]# chkconfig --list | grep autofs 
autofs          0:off   1:off   2:off   3:off   4:off   5:off   6:off 
[root@fcdfcaf370 root]# /etc/init.d/autofs stop 
[root@fcdfcaf370 root]# umount /home/cdfcaf 
[root@fcdfcaf370 root]# mount -a 
[root@fcdfcaf370 root]# df -h | grep cdfcaf 
fcdfheadt1:/home/cdfcaf 
                       37G   12G   24G  33% /home/cdfcaf 
 
on fcdfheadt1 for every node: 
[cdfcaf@fcdfheadt1:~/condor/dist/etc] $ /usr/krb5/bin/rsh -l root fcdfcaf370 "export CONDOR_CONFIG=/home/cdfcaf/condor/dist/etc/condor_config; /home/cdfcaf/condor/sh/start_condor.sh" 
This rsh session is using DES encryption for all data transmissions. 
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel) 
 
[cdfcaf@fcdfheadt1:~/condor/dist/etc] $ condor_reconfig fcdfcaf370 -subsystem startd 
Sent "Reconfig" command to startd fcdfcaf370.fnal.gov 
[cdfcaf@fcdfheadt1:~/condor/dist/etc] $ 
 
...done for fcdfcaf849 and 848 as well</pre>
<!-- Author: Marian -->


<!-- Date: Mon Oct 22 15:37:49 2007 -->
<pre>Cleaning up fcdfdata113 /export/data3 area - getting rid of files older than 356 days from SCRATCH of every user: 
 
[cdfcaf@fcdfdata113 cdfcaf]$ cd /export/data3 
 
[cdfcaf@fcdfdata113 data3]$ for USER in `ls -al | awk '{print $9}'`; do echo $USER ":" &gt;&gt; /tmp/deleted_from_USER_scratch.txt; find $USER/scratch -type f -mtime +356 -exec ls -alh {} \; | awk '{print $5 " " $6 " " $7 " " $8 " " $9}' &gt;&gt; /tmp/delete_from_USER_scratch.txt ; done; 
 
Result is also available at: 
http://www-cdf.fnal.gov/~zvada/delete_from_USER_scratch.txt 
 
[cdfcaf@fcdfdata113 data3]$ df -h | grep data3 
Filesystem            Size  Used Avail Use% Mounted on 
/dev/sdd1             1.6T  1.6T   31G  99% /export/data3 
/export/data3         1.6T  1.6T   31G  99% /var/ftp/data3 
 
[cdfcaf@fcdfdata113 data3]$ ksu 
Authenticated zvada@FNAL.GOV 
Account root: authorization for zvada@FNAL.GOV successful 
Changing uid to root (0) 
 
[root@fcdfdata113 data3]# for USER in `ls -al | awk '{print $9}'`; do find $USER/scratch -type f -mtime +356 -exec rm {} \; ; done; 
 
[root@fcdfdata113 data3]# df -h | grep data3 
Filesystem            Size  Used Avail Use% Mounted on 
/dev/sdd1             1.6T  1.3T  384G  77% /export/data3 
/export/data3         1.6T  1.3T  384G  77% /var/ftp/data3 
 
...very nice, hope, that nothing important is lost :-)</pre>
<!-- Author: Marian -->


<!-- Date: Wed Nov 14 19:22:25 2007 -->
<pre>Steps used to remove users from cdfcaf.keytab file on fcdfhead2 
0) cd ~/CafCondor/admin 
1) klist -t -k /etc/cdfcaf.keytab &gt; /tmp/cafcaf_user_list.original 
2) wget http://cdfcaf.fnal.gov/users/allUsers6M.txt 
3) cp allUsers6M.txt /tmp/ 
4) grep  -vf /tmp/allUsers6M.txt /tmp/cafcaf_user_list.original | awk '{print $4}' | sed  "{s/\/cdf\/glidecaf@FNAL.GOV/ /}" &gt; /tmp/users_to_remove.dat 
5) cp /tmp/users_to_remove.dat /tmp/fcdfhead2_users_to_remove.dat 
6) for user in `cat /tmp/fcdfhead2_users_to_remove.dat`; do ./CafAdmin delPrincipal $user; done; 
 
On fcdfhead3 used the user list from fcdfhead2  and removed users 
1) cd CafCondor/admin 
2) for user in `cat /tmp/fcdfhead2_users_to_remove.dat`; do ./CafAdmin delPrincipal $user; done; 
3) klist -t -k /etc/cdfcaf.keytab &gt; /tmp/fcdfhead3_cafcaf_user_list.original 
4) wget http://cdfcaf.fnal.gov/users/allUsers6M.txt . 
5)  grep  -vf /tmp/allUsers6M.txt /tmp/fcdfhead3_cafcaf_user_list.original | awk '{print $4}' | sed "{s/\/cdf\/h2caf@FNAL.GOV/ /}" 
 
Can not delete these users: 
 
lippi/cdf/h2caf@FNAL.GOV 
ltmsalmi/cdf/h2caf@FNAL.GOV 
s.senkin/cdf/h2caf@FNAL.GOV 
icortesm/cdf/h2caf@FNAL.GOV 
icortes/cdf/h2caf@FNAL.GOV 
icortesm/cdf/h2caf@FNAL.GOV 
icortesm/cdf/h2caf@FNAL.GOV 
 
On fcdfhead5 these steps were taken : 
1005  cd CafCondor/admin 
 1006  klist -t -k /etc/cdfcaf.keytab &gt; /tmp/fcdfhead5_cdfcaf_user_list.original 
 1007  wget http://cdfcaf.fnal.gov/users/allUsers6M.txt 
 1008  cp allUsers6M.txt /tmp/allUsers6M.txt 
 1009  grep  -vf /tmp/allUsers6M.txt /tmp/fcdfhead5_cdfcaf_user_list.original | awk '{print $4}' | sed  "{s/\/cdf\/namcaf@FNAL.GOV/ /}" &gt; /tmp/fcdfhead5_users_to_remove.dat 
 1010  cat /tmp/fcdfhead5_users_to_remove.dat 
 1011  grep benjamin /tmp/fcdfhead5_users_to_remove.dat 
 1012  for user in `cat /tmp/fcdfhead5_users_to_remove.dat`; do ./CafAdmin delPrincipal $user; done; 
 
 
</pre>
<!-- Author: Doug Benjamin -->


<!-- Date: Thu Nov 15 08:26:38 2007 -->
<pre>Current group accounting on fcdfhead3 (group CAF) 
# Specify VM count for each group. 
GROUP_QUOTA_group_alignment = 10 
GROUP_QUOTA_group_cantabria = 18 
GROUP_QUOTA_group_cmu = 24 
GROUP_QUOTA_group_dbtest = 1 
GROUP_QUOTA_group_dqm = 50 
GROUP_QUOTA_group_florida = 15 
GROUP_QUOTA_group_fnal = 200 
GROUP_QUOTA_group_geneva = 18 
GROUP_QUOTA_group_harvard = 3 
GROUP_QUOTA_group_japan = 30 
GROUP_QUOTA_group_jhu = 9 
GROUP_QUOTA_group_karlsruhe = 45 
GROUP_QUOTA_group_michigan = 9 
GROUP_QUOTA_group_mit = 30 
GROUP_QUOTA_group_penn = 45 
GROUP_QUOTA_group_pitt = 24 
GROUP_QUOTA_group_prd  = 900 
GROUP_QUOTA_group_physmon = 10 
GROUP_QUOTA_group_rochester = 30 
GROUP_QUOTA_group_sam = 75 
GROUP_QUOTA_group_ucdavis = 3 
GROUP_QUOTA_group_ucsd = 60 
GROUP_QUOTA_group_uk = 30 
GROUP_QUOTA_group_wisconsin = 12 
GROUP_QUOTA_group_yale = 3 
GROUP_QUOTA_group_italy = 174 
GROUP_QUOTA_group_highprio = 700 
GROUP_QUOTA_group_calib = 200 
 
[cdfcaf@fcdfhead3 etc]$ condor_status -totals 
 
                     Total Owner Claimed Unclaimed Matched Preempting Backfill 
 
         INTEL/LINUX  2858   487    2352        19       0          0        0 
 
               Total  2858   487    2352        19       0          0        0 
[</pre>
<!-- Author: Doug B -->


<!-- Date: Thu Nov 15 23:09:25 2007 -->
<pre>steps for getting the official user list 
 
- goto cafusers dir :  cd ~/CafCondor/cafusers 
- get the users list from past 6 months all CAF's - wget http://cdfcaf.fnal.gov/users/allUsers6M_111507.txt . 
- create a new file allUsers6M_111507_cafmon_cafexe.txt 
  add the cafmon and cafexe users 
 
- get the list of users in the /etc/cdfcaf,keytab file - 
klist -t -k /etc/cdfcaf.keytab | awk '{print $4}' | grep h2caf | sed "{s/\/cdf\/h2caf@FNAL.GOV/ /}" | sort &gt; fcdfhead2_cdfcaf_keytab_user.list_151107_2155 
 
-- create the modified user list  
 
 
Checked into CVS the revised user list: 
[cdfcaf@fcdfhead3 cafusers]$ cvs ci -m 'revise user list to include users from last 6 months of world wide CAF useage only' cafuser.list 
This rsh session is using DES encryption for all data transmissions. 
Checking in cafuser.list; 
/cdf/code/cdfcvs/run2/CafUtil/cafusers/cafuser.list,v  &lt;--  cafuser.list 
new revision: 1.226; previous revision: 1.225 
done 
 
/etc/cdfcaf.keytab file revised on all three machines fcdfhead3, fcdfhead2 and fcdfhead5 
 
</pre>
<!-- Author: Doug -->


<!-- Date: Mon Nov 26 13:25:58 2007 -->
<pre>Steps to check the high memory use. 
 
[cdfcaf@fcdfhead2:~] $ ps auxwww | sort -rn +3 | grep -v USER | head -5 
cdfcaf   17457  3.2  4.9 410688 405604 ?     Ss   Nov20 276:12 condor_schedd -f 
cdfcaf   17412  2.0  3.5 295412 290720 ?     Ss   Nov20 172:37 condor_schedd -f 
cdfcaf   17443  1.7  3.4 282928 279152 ?     Ss   Nov20 146:01 condor_schedd -f 
cdfcaf   17427  1.8  3.3 280600 275624 ?     Ss   Nov20 158:09 condor_schedd -f 
cdfcaf   17524  0.9  1.5 128680 123848 ?     S    Nov20  77:29 condor_gridmanager -f -C (Owner=?="condor"&&JobUniverse==9) -o condor -S /tmp/condor_g_scratch.0xa1c1c00.17484 
</pre>
<!-- Author: Doug -->


<!-- Date: Mon Nov 26 16:49:27 2007 -->
<pre>Steps used to update CAF userlist and CafAdmin.py code on 
fcdfhead5 (also followed for fcdfhead2) 
 
[cdfcaf@fcdfhead5:~] $ source ~cdfsoft/cdf2.shrc 
[cdfcaf@fcdfhead5:~] $ setup cdfsoft2 development 
[cdfcaf@fcdfhead5:~] $ cvs co CafUtil/cafusers/cafuser.list 
U CafUtil/cafusers/cafuser.list 
[cdfcaf@fcdfhead5:~] $ wc -l CafUtil/cafusers/cafuser.list 
438 CafUtil/cafusers/cafuser.list 
[cdfcaf@fcdfhead5:~] $ cvs co CafCondor/admin/CafAdmin.py 
cvs checkout: move away CafCondor/admin/CafAdmin.py; it is in the way 
C CafCondor/admin/CafAdmin.py 
[cdfcaf@fcdfhead5:~] $ mv CafCondor/admin/CafAdmin.py CafCondor/admin/CafAdmin.py_25Nov07 
[cdfcaf@fcdfhead5:~] $ cvs co CafCondor/admin/CafAdmin.py 
U CafCondor/admin/CafAdmin.py 
[cdfcaf@fcdfhead5:~] $ exit 
 
&lt; log out &gt; 
 
in clean login  on fcdfhead5 : 
[cdfcaf@fcdfhead5:~] $ cd ~/CafCondor/admin 
[cdfcaf@fcdfhead5:~/CafCondor/admin] $ ./CafAdmin SynchronizeFromList /home/cdfcaf/CafUtil/cafusers/cafuser.list 
 
Note - normally one would wait until the development version  
was refreshed. and  
 
&gt; cd ~/CafCondor/admin 
&gt; ./CafAdmin SynchronizeFromList ~cdfsoft/dist/releases/development/CafUtil/cafusers/cafuser.list</pre>
<!-- Author: Doug  -->


<!-- Date: Mon Nov 26 17:55:54 2007 -->
<pre>new crontab fcdfhead2 
cdfcaf@fcdfhead2:~] $ crontab -l 
54 4,16 * * * cd /home/cdfcaf/CafCondor/krbrefresh/; ./krbrefresh &gt;&gt;/tmp/cron_krbrefresh.log 2&gt;&gt; /tmp/cron_krbrefresh.err; cd /home/cdfcaf/CafCondor/glidein/scripts/;./refresh_proxy.sh &gt;&gt; /tmp/cron_krbrefresh_glidein.log 2&gt;&1 
 
10 2,14 * * * cd /home/cdfcaf/CafCondor/admin; ./CafAdmin SynchronizeFromList ~cdfsoft/dist/releases/development/CafUtil/cafusers/cafuser.list &gt;&gt; /tmp/cafuserlist.log 2&gt;&gt; /tmp/cafuserlist.log 
 
#10 2,14 * * * cd /home/cdfcaf/CafCondor/admin; ./CafAdmin addUsersFromList --condor ~cdfsoft/dist/releases/development/CafUtil/cafusers/cafuser.list &gt;&gt; /tmp/cafuserlist.log 2&gt;&gt; /tmp/cafuserlist.log 
 
 
#13 14,20,2,8 * * * /home/cdfcaf/grid/UIPnP/files/usr/sbin/fetch-crl -o /home/cdfcaf/grid/UIPnP/files/etc/grid-security/certificates &gt;&gt; /home/cdfcaf/grid/UIPnP/files/var/log/fetch-crl.log 2&gt;&1 
#22 18,20,2,8 * * * /home/cdfcaf/.globus/bin/check-UIPnP 
 
</pre>
<!-- Author: Doug -->


<!-- Date: Mon Nov 26 17:56:40 2007 -->
<pre>To test new CafAdmin checked out from CVS 
cdfcaf@fcdfhead2:~] $ cd /home/cdfcaf/CafCondor/admin; ./CafAdmin SynchronizeFromList --test ~cdfsoft/dist/releases/development/CafUtil/cafusers/cafuser.list 
/home/cdfcaf/CafCondor/admin/CafAdmin.py:11: RuntimeWarning: Python C API version mismatch for module krb5: This Python has API version 1012, module krb5 has version 1010. 
  import krb5 
SynchronizeFromList 
['--test', '/home/cdfsoft/dist/releases/development/CafUtil/cafusers/cafuser.list'] 
Official list of users taken from file =  /home/cdfsoft/dist/releases/development/CafUtil/cafusers/cafuser.list 
Would add Principal aafke 
Would add Principal aguilar 
Would add Principal apwagner 
Would add Principal agibson 
Would add Principal sineadf 
Would add Principal borgatti 
</pre>
<!-- Author: Doug B -->


<!-- Date: Tue Nov 27 10:42:52 2007 -->
<pre>steps to remove users form the cafusers.list file 
 
1) Receive e-mail from Frank Nagy with list of names. Save  
into a file. 
 
2) for user in `cat users_to_remove_271107_0900.txt`; do grep $user cafuser.list; done; 
 
to get the list of users to remove from cafuser.list 
 
cvs ci -m "Remove users chiara danbgood eagroves fernandj" cafuser.list 
cvs tag -F  CURRENT_WORKING cafuser.list 
</pre>
<!-- Author: Doug B -->


<!-- Date: Tue Dec 11 15:23:28 2007 -->
<pre>Steps to upgrade from SL3.0.x to SL3.0.9 - with yum 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
 
1) Check if the lm_sensors rpm is present: 
   a) if yes, do /etc/init.d/lm_sensors stop 
   b) yum remove lm_sensors 
Note: If lm_sensor is installed you have to remove this package, otherwise you don't succeed in next steps 
 
2) rpm -e kernel-module-sensors-2.4.* 
 
3) rpm -e kernel-module-i2c-2.4.* 
 
4) run yum: 
   a) yum clean all 
   b) yum update yum 
   c) yum upgrade 
 
This upgrade make only new kernel packages, but not whole system. Probably, it keeps some cache which doesn't allow upgrade of other packages. 
Therefore, reboot is needed and run step 4) again. After this, it applies real upgrade to the system of SL v3.0.9.</pre>
<!-- Author: Marian -->


<!-- Date: Tue Dec 11 17:53:32 2007 -->
<pre>Blacklisting of nodes fcdfcaf{351,489,534,686,847} for decommissioning: 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
 
[cdfcaf@fcdfhead3 ~]$ cd condor/dist/etc 
 
[cdfcaf@fcdfhead3 etc]$ cat fcdfcaf339.local 
############################################################ 
## 
## This file containts the variables used for automated 
## Condor pool maintanance 
## 
############################################################ 
 
# Are standard jobs allowed to run? 
CAF_DEFAULT_START     = 0 
# Are test jobs allowed to run? 
CAF_TEST_START        = 0 
# Are production jobs allowed to run? 
CAF_PRODUCTION_START  = 0 
 
START = CAF_PRODUCTION_START 
 
[cdfcaf@fcdfhead3 etc]$ source ~/condor/sh/condor_setup.sh 
[cdfcaf@fcdfhead3 etc]$ for i in 351 489 534 686 847; do cp fcdfcaf339.local fcdfcaf$i.local; echo "Reconfiguring fcdfcaf$i.fnal.gov..."; condor_reconfig fcdfcaf$i.fnal.gov -subsystem startd; done; 
Reconfiguring fcdfcaf351.fnal.gov... 
Sent "Reconfig" command to startd fcdfcaf351.fnal.gov 
Reconfiguring fcdfcaf489.fnal.gov... 
Sent "Reconfig" command to startd fcdfcaf489.fnal.gov 
Reconfiguring fcdfcaf534.fnal.gov... 
Can't find address for startd fcdfcaf534.fnal.gov 
Perhaps you need to query another pool. 
Reconfiguring fcdfcaf686.fnal.gov... 
Can't find address for startd fcdfcaf686.fnal.gov 
Perhaps you need to query another pool. 
Reconfiguring fcdfcaf847.fnal.gov... 
Sent "Reconfig" command to startd fcdfcaf847.fnal.gov 
[cdfcaf@fcdfhead3 etc]$</pre>
<!-- Author: Marian -->


<!-- Date: Wed Dec 12 10:01:15 2007 -->
<pre>Status of CDF grid computing: 
1) group CAF up and running with University group quotas turned off. Doug G. has asked that all reference to them be removed. (In process).  New SAM CAF interface deployed. - Working with KG on investigation of the restarts for his jobs 
 
2) Fermigrid - upgraded to  Condor 6.9.5 
Trying to use new SAM-CAF interface but did not run on SL4 nodes 
(ran on SL 3 nodes though???) 
 
3) NAMCAF - upgraded to Condor 6.9.5 
New kerberos ticket refresh mechanism , new submitter. 
Need to add Gaia and investigate why so little cycles from Florida and U Wisc 
 
4) PacCAF - no issues 
 
5) LCGCAF - no pending issues except MC tar problems  
 
6) CNAF - just received kerberos credentials should be on by  
Friday. 
 
Security issues - last week finished removing all users from CAFuser list who did not have valid kerberos credentials 
 
On NAMCAF, Fermigrid and Group CAF synchronize user robotic account keytab file with official user list. To be done on other CAF by mid January. 
 
Plan to migrate all remote dCAF's to NAMCAF, LCGCAF and PACCAF within 6 months. Will drop support for stand alone dCaf's at that time (including kerberos support)</pre>
<!-- Author: Doug Benjamin -->


<!-- Date: Tue Dec 18 11:02:32 2007 -->
<pre>The following nodes were decommissioned and are out of roster list: 
 
fcdfcaf324						      &lt; 
fcdfcaf327						      &lt; 
fcdfcaf334						      &lt; 
fcdfcaf340						      &lt; 
fcdfcaf341						      &lt; 
fcdfcaf344						      &lt; 
fcdfcaf352						      &lt; 
fcdfcaf355						      &lt; 
fcdfcaf374						      &lt; 
fcdfcaf395						      &lt; 
fcdfcaf396						      &lt; 
fcdfcaf404						      &lt; 
fcdfcaf405						      &lt; 
fcdfcaf416						      &lt; 
fcdfcaf426						      &lt; 
fcdfcaf427						      &lt; 
fcdfcaf434						      &lt; 
fcdfcaf438						      &lt; 
fcdfcaf458						      &lt; 
fcdfcaf473						      &lt; 
fcdfcaf474						      &lt; 
fcdfcaf476						      &lt; 
fcdfcaf479						      &lt; 
fcdfcaf483						      &lt; 
fcdfcaf486						      &lt; 
fcdfcaf487						      &lt; 
fcdfcaf492						      &lt; 
fcdfcaf497						      &lt; 
fcdfcaf503						      &lt; 
fcdfcaf507						      &lt; 
fcdfcaf509						      &lt; 
fcdfcaf525						      &lt; 
fcdfcaf536						      &lt; 
fcdfcaf541						      &lt; 
fcdfcaf543						      &lt; 
fcdfcaf562						      &lt; 
fcdfcaf563						      &lt; 
fcdfcaf564						      &lt; 
fcdfcaf577						      &lt; 
fcdfcaf600						      &lt; 
fcdfcaf626						      &lt; 
fcdfcaf659						      &lt; 
fcdfcaf672						      &lt; 
fcdfcaf682						      &lt; 
fcdfcaf684						      &lt; 
fcdfcaf702						      &lt; 
fcdfcaf721						      &lt; 
fcdfcaf722						      &lt; 
fcdfcaf836						      &lt; 
fcdfcaf841						      &lt;</pre>
<!-- Author: Marian -->


<!-- Date: Tue Jan 8 14:56:18 2008 -->
<pre>squid on fcdfsrv003 failed because store.log file grew to 2GB and the squid server could add more to log and the squid server crashed. 
 
The squid could not be restarted with these commands: 
&lt;as root&gt; 
/etc/init.d/squid status 
rm /var/lock/subsys/squid 
rm /var/run/squid.pid 
/etc/init.d/squid start 
 
So.. I made a symlink on all three squid machines  
fcdfsrv001 fcdfsrv002 fcdfsrv003 
 
rmdir /var/log/squid 
ln -s /fbsng/squid /var/log/squid  
chown squid:squid /var/log/squid 
 
The squid logrotate configuration ( /etc/logrotate.d/squid) was modified so that the log files are rotated daily with 7 old copies.  
 
The squid configuration file: ( /etc/squid/squid.conf ) is modified to write the log files in /var/log/squid. 
 
 
squid on fcdfsrv003 restarted now without problems</pre>
<!-- Author: Doug B -->


<!-- Date: Tue Mar 11 22:26:27 2008 -->
<pre>Steps done as cdfcaf to get the VDT code that we want. 
  
mkdir ~/grid/pacman 
cd grid/pacman/ 
wget http://physics.bu.edu/pacman/sample_cache/tarballs/pacman-3.21.tar.gz  
tar xzvf pacman-3.21.tar.gz  
cd pacman-3.21 
source setup.sh 
 
cd ~/grid 
mkdir ~/grid/VDT-SL3 
mv -v ~/grid/VDT/* ~/grid/VDT-SL3 
cd ~/grid/VDT 
pacman -get http://vdt.cs.wisc.edu/vdt_181_cache:VDT-Client 
 
&lt; say yes and then local &gt; 
 
edit the setup.sh file to comment out condor enviroment line 
#. /home/cdfcaf/grid/VDT/vdt/etc/condor-env.sh 
 
-- Get the CA updater 
 
 pacman -get http://vdt.cs.wisc.edu/vdt_181_cache:CA-Certificates-Updater 
 
As root - 
cd ~cdfcaf/grid/VDT 
source setup.sh 
vdt-register-service --name fetch-crl --enable 
vdt-register-service --name vdt-rotate-logs --enable 
vdt-register-service --name update-certs --enable 
vdt-control --on fetch-crl 
vdt-control --on vdt-rotate-logs 
vdt-control --on update-certs 
vdt-control --list 
</pre>
<!-- Author: Doug -->
<!-- Comment: Tue Mar 11 22;37;08 comment by...Doug -->
<pre>need also to get the VOMS-Client (as cdfcaf)

pacman -get http://vdt.cs.wisc.edu/vdt_181_cache:VOMS-Client</pre>


<!-- Date: Tue Mar 11 23:12:53 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=caf&action=view&page=-13&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=caf&action=view&page=-13  width='300' align='top'></a></td><td>after fixing/running restart_idle_workers.sh script</td></tr></table>
<!-- Author: klg -->