<!-- Title: 09:55:43  Mon Apr 14 2003 -->
<!-- Crewchief: Mary Bishai -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Offline shift reports</pre>


<!-- Date: Mon Apr 14 10:01:57 2003 -->
<pre>DAILY-JOBS-REPORT 
----------------- 
 
*Calibrations status: Runs 161408,09,11,14,15 have bad/no  
 valid calibrations yet. 
 
*Oracle DB usage: 21 connections at 9:00 CDT. Largest CPU usage 
 is 2290(cafuser). 
 
*Data-Handling:  
     -Kahuna process is running OK 
     -Tape queue is active with 0 jobs and 0 pending 
     -Last filesets written to tape succesfully at 9:53 
     -Data logger disks are OK 
</pre>
<!-- Author: M. Bishai -->


<!-- Date: Mon Apr 14 10:53:21 2003 -->
<pre>Hostname: fcdfsgi2 Time: Mon Apr 14 10:44:33 2003 
 
Message: litvinse stopped kahuna successfully</pre>
<!-- Author: Moi -->


<!-- Date: Mon Apr 14 10:53:42 2003 -->
<pre>Hostname: fcdfsgi2 Time: Mon Apr 14 10:45:06 2003 
 
Message: litvinse started kahuna OK</pre>
<!-- Author: Again -->


<!-- Date: Mon Apr 14 11:54:05 2003 -->
<pre>MID-SHIFT REPORT 
---------------- 
*Oracle status: 
   -22 connections at 11:00 
   -cafuser has used 6379 CPU seconds from 6:10 to 10:00 
 
*Data handling status:  
   -Dmitri stopped and started Kahuna it is now running  
   -There are 0 tape jobs running and 0 pending. 
   -No new filesets written to tape since last report 
   -logger disks are OK 
 
*ProductionExe patch 
   -New patches needed for Level3.  
   -Started patch 4.8.4f with 4.8.4e patches +L3 stuff 
 </pre>
<!-- Author: Hungry Mary -->


<!-- Date: Mon Apr 14 16:14:08 2003 -->
<pre>END-OF-SHIFT Report 
-------------------- 
 
*Status of Oracle: 22 connections at 16:00. 4123 seconds 
of CPU usage logged by cafuser from 10:00 to 14:00 
 
*Status of Data handling: 
  -Kahuna process is running 
  -1 job running in tape queue 
  -Filesets last spooled to tape at 15:24 
  -Logger disk look OK. Not much usage last few hours due to  
   access. 
 
*Status of 4.8.4f ProductionExe patch 
  -Patches to Framework and Level3* libraries 
   compiled successfuly. </pre>
<!-- Author: Mary who is going adios -->


<!-- Date: Tue Apr 15 08:55:34 2003 -->
<pre>DAILY-JOBS-REPORT  
-----------------  
 
*Calibrations status: Runs 161408,09,11,14,15,41 have bad/no   
 valid calibrations yet.  
 
*Oracle DB usage: 20 connections at 8:00 CDT. Largest CPU usage  
 is 7981 seconds from 02:10 to 06:00(cafuser).  
 
*Data-Handling:   
     -Kahuna process is running OK  
     -Tape queue is active with 0 jobs and 0 pending  
     -Last filesets written to tape succesfully at 8:46  
     -Data logger disks are OK </pre>
<!-- Author: Mary -->


<!-- Date: Tue Apr 15 12:00:38 2003 -->
<pre>MID-SHIFT-REPORT 
----------------- 
 
*Oracle status: 18 connections as of 11:00.  
 
*DataHandling: 
  -Kahuna process is running 
  -Tape queue is active with 0 processes running and 0 pending 
  -No new data written to tape since 8:46 
  -Data logger disks look fine</pre>
<!-- Author: Mary Bishai -->


<!-- Date: Wed Apr 16 09:43:00 2003 -->
<pre>BEGIN-SHIFT-REPORT 
------------------ 
 
*Calibration check: Only 1 run out of 7 taken in the last 72 
    hours has valid calibrations. Runs w/o calibration constants 
    are 161408,09,14,15,41,52 
 
*Oracle check: 19 connections at 09:00, largest CPU user 
     is cafuser with 5781 secpnds of CPU time from 02:10 to 06:00 
 
*DataHandling: 
   -Kahuna process is OK 
   -Tape queue is active, 0 processes running and 0 pending 
   -Last filesets successfully spooled to tape at 08:30 
   -Logger disks are OK</pre>
<!-- Author: Merry Mary -->


<!-- Date: Wed Apr 16 11:27:46 2003 -->
<pre>Status of ProductionExe patch 4.8.4f: 
------------------------------------- 
 -Compiled and linked - OK 
 -10 K test results: no crashes and the only errors reported 
   are 
     %ERLOG-s [PlugStripMaker]: 
     %ERLOG-e Err:1: Too many hits in region 
     No such path with path Id = 264 
   Experts say no problem. 
     
    *Output file test using AC++Dump: 
      %ERLOG-s SiAlignment invalid: 
      %ERLOG-e notfound: find connection: DB identifier =  NONE 
         
    *Validation Histograms OK 
 
  -Committed patch file to CVS 
  -Currently installing patched release on fcdfcode1 
</pre>
<!-- Author: Contrary Mary -->


<!-- Date: Wed Apr 16 13:38:52 2003 -->
<pre>MID-SHIFT-REPORT 
---------------- 
 
*Oracle Status: 20 connections at 13:00. Largest CPU user 
is cafuser with 2664 CPU secs logged from 06:10 to 10:00  
 
*DataHandling: 
   -Kahuna process is running OK 
   -Tape queue is active with 0 jobs running and 0 jobs pending 
   -Filesets spooled succesfully to tape at 12:21 
   -Data logger disks are fine - we are in access so no activity 
    in the last few hours</pre>
<!-- Author: Me... -->


<!-- Date: Wed Apr 16 14:02:33 2003 -->
<pre>PRODUCTIONEXE 4.8.4f patch status 
--------------------------------- 
 
The DISKCACHE_I_DIR enviroment variable is only setup when using later versions of cdfsoft2 like 4.10.2 but not 4.8.4 or 4.9.1. The FarmsControl package which contains the script that creates the tarball is in CVS but is not part of cdfsoft. I modified the file 
 
FarmsControl/fps/etc/create_tarball.csh 
 
and added the line 
 
setup diskcache_i -q $SRT_CXX 
 
Tarball has been successfully created and is at 
 
/home/cdfsoft/dist/cdfpatch/4.8.4f/ProductionExe_4.8.4f.tar.gz 
 
There was one error from the tarball script: 
 
cp: cannot stat 
`/home/cdfsoft/dist/packages/DBManager/V00-00-51/iomap_cdfofprd_direct.txt': 
No such file or directory 
</pre>
<!-- Author: Moi -->


<!-- Date: Wed Apr 16 16:39:58 2003 -->
<pre>END-OF-SHIFT Report 
-------------------- 
 
*Oracle status: 
  -29 connections at 16:00 
  -Cafuser logged in 4548 sec of CPU on cdfrep01 from 10:10 to  
   14:00 
 
*Data Handling: 
   -kahuna process is running OK 
   -Tape queue status is active with 8 jobs running and 4 jobs 
    pending 
   -Last group of files spooled to tape successfully at 12:21:06 
   -Logger disks are OK still not taking any data 
 
*ProductionExe: 
   -4.8.4f patch built OK 
   -10K test OK 
   -Declared patch and made tarball for farms...</pre>
<!-- Author: Mary - bye bye -->


<!-- Date: Thu Apr 17 09:31:47 2003 -->
<pre>BEGIN-SHIFT-REPORT  
------------------  
 
*Calibration check: All runs from 4/12 to 4/14 have valid   calibrations. Run 161552 and 161632 need to be rechecked.  
 
*Oracle check: 17 connections at 09:00, largest CPU user  
     is cafuser with 3790 secpnds of CPU time from 02:10 to 06:00  
 
*DataHandling:  
   -Kahuna process is OK  
   -Tape queue is active, 1 process running and 0 pending  
   -Last filesets successfully spooled to tape at 08:52  
   -Logger disks are OK</pre>
<!-- Author: Mary B. -->


<!-- Date: Thu Apr 17 12:35:12 2003 -->
<pre>MID-SHIFT-REPORT  
-----------------  
 
*Oracle status: 18 connections as of 12:00. Cafuser leads 
 CPU usage with 2869 secs from 06:10:14 to 10:00 
 
*DataHandling:  
  -Kahuna process is running  
  -Tape queue is active with 9 processes running and 1 pending  
  -Data written to tape successfully 11:23:05 
  -Data logger disks look fine</pre>
<!-- Author: Hungry Mary -->


<!-- Date: Thu Apr 17 12:55:08 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-59&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-59  width='300' align='top'></td><td>W -> e nu rate in recent data</a></td></tr></table>
<!-- Author: Mary Bishai -->


<!-- Date: Thu Apr 17 12:55:45 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-60&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-60  width='300' align='top'></td><td>J/psi rate in recent data - NOTE THE MISSING RUNS</a></td></tr></table>
<!-- Author: Mary Bishai -->


<!-- Date: Thu Apr 17 16:57:20 2003 -->
<pre>END-OF-SHIFT Report  
--------------------  
 
*Status of Oracle: 19 connections at 16:00. 4202 seconds  
of CPU usage logged by cafuser from 10:10 to 14:00  
 
*Status of Data handling:  
  -Kahuna process is running  
  -tape queue is active, 5 jobs running, 0 pending  
  -Filesets last spooled to tape at 15:33 
  -Logger disks look OK. 
 
*Status of 4.10.3 ProductionExe 
  -10K test is still running, but is almost 80% done.  
   No ERLOG-s or ERLOG-e errors reported so far. 
 
*Status of 4.10.3a patch 
  -patched setup_managers.tcl to pick up 08 calibrations 
  -Added Yanwens latest fix to PhotonValidation 
  -Compiled fine but couldnt find a data run with 08 calibrations    
   to run a short test with. 
  -I went ahead and made the tarball. As soon as we do a short 
   test, it can be immediately be sent to the farms </pre>
<!-- Author: Merry Mary -->


<!-- Date: Fri Apr 18 11:18:10 2003 -->
<pre>DAILY-JOBS-REPORT  
-----------------  
 
*Calibrations status: Runs 161552, 161632, 161633, 161634, 161638 have bad/no valid calibrations yet.  
 
*Oracle DB usage: 17 connections at 11:00 CDT. Largest CPU usage  
 is 7061(cafuser).  
 
*Data-Handling:   
     -Kahuna process is running OK  
     -Tape queue is active with 8 jobs running and 4 pending  
     -Last filesets written to tape succesfully at 11:01  
     -Data logger disks are OK </pre>
<!-- Author: maura -->


<!-- Date: Fri Apr 18 12:33:36 2003 -->
<pre>MID-SHIFT-REPORT   
-----------------   
 
*Oracle status: 16 connections as of 12:00. Cafuser leads  
 CPU usage with 7061 secs from 06:10:15 to 10:00:15 
 
*DataHandling:   
  -Kahuna process is running   
  -Tape queue is active with 7 processes running and 0 pending   
  -Data written to tape successfully 11:30:41 
  -Data logger disks look fine 
</pre>
<!-- Author: maura -->


<!-- Date: Fri Apr 18 13:53:02 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-61&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-61  width='300' align='top'></td><td>number of jpsi</a></td></tr></table>
<!-- Author: maura -->


<!-- Date: Fri Apr 18 16:02:08 2003 -->
<pre>END-OF-SHIFT Report   
--------------------   
 
*Status of Oracle: 18 connections at 16:00. 7731 seconds   
of CPU usage logged by cafuser from 10:10 to 14:00   
 
*Status of Data handling:   
  -Kahuna process is running   
  -tape queue is active, 6 jobs running, 4 pending   
  -Filesets last spooled to tape at 14:24 
  -Logger disks look OK.  
</pre>
<!-- Author: maura -->


<!-- Date: Sat Apr 19 08:35:28 2003 -->
<pre>BEGIN-SHIFT-REPORT  
------------------  
 
*Calibration status:  Runs 161552, 161632,33,34,38 have bad/no valid calibrations yet. 
 
*Oracle check: 27 connections at 08:00, largest CPU user  
     is cafuser with 5337 seconds of CPU time from 02:10 to 06:00  
 
*DataHandling:  
   -Kahuna process is OK  
   -Tape queue is active, 1 process running and 0 pending  
   -Last filesets successfully spooled to tape at 06:07  
   -Logger disks are OK</pre>
<!-- Author: maura -->


<!-- Date: Sat Apr 19 12:47:42 2003 -->
<pre>MID-SHIFT-REPORT   
-----------------   
 
*Oracle status: 21 connections as of 12:00. Cafuser leads  
 CPU usage with 6308 secs from 06:10 to 10:00  
 
*DataHandling:   
  -Kahuna process is running   
  -Tape queue is active with 0 processes running and 0 pending. 
   No unfinished job found in queue 
  -Data written to tape successfully 09:09  
  -Data logger disks look fine 
 
</pre>
<!-- Author: maura -->


<!-- Date: Sat Apr 19 16:05:30 2003 -->
<pre>END-OF-SHIFT Report  
--------------------  
 
*Status of Oracle:  
  - 18 connections at 16:00 
  - 2986 seconds of CPU usage logged by cafuser from 10:00 to    14:00  
 
*Status of Data handling:  
  -Kahuna process is running  
  -Tape queue is active with 0 processes running and 0 pending.  
   No unfinished job found in queue 
  -Last filesets successfully spooled to tape at 15:02  
  -Logger disks look OK  
</pre>
<!-- Author: maura -->


<!-- Date: Sun Apr 20 09:04:37 2003 -->
<pre>BEGIN-SHIFT-REPORT   
-----------------   
 
*Calibrations status: Runs 161633,34,38 have no    
 valid calibrations yet. 
 
*Oracle DB usage: 20 connections at 8:00 CDT. Largest CPU usage   
 is 5082 seconds from 02:10 to 06:00(cafuser).   
 
*Data-Handling:    
     -Kahuna process is running OK   
     -Tape queue is active with 1 running job and 0 pending   
     -Last filesets succesfully spooled to tape at 6:36  
     -Data logger disks are OK 
  
*CDF_software_Help: 
     -Status unchanged... 
</pre>
<!-- Author: maura -->


<!-- Date: Sun Apr 20 13:15:11 2003 -->
<pre>MID-SHIFT-REPORT    
-----------------    
 
*Oracle status: 16 connections at 13:00 CDT. Cafuser leads   
 CPU usage with 1301 secs from 06:10 to 10:00   
 
*DataHandling:    
  -Kahuna process is running    
  -Tape queue is active with 1 process running and 0 pending. 
  -Filesets last spooled to tape at 10:59   
  -Data logger disks are OK 
</pre>
<!-- Author: maura -->


<!-- Date: Sun Apr 20 16:03:42 2003 -->
<pre> 
END-OF-SHIFT Report   
--------------------   
 
*Status of Oracle:   
  - 16 connections at 16:00  
  - 10904 seconds of CPU usage logged by cafuser from 10:00 to    14:00   
 
*Status of Data handling:   
  -Kahuna process is running   
  -Tape queue is active with 0 processes running and 0 pending.   
   No unfinished job found in queue  
  -Last filesets successfully spooled to tape at 15:58   
  -Logger disks are OK  
</pre>
<!-- Author: maura -->


<!-- Date: Mon Apr 21 09:40:03 2003 -->
<pre>BEGIN-SHIFT-REPORT    
-----------------    
 
*Calibrations status: Runs 161633,34,38,78, 161713 have bad/no 
 valid calibrations yet. Experts informed. 
 
*Oracle DB usage: 17 connections at 9:00 CDT. Largest CPU usage    
 is 1973 seconds from 02:10 to 06:00 (cafuser).    
 
*Data-Handling:     
     -Kahuna process is running OK    
     -Tape queue is active with 2 running jobs and 0 pending    
     -Last filesets succesfully spooled to tape at 9:23   
     -Data logger disks are OK  
</pre>
<!-- Author: maura -->


<!-- Date: Mon Apr 21 12:04:36 2003 -->
<pre>MID-SHIFT-REPORT     
-----------------     
 
*Oracle status: 18 connections at 12:00 CDT. Cafuser leads    
 CPU usage with 1332 secs from 06:10 to 10:00    
 
*DataHandling:     
  -Kahuna process is running     
  -Tape queue is active with 1 process running and 0 pending.  
  -Filesets last spooled to tape at 11:25    
  -Data logger disks are OK  
</pre>
<!-- Author: maura -->


<!-- Date: Mon Apr 21 16:04:25 2003 -->
<pre>END-OF-SHIFT Report    
--------------------    
 
*Status of Oracle:    
  - 18 connections at 16:00   
  - Largest CPU usage is 2277 seconds from 10:10 to 14:00(cafuser).  
 
*Status of Data handling:    
  -Kahuna process is running OK 
  -Tape queue is active with 1 process running and 0 pending 
  -Last filesets successfully spooled to tape at 14:40   
  -Logger disks are OK   
 
*CDF_software_Help:  
     -Status unchanged...  
</pre>
<!-- Author: maura -->


<!-- Date: Tue Apr 22 08:59:35 2003 -->
<pre>BEGIN-SHIFT-REPORT     
-----------------     
 
*Calibrations status: Runs 161714,18,21,22,54,56,57,58 have no  
 calibrations yet. 
 
*Oracle DB usage: 22 connections at 8:00 CDT. Largest CPU usage     
 is 5146 seconds from 02:10 to 06:00 (cafuser).     
 
*Data-Handling:      
     -Kahuna process is running OK     
     -Tape queue is active with 0 running jobs and 0 pending 
      No unfinished job found in queue 
     -Last filesets succesfully spooled to tape at 4:55    
     -Data logger disks are OK   
 
</pre>
<!-- Author: maura -->


<!-- Date: Tue Apr 22 12:08:07 2003 -->
<pre>MID-SHIFT-REPORT      
-----------------      
 
*Oracle status: 20 connections at 12:00 CDT. Cafuser leads     
 CPU usage with 5937 secs from 06:10 to 10:00     
 
*DataHandling:  
  -Kahuna process is running 
  -Tape queue is active with 1 process running and 0 pending 
  -Filesets last spooled to tape at 10:04 
  -Data logger disks are OK 
  -Error message in tape writing (Apr 21, 16:14) reported to experts 
 
</pre>
<!-- Author: maura -->


<!-- Date: Tue Apr 22 16:06:13 2003 -->
<pre>END-OF-SHIFT Report     
--------------------     
 
*Status of Oracle:     
  - 19 connections at 16:00    
  - Largest CPU usage is 6564 seconds from 10:10 to 14:00(cafuser).   
 
*Status of Data handling:     
  -Kahuna process is running OK  
  -Tape queue is active with 6 processes running and 5 pending  
  -Last filesets successfully spooled to tape at 13:51    
  -Data logger disks are OK    
 
</pre>
<!-- Author: maura -->


<!-- Date: Wed Apr 23 09:02:06 2003 -->
<pre>BEGIN-SHIFT-REPORT      
-----------------      
 
*Calibrations status: Runs 161714,18,21,22,54,56,57 and 
 161758,60,61,62,63,78,79,80,81,83,84,86 have no   
 calibrations yet.  
 
*Oracle DB usage: 22 connections at 8:00 CDT. Largest CPU usage      
 is 6070 seconds from 22:10 to 02:00 (cafuser).      
 
*Data-Handling:       
     -Kahuna process is running OK      
     -Tape queue is active with 0 running jobs and 0 pending  
      No unfinished job found in queue  
     -Last filesets succesfully spooled to tape at 8:09     
     -Data logger disks are OK    
 
</pre>
<!-- Author: maura -->


<!-- Date: Wed Apr 23 12:26:59 2003 -->
<pre>MID-SHIFT-REPORT       
-----------------       
*Calibrations: Runs with missing calibrations (see beg-shift-rep): experts informed 
 
*Oracle status: 25 connections at 12:00 CDT. Cafuser leads      
 CPU usage with 6874 secs from 07:33 to 11:23      
 
*DataHandling:   
  -Kahuna process is running  
  -Tape queue is active with 0 processes running and 0 pending. 
   No unfinished job found in queue   
  -Filesets last spooled to tape at 08:09 
  -Data logger disks are OK  
 
 
</pre>
<!-- Author: maura -->


<!-- Date: Wed Apr 23 16:30:42 2003 -->
<pre>END-OF-SHIFT Report      
--------------------      
 
*Status of Oracle:      
  - 19 connections at 16:00     
  - Largest CPU usage is 9434 seconds from 10:44 to 14:34(cafuser).    
 
*Status of Data handling:      
  -Kahuna process is running OK   
  -Tape queue is active with 8 processes running and 27 pending   
  -Last filesets successfully spooled to tape at 14:00    
  -Data logger disks are OK     
 
*Monitor of the Physics Output 
  -Validation histograms: Runs from 161817 to 161825 have a low number of Jpsi and Wenu. These are marked as Good runs. 
 
  
 
     
 
</pre>
<!-- Author: maura -->


<!-- Date: Thu Apr 24 08:46:57 2003 -->
<pre>BEGIN-SHIFT-REPORT       
-----------------   
 
**Supervised access in progress** 
 
**fcdfsgi2 shutdown** 
 
*Calibrations status: Runs 161787,88,89,90,91,92,95, and 161817,18,20,21 and have no calibrations yet.   
 
*Oracle DB usage: 20 connections at 8:00 CDT. Largest CPU usage       
 is 9164 seconds from 02:10 to 06:00 (cafuser).       
 
*Data-Handling: 
     -No new filesets since yesterday      
     -Data logger disks are OK</pre>
<!-- Author: maura -->


<!-- Date: Thu Apr 24 09:27:04 2003 -->
<pre>Runs with missing calibrations (see beg-shift-rep): experts informed.</pre>
<!-- Author: maura -->


<!-- Date: Thu Apr 24 10:33:10 2003 -->
<pre>* /cdf/opr 100% full : there are multiple core files due to recent crashes of Production 4.8.4a on the Farm. 
Problem reported to cdf_offline_operations@fnal.gov</pre>
<!-- Author: maura -->


<!-- Date: Thu Apr 24 11:25:08 2003 -->
<pre>fcdfsgi2 is back. Root started kahuna OK 
 
Kahuna process is running 
Tape queue is active with 4 running jobs and 4 pending </pre>
<!-- Author: maura -->


<!-- Date: Thu Apr 24 12:23:22 2003 -->
<pre>MID-SHIFT-REPORT        
-----------------        
*Oracle status: 21 connections at 12:00 CDT. Cafuser leads       
 CPU usage with 9164 secs       
 
*DataHandling:    
  -Kahuna process is running   
  -Tape queue is active with 3 processes running and 0 pending 
  -No filesets spooled to tape today 
  -Data logger disks are OK</pre>
<!-- Author: maura -->


<!-- Date: Thu Apr 24 15:18:01 2003 -->
<pre>User found with 2000+ secs cpu usage on cdfrep01. Mail sent to  
alansill@ttu.edu, ahamil@phys.ualberta.ca, wicklund@cdfsga.fnal.gov. 
Reply from the user forwarded to cdfdb-support@fnal.gov. 
 
For time period 24-APR-2003 10:10:10 
             to 24-APR-2003 14:00:10 
 
Top 20 CPU users on database instance cdfrep01 by user's login name 
 
User's login name   CPU Used (sec) 
-----------------   -------------- 
cafuser                      11237 
tecchio                       2094 
aa                             435 
ratnikov                       420 
loginov                        338 
cluster                        208 
johnzhou                       182 
rlc                            127 
tafirout                        90 
rekovic                         77 
hayward                         66 
toolman                         51 
nigmanov                        47 
ahamil                          44 
ais                             32 
herndon                         19 
shapiro                         18 
hkg                             18 
vickey                          18 
oracle                          16</pre>
<!-- Author: maura -->


<!-- Date: Thu Apr 24 16:04:46 2003 -->
<pre>END-OF-SHIFT Report       
--------------------       
**Supervised access until 00:00** 
 
*Oracle DB usage:       
  - 20 connections at 16:00      
  - Largest CPU usage is 9434 seconds from 10:44 to   14:34(cafuser). User tecchio exceeded 2000 secs of CPU: informed.    
 
*Status of Data handling:       
  -Kahuna process is running OK    
  -Tape queue is active with 0 processes running and 0 pending 
   No unfinished job found in queue  
  -Filesets last spooled to tape at 15:54 
  -Data logger disks are OK     </pre>
<!-- Author: maura -->


<!-- Date: Fri Apr 25 10:00:05 2003 -->
<pre>BEGIN-SHIFT-REPORT       
-----------------       
 
*Calibrations status: Runs from 161823 to 161830 have valid calibrations. 
 
*Oracle DB usage:  18 connections at 9:00  
 		25-APR-2003 02:10:14 
             to 25-APR-2003 06:00:14 biggest user 1390 s 
 
*Data-Handling:        
     -Kahuna process is running OK       
     -Tape queue is active with 1 running jobs and 0 pending   
      - Error  FAILED TO COPY FILESET CA8852.0 reported 
     -data logger disks fine  
 
*Problem with production crashes: no reply yet. 
 
 
</pre>
<!-- Author: mike houlden -->


<!-- Date: Fri Apr 25 14:23:39 2003 -->
<pre>MID-SHIFT-REPORT         
-----------------         
*Oracle status: 18 connections at 14:00 CDT. no use &gt;2000 s        
     
 
*DataHandling:     
  -Kahuna process is running    
  -Tape queue is active with 3 processes running and 0 pending  
  -No new filesets spooled to tape   
  -Data logger disks are OK</pre>
<!-- Author: mike  Houlden -->


<!-- Date: Fri Apr 25 14:34:23 2003 -->
<pre>The tarball for version 4.10.4a was created.  
The scripts for patching have been updated by Pasha to account 
for some changes. 
The instructions for patching have been slightly modified, 
and the Web pages updated. 
cdf_offline_operations have been informed by email about the new tarball and a test of this requested. </pre>
<!-- Author: mike houlden -->


<!-- Date: Fri Apr 25 16:10:13 2003 -->
<pre>END-OF-SHIFT Report        
--------------------        
 
*Oracle DB usage:        
  - 19 connections at 16:00       
  - Largest CPU usage is 8952 seconds from 10:10 to   		      14:00(cafuser).  
 
*Status of Data handling:        
  -Kahuna process is running OK     
  -Tape queue is active with 0 processes running and 0 pending  
   No unfinished job found in queue   
  -Data logger disks are OK     </pre>
<!-- Author: mike houlden -->


<!-- Date: Sat Apr 26 07:54:22 2003 -->
<pre>BEGIN-SHIFT-REPORT        
-----------------        
* no data taken overnight. No beam yet 
 
 
*Oracle DB usage:  19 connections at 7:00   
               25-APR-2003 02:10:14  
             to 25-APR-2003 06:00:14  biggest user 98 s  
 
*Data-Handling:         
     -Kahuna process is running OK        
     -Tape queue is active with 1 running jobs and 0 pending    
     -data logger disks fine   
 
*spooling to tape working ok: filesets CA8861.0 to CA8861 (encp   	       SUCCESS) at 07:27:30 
 
 
 
</pre>
<!-- Author: mike houlden -->


<!-- Date: Sat Apr 26 11:55:02 2003 -->
<pre>* A new tarball for 4.8.4f was made at Pasha's request. The existing file was overwritten. The new file is stored on file ~cdfsoft/dist/cdfpatch. An email about this was sent to cdf_offline_operations.</pre>
<!-- Author: mike houlden -->


<!-- Date: Sat Apr 26 12:50:26 2003 -->
<pre>MID-SHIFT-REPORT          
-----------------          
*Oracle status: 17 connections at 12:00 CDT.  
 biggest user 351 s 
 
*DataHandling:      
  -Kahuna process is running     
  -Tape queue is active with 2 processes running and 0 pending   
  -No new filesets spooled to tape since this morning    
  -Data logger disks are OK</pre>
<!-- Author: mike houlden -->


<!-- Date: Sat Apr 26 14:36:11 2003 -->
<pre>I am not totally convinced that the new tarball for 4.8.4f made 
at 11:55 today is correct or indeed different from the one it  
overwrote. Afer more phone discussions with Pasha  I attempted 
to remake it again starting from step 6 in the Patching the  
Production Executable web page. However this does not run as it 
says 4.8.4f already exists. </pre>
<!-- Author: mike houlden -->


<!-- Date: Sat Apr 26 16:08:00 2003 -->
<pre>END-OF-SHIFT Report         
--------------------         
 
*Oracle DB usage:         
  - 16 connections at 16:00        
  - Largest CPU usage is 4127 seconds from 10:10 to 14:00(cafuser) 
        
*Status of Data handling:         
  -Kahuna process is running OK      
  -Tape queue is active with 4 processes running and 0 pending   
   No unfinished job found in queue    
  -Data logger disks are OK     </pre>
<!-- Author: mike houlden -->


<!-- Date: Sun Apr 27 07:59:40 2003 -->
<pre>BEGIN-SHIFT-REPORT         
-----------------         
* data taken overnight. Runs 162099 and 162100 
 
*Oracle DB usage:  19 connections at 7:00    
              25-APR-2003 02:10:14   
             to 25-APR-2003 06:00:14   biggest user 190 s   
 
*Data-Handling:          
     -Kahuna process is running OK         
     -Tape queue is active with 0 running jobs and 0 pending 
     -no unfinished jobs found in queue.     
     -data logger disks fine    
 
*problem spooling to tape: no copy jobs worked overnight.  
  sample message: 
   003-Apr-27 01:09:23 encp FAILED TO COPY FILESET CA8862.0  
   source directory: /cdf/b0/disk6/stager/CA8862.0 
   reported by email at 7:40 	 
</pre>
<!-- Author: mike houlden -->


<!-- Date: Sun Apr 27 08:06:27 2003 -->
<pre>* Reason for making the version 4.8.4.f yesterday:- 
 
Level3 is now capable of writing out a new object - L3SummaryObject and we do not want to be dropping it from the output of Production. Since L3SummaryObject was introduced after we first shipped 4.8.4 to the Farm, we obviously neeed to patch ProductionExe to allow proper I/O for this new object. This is the scope of the patch. Pasha Murat.      </pre>
<!-- Author: mike houlden -->


<!-- Date: Sun Apr 27 09:53:41 2003 -->
<pre>The web page  on b0dap18 monitoring the disk useage for the last 24 hours was not updating properly.This was spotted by Tony 
Vaiciulis who called by to report on the tape copy problem. 
He cleared the web cache and it now appears OK with the correct 
time stamps appearing. 
He asks that the disk useage be watched closly during this shift 
and the data handling group paged if the grey areas on the plots 
grow too much by the end of today.</pre>
<!-- Author: mike houlden -->
<!-- Comment: Sun Apr 27 15;32;36 comment by...murat -->
<pre>this morning Tony V. reported tape write failures to
cdfdh_oper@fnal.gov and enstore_admin@fnal.gov - no reply from
enstore-admin yet.
</pre>
<!-- Comment: Sun Apr 27 15;35;15 comment by...murat -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>at around 15:00 tape writing resumed.</pre></td></tr></td></table>
<!-- Date: Sun Apr 27 12:28:25 2003 -->
<pre>MID-SHIFT-REPORT           
-----------------           
*Oracle status: 17 connections at 12:00 CDT.   
 biggest user 575 s  
 
*DataHandling:       
  -Kahuna process is running      
  -Tape queue is active with 0 processes running and 0 pending    
  -No new filesets spooled to tape since this last night. 
  -Problem with spooling reported this morning still not 
   cured. Pasha knows about it and it is being worked on.     
  -Data logger disks are OK: time stamp was 12:17</pre>
<!-- Author: mike houlden -->




<!-- Date: Sun Apr 27 16:07:22 2003 -->
<pre>END-OF-SHIFT Report          
--------------------          
* beam is off 
  
* Run 162130 is last one through FPS =&gt; total of 3 this shift. 
 
*Oracle DB usage:          
  - 18 connections at 16:00         
  - Largest CPU usage is 5958 seconds from 10:10 to 14:00(cafuser).  
        Biggest other user 64 sec.  
 
*Status of Data handling:          
  -Kahuna process is running OK 
       
  -Tape queue is active with 2 processes running and 0 pending 
 
  -As noted above, spooling to tape has re-started at ~15:00 
    no further information about the problem has been received. 
       
  -Data logger disks are OK: Time stamp 15:47 
 
*No other offline checks (physics/ calibration) have been made this shift.</pre>
<!-- Author: mike houlden -->


<!-- Date: Mon Apr 28 08:12:50 2003 -->
<pre>BEGIN-SHIFT-REPORT          
-----------------          
* data taken overnight. Runs 162174 and 162175 processed  
 
*Oracle DB usage:  16 connections at 7:00     
              25-APR-2003 02:10:13    
             to 25-APR-2003 06:00:13   biggest user 365 s    
 
*Data-Handling:           
     -Kahuna process is running OK          
     -Tape queue is active with 2 running jobs and 0 pending     
     -data logger disks fine: time stamp 7:57     
 
*problem spooling to tape:  copy job failed overnight.   
  sample message:  
   2003-Apr-28 03:46:48 encp FAILED TO COPY FILESET CA8881.0  
   source directory: /cdf/b0/disk9/stager/CA8881.0 
   reported by email at 7:06     </pre>
<!-- Author: mike houlden -->


<!-- Date: Mon Apr 28 08:40:32 2003 -->
<pre>There are  intermittent problems with spooling.. 
 
2003-Apr-28 07:28:50 filesets CA8887.0 CA8887.1 CA8887.2  
CA8887.3 CA8887.4 spooled to tape CA8887 (encp SUCCESS) 
 
2003-Apr-28 07:46:52 encp FAILED TO COPY FILESET CA8889.4 
source directory: /cdf/b0/disk5/stager/CA8889.4                
 
reported by email at about 8:30</pre>
<!-- Author: mike houlden -->


<!-- Date: Mon Apr 28 11:06:10 2003 -->
<pre>* Possible loss of data when tape spooling fails. 
 
I see from the log that CA8852 failed to copy on 25 April 5:16:42 
I cannot see where this has been picked up later. All the other 
failures in recent days seem to be rescued,except CA8873 that failed 
2003-Apr-27 13:07:54 encp FAILED TO COPY FILESET CA8873.0 
source directory: /cdf/b0/disk4/stager/CA8873.0 
 
but has not been processed yet ( 10:56 Apr 28) </pre>
<!-- Author: mike houlden -->


<!-- Date: Mon Apr 28 12:15:26 2003 -->
<pre>MID-SHIFT-REPORT            
-----------------            
*Oracle status: 20 connections at 12:00 CDT.  
    28-APR-2003 06:10:11 
 to 28-APR-2003 10:00:1  2nd biggest user 239 s   
 
*DataHandling:        
  -Kahuna process is running       
  -Tape queue is active with 10 processes running and 2 pending     
  -Spooling OK at present: latest was fileset CA8884  OK. 
  -Data logger disks are OK: time stamp was 12:07. 
 
* Possible loss of dataset CA8852 (see above) reported by email 
  to Rob Kennedy.</pre>
<!-- Author: mike houlden -->


<!-- Date: Mon Apr 28 13:07:46 2003 -->
<pre>*Lost sheep returns: The file reported above that may have got lost has just been staged to tape....3 days late. 
  2003-Apr-28 12:54:49 filesets CA8852.0   OK</pre>
<!-- Author: mike houlden -->


<!-- Date: Mon Apr 28 16:17:57 2003 -->
<pre>END-OF-SHIFT Report           
--------------------           
* beam is off  
 
* Run 162178 is last one through FPS  
 
*Oracle DB usage:           
  - 17 connections at 16:00          
  - Largest CPU usage is 5764 seconds from 10:10 to 	14:00(cafuser).   
        Biggest other user 296 sec.   
 
*Status of Data handling:           
  -Kahuna process is running OK  
 
  -Tape queue is active with 3 processes running and 0 pending  
 
  - spooling to tape has stopped again at 16:05 
    last 3 jobs failed. Reported by email   
     
  -Data logger disks are OK: Time stamp 16.07  
 
*No other offline checks physics have been made this shift. 
 
*Pasha has updated and modified some of the Offline web pages  </pre>
<!-- Author: mike houlden -->


<!-- Date: Tue Apr 29 08:03:50 2003 -->
<pre>BEGIN-SHIFT-REPORT           
-----------------           
*  Runs 162241 and 162238 processed overnight.   
 
*Oracle DB usage:  27 connections at 7:00      
 For time period 29-APR-2003 02:10:13 
              to 29-APR-2003 06:00:13 biggest 2nd user 87 s     
 
*Data-Handling:            
     -Kahuna process is running OK           
     -Tape queue is active with 0 running jobs and 0 pending 
	No unfinished job found in queue &lt;tape&gt;          
     -Data logger disks fine: time stamp 7:37 
     -Tape spooling working well, no failures overnight.      
 
</pre>
<!-- Author: mike houlden -->


<!-- Date: Tue Apr 29 12:25:07 2003 -->
<pre>MID-SHIFT-REPORT             
-----------------             
*Oracle status: 21 connections at 12:00 CDT.   
    29-APR-2003 06:10:11  
 to 29-APR-2003 10:00:1  2nd biggest user 85 s    
 
*DataHandling:         
  -Kahuna process is running        
  -Tape queue is active with 14 jobs,  5 running and 9 pending.      
  -Spooling OK at present: latest was fileset 8909  OK.  
  -Data logger disks are OK: time stamp was 12:17.  
 
* Have checked calibration file against FPS data back to run 
  161633.  
  Have updated the file to show the calibrations are 
  mostly OK.  
  Find that runs 161638 and 161634 have "no version 
  available" for calibration data in the FPS information. 
  Is this a problem? These data taken on 17 APRIL 2003. </pre>
<!-- Author: mike houlden -->


<!-- Date: Tue Apr 29 14:39:31 2003 -->
<pre>* CPU use of database. 
   One user exceeded the 2000 second limit. Email sent, 
   and copied to alansill@ttu.edu, ahamil@phys.ualberta.ca, wicklund@cdfsga.fnal.gov 
 
For time period 29-APR-2003 10:10:11 
             to 29-APR-2003 14:00:11 
 
Top 20 CPU users on database instance cdfrep01 by user's login name 
 
User's login name   CPU Used (sec) 
-----------------   -------------- 
cafuser                       9114 
hayward                       3922 
willis                         264                 
</pre>
<!-- Author: mike houlden -->


<!-- Date: Tue Apr 29 16:18:42 2003 -->
<pre>END-OF-SHIFT Report            
--------------------            
* beam is off   
 
* Run 162252 is last one through FPS   
 
*Oracle DB usage:            
  - There are 29 connections at 16:00           
  - Problem with one user 3922 s CPU reported (see above). 
         
*Status of Data handling:            
  -Kahuna process is running OK   
  -Tape queue is active with 14 jobs, 6 running  and 8 pending   
  -Spooling to tape working OK. 
  -Data logger disks are OK: Time stamp 16.07   
 
*No other offline physics have been made for the last 3 shifts. </pre>
<!-- Author: mike houlden -->


<!-- Date: Wed Apr 30 08:06:42 2003 -->
<pre>BEGIN-SHIFT-REPORT            
-----------------            
*  Runs 162300 and 162301 processed overnight.    
 
*Oracle DB usage:  48 connections at 7:00       
 For time period 30-APR-2003 02:10:13  
              to 30-APR-2003 06:00:13 biggest 2nd user 494 s      
 
*Data-Handling:             
     -Kahuna process is running OK            
     -Tape queue is active with 11 jobs, 4 running  and 7 pending  
                  
     -Data logger disks fine: time stamp 7:47  
     -Tape spooling working well, no failures overnight. 
       latests tape 2003-Apr-30 07:51:12 filesets CA8919.0  OK  
* WEb: very slow to access the FPS data for a run; wait  
	~3  minutes for page to load. </pre>
<!-- Author: mike houlden -->


<!-- Date: Wed Apr 30 08:56:10 2003 -->
<pre>* Tape spooling failure over the weekend. 
  Learnt at the 8:00 meeting today that the spooling failure 
  over the weekend was due to a software fault in a product 
  that had never been seen to fail before. This has now been 
  fixed and should not occur again. Also, it has been realised 
  that the internal communications about these issues are not 
  optimal and need to be improved. There will be a meeting about 
  this tomorrow. For example, the CDF Offline shifter did not 
  get any e-mail feedback about the fix to this spooling problem, 
  even though he had reported it.    	 
 
</pre>
<!-- Author: mike houlden -->


<!-- Date: Wed Apr 30 08:58:35 2003 -->
<pre>*Tape spooling to be stopped tomorrow morning. 
 
 Tape spooling will be suspended tomorrow morning, 1 May from 
 8:00 until ~12:00 to allow access. Checking of the status 
 of the disk store will be important at these times. 
</pre>
<!-- Author: mike houlden -->


<!-- Date: Wed Apr 30 09:06:35 2003 -->
<pre>* Change in Luminosity data on FPS database. 
 
  While checking the status of the calibrations I noticed 
  that the FPS luminosity data presented for a run had changed 
  e.g. 
 run      Lum   date taken     GRB    date FDS data looked at 
162175  218.644 2003.04.27      1       2003.04.29         
162175  200.523 2003.04.27      1       2003.04.30  
 
Is this a problem? Why does it happen?? 
 
This is also true for run 162178 where the Lum was  697.106 
when I first looked and is now 670.806 nb-1. 
 
         </pre>
<!-- Author: mike houlden -->


<!-- Date: Wed Apr 30 12:19:49 2003 -->
<pre>*Did physics run (with Maura's help) on runs 161817 to 162257 
 
See large spike in N(Wenu events) per nb-1  vs  run number 
This is due to bad luminosity data for 2 runs, 162102 nd 162127 
The root job does not allow for Lum =0! 
 
rate for run 162100: 14.0879 
rate for run 162102: 6.42365e+08 
rate for run 162127: 8.92621e+08 
rate for run 162129: 6.22623 
   
runnumber, lumi: 162100  42.355000 
runnumber, lumi: 162102  0.000000 
runnumber, lumi: 162127  0.000000 
runnumber, lumi: 162129  15.306000 
 
Other rate plots show the same effect 
J/psi mass vs run number looks OK. 
Reported this to shift leader. 
Have not yet updated the LIST_OF_PROCESSED_RUNS file with "done" 
for these latest runs. 
</pre>
<!-- Author: mike houlden -->


<!-- Date: Wed Apr 30 12:33:36 2003 -->
<pre>MID-SHIFT-REPORT              
-----------------              
*Oracle status: 32 connections at 12:00 CDT.    
    30-APR-2003 06:10:13   
 to 30-APR-2003 10:00:13  2nd biggest user 717 s     
 
*DataHandling:          
  -Kahuna process is running         
  -Tape queue is active with 17 jobs,  10 running and 7 pending.       
  -Spooling OK at present: but nothing spooled since 9:16  
   latest 2003-Apr-30 09:16:12 filesets CA8921.0   
  -Data logger disks are OK: time stamp was 12:27.  </pre>
<!-- Author: mike houlden -->


<!-- Date: Wed Apr 30 13:13:18 2003 -->
<pre>Have now updated LIST_OF_PROCESSED_RUNS file (see above comment 
about physics check).</pre>
<!-- Author: mike houlden  -->


<!-- Date: Wed Apr 30 13:56:59 2003 -->
<pre>* The issue of the changing luminosity numbers (see above) 
 
  There are two numbers , the estimate from the TeV and the  
  luminosity written to tape. This second one is "correct". 
    run     Lum   date taken 
  162175  218.644 2003.04.27          Lum here is TeV estimate 
  162175  200.523 2003.04.27          Lum written to tape 
  So early luminosity values displayed by the FPS data will  
  change to the second value during processing.</pre>
<!-- Author: mike houlden -->


<!-- Date: Wed Apr 30 15:29:18 2003 -->
<pre>* Lost data file CA8873? 
 
  The file reported last monday as not yet copied to tape 
  after the spool failures last weekend has still not been  
  copied to tape. Did search for CA8873 string on stage log 
  web page, only find the error messages. 
 
  E-mail sent to Rob Kennedy about this.</pre>
<!-- Author: mike houlden -->


<!-- Date: Wed Apr 30 16:02:20 2003 -->
<pre>END-OF-SHIFT Report             
--------------------             
 
* Run 162310 is last one through FPS    
 
*Oracle DB usage:             
  - There are 41 connections at 15:00            
  -  30-APR-2003 10:10:12 
 to  30-APR-2003 14:00:12   2nd biggest user 1643 s. 
 
*Status of Data handling:             
  -Kahuna process is running OK    
  -Tape queue is active with 17 jobs, 10 running  and 7 pending    
  -Spooling to tape working OK.  
   last set spooled: 2003-Apr-30 14:02:04 filesets CA8924.0  
  -Data logger disks are OK: Time stamp 15:47. One empty bin in 
   the plots... update failure?    
 
* Up-to-date with physics and calibration checks. 
 
* Possible lost file CA8873. 
  The data are not lost: This from Dmitri Litvintsev  
  fixed on sunday 
 
[litvinse@fcdflnx3 submits]$ DFCFilesetTool -db production_file_catalog -locate  
-fileset CA8873.0 
FilesetName|DatasetName|TapeLabel|TapePartition|CreationTime|FileCount 
CA8873.0|atestr|CA8873|0|1051471877|2 
 
&gt; getdate \= 1051471877 
Sun Apr 27 14:31:17 CDT 2003 
 
 I encped these two files by hand (hence not in web log ..mah).                               </pre>
<!-- Author: mike houlden -->


<!-- Date: Thu May 1 08:05:03 2003 -->
<pre>BEGIN-SHIFT-REPORT             
-----------------             
*  Runs 162307,163310 and 162312 processed overnight.     
 
*Oracle DB usage:  17 connections at 7:00        
 For time period 01-MAY-2003 02:10:14 
              to 01-MAY-2003 06:00:14 
	      biggest 2nd user 164 s       
 
*Data-Handling:              
     -Kahuna process is running OK             
     -Tape queue is active with 10 jobs, 4 running  and 6 pending   
 
     -Data logger disks fine: time stamp 7:57   
     -Tape spooling working well, no failures overnight.  
       latests tape 2003-May-01 07:09:27 filesets CA8935.0 OK  
* WEb: slow to access the FPS data for a run; wait   
        ~1  minutes for page to load. </pre>
<!-- Author: mike houlden -->


<!-- Date: Thu May 1 11:50:04 2003 -->
<pre>*Physics Monitoring Web page updated 
     I have made a lot of modifications to the Offline physics 
     analysis web page to make it more understandable for novice 
     offline shifters, like me.              </pre>
<!-- Author: mike houlden -->


<!-- Date: Thu May 1 12:09:42 2003 -->
<pre>MID-SHIFT-REPORT               
-----------------               
*Oracle status: 18 connections at 12:00 CDT.     
      01-MAY-2003 06:10:11 
   to 01-MAY-2003 10:00:11 
   2nd biggest user 171 s      
 
*DataHandling:           
  -Kahuna process is running          
  -Tape queue is active with 10 jobs,  4 running and 6 pending.        
  -Spooling OK at present: 
   Latest is 2003-May-01 07:09:27 filesets CA8935.0 OK   
  -Data logger disks are OK: time stamp was 11:57.  </pre>
<!-- Author: mike houlden -->


<!-- Date: Thu May 1 12:23:14 2003 -->
<pre>*Spooling: actually spooling has been suspended as from 8:00 to  
  allow access. It should re-start about noon.</pre>
<!-- Author: mike houlden -->


<!-- Date: Thu May 1 12:59:01 2003 -->
<pre>* Problems on fcdflnx3 
  Due to problems fcdflnx3 was rebooted around 12:45</pre>
<!-- Author: mike houlden -->


<!-- Date: Thu May 1 15:27:16 2003 -->
<pre>*Modification to Physics Monitoring scripts. 
 
 The root jobs have been using all the runs, even those with 
 low or zero luminosity. This spoils the plots with bad data 
 points.The scripts have been modified to only use runs with 
 lum &gt; 10 nb-1; except for the Zee mass plot where only runs with 
 lum &gt; 100 nb-1 will be used. 
 
 In looking through the plots several runs were found that had 
 exactly zero for the number of Wenu events per nb-1; these were 
 
             					comment in 
   Run    Lum nb-1                             on-line e-log 
 155297  20.903      goodrunbit = 0   	     No COT no Si no TOF 
 159228  430.270     goodrunbit = 1087224720    not looked yet 
 159289  519.3410    goodrunbit = 1087224712       " 
 160151  509.5570    goodrunbit = 1                " 
 160227  25.124001   goodrunbit = 1                " 
 160259  57.023998   goodrunbit = 1                " 
 
 The runs not looked at were absent from the 2002 log.  
 Access to the 2003 log is not offered on the present web page. 
 
 The recent data looks OK.</pre>
<!-- Author: mike houlden -->


<!-- Date: Thu May 1 16:09:37 2003 -->
<pre>END-OF-SHIFT Report              
--------------------              
 
* Run 162312 is last one through FPS     
 
*Oracle DB usage:              
  - There are 21 connections at 16:00 
         01-MAY-2003 06:10:11 
      to 01-MAY-2003 10:00:1  
        2nd biggest user 173 s.  
 
*Status of Data handling:              
  -Kahuna process is running OK     
  -Tape queue is active with 39 jobs, 8 running  and 31 pending     
  -Spooling to tape working OK after shutdown this morning.   
   last set spooled: 2003-May-01 14:29:11 filesets CA8933.0   OK 
  -Data logger disks are OK: Time stamp 15:57.  
 
* Up-to-date with physics and calibration checks. </pre>
<!-- Author: mike houlden -->


<!-- Date: Fri May 2 08:41:24 2003 -->
<pre>BEGIN-SHIFT-REPORT              
-----------------              
*  Run 162392 processed overnight.      
 
*Oracle DB usage:  21 connections at 8:00         
 For time period 02-MAY-2003 02:10:14  
              to 02-MAY-2003 06:00:14  
              biggest 2nd user 435 s        
 
*Data-Handling:               
     -Kahuna process is running OK              
     -Tape queue is active with 10 jobs, 4 running  and 6 pending    
 
     -Data logger disks fine: time stamp 8:27    
     -Tape spooling working well, no failures overnight.   
       latests tape 2003-May-02 08:17:03 filesets CA8948.0  OK 
   
* WEb: improved  access the FPS data for a run; wait now only    
 
        ~15 secons for page to load. </pre>
<!-- Author: Ke  -->


<!-- Date: Fri May 2 09:29:59 2003 -->
<pre>*Is there a problem with Cron Job? 
The list of processed runs file has not been updated since  
yesterday. Has the cron job stopped? 
Emailed Pasha Murat about this. </pre>
<!-- Author: Ke Li -->
<!-- Comment: Fri May  2 12;08;02 comment by...Ke  -->
<pre>The latest run file from express production 
(in /cdf/data01/stream_a) is 162312 while 
there are several runs after that in the look area. 
Sent mail to knuteson@fnal.gov and cwolfe@fnal.gov. 
</pre>


<!-- Comment: Fri May  2 13;08;00 comment by...Ke  -->
<pre>got response from Bruce Knuteson, 
-----------
Hi Ke, 

 Thanks for your email.  The CAF maintenance in the past day or so 
 interrupted my express stream production -- I fixed the remaining 
 problems this morning, and resubmitted the jobs that had been held up. 
 New files should be appearing this afternoon. 

 Thanks, 
 Bruce </pre>
<!-- Date: Fri May 2 12:13:05 2003 -->
<pre>MID-SHIFT-REPORT                
-----------------                
*Oracle status: 25 connections at 12:00 CDT.      
      02-MAY-2003 06:10:11  
   to 02-MAY-2003 10:00:11  
   2nd biggest user 972 s       
 
*DataHandling:            
  -Kahuna process is running           
  -Tape queue is active with 12 jobs,  8 running and 4 pending.         
  -Spooling OK at present:  
   Latest is 2003-May-02 12:07:34 filesets CA8953.0 
  -Data logger disks are OK: time stamp was 11:57.  </pre>
<!-- Author: Ke  -->



<!-- Date: Fri May 2 16:25:37 2003 -->
<pre>END-OF-SHIFT Report               
--------------------               
 
* Run 162393 is last one through FPS      
 
*Oracle DB usage:               
  - There are 31 connections at 15:00  
         01-MAY-2003 10:10:11  
      to 01-MAY-2003 14:00:1   
        2nd biggest user 197 s.   
 
*Status of Data handling:               
  -Kahuna process is running OK      
  -Tape queue is active with 18 jobs, 10 running  and 8 pending      
  -Spooling to tape working OK after shutdown this morning.    
   last set spooled: 2003-May-02 15:44:37 filesets CA8958.0 OK  
  -Data logger disks are OK: Time stamp 15:57.   
 
* Up-to-date with physics and calibration checks.  
*  Install g patching release right now.  
 
</pre>
<!-- Author: Ke -->
<!-- Comment: Sat May  3 08;37;33 comment by...Ke  -->
<pre>The g patching release installation failed. 
Contacted Pasha and Liz. 

The error message is
---------------------------------
  bash-2.05a$ ~cdfsoft/dist/releases/development/Release/Scripts/relpatch $release $patch cdfpatch ""
 OK - running  relpatch 2002 02 28 on Fri May  2 15:55:47 CDT 2003
 OK - creating new release 4.8.4g
 OK - fetching 4.8.4g.patches from CVS
U cvstmp/4.8.4g.patches
 OK - have patch file 4.8.4g.patches
 OK - including standard packages
        Production
 OK - applying patches
 OK - applied  patches cleanly
 OK - building libraries for 4.8.4g...

 OOPS - ERROR - found errors in gmake.log

gmake[1]: *** [DataValidation.lib] Error 2

bash-2.05a$                                 
</pre>
<!-- Comment: Sat May  3 10;52;40 comment by...Ke  -->
<pre>after pasha made the necessary changes and 
reinstall it after pasha made the necessary changes
it's okay now. </pre>
<!-- Date: Sat May 3 08:36:14 2003 -->
<pre>BEGIN-SHIFT-REPORT               
-----------------               
*  Run 162399,162396 processed overnight.       
 
*Oracle DB usage:  37 connections at 8:00          
 For time period 02-MAY-2003 02:21:14   
              to 02-MAY-2003 06:11:14   
              biggest 2nd user 124 s         
 
*Data-Handling:                
     -Kahuna process is running OK               
     -Tape queue is active with 30 jobs, 15 running  and 15 pending     
 
     -Data logger disks fine: time stamp 8:17     
     -Tape spooling working well, no failures overnight.    
       latests tape 2003-May-03 08:09:02 filesets CA8971.0 OK  
 
* WEb: okay web access to the FPS data for a run; wait about    
 
        ~30 secons for page to load.  
 
  
</pre>
<!-- Author: Ke Li -->




<!-- Date: Sat May 3 12:55:54 2003 -->
<pre>MID-SHIFT-REPORT                 
-----------------                 
*Oracle status: 27 connections at 12:00 CDT.       
      02-MAY-2003 07:51:11   
   to 02-MAY-2003 11:41:11   
   2nd biggest user 142 s        
 
*DataHandling:             
  -Kahuna process is running            
  -Tape queue is active with 24 jobs,  15 running and 9 pending.          
  -Spooling OK at present:   
   Latest is 2003-May-03 12:15:59 filesets CA8976.0  
  -Data logger disks are OK: time stamp was 12:47.   
 
 </pre>
<!-- Author: Ke -->


<!-- Date: Sat May 3 15:57:51 2003 -->
<pre>END-OF-SHIFT Report                
--------------------                
 
* Run 162423 is last one through FPS       
 
*Oracle DB usage:                
  - There are 24 connections at 15:00   
         01-MAY-2003 10:10:11   
      to 01-MAY-2003 14:00:1    
        2nd biggest user 131 s.    
 
*Status of Data handling:                
  -Kahuna process is running OK       
  -Tape queue is active with 18 jobs, 10 running  and 8 pending       
  -Spooling to tape working OK after shutdown this morning.     
   last set spooled: 2003-May-03 15:16:51 filesets CA8979.0 OK   
  -Data logger disks are OK: Time stamp 15:47.    
 
* Physics monitoring up to 162392 
  calibration checks available to 162312 
     
*  rebuild 4.8.4g patch, and failed to create tar ball 
 
*   run 10K test in request from Pasha (still going)</pre>
<!-- Author: Ke  -->


<!-- Date: Sun May 4 08:08:36 2003 -->
<pre>BEGIN-SHIFT-REPORT                
-----------------                
*  Run 162453 processed overnight.        
 
*Oracle DB usage:  24 connections at 8:00           
 For time period 02-MAY-2003 02:21:14    
              to 02-MAY-2003 06:11:14    
              biggest 2nd user 295 s          
 
*Data-Handling:                 
     -Kahuna process is running OK                
     -Tape queue is active with 10 jobs, 4 running  and 6 pending      
 
     -Data logger disks fine: time stamp 7:57      
     -Tape spooling working well, no failures overnight.     
       latests tape 2003-May-04 07:29:50 filesets CA8992.0 OK 
 
* WEb: okay web access to the FPS data for a run; wait about     
 
        ~20 secons for page to load.   
 
 
</pre>
<!-- Author: Ke -->
<!-- Comment: Sun May  4 10;00;04 comment by...Ke  -->
<pre>The db usage time frame is 
   For time period 03-MAY-2003 18:10:11
                to 03-MAY-2003 22:00:11</pre>
<!-- Date: Sun May 4 09:59:04 2003 -->
<pre>The DB cpu usage webpage http://fcdfhome.fnal.gov/usr/wicklund/cpu_users.html 
doesn't seem to update.  
sent email to db_help.  
</pre>
<!-- Author: Ke  -->
<!-- Comment: Sun May  4 12;07;33 comment by...Ke -->
<pre>it's been fixed by Eric
-------------------------------------
Hi Ke,

Thanks for the message.

This was a problem with the weekly update of the CDF software.

I have run the program by hand.

Thanks, and Best Regards,

- Eric                          </pre>


<!-- Date: Sun May 4 12:09:54 2003 -->
<pre>the 4.8.4g Production tarball problem was  
fixed by Pasha. It was due to a recent added script variable 
 which was not available in 4.8.4.  
The  tarball has been created. </pre>
<!-- Author: Ke -->


<!-- Date: Sun May 4 12:14:51 2003 -->
<pre>MID-SHIFT-REPORT              
-----------------              
*Oracle status: 20 connections at 12:00 CDT.    
    04-May-2003 07:37:11   
 to 04-May-2003 11:27:1  2nd biggest user 455 s     
 
*DataHandling:          
  -Kahuna process is running         
  -Tape queue is active with 11 jobs,  4 running and 7 pending.       
  -Spooling OK at present: 2003-May-04 11:32:36 filesets CA8998.0 OK.   
  -Data logger disks are OK: time stamp was 12:07.   
 
  
</pre>
<!-- Author: Ke -->


<!-- Date: Sun May 4 15:56:21 2003 -->
<pre>END-OF-SHIFT Report                 
--------------------                 
 
* Run 162462 is last one through FPS        
 
*Oracle DB usage:                 
  - There are 22 connections at 15:00    
         01-MAY-2003 11:00:11    
      to 01-MAY-2003 15:00:1     
        2nd biggest user 121 s.     
 
*Status of Data handling:                 
  -Kahuna process is running OK        
  -Tape queue is active with 12 jobs,6 running  and 6 pending        
  -Spooling stage webpage hasn't updated since noon.         		    http://fcdfhome.fnal.gov/usr/stager/stager-fcdfsgi1.html 
    sent email to cdf_dh 
     
  -Data logger disks are OK: Time stamp 15:37.     
 
* Physics monitoring up to 162408  
  calibration checks available to 162312  
 
   
 
*   run 10K test in request from Pasha (still going) 
 
 - Ke </pre>
<!-- Author: Ke -->


<!-- Date: Mon May 5 08:25:11 2003 -->
<pre>BEGIN-SHIFT-REPORT                 
-----------------                 
*  Run 162479 processed overnight.         
 
*Oracle DB usage:  29 connections at 8:00            
 For time period 05-MAY-2003 02:24:14     
              to 05-MAY-2003 06:24:14     
              biggest 2nd user 360 s           
 
*Data-Handling:                  
     -Kahuna process is running OK                 
     -Tape queue is active with 1 jobs, 1 running  and 0 pending       
 
     -Data logger disks fine: time stamp 8:17       
     -Tape spooling working well, no failures overnight.      
       latests tape 2003-May-05 07:24:29 filesets CA9015.0  OK  
 
* WEb: okay web access to the FPS data for a run; wait about      
 
        ~15 secons for page to load.    
 
 
</pre>
<!-- Author: Ke -->


<!-- Date: Mon May 5 11:47:20 2003 -->
<pre>MID-SHIFT-REPORT               
-----------------               
*Oracle status: 25 connections at 11:00 CDT.     
    05-May-2003 06:10:17    
 to 05-May-2003 10:00:17  2nd biggest user 228 s      
 
*DataHandling:           
  -Kahuna process is running          
  -Tape queue is active with 29 jobs,  9 running and 20 pending.        
  -Spooling OK at present: 2003-May-05 11:41:21 filesets CA9018.0 OK  
  -Data logger disks are OK: time stamp was 11:37.    
 
 
</pre>
<!-- Author: Ke  -->


<!-- Date: Mon May 5 15:51:09 2003 -->
<pre>END-OF-SHIFT Report                 
--------------------                 
 
* Run 162481 is last one through FPS        
 
*Oracle DB usage:                 
  - There are 28 connections at 15:00    
         01-MAY-2003 10:21:11    
      to 01-MAY-2003 14:10:1     
        2nd biggest user 373 s.     
 
*Status of Data handling:                 
  -Kahuna process is running OK        
  -Tape queue is active with 48 jobs, 15 running  and 33 pending        
  -Spooling to tape working OK  
   last set spooled: 2003-May-05 15:03:20 filesets CA9023.0 spooled to tape CA9023 (encp SUCCESS)  
  -Data logger disks are OK: Time stamp 15:37.     
 
* Physics monitoring up to 162408  
  calibration checks available to 162399  
 
</pre>
<!-- Author: Ke  -->


<!-- Date: Tue May 6 08:16:25 2003 -->
<pre>BEGIN-SHIFT-REPORT                  
-----------------                  
*  Run 162498 processed overnight.          
 
*Oracle DB usage:  27 connections at 8:00             
 For time period 05-MAY-2003 02:11:14      
              to 05-MAY-2003 06:11:14      
              biggest 2nd user 425 s            
 
*Data-Handling:                   
     -Kahuna process is running OK                  
     -Tape queue is active with 80 jobs, 15 running  and 65 pending        
 
     -Data logger disks fine: time stamp 8:07        
     -Tape spooling working well, no failures overnight.       
       latests tape 2003-May-06 07:31:43 filesets CA9035.0 OK   
 
* WEb: okay web access to the FPS data for a run; wait about       
 
        ~1 min for page to load.     
 
</pre>
<!-- Author: Ke -->


<!-- Date: Tue May 6 11:27:27 2003 -->
<pre>MID-SHIFT-REPORT                
-----------------                
*Oracle status: 29 connections at 11:00 CDT.      
    05-May-2003 06:10:17     
 to 05-May-2003 10:00:17  2nd biggest user 606 s       
 
*DataHandling:            
  -Kahuna process is running           
  -Tape queue is active with 100 jobs,  15 running and 85 pending.         
  -Spooling OK at present: 2003-May-06 09:32:50 filesets CA9039.0 OK   
  -Data logger disks are OK: time stamp was 11:17.     
 
</pre>
<!-- Author: Ke  -->


<!-- Date: Wed May 7 08:12:58 2003 -->
<pre>END-OF-SHIFT Report                  
--------------------                  
 
* Run 162519 is last one through FPS         
 
*Oracle DB usage:                  
  - There are 29 connections at 15:00     
         06-MAY-2003 10:57:11     
      to 06-MAY-2003 14:47:1      
        2nd biggest user 211 s.      
 
*Status of Data handling:                  
  -Kahuna process is running OK         
  -Tape queue is active with 108 jobs, 15 running  and 93 pending         
  -Spooling to tape working OK   
   last set spooled: 2003-May-06 15:19:43 filesets CA9046.0 OK  
  -Data logger disks are OK: Time stamp 15:37.      
 
* Physics monitoring up to 162408   
  calibration checks available to 162423   
 
</pre>
<!-- Author: Ke -->


<!-- Date: Wed May 7 08:22:23 2003 -->
<pre>BEGIN-SHIFT-REPORT                  
-----------------                  
*  Run 162521, 162520 processed overnight.          
 
*Oracle DB usage:  39 connections at 8:00             
 For time period 05-MAY-2003 03:27:14      
              to 05-MAY-2003 07:17:14      
              biggest 2nd user 203 s            
 
*Data-Handling:                   
     -Kahuna process is running OK                  
     -Tape queue is active with 13 jobs, 5 running  and 8 pending        
 
     -Data logger disks fine: time stamp 8:17        
     -Tape spooling working well, no failures overnight.       
       latests tape 2003-May-07 02:43:23 filesets CA9052.0  OK   
 
* WEb: okay web access to the FPS data for a run; wait about       
 
        ~30 secons for page to load.     
 
</pre>
<!-- Author: KE -->


<!-- Date: Wed May 7 09:30:20 2003 -->
<pre>4.10.4b Production patches has been installed.  
The tarball has been created</pre>
<!-- Author: Ke  -->


<!-- Date: Wed May 7 09:45:09 2003 -->
<pre>A message from Mike Houlden:  
----------------------------------------- 
 I got a reply about the database access problem last week, when 
 Helen Hayward used ~3900 seconds CPU in 4 hours. This summary should go in the log? 
 
  Summary of reply from Andrew Hamilton 
 "The job's "problem" with the database was an excessive amount of calls for beamlines. Many other stntuple_maker jobs do the same thing. The fix is detailed at 
 
  http://www-cdf.fnal.gov/internal/upgrades/align/beam.html 
 
 under the heading "Database Access, Used Sets Mode". 
 
</pre>
<!-- Author: Ke  -->


<!-- Date: Wed May 7 12:07:38 2003 -->
<pre>MID-SHIFT-REPORT                 
-----------------                 
*Oracle status: 34 connections at 12:00 CDT.       
    05-May-2003 06:10:17      
 to 05-May-2003 10:00:17  2nd biggest user 221 s        
 
*DataHandling:             
  -Kahuna process is running            
  -Tape queue is active with 5 jobs,  4 running and 1 pending.          
  -Spooling OK at present: 2003-May-07 02:43:23 filesets CA9052.0 OK    
  -Data logger disks are OK: time stamp was 11:17.      
 
 
 - Ke  
</pre>
<!-- Author: Ke -->


<!-- Date: Wed May 7 15:38:43 2003 -->
<pre> 
 
END-OF-SHIFT Report                   
--------------------                   
 
* Run 162519 is last one through FPS          
 
*Oracle DB usage:                   
  - There are 27 connections at 15:00      
         06-MAY-2003 10:10:11      
      to 06-MAY-2003 14:00:1       
        2nd biggest user 463 s.       
 
*Status of Data handling:                   
  -Kahuna process is running OK          
  -Tape queue is active with 19 jobs, 13 running  and 6 pending          
  -Spooling to tape working OK    
   last set spooled: 2003-May-06 15:19:43 filesets CA9046.0 OK   
  -Data logger disks are OK: Time stamp 15:27.       
 
* Physics monitoring up to 162453    
  calibration checks available to 162480    
 
 
 - Ke </pre>
<!-- Author: Ke -->
<!-- Comment: Wed May  7 15;39;48 comment by...Ke -->
<pre>* -spolling to tape working ok 
last set spooled is 2003-May-07 15:24:51 filesets CA9057.0 OK</pre>


<!-- Date: Thu May 8 08:25:35 2003 -->
<pre>BEGIN-SHIFT-REPORT                   
-----------------                   
* no new run processed overnight.           
 
*Oracle DB usage:  35 connections at 8:00              
 For time period 05-MAY-2003 02:27:14       
              to 05-MAY-2003 06:17:14       
              biggest 2nd user 415 s             
 
*Data-Handling:                    
     -Kahuna process is running OK                   
     -Tape queue is active with 10 jobs, 7 running  and 3 pending         
 
     -Data logger disks fine: time stamp 7:57         
     -Tape spooling working well, no failures overnight.        
       latests tape 2003-May-08 07:33:26 filesets CA9073.0 OK    
 
* WEb: okay web access to the FPS data for a run; wait about        
 
        ~30 secons for page to load.      
 
</pre>
<!-- Author: Ke  -->


<!-- Date: Thu May 8 13:01:43 2003 -->
<pre>MID-SHIFT-REPORT                  
-----------------                  
*Oracle status: 30 connections at 12:00 CDT.        
    05-May-2003 06:20:17       
 to 05-May-2003 10:10:17  2nd biggest user 316 s         
 
*DataHandling:              
  -Kahuna process is running             
  -Tape queue is active with 8 jobs,  7 running and 1 pending.           
  -Spooling OK at present: 2003-May-08 11:01:33 filesets CA9076.0  OK     
  -Data logger disks are ok: time stamp was 12:57.       
 
 
</pre>
<!-- Author: Ke -->


<!-- Date: Thu May 8 17:01:52 2003 -->
<pre>ND-OF-SHIFT Report                    
--------------------                    
 
* Run 162631 is last one through FPS           
 
*Oracle DB usage:                    
  - There are 29 connections at 16:00       
         08-MAY-2003 10:10:11       
      to 08-MAY-2003 14:00:1        
        2nd biggest user 1352 s.        
 
*Status of Data handling:                    
  -Kahuna process is running OK           
  -Tape queue is active with 4 jobs, 4 running  and 0 pending           
  -Spooling to tape working OK     
   last set spooled: 2003-May-08 11:01:33 filesets CA9076.0  OK    
  -Data logger disks are OK: Time stamp 16:37.        
 
* Up-to-date with physics and calibration checks. 
 
* 4.10.4b patch crashes problem identified.  
   needs to disable PhotoValidation to creat a new tar ball 
 
</pre>
<!-- Author: Ke -->


<!-- Date: Fri May 9 08:12:34 2003 -->
<pre>Testing ProductionExe web page updated</pre>
<!-- Author: Ke -->
<!-- Comment: Fri May  9 08;13;00 comment by...Ke -->
<pre>http://www-cdf.fnal.gov/internal/upgrades/computing/operations/offline_shifts/FarmTools/doc/testing.html</pre>


<!-- Date: Fri May 9 09:03:30 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-62&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-62  width='300' align='top'></td><td>UNIX load on fcdflnx1.fnal.gov morning of May 9th, 2003</a></td></tr></table>
<!-- Author: Eric Wicklund -->


<!-- Date: Fri May 9 09:06:43 2003 -->
<pre>The load on the cdfrep01 database instance went 
up to approximately 30 at about 11 PM May 8th, 
and stayed high until about 5 AM May 9th. 
 
There are indications that the problem may have been  
caused by many ProductionExe jobs running on CAF. 
</pre>
<!-- Author: Eric Wicklund -->