<!-- Title: 13:47:00  Wed Jan 12 2005 -->
<!-- Crewchief: Ken -->
<!-- Op1: Pasha -->
<!-- Op2: Liz -->
<!-- Op3: Ashutosh -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>A new file for the year 2005. 
</pre>


<!-- Date: Thu Jan 13 11:45:48 2005 -->
<pre>Start of the shift (real time when report written: 8.00 am) 
1) PROCESSING 
   Run on Tape	stream A 192073 2005.01.07   05:39:15 
		stream B 190932 2004.12.11 
		stream G 192069  2005.01.06 
2)FPS CURRENTLY PROCESSING 
   stream A:	191640 2004.12.28 
   stream B:	191640 
   stream G:	191640 
3)CALIBRATIONS 
	10A:	last available:   192280        2005.01.13   04:46:05 
	11A:	last available:	  192073        2005.01.07   05:39:15  
	17A:	last available:	  192073        2005.01.07   05:39:15 
Comments: one hole: run 192244 (10A) not available 
4)EXPRESS ANALYSIS 
	  last available:  2005-01-11 05:01:16   5751  192216 
	  The same as yesterday.Ken found some inconsisntencies on the  
	  run/files processed. Pasha is investigating. 25 files not analized. 
 
5)DB Monitoring 
     ok 
6) DH Monitoring: everything is Ok 
7) DQM. 
   working, but nothing has changed since yesterday. Date of the 
   last cron job is Tue Jan 11. DQM people notified. 
   Last run 190936 2004.12.11  
</pre>
<!-- Author: Toni Munar -->


<!-- Date: Fri Jan 14 08:49:18 2005 -->
<pre>1) PROCESSING 
   Run on Tape	stream A 192073 2005.01.07   05:39:15 
		stream B 192073  
		stream G 192073 
2)FPS CURRENTLY PROCESSING 
 Instructions changed by Ken. Now they are the same as above 
3)CALIBRATIONS 
	10A:	last available:   192298 2005.01.13   22:08:59 
	11A:	last available:	  192073        2005.01.07   05:39:15  
	17A:	last available:	  192073        2005.01.07   05:39:15 
Comments: 10A: some holes but till 2005.01.12 everything is done. 
 (192244,192281,192282,192296,192297,192299) 
	11A:  
4)EXPRESS ANALYSIS 
	  last available:  2005-01-11 05:01:16   5751  192216 
	  The number of non-analyzed files keeps growing. 
 
5)DB Monitoring 
     ok 
6) DH Monitoring: everything is Ok 
7) DQM. Up to date now (last cron job update Fri Jan 14) 
   Last run 190936 2004.12.11  
 
</pre>
<!-- Author: George Velev -->


<!-- Date: Sat Jan 15 08:58:25 2005 -->
<pre>Reprort from 01/15/2005, Staurday  
1) PROCESSING 
   Run on Tape	stream A 192171 2005.01.15   08:01:05 
		stream B 192069  
		stream G 192069 
2)FPS CURRENTLY PROCESSING 
 THe runs are the same as above 
3)CALIBRATIONS 
	10A:	last available:   192306         PHYSICS_3_00   2 2005.01.14   09:33:59   
	11A:	last available:	  192216         PHYSICS_3_00   2 2005.01.10   19:17:13  
	17A:	last available:	  192232         PHYSICS_3_00   2 2005.01.11   17:23:56  
Comments: 10A: some holes but till 2005.01.14 everything is done. 
 (192302) 
 
4)EXPRESS ANALYSIS 
	  last available:  2005-01-15 04:44:54    135  192343     
	  The number of non-analyzed files is still large. 
 
5)DB Monitoring 
     ok 
6) DH Monitoring: everything is Ok 
7) DQM. Up to date now (last cron job update Jan 15 04:30) 
   Check DQM Histogram: Last run 192068  jan 15 2005 
</pre>
<!-- Author: G. Velev -->


<!-- Date: Sun Jan 16 08:17:54 2005 -->
<pre> 
Reprort from 01/16/2005, Sunday  
1) PROCESSING 
   Run on Tape	stream A 192171 Sun Jan 16 08:02:07 CST 2005 
		stream B 192069  
		stream G 192069 
2)FPS CURRENTLY PROCESSING 
 THe runs are the same as above 
3)CALIBRATIONS 
	10A:	last available:   192345         PHYSICS_3_00   2 2005.01.15   06:17:05 
	11A:	last available:	  192280         PHYSICS_3_00   2 2005.01.13   04:46:05 
	17A:	last available:	  192278         PHYSICS_3_00   2 2005.01.13   01:24:58  
progress on 11A & 17A 
 
4)EXPRESS ANALYSIS 
	  last available:  2005-01-15 23:43:26     38  192345  
	  The number of non-analyzed files is still large. Only two runs were analyzed from 01/15/2005 
 
5)DB Monitoring 
     ok 
6) DH Monitoring: everything is Ok 
7) DQM. Up to date now (last cron job update Last cron job run: Jan 16 04:30) 
   3 jobs from 12 are red.  
   Check DQM Histogram: Last run 192073  jan 16 2005 05:00 
</pre>
<!-- Author: G.Velev -->


<!-- Date: Mon Jan 17 09:07:58 2005 -->
<pre> 
Report:	 01/16/2005, Sunday  
1) PROCESSING 
   Run on Tape	stream A 192171 Mon Jan 17 08:02:01 CST 2005 The precessed runs are the   
		stream B 192069  same as  yesterday. Seems waiting for calibration.             
		stream G 192069 
2)FPS CURRENTLY PROCESSING 
 THe runs are the same as above 
3)CALIBRATIONS 
	10A:	last available:   192351         PHYSICS_3_00   2 2005.01.16   12:49:37 
	11A:	last available:	  192280         PHYSICS_3_00   2 2005.01.13   04:46:05 
	17A:	last available:	  192278         PHYSICS_3_00   2 2005.01.13   01:24:58  
 
4)EXPRESS ANALYSIS 
	  last available:  2005-01-16 22:20:44    926  192351   5648053  192351   5959431 
	  The number of non-analyzed files is 6. 
 
5)DB Monitoring 
     ok 
6) DH Monitoring: everything is Ok 
7) DQM. Up to date now (last cron job update Last cron job run: Jan 17 04:30 
   Check DQM Histogram: Last run 192170  17-Jan-2005 05:00  
   The plot of inst. lum. vs Nvtx qual 12 is empty. Seems not to be  
a low statistic problem. Will notify the DQm people.  
</pre>
<!-- Author: G.Velev -->


<!-- Date: Mon Jan 17 12:23:13 2005 -->
<pre>Physics Monitoring Check was done at  12:25 done and  all plots seem OK. </pre>
<!-- Author: g. Velev -->


<!-- Date: Tue Jan 18 09:17:34 2005 -->
<pre> 
Report:	 01/18/2005  
1) PROCESSING 
   Run on Tape	stream A 192280 Tue Jan 18 04:00:38 CST 2005  
		stream B 192069  The precessed runs are the  same as  two days ago. Seems waiting for calibration.             
		stream G 192069 
2)FPS CURRENTLY PROCESSING 
 THe runs are the same as above 
3)CALIBRATIONS 
	10A:	last available:   192354         PHYSICS_3_00   2 2005.01.16   23:01:10  
	11A:	last available:	  192306         PHYSICS_3_00   2 2005.01.14   09:33:59 
	17A:	last available:	  192306         PHYSICS_3_00   2 2005.01.14   09:33:59  
 
4)EXPRESS ANALYSIS 
	  last available:  2005-01-17 20:21:52     51  192371        37  192371      8077 
	  17 non-analyzed filesbefore this file 
 
5)DB Monitoring 
     ok 
6) DH Monitoring: everything is Ok 
7) DQM.  (last cron job update Last cron job run: Last cron job run: Jan 17 20:00 
   Check DQM Histogram: Last run 192171  08-Jan-2005 18:59:25 
   Histograms are okay. 
 </pre>
<!-- Author: G.Velev -->


<!-- Date: Tue Jan 18 09:26:05 2005 -->
<pre> 
Last two days the reports pointed to incorrect dates for DQM histograms. The correct ones are: 
 
 run                     date  
192170     Day:	2005.01.08 Activate:	15:56:37 
        
192073     Day:	2005.01.07 Activate:	05:39:15</pre>
<!-- Author: G. Velev -->


<!-- Date: Wed Jan 19 08:36:43 2005 -->
<pre>Report:	 01/19/2005  
1) PROCESSING 
   Run on Tape	stream A 192354 2005.01.16 
		stream B 192170 2005.01.08 
		stream G 192069 2005.01.06  
2)FPS CURRENTLY PROCESSING 
 THe runs are the same as above 
3)CALIBRATIONS 
	10A:	last available:   192354         PHYSICS_3_00   2 2005.01.16   23:01:10  no changes 2005.01.16 
	11A:	last available:	  192354         PHYSICS_3_00   2 2005.01.16   23:01:10      
	17A:	last available:	  192354         PHYSICS_3_00   2 2005.01.16   23:01:10 
 
4)EXPRESS ANALYSIS 
	  last available:  2005-01-18 15:44:16    205  192391        96  192391     31934 
	  16 non-analyzed files before this file 
 
5)DB Monitoring 
     ok 
6) DH Monitoring: everything is Ok 
7) DQM.  (Last cron job run: Jan 19 04:30 
   Check DQM Histogram: Last run 192214  2005.01.10  
   Histograms are okay.</pre>
<!-- Author: G. Velev -->


<!-- Date: Thu Jan 20 08:50:20 2005 -->
<pre>Report:	 01/20/2005  
1) PROCESSING 
   Run on Tape	stream A 192354 2005.01.16 
		stream B 192354 2005.01.16 
		stream G 192354 2005.01.16 
2)FPS CURRENTLY PROCESSING 
 THe runs are the same as above 
3)CALIBRATIONS 
	10A:	last available:   192503 PHYSICS_HIGHLUM_3_00   2 2005.01.20   05:19:24  
	11A:	last available:	  192354         PHYSICS_3_00   2 2005.01.16   23:01:10 
	17A:	last available:	  192354         PHYSICS_3_00   2 2005.01.16   23:01:10  
 
4)EXPRESS ANALYSIS 
	  last available:  2005-01-20 05:56:33    354  192503       212  192503    
	  16 non-analyzed files before this file 
 
5)DB Monitoring  
     ok 
     User rakitin notified for usage more than 10k Db connections 
6) DH Monitoring: everything is Ok 
7) DQM.  (Last cron job run: Jan 20 04:30 
   Check DQM Histogram: Last run 192230  2005.01.11  
   Histograms are okay.</pre>
<!-- Author: G.Velev -->


<!-- Date: Fri Jan 21 09:05:54 2005 -->
<pre>Report:	 01/21/2005  
1) PROCESSING 
   Run on Tape	stream A 192373 2005.01.17 
		stream B 192354 2005.01.16 
		stream G 192354 2005.01.16 
2)FPS CURRENTLY PROCESSING 
 THe runs are the same as above 
3)CALIBRATIONS 
	10A:	last available:   192505         PHYSICS_3_00   2 2005.01.20   07:53:02 
	11A:	last available:	  192373         PHYSICS_3_00   2 2005.01.17   20:22:42  
	17A:	last available:	  192373         PHYSICS_3_00   2 2005.01.17   20:22:42  
 
4)EXPRESS ANALYSIS 
	  last available:  2005.01.20   07:53:02   1174  192505   5704207  192505  
	  15 non-analyzed files before this file 
 
5)DB Monitoring  
     ok 
6) DH Monitoring: everything is Ok 
7) DQM.  (Last cron job run:  Jan 21 04:30 
   Check DQM Histogram: Last run 192244  2005.01.12 
   Histograms okay okay. 
</pre>
<!-- Author: G.Velev, Igor Gorelov -->


<!-- Date: Fri Jan 21 14:50:32 2005 -->
<pre>"Availability 10A", 1st run for which calibrations  
are not available: 192504   2005.01.20   05:32:28     
 
"Availability 11A", 1st run for which calibrations  
are not available: 192374   2005.01.17   20:39:25   
 
"Availability 17A", 1st run for which calibrations  
are not available: 192374   2005.01.17   20:39:25   
</pre>
<!-- Author: Igor V. Gorelov -->


<!-- Date: Sat Jan 22 09:01:20 2005 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Offline Shift has started.  
Do the routine checks...</pre></td></tr></td></table>
<!-- Author: Igor V. Gorelov -->


<!-- Date: Sat Jan 22 09:09:05 2005 -->
<pre>First of all: the problem with calibrations delay 
persists. Beamlines are not yet ready to process 
runs 192374 and onwards, namely: 
 
10A: 192504   2005.01.20   05:32:28   10A is not available 
 
And critical (likely holding the data processing, as now  
              Run# 192373 is being cranched ?) 
 
11A: 192374       2005.01.17   20:39:25 11A is not available 
17A: 192374       2005.01.17   20:39:25 17A is not available 
 </pre>
<!-- Author: Igor V. Gorelov -->


<!-- Date: Sat Jan 22 11:05:42 2005 -->
<pre>Checkout the status as of morning January 22, 2005:  
1) PROCESSING (CPU):  latest processed run: 
   Stream a 192373 2005.01.17 20:23:14.0 - 20:33:21.0 
   Stream b 192373 2005.01.17 20:23:14.0 - 20:33:21.0 
2) PROCESSING (TO TAPES): 
                Stream a 192373  
                Stream b 192373  
3) CALIBRATIONS are delayed: 
	10A:	1ST NOT available:   192504  2005.01.20  05:32:28 
	11A:	1ST NOT available:   192374  2005.01.17  20:39:25   
 
4)EXPRESS ANALYSIS  
	  last line:  2005-01-20 15:58:40  Run# 192505 
 
5)DB Monitoring   
     the user "lungu" has exceeded 500 connections during ~1hr.  
     CPU usage was exceeded by "ptl" 
6) DH Monitoring: everything is Ok  
7) DQM.  ( Last update: Sat Jan 22 05:00:08 2005  
           Last cron job run: Jan 22 04:30      ) 
  Check DQM Histogram: Last run 192354 taken on 
                       2005.01.16  
                       23:01:10 thru 07:48:02 
   Histograms are fine ("ElectroWeak.ps" has small statistics  
                         for PEM ).  
 
</pre>
<!-- Author: Igor V. Gorelov -->


<!-- Date: Sat Jan 22 15:51:48 2005 -->
<pre>Physics Monitoring plots have been produced according to  
a prescribed recepywas done and that all plots are OK. </pre>
<!-- Author: Igor V. Gorelov -->


<!-- Date: Sun Jan 23 08:13:52 2005 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>CDF Offline Shift has commenced... 
Online World is still missing TeV beams.  ;-)</pre></td></tr></td></table>
<!-- Author: Igor V. Gorelov -->


<!-- Date: Sun Jan 23 10:26:13 2005 -->
<pre>Checkout the status as of morning January 23, 2005:   
1) PROCESSING (CPU):  latest processed run:  
   Stream a 192373 2005.01.17 20:23:14.0 - 20:33:21.0  
   Stream b 192373 2005.01.17 20:23:14.0 - 20:33:21.0  
    The same as yesterday. 
2) PROCESSING (TO TAPES):  
                Stream a 192373   
                Stream b 192373   
   The same as yesterday. 
3) CALIBRATIONS are delayed:  
	10A:	1ST NOT available:   192504  2005.01.20  05:32:28  
	11A:	1ST NOT available:   192374  2005.01.17  20:39:25    
 
   The same as yesterday. 
 
4)EXPRESS ANALYSIS   
	  last line:  2005-01-20 15:58:40  Run# 192505  
 
5)DB Monitoring    
  
    the user "lungu" has kept exceeding 500 connections  
    on 23 Jan 05 during 4:30am thru 6:30am, i.e. within ~2hrs  
  
    the user "ptl" has kept consuming over 
    500 CPU seconds/10 elapsed minutes 
    on 23 Jan 05 during 8:30am thru 9:30am, i.e. within ~1hrs  
      
    The same pattern of "cyber-social rules violations"  
    as yesterday. I will send notifications according  
    to protocol. 
 
6) DH Monitoring: everything is Ok (   
7) DQM.  ( Last update: Sun Jan 23 05:00:07 2005 
           Last cron job run: Jan 23 04:30 ) 
  Check DQM Histogram: NO histogramm .ps files are visible, 
  will notify Mario. 
</pre>
<!-- Author: Igor  V.  Gorelov -->


<!-- Date: Mon Jan 24 08:01:20 2005 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Shift starts</pre></td></tr></td></table>
<!-- Author: Igor V. Gorelov -->


<!-- Date: Mon Jan 24 09:14:57 2005 -->
<pre>Checkout the status as of morning January 24, 2005:    
1) PROCESSING (CPU):  latest processed run:   
   Stream a 192373 2005.01.17 20:23:14.0 - 20:33:21.0   
   Stream b 192373 2005.01.17 20:23:14.0 - 20:33:21.0   
    The same as yesterday.  
2) PROCESSING (TO TAPES):   
                Stream a 192373    
                Stream b 192373    
   The same as yesterday.  
3) CALIBRATIONS are delayed:   
	10A:	1ST NOT available:   192504  2005.01.20  05:32:28   
               LaST NOT available:   192795  2005.01.24  05:42:49 
	11A:	1ST NOT available:   192374  2005.01.17  20:39:25     
               LaST NOT available:   192795  2005.01.24  05:42:49 
 
4)EXPRESS ANALYSIS    
	  last lines:  2005-01-20 15:58:40  Run# 192505   
 
... 2005-01-20 09:51:36   1381  192504   1256968  192504    
... 2005-01-20 17:14:28   5141  192505       392  192505  
... 2005-01-20 19:36:15   5252  192505    870421  192505    
... 2005-01-20 21:13:13   5314  192505   2033138  192505  
... 2005-01-20 21:43:57   5456  192505   3174218  192505    
sopr00.last    aa02eff9.023bphys.s       -1 
2005-01-20 15:58:40   1174  192505   5704207  192505    
sopr00.last    aa02f118.0001phys.s       -1 
sopr00.last    aa02f119.0001phys.s       -1 
sopr00.last    aa02f11a.0001phys.s       -1 
sopr00.last    aa02f11a.00a6phys.s       -1 
sopr00.last    aa02f11b.0001phys.s       -1 
 
Five last ROOT files have not been completed .... 
 
5)DB Monitoring     
 
   The DB usage looks nice at this time, 
   the emails sent yesterday produced a positive effect. 
   The "cyber-social" rules are respected. 
 
 6) DH Monitoring: the disk space consumption plots  
    are reasonable and fine.   
 7) DQM.  ( Last update: Mon Jan 24 05:00:08 2005 
            Last cron job run: Jan 24 04:30 ) 
 
  Check DQM Histogram: NO histogramm .ps files are visible,  
  Mario & Co. are notified. 
 
</pre>
<!-- Author: Igor  V.  Gorelov. -->


<!-- Date: Tue Jan 25 08:28:55 2005 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Offline Shift has started ...</pre></td></tr></td></table>
<!-- Author: Igor  V.  Gorelov. -->


<!-- Date: Tue Jan 25 11:21:17 2005 -->
<pre>Checkout the status as of morning January 25, 2005:     
1) PROCESSING (CPU):  latest processed run:    
   Stream a 192373 2005.01.17 20:23:14.0 - 20:33:21.0    
   Stream b 192373 2005.01.17 20:23:14.0 - 20:33:21.0    
    The same as yesterday.   
2) PROCESSING (TO TAPES):    
                Stream a 192373     
                Stream b 192373     
   The same as yesterday.   
3) CALIBRATIONS are delayed:    
	10A:	1ST NOT available:   192792  2005.01.24  01:28:49   
               LaST NOT available:   192869  2005.01.25  08:31:51 
	11A:	1ST NOT available:   192374  2005.01.17  20:39:25      
               LaST NOT available:   192869  2005.01.25  08:31:51  
 
4)EXPRESS ANALYSIS     
	  last lines:  2005-01-20 15:58:40  Run# 192505    
 
... 2005-01-20 17:14:28   5141  192505       392  192505   
... 2005-01-20 19:36:15   5252  192505    870421  192505     
... 2005-01-20 21:13:13   5314  192505   2033138  192505   
... 2005-01-20 21:43:57   5456  192505   3174218  192505     
sopr00.last    aa02eff9.023bphys.s       -1  
2005-01-20 15:58:40   1174  192505   5704207  192505     
sopr00.last    aa02f118.0001phys.s       -1  
sopr00.last    aa02f119.0001phys.s       -1  
sopr00.last    aa02f11a.0001phys.s       -1  
sopr00.last    aa02f11a.00a6phys.s       -1  
sopr00.last    aa02f11b.0001phys.s       -1  
sopr00.last    aa02f162.0001phys.s       -1 
sopr00.last    aa02f164.0001phys.s       -1 
sopr00.last    aa02f164.0067phys.s       -1 
sopr00.last    aa02f165.0001phys.s       -1 
 
Nine (9) last ROOT files have not been completed ....  
 
5)DB Monitoring      
 
 User "rakitin"  has consumed 14215 sec of CPU, to be notified 
 by email.  
 User "rakitin"  has connected  &gt;500/10minutes within  
 3am thru 10am of Jan 25, 05, email sent. 
 
 Other checks are fine 
  
 6) DH Monitoring: the disk space consumption plots   
    are reasonable and fine.    
 7) DQM.  ( Last update: Tue Jan 25 05:00:08 2005 
            Last cron job run: Jan 25 04:30 ) 
 
  Check DQM Histogram: NO histogramm .ps files are visible,   
  as corresponding runs are not processed due to lack  
  of Calibrations. Mario & Co. are aware of the issue. 
 
 
</pre>
<!-- Author: Igor  V.  Gorelov. -->


<!-- Date: Tue Jan 25 14:14:35 2005 -->
<pre>Checkout the status as of Afternoon January 25, 2005:      
1) PROCESSING (CPU):  latest processed run:     
   Stream a 192373 2005.01.17 20:23:14.0 - 20:33:21.0     
   Stream b 192373 2005.01.17 20:23:14.0 - 20:33:21.0     
2) PROCESSING (TO TAPES):     
                Stream a 192373      
                Stream b 192373      
3) CALIBRATIONS are delayed:     
	10A:	1ST NOT available:   192792  2005.01.24  01:28:49    
               LaST NOT available:   192869  2005.01.25  08:31:51  
	11A:	1ST NOT available:   192374  2005.01.17  20:39:25       
               LaST NOT available:   192869  2005.01.25  08:31:51   
 
4)EXPRESS ANALYSIS      
	  last lines:  2005-01-20 15:58:40  Run# 192505     
 
... 2005-01-20 17:14:28   5141  192505       392  192505    
... 2005-01-20 19:36:15   5252  192505    870421  192505      
... 2005-01-20 21:13:13   5314  192505   2033138  192505    
... 2005-01-20 21:43:57   5456  192505   3174218  192505      
sopr00.last    aa02eff9.023bphys.s       -1   
2005-01-20 15:58:40   1174  192505   5704207  192505      
............................................ 
sopr00.last    aa02f165.0001phys.s       -1 
sopr00.last    aa02f166.0001phys.s       -1 
sopr00.last    aa02f166.0073phys.s       -1 
sopr00.last    aa02f167.0001phys.s       -1 
 
Nine (12) last ROOT files have not been completed ....   
 
5)DB Monitoring -- looks fine.   
 
6) DH Monitoring: the disk space consumption plots    
    are reasonable and fine.     
7) DQM.  ( Last update: Tue Jan 25 05:00:08 2005  
            Last cron job run: Jan 25 04:30 )  
 
  Check DQM Histogram: NO histogramm .ps files are visible,    
  as corresponding runs are not processed due to lack   
  of Calibrations. Mario & Co. are aware of the issue.  
</pre>
<!-- Author: Igor  V.  Gorelov.  -->


<!-- Date: Wed Jan 26 09:00:25 2005 -->
<pre>Checkout the status as of Morning January 26, 2005:       
1) PROCESSING (CPU):  latest processed run:      
   Stream a 192504  2005.01.20 05:32:28 - 07:48:40      
   Stream b 192504  2005.01.20 05:32:28 - 07:48:40  
   Stream g 192504  2005.01.20 05:32:28 - 07:48:40  
     
2) PROCESSING (TO TAPES):      
   Stream a 192399  2005.01.18 15:10:49 - 17:49:25      
   Stream b 192373  2005.01.17 20:22:42 - 20:32:59 
   Stream g 192372  2005.01.17 19:56:39 - 20:10:40 
 
3) CALIBRATIONS are delayed:      
	10A:	1ST NOT available:   192792  2005.01.24  01:28:49     
               LaST NOT available:   192893  2005.01.26  07:12:38    
	11A:	1ST NOT available:   192505  2005.01.20  07:53:02        
               LaST NOT available:   192893  2005.01.26  07:12:38     
 
4)EXPRESS ANALYSIS       
 
This is the last run available as seen by one web-page: 
 
-rw-r--r--    1 cdfopr   cdfofrmd 307060736 Jan 26 08:00 aa02f17c.01d7phys  run number 192892 
Looks like th elast run# is 192892. 
 
This is the last run available as seen at another web-page: 
 
sopr00.last    aa02eff9.0192phys.s     185.691  2005-01-20 21:43:57   5456  192505   3174218  192505   4333857 
sopr00.last    aa02eff9.023bphys.s       -1 
sopr00.last    aa02eff9.02f2phys.s      38.427  2005-01-20 15:58:40   1174  192505   5704207  192505   6022234 
sopr00.last    aa02f118.0001phys.s       -1 
sopr00.last    aa02f119.0001phys.s       -1 
sopr00.last    aa02f11a.0001phys.s       -1 
sopr00.last    aa02f11a.00a6phys.s       -1 
sopr00.last    aa02f11b.0001phys.s       -1 
sopr00.last    aa02f162.0001phys.s       -1 
sopr00.last    aa02f164.0001phys.s       -1 
sopr00.last    aa02f164.0067phys.s       -1 
sopr00.last    aa02f165.0001phys.s       -1 
sopr00.last    aa02f166.0001phys.s       -1 
sopr00.last    aa02f166.0073phys.s       -1 
sopr00.last    aa02f167.0001phys.s       -1 
sopr00.last    aa02f167.0086phys.s       -1 
sopr00.last    aa02f167.011aphys.s       -1 
sopr00.last    aa02f167.01bcphys.s       -1 
sopr00.last    aa02f16e.0001phys.s       -1 
sopr00.last    aa02f178.0001phys.s       -1 
sopr00.last    aa02f17c.0001phys.s       -1 
sopr00.last    aa02f17c.00ddphys.s       -1 
sopr00.last    aa02f17c.01d7phys.s       -1 
sopr00.last    aa02f17c.02f6phys.s       -1 
sopr00.last    aa02f17c.0446phys.s       -1 
sopr00.last    aa02f17d.0001phys.s       -1 
TwentyThree (23) last ROOT files have status "-1"... 
 
Here the last run # is 192505 taken earlier than  
the previous one, #192892 .... 
What is the difference between these two pieces of info provided  
by different pages?  
 
5)DB Monitoring: 
 
   DB ## of connections/10minutes elapsed time 
   User "rakitin" exceeded 500 over 1:30 thru 8:30. 
   Petar to be notified by email. 
 
   Other checks are fine. 
 
6) DH Monitoring: the disk space consumption plots     
    are reasonable and fine.      
7) DQM.  ( Last update: Wed Jan 26 05:00:07 2005 
          Last cron job run: Jan 26 04:30 ) 
 
  Check DQM Histograms have appeared finally as 
  calibrations are advancing.... 
  Histogramms look fine for now ... 
 </pre>
<!-- Author: Igor  V. Gorelov.  -->


<!-- Date: Wed Jan 26 13:39:04 2005 -->
<pre>DQM and Physics Output checks: 
 
 o 192370_aexp0d_psfiles 
 
    ElectroWeak.ps  
 
     o Suggestion: plot TrkIso4CEMAll in Logy scale 
       due to huge spike at 0 and ralatively small statistics  
       in the rest of bins .... 
 
     o PEMMon histogramms have 1 or 0  
       entries. 
 
  o 192370_bhmu0d_psfiles 
 
    Muon.ps  
    Histos BMUdX, BMUdZ have a little of entries... 
 
  o 192373_aexp0d_psfiles 
 
    ElectroWeak.ps  
    PEM histogramms are empty  
    due to small statistics of the run.  
 
  o 192373_bhmu0d_psfiles 
 
    Muon.ps  
    Histos BMUdX, BMUdZ have a little of entries. 
 
  o The ROOT files for Physics Output Analysis  
    are not yet available after run #192505  
#--------------------------------------------------------------------- 
#  files on disk 
#--------------------------------------------------------------------- 
sopr00.last  fcdfdata033.fnal.gov   /cdf/scratch/cdfopr/val/results   
   1   190697   192505 
#---------------------------------------------------------------------- 
</pre>
<!-- Author: Igor  V.  Gorelov.   -->


<!-- Date: Thu Jan 27 10:16:15 2005 -->
<pre>Checkout the status as of Morning January 27, 2005:        
1) PROCESSING (CPU):  latest processed run:       
   Stream a 192505  2005.01.20 07:53:02 - 14:41:07       
   Stream b 192505  2005.01.20 07:53:02 - 14:41:07 
   Stream g 192505  2005.01.20 07:53:02 - 14:41:07 
 
2) PROCESSING (TO TAPES):       
   Stream a 192399  2005.01.18 15:10:49 - 17:49:25       
   Stream b 192390  2005.01.18 12:40:36 - 13:04:36   
   Stream g 192372  2005.01.17 19:56:39 - 20:10:40  
 
3) CALIBRATIONS are delayed:       
   10A:	 1st NOT available:  192793  2005.01.24  01:56:55      
        Next NOT available:  192878  2005.01.25  16:06:47     
       2Next NOT available:  192892  2005.01.25  18:32:49     
       3Next NOT available:  192918  2005.01.26  12:29:46  
        Last NOT available:  192982  2005.01.27  07:05:00      
   11A:	 1st NOT available:  192792  2005.01.24  01:28:49         
        Last NOT available:  192984  2005.01.27  07:39:58      
 
4)EXPRESS ANALYSIS        
 
This is the last run available as seen by one web-page:  
-rw-r--r--    1 cdfopr   cdfofrmd 293363716 Jan 27 07:56 aa02f1d4.0001phys  run number 192980 
 
This is the last run available as seen at another web-page:  
 
sopr00.last  aa02eff9.0192phys.s  185.691  2005-01-20 21:43:57  
5456  192505   3174218  192505  4333857  
 
sopr00.last  aa02eff9.023bphys.s       -1  
sopr00.last  aa02eff9.02f2phys.s  38.427  2005-01-20 15:58:40  
1174  192505   5704207  192505 6022234  
 
sopr00.last    aa02f118.0001phys.s       -1 
sopr00.last    aa02f119.0001phys.s       -1 
sopr00.last    aa02f11a.0001phys.s       -1 
sopr00.last    aa02f11a.00a6phys.s       -1 
sopr00.last    aa02f11b.0001phys.s       -1 
sopr00.last    aa02f162.0001phys.s       -1 
sopr00.last    aa02f164.0001phys.s       -1 
sopr00.last    aa02f164.0067phys.s       -1 
sopr00.last    aa02f165.0001phys.s       -1 
sopr00.last    aa02f166.0001phys.s       -1 
sopr00.last    aa02f166.0073phys.s       -1 
sopr00.last    aa02f167.0001phys.s       -1 
sopr00.last    aa02f167.0086phys.s       -1 
sopr00.last    aa02f167.011aphys.s       -1 
sopr00.last    aa02f167.01bcphys.s       -1 
sopr00.last    aa02f16e.0001phys.s       -1 
sopr00.last    aa02f178.0001phys.s       -1 
sopr00.last    aa02f17c.0001phys.s       -1 
sopr00.last    aa02f17c.00ddphys.s       -1 
sopr00.last    aa02f17c.01d7phys.s       -1 
sopr00.last    aa02f17c.02f6phys.s       -1 
sopr00.last    aa02f17c.0446phys.s       -1 
sopr00.last    aa02f17d.0001phys.s       -1 
sopr00.last    aa02f187.0001phys.s       -1 
sopr00.last    aa02f188.0001phys.s       -1 
sopr00.last    aa02f18c.0001phys.s       -1 
sopr00.last    aa02f18d.0001phys.s       -1 
sopr00.last    aa02f18e.0001phys.s       -1 
sopr00.last    aa02f18f.0001phys.s       -1 
sopr00.last    aa02f190.0001phys.s       -1 
sopr00.last    aa02f192.0001phys.s       -1 
sopr00.last    aa02f193.0001phys.s       -1 
sopr00.last    aa02f195.0001phys.s       -1 
sopr00.last    aa02f196.0001phys.s       -1 
sopr00.last    aa02f197.0001phys.s       -1 
sopr00.last    aa02f19a.0001phys.s       -1 
sopr00.last    aa02f19b.0001phys.s       -1 
 
ThirtySeven (37) last ROOT files have status "-1"...  
 
Here the last run # is 192505 taken earlier than   
the previous one, #192980 ....  
What is the difference between these two pieces  
of info provided by different pages?   
 
5)DB Monitoring:  
 
   DB ## of connections/10minutes elapsed time: 
     User "maxi" exceeded 500 over 1:00am thru 7:00am. 
     User "rakitin" exceeded 500 over 7:00am thru 8:30am 
   Petar to be notified by email.  
 
   Other plots/tables are within advised ranges.  
 
6) DH Monitoring: the disk space consumption plots      
    are reasonable and fine.       
7) DQM.  ( Last update: Thu Jan 27 05:00:07 2005 
           Last cron job run: Jan 27 04:30 ) 
 
  Check DQM Histograms: made for run# 192399 
  which is the last one processed && written to tape. 
  
 
</pre>
<!-- Author: Igor  V.  Gorelov. -->


<!-- Date: Fri Jan 28 08:44:49 2005 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>The new Offline Shift has started. 
The new shifter: Jose Garcia.</pre></td></tr></td></table>
<!-- Author: Igor  V.  Gorelov, Jose Garcia. -->


<!-- Date: Fri Jan 28 10:59:11 2005 -->
<pre>Checkout the status as of Morning January 27, 2005:         
1) PROCESSING (CPU):  latest processed run:        
   Stream a 192870  2005.01.25  
   Stream b 192870  2005.01.25  
   Stream g 192870  2005.01.25  
 
2) PROCESSING (TO TAPES):        
   Stream a 192505  2005.01.20 07:53:02 - 14:41:07       
   Stream b 192390  2005.01.18 12:40:36 - 13:04:36    
   Stream g 192372  2005.01.17 19:56:39 - 20:10:40   
 
3) CALIBRATIONS are delayed:        
   10A:	 1st NOT available:  192982  2005.01.27  07:05:00               
   11A:	 1st NOT available:  192871  2005.01.25   11:04:37 
 
4)EXPRESS ANALYSIS         
 
 
This is the last run available as seen at another web-page:   
 
sopr00.last  aa02eff9.0192phys.s  185.691  2005-01-20 21:43:57   
5456  192505   3174218  192505  4333857   
 
sopr00.last  aa02eff9.023bphys.s       -1   
sopr00.last  aa02eff9.02f2phys.s  38.427  2005-01-20 15:58:40   
1174  192505   5704207  192505 6022234   
 
 
FourtyFive (45) last ROOT files have status "-1"...   
 
 
5)DB Monitoring:   
 
   DB ## of connections/10minutes elapsed time:  
     User "maxi" exceeded 500 over 1:00am thru 8:30am.  
   Petar to be notified by email.   
 
   Other plots/tables are within advised ranges.   
 
6) DH Monitoring: the disk space consumption plots       
    are reasonable and fine.        
7) DQM.  ( Last update: Fri Jan 28 05:00:08 2005 
           Last cron job run: Jan 28 04:30)  
 
  Check DQM Histograms: made for run# 192424  
  which is the last one processed && written to tape.  
 
 
 
</pre>
<!-- Author: Jose E. Garcia -->


<!-- Date: Sat Jan 29 08:38:42 2005 -->
<pre>Checkout the status as of Morning January 29, 2005:          
1) PROCESSING (CPU):  latest processed run:         
   Stream a 192870  2005.01.25   
   Stream b 192870  2005.01.25   
   Stream g 192870  2005.01.25   
 
2) PROCESSING (TO TAPES):         
   Stream a 192870          
   Stream b 192870      
   Stream g 192372     
 
3) CALIBRATIONS are delayed (from yesterday):         
   10A:	     NOT available:  193004  2005.01.28   11:10:08 
             NOT available:  193007  2005.01.28   12:37:20 
             NOT available:  193008  2005.01.28   14:59:48 
 
   11A:	 1st NOT available:  192988  2005.01.27   10:00:52 
 
4)EXPRESS ANALYSIS          
 
 
This is the last run available as seen at another web-page:    
 
sopr00.last  aa02eff9.0192phys.s  185.691  2005-01-20 21:43:57    
5456  192505   3174218  192505  4333857    
 
sopr00.last  aa02eff9.023bphys.s       -1    
sopr00.last  aa02eff9.02f2phys.s  38.427  2005-01-20 15:58:40    
1174  192505   5704207  192505 6022234    
 
 
FourtyFive (52) last ROOT files have status "-1"...    
 
 
5)DB Monitoring:    
 
   DB ## of connections/10minutes elapsed time:   
     User "maxi" exceeded 500 over 0:00am thru 7:40am. 
     User "rakitin" exceeded 500 over 3:30am thru 7:40am.   
   Petar to be notified by email.    
 
   Other plots/tables are within advised ranges.    
 
6) DH Monitoring: the disk space consumption plots        
    are reasonable and fine.         
7) DQM.  ( Last update: Sat Jan 29 05:00:08 2005 
           Last cron job run: Jan 29 04:30)   
 
  Check DQM Histograms: made for run# 192374,   
                        192376, 192384, 192399, 192505</pre>
<!-- Author: Jose E. Garcia -->


<!-- Date: Sat Jan 29 08:42:59 2005 -->
<pre>Correction 
 
1) PROCESSING (CPU):  latest processed run:         
   Stream a 192987 
   Stream b 192987  
   Stream g 192987 </pre>
<!-- Author: Jose E. Garcia -->


<!-- Date: Sat Jan 29 08:53:55 2005 -->
<pre>CALIBRATIONS 
   17A:	 1st NOT available entry:  192988 2005.01.27   10:00:52 
 
</pre>
<!-- Author: Jose E. Garcia -->


<!-- Date: Sun Jan 30 08:28:19 2005 -->
<pre>Checkout the status as of Morning January 30, 2005:           
1) PROCESSING (CPU):  latest processed run:          
   Stream a 192987     
   Stream b 192987    
   Stream g 192987   
 
2) PROCESSING (TO TAPES):          
   Stream a 192870        
   Stream b 192870   
   Stream g 192372  
 
3) CALIBRATIONS are delayed:          
   10A:	 1st NOT available:  193013  2005.01.28   17:24:52 
 
   11A:	 1st NOT available:  192988  2005.01.27   10:00:52 
 
   17A:  1st NOT available:  192988  2005.01.27   10:00:52 
 
4)EXPRESS ANALYSIS           
 
 
This is the last run available as seen at another web-page:     
 
sopr00.last  aa02eff9.0192phys.s  185.691  2005-01-20 21:43:57     
5456  192505   3174218  192505  4333857     
 
sopr00.last  aa02eff9.023bphys.s       -1     
sopr00.last  aa02eff9.02f2phys.s  38.427  2005-01-20 15:58:40     
1174  192505   5704207  192505 6022234     
 
 
FourtyFive (75) last ROOT files have status "-1"...     
 
 
5)DB Monitoring:     
 
   DB ## of connections/10minutes elapsed time:    
     User "maxi" exceeded 500 over 0:00am thru 7:40am.  
   Petar to be notified by email.     
 
   Other plots/tables are within advised ranges.     
 
6) DH Monitoring: the disk space (3-9) consumption plots         
    are reasonable and fine. 
    "home" is almost full. 
          
7) DQM.  ( Last update: Sun Jan 30 05:00:07 2005 
           Last cron job run: Jan 30 04:30)    
 
  Check DQM Histograms:  No histogram files in the directory!! 
                         probably nothing has been processed 
 
</pre>
<!-- Author: Jose E. Garcia -->


<!-- Date: Sun Jan 30 08:48:51 2005 -->
<pre> DH Monitoring:   
  "home" is almost full. CSL people already know about the problem. 
 
</pre>
<!-- Author: Jose E. Garcia -->


<!-- Date: Sun Jan 30 10:37:22 2005 -->
<pre>Calibration 10A: 
 
  192988   2005.01.27   10:00:52     10A is not available 
 
 still not available. A mail has been sent to experts.</pre>
<!-- Author: Jose E. Garcia -->


<!-- Date: Mon Jan 31 09:03:19 2005 -->
<pre>Report from morning 31/01/2005  
 
1) PROCESSING (CPU):  latest processed run:       
   Last Run  stream A 192987  - 2005.01.27 
             stream B 192987 
             stream G 192987 
 
2) PROCESSING (TO TAPES): 
   Stream A 192870  - 2005.01.25 
   Stream B 192870  - 2005.01.25 
   Stream G 192372  - 2005.01.17 
 
 
3) CALIBRATIONS 
        10A:    Still not available:   192988  2005.01.27  (hole) 
		Still not available:   193013  2005.01.28  (hole) 
		 
		First not available:   193032  2005.01.30     
 
        11A:    First not available:   192988  2005.01.27    
 
 People from calibration has been notified by email.  
 
4) EXPRESS ANALYSIS 
	  
	 Procecessed yesterday up to here. Still some holes to fill in previous runs. 
        sopr00.last    aa02f205.0117phys.s   2005-01-30 03:20:19   193029 
        sopr00.last    aa02f206.0001phys.s       -1 
        sopr00.last    aa02f206.0063phys.s   2005-01-30 03:12:06   193030 
        sopr00.last    aa02f207.0001phys.s       -1 
        sopr00.last    aa02f207.006dphys.s   2005-01-30 09:39:03   193031 
        sopr00.last    aa02f207.00e7phys.s   2005-01-30 10:50:20   193031 
        sopr00.last    aa02f207.016dphys.s       -1 
 
 
5) DB Monitoring 
     ok 
6) DH Monitoring: everything is Ok 
  
7) DQM.  ( Last update: Mon Jan 31 05:00:07 2005 
           Last cron job run: Jan 31 04:30)  
 
  for run 192795 (31-Jan-2005 05:00) and directory 192795_aexp0d_psfiles, files D0.ps, SiCt.ps and ElectroWeak.ps have empty histograms.</pre>
<!-- Author: Jose E. Garcia -->


<!-- Date: Mon Jan 31 09:43:36 2005 -->
<pre>DQM.  
 
  for run 192795 (31-Jan-2005 05:00) and directory 192795_aexp0d_psfiles, files D0.ps, SiCt.ps and 
ElectroWeak.ps have empty histograms. 
 
 DQM experts: Empty histograms in this case are caused by no silicon present in the run.</pre>
<!-- Author: Jose E. Garcia -->


<!-- Date: Tue Feb 1 09:07:01 2005 -->
<pre>Report from morning 01/02/2005  
 
1) PROCESSING (CPU):  latest processed run:       
   Last Run  stream A 193008 2005.01.28 
             stream B 193008 
             stream G 193008 
 
2) PROCESSING (TO TAPES): 
   Stream A 193005 2005.01.28 
   Stream B 192870 2005.01.25 
   Stream G 192372 2005.01.17 
 
 
3)CALIBRATIONS 
        10A:    Still not available:   193032  2005.01.30   10:58:24 
        11A:    First not available:   193013  2005.01.28   17:24:52 
 
 
4)EXPRESS ANALYSIS 
	  
	 Still some holes to fill in previous runs. 
        sopr00.last    aa02f205.0117phys.s   2005-01-30 03:20:19   193029 
        sopr00.last    aa02f206.0001phys.s       -1 
        sopr00.last    aa02f206.0063phys.s   2005-01-30 03:12:06   193030 
        sopr00.last    aa02f207.0001phys.s       -1 
        sopr00.last    aa02f207.006dphys.s   2005-01-30 09:39:03   193031 
        sopr00.last    aa02f207.00e7phys.s   2005-01-30 10:50:20   193031 
        sopr00.last    aa02f207.016dphys.s       -1 
 
	14 last ROOT files have status "-1". 
 
 
5) DB Monitoring 
     ok 
6) DH Monitoring: the disk space consumption plots      
    are reasonable and fine 
 
7) DQM.  (Last update: Tue Feb 1 05:00:05 2005 
	  Last cron job run: Feb 1 04:30) 
 
	  Checked run # 192869. Plots are not empty and look ok.</pre>
<!-- Author: Jose E. Garcia -->


<!-- Date: Tue Feb 1 09:08:11 2005 -->
<pre>3)CALIBRATIONS  
        10A:    First not available:   193032  2005.01.30   10:58:24</pre>
<!-- Author: Jose E. Garcia -->


<!-- Date: Wed Feb 2 08:00:36 2005 -->
<pre>Report from morning 02/02/2005  
 
1) PROCESSING (CPU):  latest processed run:       
   Last Run  stream A 193031 2005.01.30 
             stream B 193031 
             stream G 193031 
 
2) PROCESSING (TO TAPES): 
   Stream A 193031 2005.01.30 
   Stream B 193006 2005.01.28 
   Stream G 192372 2005.01.17 
 
 
3)CALIBRATIONS 
        10A:    First not available:   193061 2005.02.01   07:08:38 
        11A:    First not available:   193032 2005.01.30   10:58:24 
	17A:    First not available:   193032 2005.01.30   10:58:24 
 
4)EXPRESS ANALYSIS 
	  
	 Still some holes to fill in previous runs. 
        sopr00.last    aa02f205.0117phys.s   2005-01-30 03:20:19   193029 
        sopr00.last    aa02f206.0001phys.s       -1 
        sopr00.last    aa02f206.0063phys.s   2005-01-30 03:12:06   193030 
        sopr00.last    aa02f207.0001phys.s       -1 
        sopr00.last    aa02f207.006dphys.s   2005-01-30 09:39:03   193031 
        sopr00.last    aa02f207.00e7phys.s   2005-01-30 10:50:20   193031 
        sopr00.last    aa02f207.016dphys.s       -1 
 
	14 last ROOT files have status "-1". 
 
 
5) DB Monitoring 
     ok 
6) DH Monitoring: the disk space consumption plots      
    are reasonable and fine 
 
7) DQM.  (Last update: Wed Feb 2 05:00:07 2005 
          Last cron job run: Feb 2 04:30) 
 
	  Checked run # 192893. Plots are not empty and look ok. 
	  Only PEMMonMETPEMTrigs and PEMMonPLUGZTrigs are empty.</pre>
<!-- Author: Jose E. Garcia -->


<!-- Date: Wed Feb 2 08:27:15 2005 -->
<pre>7) DQM. 
	  Checked run # 192893. Plots are not empty and look ok.  
	  Only PEMMonMETPEMTrigs and PEMMonPLUGZTrigs are empty. 
 
 Expert says that it is due to statistics.</pre>
<!-- Author: Jose E. Garcia -->


<!-- Date: Thu Feb 3 08:40:02 2005 -->
<pre>Report from morning 03/02/2005  
 
1) PROCESSING (CPU):  latest processed run:       
   Last Run  stream A 193032 2005.01.30 
             stream B 193032 
             stream G 193032 
 
2) PROCESSING (TO TAPES): 
   Stream A 193031 2005.01.30 
   Stream B 193030 2005.01.30 
   Stream G 192372 2005.01.17 
 
 
3)CALIBRATIONS 
        10A:    First not available:   193064 2005.02.01   10:12:49 
	....... 
        10A:    Not available	       193107 2005.02.02   08:38:15 
	10A:    Available              193109 2005.02.02   10:29:35 
        10A:    First not available:   193110 2005.02.02   10:50:20 
        ....... 
 
        11A:    First not available:   193049 2005.01.31   08:26:14 
        ....... 
 
    Experts have been contacted to solve the problem. 
 
4)EXPRESS ANALYSIS 
	  
	 Still some holes to fill in previous runs. 
        sopr00.last    aa02f205.0117phys.s   2005-01-30 03:20:19   193029 
        sopr00.last    aa02f206.0001phys.s       -1 
        sopr00.last    aa02f206.0063phys.s   2005-01-30 03:12:06   193030 
        sopr00.last    aa02f207.0001phys.s       -1 
        sopr00.last    aa02f207.006dphys.s   2005-01-30 09:39:03   193031 
        sopr00.last    aa02f207.00e7phys.s   2005-01-30 10:50:20   193031 
        sopr00.last    aa02f207.016dphys.s       -1 
 
	14 last ROOT files have status "-1". 
 
 
5) DB Monitoring 
     ok 
6) DH Monitoring: the disk space consumption plots      
    are reasonable and fine 
 
7) DQM.  (Last update: Thu Feb 3 05:00:05 2005 
	  Last cron job run: Feb 3 04:30) 
 
	  Checked run # 192987. Plots are not empty and look ok. 
</pre>
<!-- Author: Jose E. Garcia -->


<!-- Date: Fri Feb 4 09:14:48 2005 -->
<pre>Report from morning 04/02/2005  
 
1) PROCESSING (CPU):  latest processed run:       
   Last Run  stream A 193061 2005.02.01 
             stream B 193061 
             stream G 193061 
 
2) PROCESSING (TO TAPES): 
   Stream A 193049 2005.01.31 
   Stream B 193030 2005.01.29 
   Stream G 192372 
 
 
3)CALIBRATIONS 
        10A:    Not available:         193105 2005.02.02   00:58:42  
	10A:    Done                   193106-193110  
	10A:    Not available	       193111 2005.02.02   13:39:19 
	10A:    Not available          193112-193164 
	10A:    Done		       193166 2005.02.03   14:40:58 
	10A: 	Not available          193167 2005.02.03   14:56:18 
 
        11A:    First not available:   193064 2005.02.01   10:12:49 
 
 
4)EXPRESS ANALYSIS 
	  
	 Still some holes to fill in previous runs. 
        sopr00.last    aa02f205.0117phys.s   2005-01-30 03:20:19   193029 
        sopr00.last    aa02f206.0001phys.s       -1 
        sopr00.last    aa02f206.0063phys.s   2005-01-30 03:12:06   193030 
        sopr00.last    aa02f207.0001phys.s       -1 
        sopr00.last    aa02f207.006dphys.s   2005-01-30 09:39:03   193031 
        sopr00.last    aa02f207.00e7phys.s   2005-01-30 10:50:20   193031 
        sopr00.last    aa02f207.016dphys.s       -1 
 
	14 last ROOT files have status "-1". 
 
 
5) DB Monitoring 
     ok 
6) DH Monitoring: the disk space consumption plots      
    are reasonable and fine 
 
7) DQM.  (Last update: Fri Feb 4 05:00:06 2005 
	  Last cron job run: Feb 4 04:30) 
 
	  Checked run # 192994. Plots are not empty and look ok. 
</pre>
<!-- Author: Eric & Jose E. -->


<!-- Date: Sat Feb 5 10:16:25 2005 -->
<pre>Report from morning 02/05/2005  
 
 
CALIBRATIONS 
    10A:    First not available    193167 2005.02.03   14:56:18 
    11A:    First not available:   193104 2005.02.01   23:07:23 
 
    11A, which is used by production farms, is a bit behind.   
         Will watch this. 
 
 
EXPRESS ANALYSIS 
	  
   Last file aa02f207.00e7phys  2005-01-30 10:50:20  
   Seems to have problems. Will pursue with Pasha. 
 
DB Monitoring 
      
   In good shape.  Database is working hard, but not overloaded. 
 
DH Monitoring 
 
   The disk space consumption plots are good. 
 
DQM.   
 
   Last update: Sat Feb 5 05:00:14  
   Checked run # 193007 dataset aexp0d, plots are not  
   empty and look ok.</pre>
<!-- Author: Eric Wicklund -->


<!-- Date: Sun Feb 6 08:35:52 2005 -->
<pre>PROCESSING (CPU):  latest processed run:       
   Last Run  stream A 193067 2005.02.01 
             stream B 193067 
             stream G 193067 
 
   Farms have a hardware problem for node cdffarm2.fnal.gov and  
   have fallen behind. 
 
 
CALIBRATIONS 
 
  10A:    First not available    193167 2005.02.03   14:56:18 
  11A:    First not available:   193125 2005.02.03   06:46:42  
 
 
EXPRESS-ANALYSIS 
          
  Last file aa02f207.00e7phys  2005-01-30 10:50:20  
  Pasha is merging Express-Analysis with Physmon,  
  Express-Analysis is down for the time being.  
 
 
DB Monitoring 
      
  In good shape.  Database is working hard, but not overloaded. 
 
 
DH Monitoring 
 
  The disk space consumption plots are good. 
  Raw data is going to tapes okay. 
 
 
DQM.   
 
  Last update: Sun Feb 6 05:00:07 2005 
  Checked run # 193030 dataset aexp0d, plots are not empty  
  and look ok. 
</pre>
<!-- Author: Eric Wicklund -->


<!-- Date: Mon Feb 7 09:46:49 2005 -->
<pre>Report from morning 02/07/2005  
 
1) PROCESSING (CPU):  latest processed run:       
   Last Run  stream A 193067 2005.02.01 
             stream B 193067 
             stream G 193067 
 
   Farms have a hardware problem for node cdffarm2.fnal.gov and  
   have fallen behind. 
 
 
2) PROCESSING (TO TAPES): 
   Stream A 193049 2005.01.31 
   Stream B 193030 2005.01.29 
   Stream G 192372 
 
 
3) CALIBRATIONS 
 
    10A:    First not available    193242   2005.02.04   08:10:17 
    11A:    First not available:   193125   2005.02.03   06:46:42  
 
 
4) DB Monitoring 
      
   In good shape.  Database is working hard, but not overloaded. 
 
 
5) DH Monitoring 
 
    The disk space consumption plots are good. 
    Most recent raw data tape written at 2005-Feb-07 08:25:06  
 
 
6) DQM.   
 
    Last update: Mon Feb 7 05:00:07 2005  
    Checked run # 193013 dataset aexp0d, plots are not empty  
    and look ok. 
</pre>
<!-- Author: Eric Wicklund -->


<!-- Date: Tue Feb 8 10:00:49 2005 -->
<pre>Report from morning 02/08/2005  
 
1) PROCESSING (CPU):    
      
   Last Run  stream A 193167    2005.02.03   14:56:18 
             stream B 193167 
             stream G 193167  
 
   Farms have recovered from the hardware problem, and  
   are caught up to to the the calibrations (see below). 
 
 
2) PROCESSING (TO TAPES): 
 
   Stream A 193061 
   Stream B 193049 
   Stream G 192372 
 
 
3) CALIBRATIONS 
 
    10A:    First not available    193272   2005.02.05   03:38:47 
    11A:    First not available:   193242   2005.02.04   08:10:17  
 
    SAM-farm is having problems, and so BeamExe which runs  
    on SAM-farm is delayed. 
 
 
4) DB Monitoring 
      
   In good shape.  Database is working hard, but not overloaded. 
 
 
5) DH Monitoring 
 
    The disk space consumption plots are good. 
    Most recent raw data tape written at 2005-Feb-08 08:27:51  
    dCache looks good. 
 
 
6) DQM.   
 
    Last update: Tue Feb 8 05:00:06 2005  
    Checked run # 193031 dataset aexp0d,  
    plots are not empty and look ok.</pre>
<!-- Author: Eric Wicklund -->


<!-- Date: Wed Feb 9 15:12:33 2005 -->
<pre>Report from afternoon 02/09/2005  
 
1) PROCESSING (CPU):    
      
   Last Run   stream A 193167    2005.02.03   14:56:18 
              stream B 193167 
              stream G 193167  
 
   Farms are caught up to the calibrations (see below). 
 
 
2) PROCESSING (TO TAPES): 
 
   Last Run   stream A 193061    2005.02.01   07:08:38 
              stream B 193104    2005.02.01   23:07:23         
              stream G 192399    2005.01.18   15:10:49           
 
 
3) CALIBRATIONS 
 
   10A:    First not available    193272   2005.02.05   03:38:47 
   11A:    First not available:   193271   2005.02.04   22:20:52 
 
   Waiting for SVX calibrations. 
 
 
4) DB Monitoring 
      
   Database is heavily loaded at present. 
   Investigating. 
 
 
5) DH Monitoring 
 
   The disk space consumption plots are good. 
   Most recent raw data tape written at 2005-Feb-09 13:28:21 
   dCache looks good. 
 
 
6) DQM.   
 
   Last update: Wed Feb 9 05:00:25 2005  
   Checked run # 193061  2005.02.01 07:08:38  dataset aexp0d, 
   plots are not empty and look ok. 
</pre>
<!-- Author: Eric Wicklund -->


<!-- Date: Thu Feb 10 08:34:59 2005 -->
<pre>Report from morning 02/10/2005  
 
1) PROCESSING (CPU):    
      
   Last Run   stream A 193247  2005.02.04   13:31:58   
              stream B 193247 
              stream G 193247 
 
   Farms are caught up to the calibrations (see below). 
 
 
2) PROCESSING (TO TAPES): 
 
   Last Run   stream A 193167  2005.02.03   14:56:18  
              stream B 193104  2005.02.01   23:07:23   
              stream G 192399  2005.01.18   15:10:49         
 
 
3) CALIBRATIONS 
 
   10A:    First not available    193272   2005.02.05   03:38:47 
   11A:    First not available:   193271   2005.02.04   22:20:52 
 
   Waiting for SVX calibrations. 
 
 
4) DB Monitoring 
      
   In good shape.  Database is working hard, but not overloaded. 
 
 
5) DH Monitoring 
 
   The disk space consumption plots are good. 
   Most recent raw data tape written at 2005-Feb-10 08:09:01  
   dCache looks good. 
 
 
6) DQM.   
 
   Last update: Last update: Thu Feb 10 05:00:23 2005 
   Checked run # 193107 2005.02.02   08:38:15  dataset aexp0d, 
   plots are not empty and look ok.</pre>
<!-- Author: Eric Wicklund -->


<!-- Date: Fri Feb 11 10:06:22 2005 -->
<pre>Report from morning 02/11/2005  
 
1) PROCESSING (CPU):    
      
   Last Run   stream A 193247  2005.02.04   13:31:58   
              stream B 193247 
              stream G 193247 
 
   Farms are caught up to the calibrations (see below). 
 
 
2) PROCESSING (TO TAPES): 
 
   Last Run   stream A 193167  2005.02.03   14:56:18  
              stream B 193104  2005.02.01   23:07:23   
              stream G 192399  2005.01.18   15:10:49         
 
 
3) CALIBRATIONS 
 
   10A:    First not available    193377   2005.02.06   14:42:50  
   11A:    First not available:   193271   2005.02.04   22:20:52 
 
   BeamExe runs on SAM-Farm and the output of BeamExe is  
   needed for calibration 11A; there are currently problems. 
 
 
4) DB Monitoring 
      
   In good shape.  Database is working hard, but not overloaded. 
 
 
5) DH Monitoring 
 
   The disk space consumption plots are good. 
   Most recent raw data tape written at 2005-Feb-11 01:13:23  
   dCache is running well. 
 
 
6) DQM.   
 
   Last update: Last update: Fri Feb 11 05:00:14 2005 
   Checked run # 193167 2005.02.03   14:56:18 dataset aexp0d, 
   D0 plots are empty, but others look okay.  This 
   appears to be a side effect of the PHYSICS_HIGHLUM 
   trigger table. 
</pre>
<!-- Author: Eric Wicklund -->


<!-- Date: Sat Feb 12 09:20:11 2005 -->
<pre>   Last Run  stream A 193372  2005.02.06   01:20:53 
             stream B 193372 
             stream G 193372 
 
2) PROCESSING (TO TAPES): 
   Stream A 193247 2005.02.04   13:31:58 
   Stream B 193245 2005.02.04   12:30:33 
   Stream G 192399 2005.01.18   15:10:49 
 
 
3)CALIBRATIONS 
        10A:    First not available:   193463   2005.02.08   14:06:13 
 
        11A:    First not available:   193377   2005.02.06   14:42:50 
 
 
 
 
4)DB Monitoring 
     ok 
5) DH Monitoring: everything is Ok 
 
6) DQM is ok 
        193245_aexp0d_psfiles/ look ok 
        193245_bhmu0d_psfiles/ look ok 
</pre>
<!-- Author: Vladimir Rekovic -->


<!-- Date: Sat Feb 12 16:02:43 2005 -->
<pre> 
Reprort for afternoon 2005/02/12 
 
 **the only thing different from the morning is 3) 11A calibration run.  See bellow 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 193372  2005.02.06   01:20:53 
             stream B 193372 
             stream G 193372 
 
2) PROCESSING (TO TAPES): 
   Stream A 193247 2005.02.04   13:31:58 
   Stream B 193245 2005.02.04   12:30:33 
   Stream G 192399 2005.01.18   15:10:49 
 
 
3)CALIBRATIONS 
        10A:    First not available:   193463   2005.02.08   14:06:13 
 
        11A:    First not available:   193399   2005.02.07   09:51:09 
 
 
 
 
4)DB Monitoring 
     ok 
5) DH Monitoring: everything is Ok 
 
6) DQM is ok 
        193245_aexp0d_psfiles/ look ok 
        193245_bhmu0d_psfiles/ look ok 
</pre>
<!-- Author: Vladimir Rekovic -->


<!-- Date: Sun Feb 13 09:09:59 2005 -->
<pre>Reprort for morning 2005/02/12 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 193398    2005.02.07 08:23:31 
             stream B 193398 
             stream G 193398 
 
2) PROCESSING (TO TAPES): 
   Stream A 193377      2005.02.06   14:42:50 
   Stream B 193271      2005.02.04   22:20:52 
   Stream G 192399      2005.01.18   15:10:49 
 
 
 
3)CALIBRATIONS 
        10A:    First not available:   193463         2005.02.08   14:06:13 
 
        11A:    First not available:   193399         2005.02.07   09:51:09 
 
 
 
 
4)DB Monitoring 
 
        For time period 12-FEB-2005 18:10:02 to 12-FEB-2005 22:00:02 
the following two users had a high CPU usage: 
 
User's login name   CPU Used (sec)   Cpu per session (sec) 
-----------------   --------------   --------------------- 
bello                        11873                   0.859 
ruiza                        10012                   0.847 
 
 
After this period, and at the beginning of this shift their CPU usage went to a reasonable level.  No message sent to the users. 
 
5) DH Monitoring: everything is Ok 
 
6) DQM is ok 
        Hisotgrams for the run 193275 2005.02.05   08:36:40 
look ok. 
        193275_aexp0d_psfiles are OK : D0 plots not empty.  ElectroWeak (PEMMu 
ons) 
plots empty. 
        193275_gjt10d_psfiles are OK: 
</pre>
<!-- Author: Vladimir Rekovic -->


<!-- Date: Sun Feb 13 16:30:52 2005 -->
<pre> 
Reprort for afternoon 2005/02/13 
 
 
 The only thing changed from the morning is 
 
3)CALIBRATIONS 
        10A:    First not available: 193488   2005.02.09   14:40:31 
 
4)DB Monitoring: OK 
 
5) DH Monitoring: OK 
 
6) DQM is OK 
 
</pre>
<!-- Author: Vladimir Rekovic -->


<!-- Date: Mon Feb 14 16:10:25 2005 -->
<pre>Info under 1), 2), and 3) did not change since the morning. 
 
Reprort for afternoon 2005/02/14 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 193398    2005.02.07 08:23:31 
             stream B 193398 
             stream G 193398 
 
2) PROCESSING (TO TAPES): 
   Stream A 193377      2005.02.06   14:42:50 
   Stream B 193271      2005.02.04   22:20:52 
   Stream G 192399      2005.01.18   15:10:49 
 
Contacted Suen about Stream G.  His response was: 
The production is up to the DSP new run, 
and is essentially shutdown waiting for new patch. 
The tape writting won't be resumed anytime soon.  
 
 
 
3)CALIBRATIONS 
        10A:    First not available:   193488         2005.02.09   14:40:31 
 
        11A:    First not available:   193399         2005.02.07   09:51:09 
 
 
 
 
4)DB Monitoring 
   OK 
   Special comment:  user bello had over 1800 connections for the period 
10am-11am.  User not notified, since the number of connections decreased 
rapidly. 
 
5) DH Monitoring: everything is Ok 
 
6) DQM is ok 
        Checked plot in 
        193377_aexp0d_psfiles/ 
        193377_gjt10d_psfiles/ 
</pre>
<!-- Author: Vladimir Rekovic -->


<!-- Date: Tue Feb 15 10:36:10 2005 -->
<pre> 
Reprort for morning 2005/02/15 
 
Farms waiting for more data, so the info under 1) and 2) are same as yesterday. 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 193398    2005.02.07 08:23:31 
             stream B 193398 
             stream G 193398 
 
2) PROCESSING (TO TAPES): 
   Stream A 193377      2005.02.06   14:42:50 
   Stream B 193271      2005.02.04   22:20:52 
   Stream G 192399      2005.01.18   15:10:49 
 
 
 
3)CALIBRATIONS 
        10A:    First not available:   193846         2005.02.13   01:05:02 
 
        11A:    First not available:   193469         2005.02.08   19:33:05 
 
 
 
 
4)DB Monitoring 
   OK 
 
5) DH Monitoring: everything is Ok 
 
6) DQM is ok 
</pre>
<!-- Author: Vladimir Rekovic -->


<!-- Date: Wed Feb 16 08:22:54 2005 -->
<pre> 
Reprort for morning 2005/02/16 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 193472    2005.02.09   00:05:36 
             stream B 193398    2005.02.07   08:23:31 
             stream G 193398 
 
2) PROCESSING (TO TAPES): 
   Stream A 193398      2005.02.07   08:23:31 
   Stream B 193398      2005.02.07   08:23:31 
   Stream G 192399      2005.01.18   15:10:49 
 
 
 
3)CALIBRATIONS 
        10A:    First not available:   193846           2005.02.13   01:05:02 
 
        11A:    First not available:   193475           2005.02.09   04:30:32 
 
 
 
 
4)DB Monitoring 
   OK 
 
5) DH Monitoring: everything is Ok 
 
6) DQM 
        Highest luminosity run folder is empty. !!!!</pre>
<!-- Author: Vladimir Rekvovic -->


<!-- Date: Thu Feb 17 08:31:14 2005 -->
<pre>Reprort for morning 2005/02/17 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 193475    2005.02.09   04:30:32 
             stream B 193398    2005.02.07   08:23:31 
             stream G 193398 
 
2) PROCESSING (TO TAPES): 
   Stream A 193472      2005.02.09   00:05:36 
   Stream B 193398      2005.02.07   08:23:31 
   Stream G 192399      2005.01.18   15:10:49 
 
 
 
3)CALIBRATIONS 
        10A:    First not available:   193913         2005.02.14   22:29:33 
 
        11A:    First not available:   193479         2005.02.09   05:16:48 
 
 
 
 
4)DB Monitoring 
   OK 
 
5) DH Monitoring: everything is Ok 
        Note:  Around 5 am, noticed lots of idle recources. 
        For the same period, "wasted line" inclreaseed, and dropped at the 
        same time idle recources disappeared. 
 
6) DQM 
        AGAIN, same as yesterday, Highest luminosity run folder is empty. !!!! 
        email sent to mmp@fnal.gov,regis@fnal.gov,portell@fnal.gov 
</pre>
<!-- Author: Vladimir Rekovic -->


<!-- Date: Fri Feb 18 10:02:23 2005 -->
<pre>Reprort for morning 2005/02/18 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 193487 2005.02.09   11:50:54 
             stream B 193479 2005.02.09   05:16:48 
             stream G 193398 2005.02.07   08:23:31 
 
2) PROCESSING (TO TAPES): 
   Stream A 193472      2005.02.09   00:05:36 
   Stream B 193398      2005.02.07   08:23:31 
   Stream G 192399      2005.01.18   15:10:49 
 
 
 
3)CALIBRATIONS 
        10A:    First not available:   193913         2005.02.14   22:29:33 
 
        11A:    First not available:   193855         2005.02.13   03:05:54 
 
 
 
 
4)DB Monitoring 
   OK 
 
5) DH Monitoring: CAF  is down since 9 pm last night.  Eliliot is working on it 
CAFCONDOR is OK. 
</pre>
<!-- Author: Vladimir Rekovic -->


<!-- Date: Sat Feb 19 09:43:46 2005 -->
<pre>Report from morning 02/19/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194028  2005.02.17   16:58:08 
             stream B 194032  2005.02.18   00:17:59 
             stream G 194032  2005.02.18   00:17:59 
 
2) PROCESSING (TO TAPES): 
   Stream A 193488  2005.02.09   14:40:31 
   Stream B 193398  2005.02.07   08:23:31 
   Stream G 192399  2005.01.18   15:10:49 
 
 
3)CALIBRATIONS 
        10A:    First not available:   194043 2005.02.18   08:43:02 
        11A:    First not available:   193911 2005.02.14   17:54:52 
 
 
5) DB Monitoring 
     ok 
6) DH Monitoring: ok 
 
 
7) DQM. 
   Last update: Sat Feb 19 05:00:08 2005 
   Last cron job run: Feb 19 04:30 
 
   not find entries in the table it probably means that nothing was processed.</pre>
<!-- Author: Jieun Kim -->


<!-- Date: Sun Feb 20 08:27:25 2005 -->
<pre>Report from morning 02/20/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194028  2005.02.17   16:58:08 
             stream B 194032  2005.02.18   00:17:59 
             stream G 194032  2005.02.18   00:17:59 
 
2) PROCESSING (TO TAPES): 
   Stream A 193488  2005.02.09   14:40:31 
   Stream B 193398  2005.02.07   08:23:31 
   Stream G 192399  2005.01.18   15:10:49 
 
 
3)CALIBRATIONS 
        10A:    First not available:   194146 2005.02.18   17:06:08 
        11A:    First not available:   194033 2005.02.18   05:43:59 
 
 
5) DB Monitoring 
     ok 
6) DH Monitoring: ok 
 
 
7) DQM. 
   Last update: Sun Feb 20 05:00:14 2005 
   Last cron job run: Feb 20 04:30 
 
   not find entries in the table it probably means that nothing was processed.</pre>
<!-- Author: Jieun Kim -->


<!-- Date: Mon Feb 21 07:47:02 2005 -->
<pre>Report from morning 02/21/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194028  2005.02.17   16:58:08 
             stream B 194032  2005.02.18   00:17:59 
             stream G 194032  2005.02.18   00:17:59 
 
2) PROCESSING (TO TAPES): 
   Stream A 193488  2005.02.09   14:40:31 
   Stream B 193398  2005.02.07   08:23:31 
   Stream G 192399  2005.01.18   15:10:49 
 
 
3)CALIBRATIONS 
        10A:    First not available:   194146 2005.02.18   17:06:08 
        11A:    First not available:   194033 2005.02.18   05:43:59 
 
 
5) DB Monitoring : ok 
 
6) DH Monitoring: ok 
 
 
7) DQM. 
   Last update: Mon Feb 21 05:00:06 2005 
   Last cron job run: Feb 21 04:30 
 
   Highest luminosity run folder is empty.</pre>
<!-- Author: Jieun Kim -->


<!-- Date: Tue Feb 22 08:46:55 2005 -->
<pre>Report from morning 02/22/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194028  2005.02.17   16:58:08 
             stream B 194032  2005.02.18   00:17:59 
             stream G 194032  2005.02.18   00:17:59 
 
2) PROCESSING (TO TAPES): 
   Stream A 193488  2005.02.09   14:40:31 
   Stream B 193398  2005.02.07   08:23:31 
   Stream G 192399  2005.01.18   15:10:49 
 
 
3)CALIBRATIONS 
        10A:    First not available:   194146 2005.02.18   17:06:08 
        11A:    First not available:   194033 2005.02.18   05:43:59 
 
 
5) DB Monitoring : ok 
 
6) DH Monitoring: ok 
 
 
7) DQM. 
   Last update: Tue Feb 22 05:00:07 2005 
   Last cron job run: Feb 22 04:30 
 
   Check DQM Histogram: Last run 193479 (2005.02.09   05:16:48) 
</pre>
<!-- Author: Jieun Kim -->


<!-- Date: Wed Feb 23 08:45:14 2005 -->
<pre>Report from day: 2005/02/23 
 
   1. PROCESSING (CPU): latest processed run: 
 
   Last Run  stream A 194028    2005.02.17   16:58:08 
             stream B 194032    2005.02.18   00:17:59 
             stream G 194032    2005.02.18   00:17:59 
 
 
 
   2. PROCESSING (TO TAPES): 
 
   Stream A 193488      2005.02.09   14:40:31 
   Stream B 193398      2005.02.07   08:23:31 
   Stream G 192399      2005.01.18   15:10:49 
 
 
   3. CALIBRATIONS: 
 
        10A:    First not available:   194316           2005.02.22   08:17:07 
        11A:    First not available:   194265           2005.02.21   10:09:31 
 
 
   4. DB MONITORING: 
 
     In good shape.  Database is working hard, but not overloaded. 
 
 
   5. DH Monitoring: 
 
    The disk space consumption plots are good. 
    dCache looks good. 
 
 
   6. DQM: 
 
    Last update: Wed Feb 23 05:00:07 2005 
    Last cron job run: Feb 23 04:30 
     
    Directory for  Highest luminosity run  is empty.</pre>
<!-- Author: Jieun Kim -->


<!-- Date: Thu Feb 24 08:50:37 2005 -->
<pre>Report from morning 02/24/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194266   2005.02.21   11:22:10 
             stream B 194266   2005.02.21   11:22:10 
             stream G 194266   2005.02.21   11:22:10 
 
2) PROCESSING (TO TAPES): 
   Stream A 194266  2005.02.21   11:22:10 
   Stream B 193889  2005.02.14   00:18:49 
   Stream G 192399  2005.01.18   15:10:49 
 
 
3)CALIBRATIONS 
        10A:    First not available:   194318   2005.02.22   08:48:41 
        11A:    First not available:   194315   2005.02.22   05:52:14 
 
 
5) DB Monitoring : ok 
 
6) DH Monitoring: ok 
 
 
7) DQM. 
   Last update: Thu Feb 24 05:00:07 2005 
   Last cron job run: Feb 24 04:30 
 
   Directory for  Highest luminosity run  is empty. 
</pre>
<!-- Author: Jieun Kim -->


<!-- Date: Fri Feb 25 09:05:43 2005 -->
<pre>Report from morning 02/25/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194266   2005.02.21   11:22:10 
             stream B 194266   2005.02.21   11:22:10 
             stream G 194266   2005.02.21   11:22:10 
 
2) PROCESSING (TO TAPES): 
   Stream A 194266  2005.02.21   11:22:10 
   Stream B 194266  2005.02.21   11:22:10 
   Stream G 192399  2005.01.18   15:10:49 
 
 
3)CALIBRATIONS 
        10A:    First not available:   194320   2005.02.22   13:11:57 
        11A:    First not available:   194315   2005.02.22   05:52:14 
 
 
5) DB Monitoring : ok 
 
6) DH Monitoring: ok 
 
 
7) DQM. 
Last update: Fri Feb 25 05:00:10 2005 
Last cron job run: Feb 25 04:30 
 
  Histograms for the latest runs are OK 
  in runs 193846, 193910 and 194030 (all aexp0d) the PEMMONPLUGZTRIGS histogram 
  is empty. 
</pre>
<!-- Author: Alex -->
<!-- Comment: Fri Feb 25 15;54;14 comment by...Alex -->
<pre>End of shift note: calibrations are still stuck where they
were this morning. Jason was notified.
DQM histograms checked with Mircea: the empty PEM histograms
are presumably due to lack of statistics.</pre>


<!-- Date: Sat Feb 26 08:32:49 2005 -->
<pre>Report from morning 02/26/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194318   2005.02.22   08:48:41 
             stream B 194318   2005.02.22   08:48:41 
             stream G 194318   2005.02.22   08:48:41 
 
2) PROCESSING (TO TAPES): 
   Stream A 194266  2005.02.21   11:22:10 
   Stream B 194023  2005.02.17   13:08:42 
   Stream G 192399  2005.01.18   15:10:49 
 
 
3)CALIBRATIONS 
        10A:    First not available:   194459 2005.02.25   03:43:01 
        11A:    First not available:   194319 2005.02.22   12:43:48 
 
 
5) DB Monitoring : ok 
 
6) DH Monitoring: ok 
 
CNAFCAF shows again an interruption of head node communications around 
6 AM... a feature? Also, at 4:30 am it shows some kind of spike in 
load=0 together with "wasted" and a load drop: network problems while running 
data jobs? 
 
7) DQM. 
Last update: Sat Feb 26 05:00:08 2005 
Last cron job run: Feb 26 04:30 
 
  Histograms for the latest runs are OK 
</pre>
<!-- Author: Alex -->
<!-- Comment: Sat Feb 26 15;54;40 comment by...Alex -->
<pre>Last round of checks before end of shift: everything ok.</pre>


<!-- Date: Sun Feb 27 09:18:01 2005 -->
<pre>Report from morning 02/27/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194458 2005.02.25   03:13:04 
             stream B 194458 2005.02.25   03:13:04 
             stream G 194458 2005.02.25   03:13:04 
 
   Farms have a hardware problem for node cdffarm2.fnal.gov and 
   have fallen behind. 
 
 
2) PROCESSING (TO TAPES): 
   Stream A 194379 2005.02.23   19:23:33 
   Stream B 194379 2005.02.23   19:23:33 
   Stream G 192399 2005.02.23   19:23:33 
 
 
3) CALIBRATIONS 
 
    10A:    First not available:   194459 2005.02.25   03:43:01 
    11A:    First not available:   194459 2005.02.25   03:43:01 
 
 
4) DB Monitoring 
 
   In good shape. 
 
 
5) DH Monitoring 
 
    The disk space consumption plots are good. 
    Most recent raw data tape written at 2005-Feb-27 08:31:09 
 
 
6) DQM. 
    Last update: Sun Feb 27 05:00:10 2005 
    Last cron job run: Feb 27 04:31 
    Checked run # 194149,194152,194161 
    plots are not empty and look ok. 
 
</pre>
<!-- Author: Alex -->


<!-- Date: Sun Feb 27 16:04:27 2005 -->
<pre>Last round of checks at the end of CO shift. All ok. 
Even if below threshold, there are a couple of oddities 
in the DB connections plot (showing a large usage by Varganov 
and cdfmcprod) and in the cpu history, showing a couple of 
spikes from cdfdqm. It is all within boundaries anyway, so 
I decided not to send out the usual warnings. We will see 
if the behaviour presists...</pre>
<!-- Author: Alex -->


<!-- Date: Mon Feb 28 07:50:23 2005 -->
<pre>Report from morning 02/28/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194458   2005.02.25   03:13:04 
             stream B 194458   2005.02.25   03:13:04 
             stream G 194458   2005.02.25   03:13:04 
 
2) PROCESSING (TO TAPES): 
   Stream A 194379  2005.02.23   19:23:33 
   Stream B 194379  2005.02.23   19:23:33 
   Stream G 192399  2005.01.18   15:10:49 
 
 
3)CALIBRATIONS 
        10A:    First not available:   194459 2005.02.25   03:43:01 
        11A:    First not available:   194459 2005.02.25   03:43:01 
 
 
5) DB Monitoring : ok 
   user Rakitin has spikes above 1200 connections for ~1hr, with ~1000 CPU seconds on 
   the same time range, the appropriate email was sent out. 
 
6) DH Monitoring: ok 
   Data logger Disk 6 is rather crowded, even though still within the recommended 75% 
 
7) DQM. 
Last update: Mon Feb 28 05:00:08 2005 
Last cron job run: Feb 28 04:58 
 
No DQM histograms available 
</pre>
<!-- Author: Alex -->
<!-- Comment: Mon Feb 28 16;08;54 comment by...Alex -->
<pre>End of shift check shows all OK.</pre>


<!-- Date: Tue Mar 1 10:40:58 2005 -->
<pre>Shutdown in progress, several items couldn't be checked. Will post an update as soon aseverything is back up! 
 
Report from morning 03/01/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194458   2005.02.25   03:13:04 
             stream B 194458   2005.02.25   03:13:04 
             stream G 194458   2005.02.25   03:13:04 
 
2) PROCESSING (TO TAPES): 
   Stream A 194458  2005.02.25   03:13:04 
   Stream B 194458  2005.02.25   03:13:04 
   Stream G 192399  2005.01.18   15:10:49 
 
 
3)CALIBRATIONS 
        10A:    First not available:   Cannot check because of shutdown 
        11A:    First not available:   Cannot check because of shutdown 
 
 
5) DB Monitoring : ok 
   cdfopr is using 13495 hours of database cpu 
   cdfopr and toolman have spikes in the cpu history graph as well... 
 
6) DH Monitoring: ok 
   Data logger Disk 6 is ok now. Most DH monitoring tasks not accomplished 
   because of the shutdown in progress... 
 
7) DQM. 
   Not accessible because of shutdown 
</pre>
<!-- Author: Alex -->


<!-- Date: Wed Mar 2 08:14:31 2005 -->
<pre>Report from morning 03/01/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194458   2005.02.25   03:13:04 
             stream B 194458   2005.02.25   03:13:04 
             stream G 194458   2005.02.25   03:13:04 
 
2) PROCESSING (TO TAPES): 
   Stream A 194458  2005.02.25   03:13:04 
   Stream B 194458  2005.02.25   03:13:04 
   Stream G 192399  2005.01.18   15:10:49 
 
 
3)CALIBRATIONS 
        10A:    First not available:   194590 2005.02.28   13:58:12 
        11A:    First not available:   194459 2005.02.25   03:43:01 
 
 
5) DB Monitoring : ok 
   cdfrep01 current usage table not loading? 
 
6) DH Monitoring: ok 
 
 
7) DQM. 
Last update: Wed Mar 2 07:47:14 2005 
Last cron job run: Mar 2 07:40 
 
Histograms: 
          193913 aexp0d  ok 
          194005 aexp0d  ok 
          194023 aexp0d  ok 
          194266 bhmu0d  ok 
          194323 gjt10d  ok 
          194382 gjt10d  ok 
</pre>
<!-- Author: Alex -->
<!-- Comment: Wed Mar  2 08;42;22 comment by...Alex -->
<pre>Well, the above entry's date is 03/02/05 :p</pre>

<!-- Comment: Wed Mar  2 16;28;24 comment by...Alex -->
<pre>Last round of check after end of shift: all good.</pre>


<!-- Date: Thu Mar 3 08:13:42 2005 -->
<pre>Report from morning 03/03/2005  
 
1) PROCESSING (CPU):  latest processed run:       
   Last Run  stream A 194458   2005.02.25   03:13:04 
             stream B 194458   2005.02.25   03:13:04 
             stream G 194458   2005.02.25   03:13:04 
 
2) PROCESSING (TO TAPES): 
   Stream A 194458  2005.02.25   03:13:04 
   Stream B 194458  2005.02.25   03:13:04 
   Stream G 194458  2005.02.25   03:13:04  
 
 
3)CALIBRATIONS 
        10A:    First not available:   194590 2005.02.28   13:58:12 
        11A:    First not available:   194459 2005.02.25   03:43:01 
 
 
5) DB Monitoring : ok 
    
 
6) DH Monitoring: ok 
 
 
7) DQM.   
Last update: Thu Mar 3 05:00:14 2005 
Last cron job run: Mar 3 04:30 
 
Histograms: 
          194382 aexp0d  ok 
 
 
</pre>
<!-- Author: Alex -->


<!-- Date: Fri Mar 4 09:15:07 2005 -->
<pre>Report from morning 03/04/2005  
 
1) PROCESSING (CPU):  latest processed run:       
   Last Run  stream A 194504   2005.02.26   04:59:20 
             stream B 194504   2005.02.26   04:59:20 
             stream G 194504   2005.02.26   04:59:20 
 
2) PROCESSING (TO TAPES): 
   Stream A 194458  2005.02.25   03:13:04 
   Stream B 194458  2005.02.25   03:13:04 
   Stream G 194458  2005.02.25   03:13:04  
 
 
3)CALIBRATIONS 
        10A:    First not available:   194590 2005.02.28   13:58:12 
        11A:    First not available:   194535 2005.02.26   14:44:37 
 
 
5) DB Monitoring : ok 
   User matthias had an abnormal number of inactive connections. 
   Sending the standard email ccd to Petar, Susana and Mircea. 
 
6) DH Monitoring: ok 
 
 
7) DQM.   
Last update: Fri Mar 4 05:00:05 2005 
Last cron job run: Mar 4 04:30 
 
Histograms: 
          No New Histograms to look at. 
</pre>
<!-- Author: lys -->


<!-- Date: Sat Mar 5 15:50:06 2005 -->
<pre>Report from morning 03/05/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194504   2005.02.26   04:59:20 
             stream B 194504   2005.02.26   04:59:20 
             stream G 194504   2005.02.26   04:59:20 
 
2) PROCESSING (TO TAPES): 
   Stream A 194504  2005.02.26   04:59:20 
   Stream B 194502  2005.02.26   04.29.07 
   Stream G 194502  2005.02.26   04.29.07 
 
 
3)CALIBRATIONS 
        10A:    First not available:   194590 2005.02.28   13:58:12 
        11A:    First not available:   194535 2005.02.26   14:44:37 
 
 
5) DB Monitoring : ok 
        Murat near "limits", cdfrep01 usage. 
 
6) DH Monitoring: ok 
 
 
7) DQM. 
Last update: Sat Mar 5 05:00:06 2005 
Last cron job run: Mar 5 04:30 
 
------- 
  No change in calibrations. Sent email to 10A and 11A people 
 
</pre>
<!-- Author: lys -->


<!-- Date: Sun Mar 6 16:13:21 2005 -->
<pre>Report from morning 03/06/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194504   2005.02.26   04:59:20 
             stream B 194504   2005.02.26   04:59:20 
             stream G 194504   2005.02.26   04:59:20 
 
2) PROCESSING (TO TAPES): 
   Stream A 194504  2005.02.26   04:59:20 
   Stream B 194502  2005.02.26   04.29.07 
   Stream G 194502  2005.02.26   04.29.07 
 
 
3)CALIBRATIONS 
        10A:    First not available:   194590 2005.02.28   13:58:12 
        11A:    First not available:   194590 2005.02.28   13:58:12 
 
 
5) DB Monitoring : 
        cdfrep01 current usage shows Vivek has 26 inactive connections 
of &gt; 100 secs each. Sent email. 
 
6) DH Monitoring: ok 
 
 
7) DQM. 
Last update: Sun Mar 6 05:00:08 2005 
Last cron job run: Mar 6 04:30 
 
Histograms: 
          No New Histograms to look at. 
 
--------------------------- 
By 2 pm vivek problem had gone away. 
 
Around 2pm Fermi time CNAFCAF showed "unusual" behavior, is it 
"abnormal"? - no message sent. 
 
Started 10KTest with 6.1.0pre6, pass1 (standard) and pass2 (-o maxopt). 
 
</pre>
<!-- Author: lys -->


<!-- Date: Mon Mar 7 15:58:47 2005 -->
<pre> 
Report from morning 03/07/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194504   2005.02.26   04:59:20 
             stream B 194504   2005.02.26   04:59:20 
             stream G 194504   2005.02.26   04:59:20 
 
2) PROCESSING (TO TAPES): 
   Stream A 194504  2005.02.26   04:59:20 
   Stream B 194502  2005.02.26   04.29.07 
   Stream G 194502  2005.02.26   04.29.07 
 
 
3)CALIBRATIONS 
        10A:    First not available:   194590 2005.02.28   13:58:12 
        11A:    First not available:   194590 2005.02.28   13:58:12 
 
 
5) DB Monitoring : ok 
 
 
6) DH Monitoring: ok 
 
 
7) DQM. 
Last update: Mon Mar 7 05:00:05 2005 
Last cron job run: Mar 7 04:30 
 
Histograms: 
          No New Histograms to look at. 
 
--------------------------- 
  As of 12 noon, 10A calibrations all available (up to run 194903) 
  11A calibrations still not avaliable from 194950 on. 
 
</pre>
<!-- Author: lys -->


<!-- Date: Tue Mar 8 16:20:54 2005 -->
<pre>Report from morning 03/08/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194562   2005.02.28   00:04:45 
             stream B 194562   2005.02.28   00:04:45 
             stream G 194562   2005.02.28   00:04:45 
 
2) PROCESSING (TO TAPES): 
   Stream A 194535  2005.02.26   14:44:37 
   Stream B 194502  2005.02.26   04.29.07 
   Stream G 194502  2005.02.26   04.29.07 
 
 
3)CALIBRATIONS 
        10A:    First not available:   194907 2005.03.07   15:33:42 
        11A:    First not available:   194899 2005.03.07   04:42:31 
 
 
5) DB Monitoring : ok 
          High CPU used by anantg 
 
6) DH Monitoring: ok 
          Predator status "unusual" 
 
7) DQM. 
Last update: Tue Mar 8 05:00:07 2005 
Last cron job run: Mar 8 04:30 
 
Histograms: 
          New hists to look at 
 
----------------- 
predator status remained unusual, emailed Mircea 
 
emailed anantg concerning high DB CPU usage 
 
emailed scuri concerning many inactive DB connections 
 
checked DQM histograms, looked OK. 
 
 
</pre>
<!-- Author: lys -->


<!-- Date: Wed Mar 9 15:50:19 2005 -->
<pre> 
Report from morning 03/09/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194862   2005.03.06   05:27:01 
             stream B 194845   2005.03.04   23:23:50 
             stream G 194862   2005.03.06   05:27:01 
 
2) PROCESSING (TO TAPES): 
   Stream A 194562  2005.02.28   00:04:45 
   Stream B 194562  2005.02.28   00.04.45 
   Stream G 194562  2005.02.28   00.04.45 
 
 
3)CALIBRATIONS 
        10A:    None not available: 
        11A:    First not available:   194899 2005.03.07   04:42:31 
 
 
5) DB Monitoring : ok 
          High CPU used by anantg 
 
6) DH Monitoring: ok 
          Predator status "unusual" 
 
7) DQM. 
Last update: Wed Mar 9 05:00:14 2005 
Last cron job run: Mar 9 04:30 
 
Histograms: 
          New hists look OK (only an aexp0d to look at) 
 
--------------------- 
16:00  All OK 
 
Randy Herber is the person to consult on Predator "unusual" problem. He says he understands why we get this unusual pattern. 
He wants to keep being told of such occurrences. 
</pre>
<!-- Author: lys -->


<!-- Date: Thu Mar 10 16:25:49 2005 -->
<pre>Report from morning 03/10/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194862   2005.03.06   05:27:01 
             stream B 194862   2005.03.06   05:27:01 
             stream G 194862   2005.03.06   05:27:01 
 
2) PROCESSING (TO TAPES): 
   Stream A 194562  2005.02.28   00:04:45 
   Stream B 194562  2005.02.28   00.04.45 
   Stream G 194562  2005.02.28   00.04.45 
 
 
3)CALIBRATIONS 
        10A:    None not available: 
        11A:    First not available:   194899 2005.03.07   04:42:31 
 
 
5) DB Monitoring : ok 
          High CPU used by Rakitin 
          cdfrep01 Connections plot - no plot! 
 
6) DH Monitoring: ok 
          Predator status "unusual" - no change from yesterday 
      - Randy Herber was told. 
          CAF plots: CAF "wasted" line high, green area low, 
      white area high, 00:00 to 08:00 . 
 
7) DQM. 
Last update: Thu Mar 10 05:00:07 2005 
Last cron job run: Mar 10 04:30 
 
Histograms: 
          No new histograms. 
 
--------------------- 
 
email Ratikin re DB usage. 
email Petar and Cabrera re DB connection plot not there. 
2.00 pm many (26) inactive connections to cdfrep01, Matthias, send email. 
4.00 pm: CAL plot looks better now. 
</pre>
<!-- Author: lys -->


<!-- Date: Fri Mar 11 09:34:03 2005 -->
<pre>Report from morning 03/11/2005 
 
1) PROCESSING (CPU):  latest processed run: 
   Last Run  stream A 194862   2005.03.06   05:27:01 
             stream B 194862   2005.03.06   05:27:01 
             stream G 194862   2005.03.06   05:27:01 
 
2) PROCESSING (TO TAPES): 
   Stream A 194562  2005.02.28   00:04:45 
   Stream B 194562  2005.02.28   00.04.45 
   Stream G 194562  2005.02.28   00.04.45 
 
 
3)CALIBRATIONS 
        10A:    First not available:   195134 2005.03.10   21:00:32 
        11A:    First not available:   194899 2005.03.07   04:42:31 
 
 
5) DB Monitoring : ok 
 
 
6) DH Monitoring: ok 
          Predator status "unusual" - no change from yesterday 
 
7) DQM. 
Last update: Fri Mar 11 05:00:08 2005 
Last cron job run: Mar 11 04:30 
 
Histograms: 
          No new histograms. 
</pre>
<!-- Author: toru -->


<!-- Date: Sat Mar 12 12:04:01 2005 -->
<pre>Report from morning 03/12/2005  
 
1) PROCESSING (CPU):  latest processed run:       
   Last Run  stream A 194862   2005.03.06   05:27:21 
             stream B 194862   2005.03.06   05:27:21 
             stream G 194862   2005.03.06   05:27:21 
 
2) PROCESSING (TO TAPES): 
   Stream A 194562  2005.02.28   00:04:45 
   Stream B 194562  2005.02.28   00.04.45 
   Stream G 194562  2005.02.28   00.04.45  
 
 
3)CALIBRATIONS 
        10A:    First not available:   195134 2005.03.10   21:00:32 
        11A:    First not available:   194899 2005.03.07   04:42:31 
 
 
5) DB Monitoring : ok 
 
 
6) DH Monitoring: ok 
      Randy's comment for "Predator status" see below  
 
	The paired numbers are days of months and hours of the day 
	of when predator runs.  When predator runs and finds nothing 
	to do, the monitoring script just adds the time to the list. 
	Only when predator finds some to do is a new-line appended, 
 
	It is reasonable and proper to call it to data-handling's 
	attention when more than a few times are stacked up in 
	the report.  Please, to ask about it when this happens, 
	even if it turns out not to be an actual predator problem 
	(as it is in this case). 
 
	The reason for the long list of times this time is that 
	the Enstore has not added any files to its Complete File 
	Listing in about a day.  Someone may want to ask the 
	Enstore people whether that report has stopped (as has 
	happened in the pass).  But, I have seen this happen when 
	no files were coming out of the CAF for other reasons. 
 
7) DQM.   
Last update: Sat Mar 12 05:00:05 2005 
Last cron job run: Mar 12 04:30 
 
Histograms: 
          No new histograms. 
</pre>
<!-- Author: toru -->


<!-- Date: Sat Mar 12 13:33:30 2005 -->
<pre>e-mail to Mircea: BeamExe job status has long "FILE NOT READY",more than 3 days</pre>
<!-- Author: Toru -->


<!-- Date: Sun Mar 13 08:28:30 2005 -->
<pre>email to experts respect with calibration delay.</pre>
<!-- Author: Toru -->


<!-- Date: Sun Mar 13 13:01:16 2005 -->
<pre>Report from morning 03/13/2005  
 
1) PROCESSING (CPU):  latest processed run:       
   Last Run  stream A 194862   2005.03.06   05:27:21 
             stream B 194862   2005.03.06   05:27:21 
             stream G 194862   2005.03.06   05:27:21 
 
2) PROCESSING (TO TAPES): 
   Stream A 194562  2005.02.28   00:04:45 
   Stream B 194562  2005.02.28   00.04.45 
   Stream G 194562  2005.02.28   00.04.45  
 
 
3)CALIBRATIONS 
        10A:    First not available:   195134 2005.03.10   21:00:32 
        11A:    First not available:   194899 2005.03.07   04:42:31 
 
 
5) DB Monitoring : ok 
 
 
6) DH Monitoring: ok 
 
7) DQM.   
Last update: Sun Mar 13 05:00:06 2005 
Last cron job run: Mar 13 04:30 
 
Histograms: 
          No new histograms. 
</pre>
<!-- Author: Toru -->


<!-- Date: Mon Mar 14 13:12:37 2005 -->
<pre>email to Mircea as "http://fnpcc.fnal.gov/process_stat.txt" 
is not accessed!!</pre>
<!-- Author: Toru -->


<!-- Date: Mon Mar 14 15:08:15 2005 -->
<pre>Report from morning 03/14/2005  
 
1) ROCESSING (CPU):  latest processed run: 
 
   Last Run  stream A 194862   2005.03.06   05:27:21 
             stream B 194862   2005.03.06   05:27:21 
             stream G 194862   2005.03.06   05:27:21 
 
2) PROCESSING (TO TAPES): 
 
	     Stream A 194562  2005.02.28   00:04:45 
	     Stream B 194562  2005.02.28   00.04.45 
	     Stream G 194562  2005.02.28   00.04.45  
 
 
3)CALIBRATIONS 
        10A:    First not available:   195203 2005.03.11   19:35:18 
        11A:    First not available:   194899 2005.03.07   04:42:31 
 
 
5) DB Monitoring : ok 
 
 
6) DH Monitoring: ok 
      Predator status unusual 
 
7) DQM.   
Last update: Mon Mar 14 05:00:04 2005 
Last cron job run: Mar 14 04:30 
 
Histograms: 
          No new histograms</pre>
<!-- Author: Toru -->


<!-- Date: Tue Mar 15 14:25:30 2005 -->
<pre>Report from morning 03/15/2005  
 
1) ROCESSING (CPU):  latest processed run: 
 
   Last Run  stream A 194862   2005.03.06   05:27:21 
             stream B 194862   2005.03.06   05:27:21 
             stream G 194862   2005.03.06   05:27:21 
 
2) PROCESSING (TO TAPES): 
 
	     Stream A 194562  2005.02.28   00:04:45 
	     Stream B 194562  2005.02.28   00.04.45 
	     Stream G 194562  2005.02.28   00.04.45  
 
 
3)CALIBRATIONS 
        10A:    First not available:   195203 2005.03.11   19:35:18 
        11A:    First not available:   194911 2005.03.08   01:28:30 
 
 
5) DB Monitoring : ok 
 
 
6) DH Monitoring: ok 
      Predator status unusual 
 
7) DQM.   
Last update: Tue Mar 15 05:00:05 2005  
Last cron job run: Mar 15 04:30 
 
Histograms: 
          No new histograms. 
</pre>
<!-- Author: Toru -->


<!-- Date: Wed Mar 16 13:38:07 2005 -->
<pre>Report from morning 03/16/2005  
 
1) ROCESSING (CPU):  latest processed run: 
 
   Last Run  stream A 194909   2005.03.07   22:14:04 
             stream B 194909   2005.03.07   22:14:04 
             stream G 194909   2005.03.07   22:14:04 
 
2) PROCESSING (TO TAPES): 
 
	     Stream A 194562  2005.02.28   00:04:45 
	     Stream B 194562  2005.02.28   00.04.45 
	     Stream G 194562  2005.02.28   00.04.45  
 
 
3)CALIBRATIONS 
        10A:    First not available:   195203 2005.03.11   19:35:18 
        11A:    First not available:   194911 2005.03.08   01:28:30 
 
 
5) DB Monitoring : ok 
 
 
6) DH Monitoring: ok 
 
 
7) DQM.   
Last update: Wed Mar 16 05:00:05 2005 
Last cron job run: Mar 16 04:30 
 
Histograms: 
          No new histograms.</pre>
<!-- Author: Toru -->


<!-- Date: Wed Mar 16 15:23:59 2005 -->
<pre>Predator works well!</pre>
<!-- Author: Toru -->


<!-- Date: Thu Mar 17 12:37:37 2005 -->
<pre>called through SCICO and mailed as: 
 
The sysadmin have contacted Sun and confirmed 
we need to replace some memory on fcdfora4. 
 
We have experienced to interruptions & at this 
point they are willing to swap out the mem. 
 
Expected downtime is ~1 hour & the sysadmin would 
like to try to schedule it for today at 14:00 hours. 
</pre>
<!-- Author: Toru -->


<!-- Date: Thu Mar 17 15:42:58 2005 -->
<pre>Report from morning 03/17/2005  
 
1) ROCESSING (CPU):  latest processed run: 
 
   Last Run  stream A 194909   2005.03.07   22:14:04 
             stream B 194909   2005.03.07   22:14:04 
             stream G 194909   2005.03.07   22:14:04 
 
2) PROCESSING (TO TAPES): 
 
	     Stream A 194810  2005.03.03   15:39:36 
	     Stream B 194562  2005.02.28   00.04.45 
	     Stream G 194562  2005.02.28   00.04.45  
 
 
3)CALIBRATIONS 
        10A:    First not available:   195343 2005.03.15   22:08:44 
        11A:    First not available:   194911 2005.03.08   01:28:30 
 
 
5) DB Monitoring : ok 
 
 
6) DH Monitoring: ok 
 
 
7) DQM.   
Last update: Thu Mar 17 05:00:05 2005 
Last cron job run: Mar 17 04:30 
 
Histograms: 
          No new histograms. 
</pre>
<!-- Author: Toru -->


<!-- Date: Fri Mar 18 09:02:28 2005 -->
<pre>Report from morning 03/18/2005  
 
1) ROCESSING (CPU):  latest processed run: 
 
   Last Run  stream A 194909   2005.03.07   22:14:04 
             stream B 194909   2005.03.07   22:14:04 
             stream G 194909   2005.03.07   22:14:04 
 
2) PROCESSING (TO TAPES): 
 
	     Stream A 194909 2005.03.07   22:14:04 
	     Stream B 194809 2005.03.03   14:55:23 
	     Stream G 194562 2005.02.28   00:04:45 
 
 
3)CALIBRATIONS 
        10A:    First not available:   195343 2005.03.15   22:08:44 
        11A:    First not available:   195343 2005.03.15   22:08:44 
 
 
5) DB Monitoring : ok 
 
 
6) DH Monitoring: ok 
 
 
7) DQM.   
Last update: Fri Mar 18 05:00:05 2005 
Last cron job run: Mar 18 04:30 
 
Histograms: 
          No new histograms. 
</pre>
<!-- Author: Jonas -->


<!-- Date: Fri Mar 18 09:43:30 2005 -->
<pre>Trying to check MONITOR DCACHE load (clicking on the link to URL:  
http://cdfkits.fnal.gov/DIST/logs/dcache/load/NOW.txt) 
I get a webpage with the following error message: 
 
./load_log: /home/cdfsoft/maint/dcache/load/load: /bin/bash: bad interpreter: Permission denied 
 
Don't quite know who to bother with this as it probably isn't a DCACHE problem, but a scripting error or something of that sort - 
sent email to cmircea@fnal.gov anyway.</pre>
<!-- Author: Jonas -->


<!-- Date: Sat Mar 19 08:32:29 2005 -->
<pre>   OFFLINE REPORT: Sat Mar 19 08:08:59 CST 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 195341  2005.03.15 16:32:51 
    Stream B: 195341  2005.03.15 16:32:51 
    Stream G: 195341  2005.03.15 16:32:51 
 
2) PROCESSING (TO TAPE): 
    Stream A: 194909  2005.03.07 22:14:04 
    Stream B: 194909  2005.03.07 22:14:04 
    Stream G: 194909  2005.03.07 22:14:04 
 
3) CALIBRATIONS 
    10A:   195410 PHYSICS_HIGHLUM_3_00   7 2005.03.18   07:09:59     PROD_PHYSICS_CDF        10A is not available 
         this is 1.0 days old. 
    11A:   195343         PHYSICS_3_00   3 2005.03.15   22:08:44     PROD_PHYSICS_CDF        11A is not available 
         this is 3.4 days old. 
    17A:   195343         PHYSICS_3_00   3 2005.03.15   22:08:44     PROD_PHYSICS_CDF        17A is not available 
         this is 3.4 days old. 
 
4) DB MONITORING 
  OK. Large spike of in "Top UsersConnection for nonFarms" by cdfdqm for about 1/2 hour, between 0:00am and 1:00am. 
 
5) DH MONITORING 
  OK except CAF. 
  CAF: CNAFCAF: No monitoring plots 
       MITCAF: "all waiting sections" at zero from 21:45. 
                System infor: Looks bad - yellow and green steadily decreasing towards zero, pink line going up (batch system problem?). 
 
7) DQM MONITORING 
OK 
 
Last update: Sat Mar 19 05:00:05 2005 
Last cron job run: Mar 19 04:30 
 
Histograms OK. 
</pre>
<!-- Author: Jonas -->


<!-- Date: Sat Mar 19 08:50:40 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-188&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-188  width='300' align='top'></a></td><td>Regarding the CAF issues mentioned in today's offline report:<BR> 
 CNAFCAF: Apart from the missing monitoring plots, it seems to be working OK (or at least working - jobs are running etc)<BR> 
 MITCAF: Seems dead (accoring to its web monitor, no jobs are running). Attached the monitoring plot for MITCAF.</td></tr></table>
<!-- Author: Jonas -->


<!-- Date: Sat Mar 19 14:04:03 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-189&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-189  width='300' align='top'></a></td><td>MITCAF is back up.</td></tr></table>
<!-- Author: Jonas -->


<!-- Date: Sun Mar 20 08:52:19 2005 -->
<pre>When trying to complete the usual offline report, I found that several webpages aren't working: 
http://fnpcc.fnal.gov/process_stat.txt 
gives me: 
 
Sun Mar 20 08:00:28 CST 2005 
Start run is 190697 
 
(this is the entire content of the webpage) 
 
similarly,  clicking on 
"Availability of 10A" (http://www-cdf.fnal.gov/~cdfopr/calibrations/prod_physics_cdf_10A.html) 
 
I find this: 
 
Sun Mar 20 08:00:06 CST 2005 
 
and 
 
"Availability of 11A" (http://www-cdf.fnal.gov/~cdfopr/calibrations/prod_physics_cdf_11A.html) 
 
gives me the following output: 
 
Sun Mar 20 08:00:34 CST 2005 
 
ORA-01089: immediate shutdown in progress - no operations are permitte 
Parse error at character offset 0 in SQL statement. 
 
The first two are not very informative, but the last one suggests an oracle problem. Sent mail to oracle people and to Mircea. 
 
 
</pre>
<!-- Author: Jonas -->


<!-- Date: Sun Mar 20 13:23:56 2005 -->
<pre>   OFFLINE REPORT: Sun Mar 20 13:09:17 CST 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 195402  2005.03.17 17:05:57  (2 day(s), 20 hours ago) 
    Stream B: 195402  2005.03.17 17:05:57  (2 day(s), 20 hours ago) 
    Stream G: 195402  2005.03.17 17:05:57  (2 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 194909  2005.03.07 22:14:04  (12 day(s), 14 hours ago) 
    Stream B: 194909  2005.03.07 22:14:04  (12 day(s), 14 hours ago) 
    Stream G: 194909  2005.03.07 22:14:04  (12 day(s), 14 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for:   195449 PHYSICS_HIGHLUM_3_00   7 2005.03.18   13:59:25     PROD_PHYSICS_C 
           This is 1 day(s), 23 hours old. 
   11A is not available for:   195405         PHYSICS_3_00   3 2005.03.17   18:26:30     PROD_PHYSICS_C 
           This is 2 day(s), 18 hours old. 
   17A is not available for:   195405         PHYSICS_3_00   3 2005.03.17   18:26:30     PROD_PHYSICS_C 
           This is 2 day(s), 18 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK, except CAF 
 o MITCAF appears to be going down again. 
 o CNAFCAF are also down (but that is a known problem) 
 
7) DQM MONITORING 
o LAST DAY RUN STATUS: OK-ish. 
 
  Last update: Sun Mar 20 05:00:06 2005 
  Last cron job run: Mar 20 04:30 
 
  All entries from run 195209 onwards report N_ACCESS.  
  Of the other 8 runs (194535-194845) 7 have at least one red box (N_STARTED). 
  Only 2 have at least one green box. 
 
Histograms: OK 
</pre>
<!-- Author: Jonas -->


<!-- Date: Sun Mar 20 13:25:55 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-190&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-190  width='300' align='top'></a></td><td>MITCAF appears to be going down.</td></tr></table>
<!-- Author: Jonas -->


<!-- Date: Mon Mar 21 07:59:44 2005 -->
<pre>   OFFLINE REPORT: Mon Mar 21 07:43:33 CST 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 195402  2005.03.17 17:05:57  (3 day(s), 14 hours ago) 
    Stream B: 195402  2005.03.17 17:05:57  (3 day(s), 14 hours ago) 
    Stream G: 195402  2005.03.17 17:05:57  (3 day(s), 14 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 194909  2005.03.07 22:14:04  (13 day(s), 9 hours ago) 
    Stream B: 194909  2005.03.07 22:14:04  (13 day(s), 9 hours ago) 
    Stream G: 194909  2005.03.07 22:14:04  (13 day(s), 9 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       195573         PHYSICS_3_00   3 2005.03.20   20:29:17     PROD_PHYSICS_CD 
       This is 11 hours old. 
   11A is not available for: 
       195449 PHYSICS_HIGHLUM_3_00   7 2005.03.18   13:59:25     PROD_PHYSICS_CD 
       This is 2 day(s), 17 hours old. 
   17A is not available for: 
       195449 PHYSICS_HIGHLUM_3_00   7 2005.03.18   13:59:25     PROD_PHYSICS_CD 
       This is 2 day(s), 17 hours old. 
 
4) DB MONITORING 
 OK 
 
5) DH MONITORING 
 o Raw Data tapes: 5 failures (but most recent 2 are success): 
 
2005-Mar-21 06:00:07 failed to catalog fileset CB5859.0 
2005-Mar-21 06:02:39 failed to catalog fileset CB5859.1 
2005-Mar-21 06:05:10 failed to catalog fileset CB5859.2 
2005-Mar-21 06:07:42 failed to catalog fileset CB5859.3 
2005-Mar-21 06:10:13 failed to catalog fileset CB5859.4 
2005-Mar-21 07:21:33 filesets CB5860.0 CB5860.1 CB5860.2 CB5860.3 CB5860.4 spooled to tape CB5860 (encp SUCCESS) 
2005-Mar-21 07:38:49 filesets CB5861.0 CB5861.1 spooled to tape CB5861 (encp SUCCESS) 
 
 o Predator status: 
   One dodgy entry at 7am (find below the 4 most recent entries): 
21-07  Number of data files added: 0 
21-06  21-05  21-04  Number of data files added: 0 
21-03  Number of data files added: 0 
21-02  Number of data files added: 0 
 
 o CAF: 
  - SDSCCAF down (new!) 
  - MITCAF still not well (but running) 
 
 The rest is OK. 
 
7) DQM MONITORING 
 o DQM Status for last 24 hours  
 
Last update: Mon Mar 21 05:00:05 2005 
Last cron job run: Mar 21 04:30 
 
  all entries report N_ACCESS (runs 195242, 195244-6) 
 
 o No histograms. 
</pre>
<!-- Author: Jonas -->
<!-- Comment: Mon Mar 21 08;22;20 comment by...Jonas -->
<pre>Actually, SCSCCAF is OK - it's just not being used. I misinterpreted the plot.</pre>

<!-- Comment: Mon Mar 21 08;54;01 comment by...Jonas -->
<pre>Art Kreymer just informed us that what I put under the heading "failures to write raw data to tape" are in fact less harmful failures to catalogue. This happened because the cdfofprd (DFC) database is down. The data are on tape and will be cataloged once cdfofprd is back up.</pre>


<!-- Date: Mon Mar 21 13:41:18 2005 -->
<pre>The following are presumable effects of the data base outage: 
 
o websites for processing and calibration status down: 
http://fnpcc.fnal.gov/process_stat.txt 
http://www-cdf.fnal.gov/~cdfopr/calibrations/prod_physics_cdf_10A.html 
http://www-cdf.fnal.gov/~cdfopr/calibrations/prod_physics_cdf_11A.html 
 
o Odd entries in Predetor status: 
 
21-13  Number of data files added: 8 
21-12  21-11  21-10  21-09  21-08  21-07  Number of data files added: 0 
21-06  21-05  21-04  Number of data files added: 0 
21-03  Number of data files added: 0</pre>
<!-- Author: Jonas -->


<!-- Date: Tue Mar 22 07:58:31 2005 -->
<pre>   OFFLINE REPORT: Tue Mar 22 07:48:04 CST 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 195570  2005.03.20 03:36:30  (2 day(s), 4 hours ago) 
    Stream B: 195570  2005.03.20 03:36:30  (2 day(s), 4 hours ago) 
    Stream G: 195570  2005.03.20 03:36:30  (2 day(s), 4 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 194909  2005.03.07 22:14:04  (14 day(s), 9 hours ago) 
    Stream B: 194909  2005.03.07 22:14:04  (14 day(s), 9 hours ago) 
    Stream G: 194909  2005.03.07 22:14:04  (14 day(s), 9 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       195573         PHYSICS_3_00   3 2005.03.20   20:29:17     PROD_PHYSICS_CD 
       This is 1 day(s), 11 hours old. 
   11A is not available for: 
       195571         PHYSICS_3_00   3 2005.03.20   03:49:18     PROD_PHYSICS_CD 
       This is 2 day(s), 3 hours old. 
   17A is not available for: 
       195571         PHYSICS_3_00   3 2005.03.20   03:49:18     PROD_PHYSICS_CD 
       This is 2 day(s), 3 hours old. 
 
4) DB MONITORING 
 OK 
 
5) DH MONITORING 
 OK 
 
7) DQM MONITORING 
 o LAST DAY RUN STATUS 
 
Last update: Tue Mar 22 05:00:05 2005 
Last cron job run: Mar 22 04:30 
 
All fields report N_ACCESS (runs 195249, 195261, 195267, 195296, 195305, 195306) 
  
 o Histograms: No histgrams. 
</pre>
<!-- Author: Jonas -->


<!-- Date: Wed Mar 23 07:57:34 2005 -->
<pre>   OFFLINE REPORT: Wed Mar 23 07:46:16 CST 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 195570  2005.03.20 03:36:30  (3 day(s), 4 hours ago) 
    Stream B: 195570  2005.03.20 03:36:30  (3 day(s), 4 hours ago) 
    Stream G: 195570  2005.03.20 03:36:30  (3 day(s), 4 hours ago) 
 
2) PROCESSING (TO TAPE):  
(Some old runs are being reprocessed, explaining the apparent the jump 
from 14 days since yesterday to 21 days this morning, for stream b) 
 
    Stream A: 194909  2005.03.07 22:14:04  (15 day(s), 9 hours ago) 
    Stream B: 194630  2005.03.01 15:52:09  (21 day(s), 15 hours ago) 
    Stream G: 194909  2005.03.07 22:14:04  (15 day(s), 9 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       195573         PHYSICS_3_00   3 2005.03.20   20:29:17     PROD_PHYSICS_CD 
       This is 2 day(s), 11 hours old. 
   11A is not available for: 
       195571         PHYSICS_3_00   3 2005.03.20   03:49:18     PROD_PHYSICS_CD 
       This is 3 day(s), 3 hours old. 
   17A is not available for: 
       195571         PHYSICS_3_00   3 2005.03.20   03:49:18     PROD_PHYSICS_CD 
       This is 3 day(s), 3 hours old. 
 
4) DB MONITORING 
 OK 
 
5) DH MONITORING 
 OK 
 
7) DQM MONITORING 
o LAST DAY RUN STATUS 
Last update: Wed Mar 23 05:00:05 2005 
Last cron job run: Mar 23 04:30 
 
All fields report N_ACCESS 
(runs 195308, 195309, 195313, 195316, 195332, 195337, 195341) 
 
o HISTOGRAMS 
No histrograms 
</pre>
<!-- Author: Jonas -->


<!-- Date: Wed Mar 23 15:37:13 2005 -->
<pre>I'm not sure 26 waiting Pnfs is large enough to worry about, but it is certainly more than usual: 
 
Waiting for 
    1200 Door 
      26 Pnfs 
      33 Pool 
      22 Limbo 
 
I also found that the last 30 doors took longer than usual (3min rather than ca 1/2 min) - maybe that's related? 
Wed Mar 23 12:22:29 CST 2005 30 doors in 22 sec 
Wed Mar 23 13:22:37 CST 2005 30 doors in 30 sec 
Wed Mar 23 14:22:24 CST 2005 30 doors in 17 sec 
Wed Mar 23 15:24:58 CST 2005 30 doors in 171 sec 
 
emailed Mircea.</pre>
<!-- Author: Jonas -->


<!-- Date: Thu Mar 24 08:04:42 2005 -->
<pre>   OFFLINE REPORT: Thu Mar 24 07:53:07 CST 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 195570  2005.03.20 03:36:30  (4 day(s), 4 hours ago) 
    Stream B: 195570  2005.03.22 11:44:13  (1 day(s), 20 hours ago) 
    Stream G: 195570  2005.03.20 03:36:30  (4 day(s), 4 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 195570  2005.03.20 03:36:30  (4 day(s), 4 hours ago) 
    Stream B: 195408  2005.03.18 03:43:24  (6 day(s), 4 hours ago) 
    Stream G: 194909  2005.03.07 22:14:04  (16 day(s), 9 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       195624         PHYSICS_3_00   3 2005.03.22   14:09:42     PROD_PHYSICS_CD 
       This is 1 day(s), 17 hours old. 
   11A is not available for: 
       195624         PHYSICS_3_00   3 2005.03.22   14:09:42     PROD_PHYSICS_CD 
       This is 1 day(s), 17 hours old. 
   17A is not available for: 
       195624         PHYSICS_3_00   3 2005.03.22   14:09:42     PROD_PHYSICS_CD 
       This is 1 day(s), 17 hours old. 
 
4) DB MONITORING 
 OK 
 
5) DH MONITORING 
o  Check record of the raw data tapes written: OK 
 
o predator status: OK 
 
o DCACHE is having problems: 
 DCACHE doors: 
Thu Mar 24 06:22:41 CST 2005 30 doors in 32 sec 
 OOPS, bad door 14  
 OOPS, bad door 15  
 OOPS, bad door 16  
 OOPS, bad door 17  
 OOPS, bad door 18  
 OOPS, bad door 19  
 OOPS, bad door 28  
 OOPS, bad door 29  
Thu Mar 24 07:27:11 CST 2005 30 doors in 301 sec 
 
 
 DCACHE ping: 
Thu Mar 24 07:34:13 CST 2005 0:09.54 
Thu Mar 24 07:44:22 CST 2005 Command failed! 
Server error message for [1]: "java.lang.NullPointerException" (errno 2). 
Failed open file in the dCache. 
dc_stage fail : "java.lang.NullPointerException" 
System error: Input/output error 
Command exited with non-zero status 255 
0:20.97 
 
o CAF: 
 MITCAF down 
 
7) DQM MONITORING 
Last update: Thu Mar 24 05:00:06 2005 
Last cron job run: Mar 24 04:30 
 
aexp0d and bhmu0d being processed,  gjt10d gmbs0d still N_ACCESS, for runs  
195343 (2005.03.15), 195346, 195354, 195360 
 
No histograms</pre>
<!-- Author: Jonas -->


<!-- Date: Thu Mar 24 08:24:34 2005 -->
<pre>More on DCACHE (in addition to this morning's offline report): This is how MONITOR DCACHE load page (http://cdfkits.fnal.gov/DIST/logs/dcache/load/NOW.txt) looks like: 
============================================================ 
 
Run Thu Mar 24 08:06:40 CST 2005      Data from Thu Mar 24 08:05:58 2005 
basename: too few arguments 
Try `basename --help' for more information. 
       0 Logins 
       0 Active 
       0 GridFTP 
       0 Killed 
 
Waiting for 
       0 Door 
       0 Pnfs 
       0 Pool 
       0 Limbo 
 
Users 
sort: open failed: nusers.lis: No such file or directory 
 
Datasets 
sort: open failed: ndsets.lis: No such file or directory 
 
============================================================ 
 
o The   DCA System Status/Cell Services link (http://cdfdca.fnal.gov:443/cellInfo) doesn't respond. 
 
o DCACHE doors and DCACHE ping still as in this morning's offline report. 
</pre>
<!-- Author: Jonas -->


<!-- Date: Thu Mar 24 12:33:10 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-191&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-191  width='300' align='top'></a></td><td>Number of connections seems very high. Notified Petar and Susana</td></tr></table>
<!-- Author: Jonas -->


<!-- Date: Thu Mar 24 12:37:04 2005 -->
<pre>DCACHE has been fixed and seems to be working fine.</pre>
<!-- Author: Jonas -->


<!-- Date: Fri Mar 25 09:10:37 2005 -->
<pre>   OFFLINE REPORT: Fri Mar 25 07:50:58 CST 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 195623  2005.03.22 11:44:13  (2 day(s), 20 hours ago) 
    Stream B: 195623  2005.03.22 11:44:13  (2 day(s), 20 hours ago) 
    Stream G: 195623  2005.03.22 11:44:13  (2 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 195623  2005.03.22 11:44:13  (2 day(s), 20 hours ago) 
    Stream B: 195623  2005.03.22 11:44:13  (2 day(s), 20 hours ago) 
    Stream G: 195623  2005.03.22 11:44:13  (2 day(s), 20 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       195689         PHYSICS_3_00   3 2005.03.24   11:28:31     PROD_PHYSICS_CD 
       This is 20 hours old. 
   11A is not available for: 
       195624         PHYSICS_3_00   3 2005.03.22   14:09:42     PROD_PHYSICS_CD 
       This is 2 day(s), 17 hours old. 
   17A is not available for: 
       195624         PHYSICS_3_00   3 2005.03.22   14:09:42     PROD_PHYSICS_CD 
       This is 2 day(s), 17 hours old. 
 
4) DB MONITORING 
 OK 
 
5) DH MONITORING 
 OK 
 (MITCAF still down) 
 
7) DQM MONITORING 
o LAST DAY RUN STATUS 
Last update: Fri Mar 25 05:00:05 2005 
Last cron job run: Mar 25 04:30 
 
All fields either green (processing sucessfully finished) or blue 
(processing in progress). Some of the green fields report N_ABORTED 
though (N_ABORTED for runs 195343, 195346, 195354, 195382). 
 
o Histograms 
 OK 
 (but D0.ps histogram empty for run 195360_aexp0d) 
</pre>
<!-- Author: rakitin -->


<!-- Date: Fri Mar 25 13:43:47 2005 -->
<pre>CNAFCAF seems to have a short lack of activity about 3 hours ago. 
MIT CAF is still down. Everything else is OK. 
</pre>
<!-- Author: rakitin -->


<!-- Date: Fri Mar 25 16:27:31 2005 -->
<pre>User maxi has very high CPU usage - warning email sent</pre>
<!-- Author: rakitin -->


<!-- Date: Sat Mar 26 09:22:21 2005 -->
<pre>   OFFLINE REPORT: Sat Mar 26 07:44:49 CST 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 195623  2005.03.22 11:44:13  (3 day(s), 20 hours ago) 
    Stream B: 195623  2005.03.22 11:44:13  (3 day(s), 20 hours ago) 
    Stream G: 195623  2005.03.22 11:44:13  (3 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 195623  2005.03.22 11:44:13  (3 day(s), 20 hours ago) 
    Stream B: 195623  2005.03.22 11:44:13  (3 day(s), 20 hours ago) 
    Stream G: 195623  2005.03.22 11:44:13  (3 day(s), 20 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       195720         PHYSICS_3_00   3 2005.03.25   09:44:46     PROD_PHYSICS_CD 
       This is 22 hours old. 
   11A is not available for: 
       195624         PHYSICS_3_00   3 2005.03.22   14:09:42     PROD_PHYSICS_CD 
       This is 3 day(s), 17 hours old. 
   17A is not available for: 
       195624         PHYSICS_3_00   3 2005.03.22   14:09:42     PROD_PHYSICS_CD 
       This is 3 day(s), 17 hours old. 
 
4) DB MONITORING 
   OK 
 
5) DH MONITORING 
   OK 
   (MIT CAF is still down)  
 
7) DQM MONITORING 
 
   o LAST DAY RUN STATUS 
   Last update: Sat Mar 26 05:00:06 2005 
   Last cron job run: Mar 26 04:30 
 
   There was an aborted (red) run 194315 
 
   Most of the fields are green (processing sucessfully finished)  
or blue (processing in progress). There are 12 runs for which 
the green fields report N_ABORTED =&gt; low luminosity 
 
   o Histograms 
 
    - Generally - OK 
 
    - all histograms in D0.ps file are empty for runs 194862_aexp0d, 19520_aexp0d9 and  
      195245_aexp0d which are regular PHYSICS_3_00, not PHYSICS_HIGHLUM_3_00)  
 
   o Full list of empty histograms:  
     - run 194862_aexp0d 
         - file D0.ps:          all histograms except d0Trk and NHitSI 
         - file Electroweak.ps: PEMMonPLUGZTrigs  
         - file SiCt.ps:        SiSVXIIEffVsProfile, all OI, SA and IO histograms  
    - run 195209_aexp0d 
         - file D0.ps:          all histograms  
         - file SiCt.ps:        all COT histograms, SiSVXIIEffVsProfile,  
                                all OI, SA and IO histograms  
    - run 195245_aexp0d 
         - file D0.ps:          all histograms  
         - file SiCt.ps:        SiSVXIIEffVsProfile, all OI, SA and IO histograms 
</pre>
<!-- Author: rakitin -->


<!-- Date: Sat Mar 26 16:13:00 2005 -->
<pre>End of shift - everything looks Ok, except for 11A calibration 
(last file is more than 4 days old)</pre>
<!-- Author: rakitin -->


<!-- Date: Sun Mar 27 07:58:15 2005 -->
<pre>  OFFLINE REPORT: Sun Mar 27 07:43:45 CST 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 195623  2005.03.22 11:44:13  (4 day(s), 19 hours ago) 
    Stream B: 195623  2005.03.22 11:44:13  (4 day(s), 19 hours ago) 
    Stream G: 195623  2005.03.22 11:44:13  (4 day(s), 19 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 195623  2005.03.22 11:44:13  (4 day(s), 19 hours ago) 
    Stream B: 195623  2005.03.22 11:44:13  (4 day(s), 19 hours ago) 
    Stream G: 195623  2005.03.22 11:44:13  (4 day(s), 19 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       195745         PHYSICS_3_01   1 2005.03.26   13:21:53     PROD_PHYSICS_CD 
       This is 18 hours old. 
   11A is not available for: 
       195624         PHYSICS_3_00   3 2005.03.22   14:09:42     PROD_PHYSICS_CD 
       This is 4 day(s), 17 hours old. 
   17A is not available for: 
       195624         PHYSICS_3_00   3 2005.03.22   14:09:42     PROD_PHYSICS_CD 
       This is 4 day(s), 17 hours old. 
 
4) DB MONITORING 
  OK 
 
5) DH MONITORING 
  OK (MIT CAF is still down) 
 
7) DQM MONITORING 
 
   o LAST DAY RUN STATUS 
     Last update: Sun Mar 27 05:00:06 2005 
     Last cron job run: Mar 27 04:30 
 
     Only one run present (195341) - it is fine 
 
   o Histograms - plots for only one run (195341) are available 
     All are OK except PEMMonPLUGZTrigs in file Electroweak.ps 
</pre>
<!-- Author: rakitin -->


<!-- Date: Sun Mar 27 15:48:02 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-192&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-193  width='300' align='top'></a></td><td>cdfdqm is going crazy!</td></tr></table>
<!-- Author: rakitin -->


<!-- Date: Mon Mar 28 09:20:25 2005 -->
<pre>   
OFFLINE REPORT: Mon Mar 28 07:53:22 CST 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 195623  2005.03.22 11:44:13  (5 day(s), 20 hours ago) 
    Stream B: 195623  2005.03.22 11:44:13  (5 day(s), 20 hours ago) 
    Stream G: 195623  2005.03.22 11:44:13  (5 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 195623  2005.03.22 11:44:13  (5 day(s), 20 hours ago) 
    Stream B: 195623  2005.03.22 11:44:13  (5 day(s), 20 hours ago) 
    Stream G: 195623  2005.03.22 11:44:13  (5 day(s), 20 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       195759         PHYSICS_3_01   1 2005.03.27   14:00:29     PROD_PHYSICS_CD 
       This is 17 hours old. 
   11A is not available for: 
       195624         PHYSICS_3_00   3 2005.03.22   14:09:42     PROD_PHYSICS_CD 
       This is 5 day(s), 17 hours old. 
   17A is not available for: 
       195624         PHYSICS_3_00   3 2005.03.22   14:09:42     PROD_PHYSICS_CD 
       This is 5 day(s), 17 hours old. 
 
4) DB MONITORING 
   satoru has very many inactive connections, zero seconds each 
 
5) DH MONITORING 
   OK (MIT CAF is still down)    
 
7) DQM MONITORING 
   Last update: Mon Mar 28 05:00:06 2005 
   Last cron job run: Mar 28 04:30 
 
   Run 194536 was aborted, others are OK    
 
   Empty histograms: 
   - run 195569 
      - file D0.ps:          all histograms 
      - file Electroweak.ps: PEMMonPLUGZTrigs  
      - file SiCt.ps:        SiSVXIIEffVsProfile, all OI, SA and IO histograms  
 
   - run 195570 
      - file D0.ps:          all histograms 
      - file Electroweak.ps: PEMMonPLUGZTrigs  
      - file SiCt.ps:        SiSVXIIEffVsProfile, all OI, SA and IO histograms  
 
   All D0 histograms are OK for runs 195573 
   The D0 histrograms for other runs (195571, 195572) are not present, 
   because aexp0d directory is missing for them. 
 
</pre>
<!-- Author: rakitin -->


<!-- Date: Mon Mar 28 11:25:24 2005 -->
<pre>Hi, 
 
 
&gt;&gt;      The runs 195569 and 195570 have all the D0 histograms  
&gt;&gt; empty and OI, SA and IO histograms are empty too, as well as 
&gt;&gt; SiSVXIIEffVsProfile. 
&gt;&gt; Is there any way to check if these runs have AAA_NoSilicon? 
 
 
To check if AAA_NoSilicon go to the run summary database: 
http://www-cdfonline.fnal.gov/cdfdb/servlet/RunSummary 
-&gt; 195569 and 195570 are AAA_NoSilicon 
-&gt; OK 
 
 
&gt;&gt; The runs 195571 and 195572 are not finished yet, so its hard &gt;&gt; to tell anything about their D0. 
 
 
... 
-&gt; OK 
 
 
&gt;&gt; The runs 195569 and 195570 are missing directories bhmu0d and 
&gt;&gt; gmbs0d, also the run 195570 is missing gjt10d, though these  
&gt;&gt; three datasets are claimed to be finished OK. 
 
 
For a given dataset, only the 3 runs with highest luminosity  
within the runs that have been processed in the dataset in the  
last 24h are reported 
-&gt; only 3 runs reported per dataset 
-&gt; may be not the same runs for different datasets 
-&gt; OK 
 
 
&gt;&gt;      The run 194536 was aborted. 
 
 
I will check and resubmit if needed, 
 
Thanks, 
 
	Regis 
 
</pre>
<!-- Author: rakitin -->


<!-- Date: Mon Mar 28 13:24:44 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-194&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-194  width='300' align='top'></a></td><td>DCache problesm on CNAFCAF?  
No actions taken 
</td></tr></table>
<!-- Author: rakitin -->


<!-- Date: Mon Mar 28 13:40:38 2005 -->
<pre>   Check run conditions to make sure the empty histograms present only 
in AAA_NOSILICON runs: 
 
   195569 - AAA_NOSILICON - empty D0 histograms, all OI, SA and IO histograms 
   195570 - AAA_NOSILICON - empty D0 histograms, all OI, SA and IO histograms 
   195571 - AAA_NOSILICON - empty D0 histograms, all OI, SA and IO histograms 
   195572 - AAA_NOSILICON - empty D0 histograms, all OI, SA and IO histograms 
   195573 - AAA_CURRENT   - all D0 histograms, OI, SA and IO histograms are OK 
</pre>
<!-- Author: rakitin -->


<!-- Date: Mon Mar 28 16:29:47 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-195&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-195  width='300' align='top'></a></td><td>Head node on CAF seems to have a short 
timeout... :-)</td></tr></table>
<!-- Author: rakitin -->


<!-- Date: Mon Mar 28 16:32:41 2005 -->
<pre>The 11A production seems to start working again. Mircea 
said the problem was version incompatibility between 
what was setup and the executable. This caused the  
incompatibilities in the ntuples for different runs, 
which could not be properly merged together to determine 
the beamline. 
 
End of shift - everything looks OK</pre>
<!-- Author: rakitin -->


<!-- Date: Tue Mar 29 08:36:01 2005 -->
<pre>  OFFLINE REPORT: Tue Mar 29 07:47:20 CST 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 195742  2005.03.26 03:06:03  (3 day(s), 4 hours ago) 
    Stream B: 195740  2005.03.26 03:06:03  (3 day(s), 4 hours ago) 
    Stream G: 195740  2005.03.25 16:56:07  (3 day(s), 14 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 195623  2005.03.22 11:44:13  (6 day(s), 20 hours ago) 
    Stream B: 195623  2005.03.22 11:44:13  (6 day(s), 20 hours ago) 
    Stream G: 195623  2005.03.22 11:44:13  (6 day(s), 20 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       195787         PHYSICS_3_01   1 2005.03.28   05:44:56     PROD_PHYSICS_CD 
       This is 1 day(s), 2 hours old. 
   11A is not available for: 
       195745         PHYSICS_3_01   1 2005.03.26   13:21:53     PROD_PHYSICS_CD 
       This is 2 day(s), 18 hours old. 
   17A is not available for: 
       195745         PHYSICS_3_01   1 2005.03.26   13:21:53     PROD_PHYSICS_CD 
       This is 2 day(s), 18 hours old. 
 
   The 11A calibrations are working again, now we are processing the files 
   which were not processed in the previous a few days. The writing to tape 
   is still behind, the last file processed to tape is almost 7 days old. Mircea says this is not a big deal. 
 
4) DB MONITORING 
   OK 
 
5) DH MONITORING 
   OK. MIT CAF is back up. It seems to be a slight problem with DCache 
   on CAF last night and a short power outage for the head node on CondorCAF 
   this morning. 
 
7) DQM MONITORING 
   Last update: Tue Mar 29 05:00:06 2005 
   Last cron job run: Mar 29 04:30 
   There are quite a few runs with red marks with flag N_STARTED 
 
   Histograms are OK 
   The dataset aexp0d for the runs 195571 195572 195574 has 
   empty D0 histograms, SiSVXIIEffVsProfile, all OI, SA and IO histograms 
 
   The RunSet for the runs above is AAA_NOSILICON =&gt; Empty histograms are fine 
 
   The runs 195571_aexp0d and 195574_aexp0d also have empty PEMMonPLUGZTrigs, 
   the run 195572_aexp0d has only one entry for this histogram, while other 
   histograms have quite high occupancy. 
 
</pre>
<!-- Author: rakitin -->


<!-- Date: Wed Mar 30 08:29:26 2005 -->
<pre>   OFFLINE REPORT: Wed Mar 30 07:33:25 CST 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 195780  2005.03.28 00:35:03  (2 day(s), 6 hours ago) 
    Stream B: 195780  2005.03.28 00:35:03  (2 day(s), 6 hours ago) 
    Stream G: 195780  2005.03.28 00:35:03  (2 day(s), 6 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 195746  2005.03.26 15:42:28  (3 day(s), 15 hours ago) 
    Stream B: 195746  2005.03.26 15:42:28  (3 day(s), 15 hours ago) 
    Stream G: 195623  2005.03.22 11:44:13  (7 day(s), 19 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       195804         PHYSICS_3_01   1 2005.03.29   07:53:10     PROD_PHYSICS_CD 
       This is 23 hours old. 
   11A is not available for: 
       195781         PHYSICS_3_01   1 2005.03.28   01:35:06     PROD_PHYSICS_CD 
       This is 2 day(s), 5 hours old. 
   17A is not available for: 
       195781         PHYSICS_3_01   1 2005.03.28   01:35:06     PROD_PHYSICS_CD 
       This is 2 day(s), 5 hours old. 
 
   The 11A calibrations are working now, we keep processing the files 
   which were not processed earlier. The writing to tape is still a  
   little bit behind for stream G, the last file processed to tape is  
   more than 7 days old. 
 
4) DB MONITORING 
   OK 
 
5) DH MONITORING 
   The head node on CondorCAF seems to have short periods of hybernation 
   CNAFCAF is idling (white spaces). Also, red line (Load=0) is above  
           yellow line (Load) -- DCache problems? 
   MIT CAF -- red line is above yellow line too 
 
7) DQM MONITORING 
   Last update: Wed Mar 30 05:00:07 2005 
   Last cron job run: Mar 30 04:30 
   OK, all the fields are green (finished processing OK) 
 
   Run 195623 has a flag N_ABORTED 
 
   Histograms are OK. 
 
   Most of the runs have RunSet AAA_NOSILICON =&gt; all D0 histograms, 
   SiSVXIIEffVsProfile, all OI, SA and IO histograms are be empty. 
 
   The PEMMonPLUGZTrigs is also empty or has 1 entry, while others --  
   PEMMonTrigs and PEMMonMETPEMTrig -- have quite high occupancy. 
   This behavior is pretty consistent from day to day. 
 
   </pre>
<!-- Author: rakitin -->


<!-- Date: Wed Mar 30 13:45:19 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-197&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-197  width='300' align='top'></a></td><td>San Diego CAF has problems...</td></tr></table>
<!-- Author: rakitin -->


<!-- Date: Wed Mar 30 16:10:15 2005 -->
<pre>End of shift - everything looks fine</pre>
<!-- Author: rakitin -->


<!-- Date: Thu Mar 31 08:25:51 2005 -->
<pre>   OFFLINE REPORT: Thu Mar 31 07:31:02 CST 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 195790  2005.03.28 21:53:44  (2 day(s), 9 hours ago) 
    Stream B: 195790  2005.03.28 21:53:44  (2 day(s), 9 hours ago) 
    Stream G: 195790  2005.03.28 21:53:44  (2 day(s), 9 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 195790  2005.03.28 21:53:44  (2 day(s), 9 hours ago) 
    Stream B: 195780  2005.03.28 00:35:03  (3 day(s), 6 hours ago) 
    Stream G: 195780  2005.03.28 00:35:03  (3 day(s), 6 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       195930         PHYSICS_3_01   2 2005.03.30   15:47:04     PROD_PHYSICS_CD 
       This is 15 hours old. 
   11A is not available for: 
       195791         PHYSICS_3_01   1 2005.03.28   23:46:36     PROD_PHYSICS_CD 
       This is 2 day(s), 7 hours old. 
   17A is not available for: 
       195791         PHYSICS_3_01   1 2005.03.28   23:46:36     PROD_PHYSICS_CD 
       This is 2 day(s), 7 hours old. 
 
4) DB MONITORING 
   OK 
 
5) DH MONITORING 
   Something strange is going on with MIT CAF.  
   The jobs on SDSCCAF seem to be doing nothing (red line is above yellow).  
   CondorCAF seems to begin having idle resources - keep checking on it. 
 
7) DQM MONITORING 
   Last update: Thu Mar 31 05:00:06 2005 
   Last cron job run: Mar 31 04:30 
 
   Only two runs available: 195677 - green (OK), 195651 - green with flag N_ABORTED 
 
   Both runs have RunSet AAA_NOSILICON =&gt; all D0 histograms, 
   SiSVXIIEffVsProfile, all OI, SA and IO histograms are empty. 
 
   The PEMMonPLUGZTrigs are empty too for both runs. 
 
 
   </pre>
<!-- Author: rakitin -->


<!-- Date: Thu Mar 31 16:10:04 2005 -->
<pre>End of shift - everything is fine. 
 
MIT CAF still has its problems. 
 
I wonder if herber is a "general user" connecting to cdfofprd. 
</pre>
<!-- Author: rakitin -->


<!-- Date: Sat Apr 2 08:54:23 2005 -->
<pre> 
OFFLINE REPORT: Sat Apr 2 07:55:41 CST 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 195805  2005.03.29 09:34:27  (3 day(s), 22 hours ago) 
    Stream B: 195805  2005.03.29 09:34:27  (3 day(s), 22 hours ago) 
    Stream G: 195805  2005.03.29 09:34:27  (3 day(s), 22 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 195805  2005.03.29 09:34:27  (3 day(s), 22 hours ago) 
    Stream B: 195805  2005.03.29 09:34:27  (3 day(s), 22 hours ago) 
    Stream G: 195805  2005.03.29 09:34:27  (3 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196045         PHYSICS_3_01   2 2005.04.01   05:57:01     PROD_PHYSICS_CD 
       This is 1 day(s), 1 hours old. 
   11A is not available for: 
       195809         PHYSICS_3_01   2 2005.03.29   23:07:06     PROD_PHYSICS_CD 
       This is 3 day(s), 8 hours old. 
   17A is not available for: 
       195809         PHYSICS_3_01   2 2005.03.29   23:07:06     PROD_PHYSICS_CD 
       This is 3 day(s), 8 hours old. 
 
There is no significant progress in offline production since April 1, 2005 
because unavailability of the 11A and 17A calibrations 
 
 
4) DB MONITORING 
   O'K 
 
5) DH MONITORING 
   Starting 4am April 2, 2005 CondorCAF shows significant idle resources (50%) 
 
7) DQM MONITORING 
   Last update: Sat Apr 2 05:00:06 2005 
   Last cron job run: Apr 2 04:30 
 
   12 runs. 8 - green, 3 -blue, 1 - white, 4 - N_ABORTED 
 
   195739_aexp0d_psfiles - no Silicon 
 
</pre>
<!-- Author: Sergey Klimenko (offline operator) -->


<!-- Date: Sat Apr 2 09:27:47 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-198&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-198  width='300' align='top'></a></td><td>Idle CondorCAF starting 4am April 2, 2005</td></tr></table>
<!-- Author: Sergey Klimenko -->


<!-- Date: Sat Apr 2 15:36:30 2005 -->
<pre>I can see that there is some improvement with calibration, 
but still there is no any progress with the data processing. 
Below I put recent report and one can see both CPU and TAPE 
processing stack at run  195805. Mircea Coca is notified. 
 
 
  OFFLINE REPORT: Sat Apr 2 15:29:40 CST 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 195805  2005.03.29 09:34:27  (4 day(s), 5 hours ago) 
    Stream B: 195805  2005.03.29 09:34:27  (4 day(s), 5 hours ago) 
    Stream G: 195805  2005.03.29 09:34:27  (4 day(s), 5 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 195805  2005.03.29 09:34:27  (4 day(s), 5 hours ago) 
    Stream B: 195805  2005.03.29 09:34:27  (4 day(s), 5 hours ago) 
    Stream G: 195805  2005.03.29 09:34:27  (4 day(s), 5 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196063    PHYSICS_TEST_3_02   3 2005.04.01   19:33:35     PROD_PHYSICS_CD 
       This is 19 hours old. 
   11A is not available for: 
       196045         PHYSICS_3_01   2 2005.04.01   05:57:01     PROD_PHYSICS_CD 
       This is 1 day(s), 9 hours old. 
   17A is not available for: 
       196045         PHYSICS_3_01   2 2005.04.01   05:57:01     PROD_PHYSICS_CD 
       This is 1 day(s), 9 hours old. 
</pre>
<!-- Author: Sergey Klimenko (offline operator) -->


<!-- Date: Sun Apr 3 08:39:38 2005 -->
<pre>   OFFLINE REPORT: Sun Apr 3 08:07:56 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196044  2005.04.01 05:04:38  (2 day(s), 2 hours ago) 
    Stream B: 196044  2005.04.01 05:04:38  (2 day(s), 2 hours ago) 
    Stream G: 196044  2005.04.01 05:04:38  (2 day(s), 2 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 195805  2005.03.29 09:34:27  (4 day(s), 21 hours ago) 
    Stream B: 195805  2005.03.29 09:34:27  (4 day(s), 21 hours ago) 
    Stream G: 195805  2005.03.29 09:34:27  (4 day(s), 21 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196085         PHYSICS_3_01   2 2005.04.02   06:02:15     PROD_PHYSICS_CD 
       This is 1 day(s), 1 hours old. 
   11A is not available for: 
       196045         PHYSICS_3_01   2 2005.04.01   05:57:01     PROD_PHYSICS_CD 
       This is 2 day(s), 1 hours old. 
   17A is not available for: 
       196045         PHYSICS_3_01   2 2005.04.01   05:57:01     PROD_PHYSICS_CD 
       This is 2 day(s), 1 hours old. 
 
    BeamExe job status reports one suspicious run 
    196045      2005/04/02 07:11:31      2454340         10A     BeamExe_1_gphysr_196045_196066 
                2005/04/02 14:33:07    30(30) / 30    237085(237085) / 236756    0 
    Mircea Coca is notified 
 
4) DB MONITORING 
   O'K 
5) DH MONITORING 
   O'K 
 
   Idle resorces on CondorCAF 
 
7) DQM MONITORING 
   Last update: Sun Apr 3 05:00:06 2005 
   Last cron job run: Apr 3 04:30 
 
   4 runs: 2 - green , 2 - white, 2 - N_ABORTED 
 
   All histograms are O'K</pre>
<!-- Author: Sergey Klimenko (offline operator) -->


<!-- Date: Sun Apr 3 15:57:35 2005 -->
<pre>Shift summary. 
 
1) The Beamline production reported two problem runs: 
 
 196045  	 2005/04/02 07:11:31  	 2454340  	 10A  	 BeamExe_1_gphysr_196045_196066  	 2005/04/02 14:33:07  	 30(30) / 30  	 237085(237085) / 236756  	 
 
and 
 
 196079  	 2005/04/02 06:11:47  	 27741  	 10A  	 BeamExe_1_gphysr_196067_196082  	 2005/04/03 11:33:05  	 1(1) / 1  	 5418(5418) / 5378 
 
Mircea Coca has been notified 
 
2) CondorCAF had a lot of idle nodes (white spots) 
 
Mircea Coca has been notified 
 
3) There was no any progres on RPOCESSING TO TAPE for the last  two days 
 
 2) PROCESSING (TO TAPE): 
    Stream A: 195805  2005.03.29 09:34:27  (5 day(s), 5 hours ago) 
    Stream B: 195805  2005.03.29 09:34:27  (5 day(s), 5 hours ago) 
    Stream G: 195805  2005.03.29 09:34:27  (5 day(s), 5 hours ago) 
 
Lets wait until tomorrow. 
 	 
 
</pre>
<!-- Author: Sergey Klimenko (offline operator) -->


<!-- Date: Sun Apr 3 16:02:24 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-199&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-199  width='300' align='top'></a></td><td>Idle nodes on CondorCAF</td></tr></table>
<!-- Author: Sergey Klimenko -->


<!-- Date: Mon Apr 4 08:27:26 2005 -->
<pre>   OFFLINE REPORT: Mon Apr 4 07:40:10 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196044  2005.04.01 05:04:38  (3 day(s), 1 hours ago) 
    Stream B: 196044  2005.04.01 05:04:38  (3 day(s), 1 hours ago) 
    Stream G: 196044  2005.04.01 05:04:38  (3 day(s), 1 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196044  2005.04.01 05:04:38  (3 day(s), 1 hours ago) 
    Stream B: 196044  2005.04.01 05:04:38  (3 day(s), 1 hours ago) 
    Stream G: 196044  2005.04.01 05:04:38  (3 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196100         PHYSICS_3_01   2 2005.04.03   17:19:39     PROD_PHYSICS_CD 
       This is 14 hours old. 
   11A is not available for: 
       196045         PHYSICS_3_01   2 2005.04.01   05:57:01     PROD_PHYSICS_CD 
       This is 3 day(s), 0 hours old. 
   17A is not available for: 
       196045         PHYSICS_3_01   2 2005.04.01   05:57:01     PROD_PHYSICS_CD 
       This is 3 day(s), 0 hours old. 
 
    Beamline Production prduced large number of files for 
    196099      2005/04/04 03:11:33      3714899         10A     BeamExe_1_gphysr_196099_196099 
    2005/04/04 04:33:11    20(20) / 51    140367(140367) / 372238 
 
Mircea Coca is notified 
 
4) DB MONITORING 
   O'K 
 
5) DH MONITORING 
   CAF - 30% of idle nodes (white spots) between 20pm-24pm April 3,2005 
   CondorCAF - increaseng number of idle nodes since 5am April 4 
   SDSCCAF - most of the night red line is above the yellow line - Dcache problem 
 
Mircea Coca is notified 
 
7) DQM MONITORING 
Last update: Mon Apr 4 05:00:05 2005 
Last cron job run: Apr 4 04:30 
 
DQM status: 11 runs: 7 - green (O'K), 4 - blue (submitted not finished) 
                     6 N_ABORTED 
</pre>
<!-- Author: Sergey Klimenko (offline operator) -->


<!-- Date: Mon Apr 4 08:36:13 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-202&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-202  width='300' align='top'></a></td><td>SDSCCAF report</td></tr></table>
<!-- Author: Sergey Klimenko -->


<!-- Date: Mon Apr 4 08:37:07 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-203&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-203  width='300' align='top'></a></td><td>CondorCAF</td></tr></table>
<!-- Author: Sergey Klimenko -->


<!-- Date: Mon Apr 4 08:37:35 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-204&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-204  width='300' align='top'></a></td><td>CAF</td></tr></table>
<!-- Author: Sergey Klimenko -->


<!-- Date: Tue Apr 5 08:40:58 2005 -->
<pre>   OFFLINE REPORT: Tue Apr 5 07:50:50 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196044  2005.04.01 05:04:38  (4 day(s), 1 hours ago) 
    Stream B: 196044  2005.04.01 05:04:38  (4 day(s), 1 hours ago) 
    Stream G: 196044  2005.04.01 05:04:38  (4 day(s), 1 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196044  2005.04.01 05:04:38  (4 day(s), 1 hours ago) 
    Stream B: 196044  2005.04.01 05:04:38  (4 day(s), 1 hours ago) 
    Stream G: 196044  2005.04.01 05:04:38  (4 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196152 PHYSICS_HIGHLUM_3_01   1 2005.04.04   11:04:16     PROD_PHYSICS_CD 
       This is 20 hours old. 
   11A is not available for: 
       196045         PHYSICS_3_01   2 2005.04.01   05:57:01     PROD_PHYSICS_CD 
       This is 4 day(s), 0 hours old. 
   17A is not available for: 
       196045         PHYSICS_3_01   2 2005.04.01   05:57:01     PROD_PHYSICS_CD 
       This is 4 day(s), 0 hours old. 
 
   11A and 17A calibration delay, experts are notified 
 
4) DB MONITORING 
   O'K 
 
5) DH MONITORING 
   O'K 
 
7) DQM MONITORING 
Last update: Tue Apr 5 05:00:06 2005 
Last cron job run: Apr 5 04:30 
 
7 runs: 2 -green, 2 - blue, 3 - white, 5 - N_ABortery 
</pre>
<!-- Author: Sergey Klimenko -->


<!-- Date: Tue Apr 5 12:57:30 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-205&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-205  width='300' align='top'></a></td><td>possible problem with CAF 
Mircea is notified</td></tr></table>
<!-- Author: Sergey Klimenko -->


<!-- Date: Tue Apr 5 14:11:21 2005 -->
<pre>Beamline production reported bad run 196100: 
 196100  	 2005/04/04 18:11:47  	 1653624  	 10A  	 BeamExe_1_gphysr_196100_196152  	 2005/04/04 19:33:23  	 17(17) / 21  	 129837(129837) / 162746  	 0  	 </pre>
<!-- Author: Sergey Klimenko -->


<!-- Date: Wed Apr 6 08:42:26 2005 -->
<pre>   OFFLINE REPORT: Wed Apr 6 07:45:17 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196099  2005.04.03 01:09:29  (3 day(s), 5 hours ago) 
    Stream B: 196099  2005.04.03 01:09:29  (3 day(s), 5 hours ago) 
    Stream G: 196099  2005.04.03 01:09:29  (3 day(s), 5 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196044  2005.04.01 05:04:38  (5 day(s), 1 hours ago) 
    Stream B: 196044  2005.04.01 05:04:38  (5 day(s), 1 hours ago) 
    Stream G: 196044  2005.04.01 05:04:38  (5 day(s), 1 hours ago) 
 
(There was a problem with 11A/17A calibrations availability a day before, 
which held the data processing. Since yesterday (April 5th), the data processing 
is catching up) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196169         PHYSICS_3_01   2 2005.04.05   07:54:20     PROD_PHYSICS_CD 
       This is 23 hours old. 
   11A is not available for: 
       196100         PHYSICS_3_01   2 2005.04.03   17:19:39     PROD_PHYSICS_CD 
       This is 2 day(s), 14 hours old. 
   17A is not available for: 
       196100         PHYSICS_3_01   2 2005.04.03   17:19:39     PROD_PHYSICS_CD 
       This is 2 day(s), 14 hours old. 
 
4) DB MONITORING 
   O'K 
 
5) DH MONITORING 
 
   Head node or batch system problem on CAF at 4 am in the morning 
 
7) DQM MONITORING 
 
   Last update: Wed Apr 6 05:00:07 2005 
   Last cron job run: Apr 6 04:30 
 
RUN\DATASET status 
195930       blue 
195814       green 
195809       green  N_ABORTED 
195805       green  N_ABORTED 
195804       green  N_ABORTED 
</pre>
<!-- Author: Sergey Klimenko (offline operator) -->


<!-- Date: Wed Apr 6 08:44:37 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-206&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-206  width='300' align='top'></a></td><td>CAF glitch at 4 am</td></tr></table>
<!-- Author: Sergey Klimenko -->


<!-- Date: Thu Apr 7 08:30:57 2005 -->
<pre>  OFFLINE REPORT: Thu Apr 7 07:48:18 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196153  2005.04.04 13:02:05  (2 day(s), 18 hours ago) 
    Stream B: 196153  2005.04.04 13:02:05  (2 day(s), 18 hours ago) 
    Stream G: 196153  2005.04.04 13:02:05  (2 day(s), 18 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196100  2005.04.03 17:19:39  (3 day(s), 14 hours ago) 
    Stream B: 196099  2005.04.03 01:09:29  (4 day(s), 5 hours ago) 
    Stream G: 196044  2005.04.01 05:04:38  (6 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196228         PHYSICS_3_01   2 2005.04.06   07:07:01     PROD_PHYSICS_CD 
       This is 1 day(s), 0 hours old. 
   11A is not available for: 
       196162         PHYSICS_3_01   2 2005.04.04   18:54:09     PROD_PHYSICS_CD 
       This is 2 day(s), 12 hours old. 
   17A is not available for: 
       196162         PHYSICS_3_01   2 2005.04.04   18:54:09     PROD_PHYSICS_CD 
       This is 2 day(s), 12 hours old. 
 
Beam line production reported one bad run: 
 196170      2005/04/06 09:11:42   1254877   10A  BeamExe_1_gphysr_196169_196183 
             2005/04/06 19:33:21    12(12) / 15    95353(95353) / 119286    0 
 
4) DB MONITORING 
   O'K 
 
5) DH MONITORING 
 
   Periodic (once per 6 hours) glitches in CAF 
 
7) DQM MONITORING 
   Last update: Thu Apr 7 05:00:06 2005 
   Last cron job run: Apr 7 04:30 
 
   RUN\DATASET    status 
     196042        green 
     196041        green 
     195930        green 
 
     196041 shows empty histograms, but it's a very short run (40nb-1), and silicon was off 
 
</pre>
<!-- Author: Sergey Klimenko (offline operator) -->


<!-- Date: Thu Apr 7 11:46:34 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-207&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-207  width='300' align='top'></a></td><td>MIRCAF problem: too many timeout jobs by user bello</td></tr></table>
<!-- Author: Sergey Klimenko -->


<!-- Date: Thu Apr 7 11:47:52 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-208&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-208  width='300' align='top'></a></td><td>SDSCCAF: too many timeout jobs by user bello. bello is notified</td></tr></table>
<!-- Author: Sergey Klimenko -->


<!-- Date: Fri Apr 8 09:09:52 2005 -->
<pre>  OFFLINE REPORT: Fri Apr 8 07:52:50 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196153  2005.04.04 13:02:05  (3 day(s), 18 hours ago) 
    Stream B: 196153  2005.04.04 13:02:05  (3 day(s), 18 hours ago) 
    Stream G: 196153  2005.04.04 13:02:05  (3 day(s), 18 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196100  2005.04.03 17:19:39  (4 day(s), 14 hours ago) 
    Stream B: 196099  2005.04.03 01:09:29  (5 day(s), 5 hours ago) 
    Stream G: 196044  2005.04.01 05:04:38  (7 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196275         PHYSICS_3_01   2 2005.04.07   06:38:02     PROD_PHYSICS_CD 
       This is 1 day(s), 1 hours old. 
   11A is not available for: 
       196162         PHYSICS_3_01   2 2005.04.04   18:54:09     PROD_PHYSICS_CD 
       This is 3 day(s), 12 hours old. 
   17A is not available for: 
       196162         PHYSICS_3_01   2 2005.04.04   18:54:09     PROD_PHYSICS_CD 
       This is 3 day(s), 12 hours old. 
 
Beamline production reported two bad runs: 
196228       2005/04/07 08:40:09         1718366         10A     BeamExe_1_gphysr_196228_196228 
2005/04/06 19:33:49    29(29) / 34    193651(193651) / 226975    2005/04/07 07:13:26    0 
 
196170      2005/04/06 09:11:40        1254877        10A         BeamExe_1_gphysr_196169_196183 
2005/04/06 19:33:21    12(12) / 15    95353(95353) / 119286    2005/04/06 16:53:31    0 
 
4) DB MONITORING 
 
   user litvise has around 100 inactive connections 13,000 sec each 
   user and experts are notified 
 
5) DH MONITORING 
 
   starting Fri Apr  8 01:14:20 CDT 2005 DCache monitor reports a problem 
   with DCache 
Fri Apr  8 02:07:01 CDT 2005 Command failed! 
Server error message for [1]: "java.lang.NullPointerException" (errno 2). 
Failed open file in the dCache. 
dc_stage fail : "java.lang.NullPointerException" 
System error: Input/output error 
Command exited with non-zero status 255 
0:22.57 
Fri Apr  8 02:17:23 CDT 2005 Command failed! 
Server error message for [1]: "java.lang.NullPointerException" (errno 2). 
Failed open file in the dCache. 
dc_stage fail : "java.lang.NullPointerException" 
System error: Input/output error 
Command exited with non-zero status 255 
0:30.31 
Fri Apr  8 02:27:54 CDT 2005 Command failed! 
Server error message for [1]: "internalError : User cdfsoft/cdf/cdfpca.fnal.gov@FNAL.GOV is not authorized" (errno 2). 
Failed open file in the dCache. 
dc_stage fail : "internalError : User cdfsoft/cdf/cdfpca.fnal.gov@FNAL.GOV is not authorized" 
System error: Input/output error 
Command exited with non-zero status 255 
2:00:02 
Fri Apr  8 04:37:57 CDT 2005 Command failed! 
Server error message for [1]: "internalError : User cdfsoft/cdf/cdfpca.fnal.gov@FNAL.GOV is not authorized" (errno 2). 
Failed open file in the dCache. 
dc_stage fail : "internalError : User cdfsoft/cdf/cdfpca.fnal.gov@FNAL.GOV is not authorized" 
System error: Input/output error 
Command exited with non-zero status 255 
2:00:01 
Fri Apr  8 06:47:59 CDT 2005 Command failed! 
Server error message for [1]: "internalError : User cdfsoft/cdf/cdfpca.fnal.gov@FNAL.GOV is not authorized" (errno 2). 
Failed open file in the dCache. 
dc_stage fail : "internalError : User cdfsoft/cdf/cdfpca.fnal.gov@FNAL.GOV is not authorized" 
System error: Input/output error 
Command exited with non-zero status 255 
2:00:01 
 
 
   Becouse of Dcache problem the CAF and CondorCAF statistics is also 
   affected 
 
Mircea Coca is notified 
 
 
7) DQM MONITORING 
 
Last update: Fri Apr 8 05:00:06 2005 
Last cron job run: Apr 8 04:30 
 
Due to DCache problem 
No runs were reported 
No DQM statistics was reported 
 
</pre>
<!-- Author: silvia amerio (offline operator) -->


<!-- Date: Fri Apr 8 09:19:58 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-209&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-209  width='300' align='top'></a></td><td>DCache problem on CAF</td></tr></table>
<!-- Author: Silvia Amerio -->


<!-- Date: Fri Apr 8 09:23:24 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-210&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-210  width='300' align='top'></a></td><td>DCache problem on Condor Caf</td></tr></table>
<!-- Author: Silvia Amerio -->


<!-- Date: Sat Apr 9 10:08:20 2005 -->
<pre>  OFFLINE REPORT: Sat Apr 9 08:01:20 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196183  2005.04.05 14:27:05  (3 day(s), 17 hours ago) 
    Stream B: 196183  2005.04.05 14:27:05  (3 day(s), 17 hours ago) 
    Stream G: 196183  2005.04.05 14:27:05  (3 day(s), 17 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196153  2005.04.04 13:02:05  (4 day(s), 18 hours ago) 
    Stream B: 196153  2005.04.04 13:02:05  (4 day(s), 18 hours ago) 
    Stream G: 196153  2005.04.04 13:02:05  (4 day(s), 18 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196276         PHYSICS_3_01   2 2005.04.07   14:07:34     PROD_PHYSICS_CD 
       This is 1 day(s), 17 hours old. 
   11A is not available for: 
       196228         PHYSICS_3_01   2 2005.04.06   07:07:01     PROD_PHYSICS_CD 
       This is 3 day(s), 0 hours old. 
   17A is not available for: 
       196228         PHYSICS_3_01   2 2005.04.06   07:07:01    PROD_PHYSICS_CD 
       This is 3 day(s), 0 hours old. 
 
4) DB MONITORING 
   Ok 
5) DH MONITORING 
   Everything ok except that the following tape write errors were found: 
 
   2005-Apr-09 02:20:47 failed to catalog fileset CB6149.0 
   2005-Apr-09 02:23:18 failed to catalog fileset CB6149.1 
   2005-Apr-09 02:25:50 failed to catalog fileset CB6149.2 
   2005-Apr-09 02:28:21 failed to catalog fileset CB6149.3 
   2005-Apr-09 02:33:22 failed to catalog fileset CB6148.0 
   2005-Apr-09 02:35:55 failed to catalog fileset CB6148.1 
   2005-Apr-09 02:38:28 failed to catalog fileset CB6148.2 
   2005-Apr-09 02:40:14 failed to catalog fileset CB6148.3 
   2005-Apr-09 02:40:16 failed to catalog fileset CB6148.4 
 
   Mircea was notified. 
 
7) DQM MONITORING 
   Last update: Sat Apr 9 05:00:06 2005 
   Last cron job run: Apr 9 04:30 
 
   RUN/DATASET  status 
    196067      aexp0d, bhmu0d, gjt10d,gmbs0d finished and ok, the others not out of production 
    196066      aexp0d, bhmu0d, gjt10d,gmbs0d finished and ok, the others not out of production 
    196064      aexp0d, bhmu0d, gjt10d,gmbs0d finished and ok, the others not out of production 
    196045      aexp0d  submitted but not finished, bhmu0d, gjt10d,gmbs0d finished and ok, the others not out of production 
    196044      aexp0d, bhmu0d, gjt10d,gmbs0d finished and ok, the others not out of production 
    196043      aexp0d  submitted but not finished, bhmu0d, gjt10d,gmbs0d finished but processed online luminosity is below 90%,the others not out of production 
 
 
    Run 196044 : empty histograms in SiCt.ps but SVX,ISL,L00 and SVT GoodRun bits are 0. Also PEMMonPLUGZTrigs is empty. 
    Run 196064 : PEMMonMETPEMTrigs and PEMMonPLUGZTrigs are empty. 
    Run 196066 : empty histograms in D0.ps and SiCt.ps (CMP and   SVX GoodRun bits = 0).Also PEMMonMETPEMTrigs and PEMMonPLUGZTrigs are empty. 
 
</pre>
<!-- Author: Silvia Amerio -->


<!-- Date: Sat Apr 9 15:48:35 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-213&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-213  width='300' align='top'></a></td><td>CAF status</td></tr></table>
<!-- Author: Silvia Amerio -->


<!-- Date: Sat Apr 9 15:49:41 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-214&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-214  width='300' align='top'></a></td><td>MITCAF status</td></tr></table>
<!-- Author: Silvia Amerio -->


<!-- Date: Sun Apr 10 09:25:12 2005 -->
<pre>  OFFLINE REPORT: Sun Apr 10 08:04:05 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196275  2005.04.07 06:38:02  (3 day(s), 1 hours ago) 
    Stream B: 196275  2005.04.07 06:38:02  (3 day(s), 1 hours ago) 
    Stream G: 196275  2005.04.07 06:38:02  (3 day(s), 1 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196153  2005.04.04 13:02:05  (5 day(s), 19 hours ago) 
    Stream B: 196153  2005.04.04 13:02:05  (5 day(s), 19 hours ago) 
    Stream G: 196153  2005.04.04 13:02:05  (5 day(s), 19 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196344         PHYSICS_3_01   2 2005.04.09   05:44:16     PROD_PHYSICS_CD 
       This is 1 day(s), 2 hours old. 
   11A is not available for: 
       196276         PHYSICS_3_01   2 2005.04.07   14:07:34     PROD_PHYSICS_CD 
       This is 2 day(s), 17 hours old. 
   17A is not available for: 
       196276         PHYSICS_3_01   2 2005.04.07   14:07:34     PROD_PHYSICS_CD 
       This is 2 day(s), 17 hours old. 
 
4) DB MONITORING 
   OK. 
 
5) DH MONITORING 
   CAF glitches at 00:00, 2 and 4 am. 
   CONDORCAF has idle resources starting from 3 am. 
   CNAFCAF plot shows idle resources and red line (load = 0) is     above yellow one (load) since 2 pm (8 am Fnal time) 
7) DQM MONITORING 
   Last update: Sun Apr 10 05:00:06 2005 
   Last cron job run: Apr 10 04:30 
 
   RUN/DATASET 
    196085   aexp0d submitted but not finished,   bhmu0d, gjt10d, 
             gmbs0d finished but with processed online luminosity 
             below 90%. 
 
    196082   aexp0d red and aborted,   bhmu0d, gjt10d,gmbs0d finished but with processed online luminosity below 90% 
 
    196079   aexp0d, bhmu0d, gjt10d,gmbs0d finished but with processed online luminosity below 90% 
 
    196077   aexp0d, bhmu0d, gjt10d,gmbs0d finished 
 
    196045   aexp0d finished 
 
    196043   aexp0d finished but with processed online luminosity 
             below 90% 
 
 
    Run 196045 : PEMMonPLUGZTrigs histogram empty 
    Run 196077 : PEMMonPLUGZTrigs histogram empty 
    Run 196079 : RunSet: AAA_NOSILICON --&gt; all D0 histograms, SiSVXIIEffProfile, all OI, SA and IO histograms are empty; PEM MonTrigs and PEMMonPLUGZTrigs  are empty too. 
 
 
</pre>
<!-- Author: Silvia Amerio -->


<!-- Date: Sun Apr 10 09:29:18 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-215&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-215  width='300' align='top'></a></td><td>CAF status</td></tr></table>
<!-- Author: Silvia Amerio -->


<!-- Date: Sun Apr 10 09:30:09 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-216&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-216  width='300' align='top'></a></td><td>CONDORCAF status</td></tr></table>
<!-- Author: Silvia Amerio -->


<!-- Date: Sun Apr 10 09:30:46 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-217&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-217  width='300' align='top'></a></td><td>CNAFCAF status</td></tr></table>
<!-- Author: Silvia Amerio -->


<!-- Date: Sun Apr 10 14:27:24 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-218&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-218  width='300' align='top'></a></td><td> Large CPU usage by cdfdqm between 12:00 and 14:00; Susana and Petar were notified </td></tr></table>
<!-- Author: Silvia Amerio -->


<!-- Date: Mon Apr 11 08:19:56 2005 -->
<pre>fcdfora4 went down this morning at 2:47. 
FBS, SAM farms are shut. 
</pre>
<!-- Author: Silvia Amerio -->


<!-- Date: Mon Apr 11 09:03:15 2005 -->
<pre>Since the db machine is down, Dehong just stopped the raw data logging. </pre>
<!-- Author: Silvia -->


<!-- Date: Mon Apr 11 14:23:58 2005 -->
<pre>All work on fcdfora4 has been completed. More memory DIMMs have been replaced. fcdfora4 is up and all services are running.  
 
Dehong re-activated the raw data logging. 
</pre>
<!-- Author: Silvia -->


<!-- Date: Mon Apr 11 14:57:08 2005 -->
<pre>  OFFLINE REPORT: Mon Apr 11 14:30:19 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196275  2005.04.07 06:38:02  (4 day(s), 7 hours ago) 
    Stream B: 196275  2005.04.07 06:38:02  (4 day(s), 7 hours ago) 
    Stream G: 196275  2005.04.07 06:38:02  (4 day(s), 7 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196232  2005.04.06 20:04:40  (4 day(s), 18 hours ago) 
    Stream B: 196229  2005.04.06 13:34:13  (5 day(s), 0 hours ago) 
    Stream G: 196153  2005.04.04 13:02:05  (7 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196344         PHYSICS_3_01   2 2005.04.09   05:44:16     PROD_PHYSICS_CD 
       This is 2 day(s), 8 hours old. 
   11A is not available for: 
       196276         PHYSICS_3_01   2 2005.04.07   14:07:34     PROD_PHYSICS_CD 
       This is 4 day(s), 0 hours old. 
   17A is not available for: 
       196276         PHYSICS_3_01   2 2005.04.07   14:07:34     PROD_PHYSICS_CD 
       This is 4 day(s), 0 hours old. 
 
  NB: fcdfora4 went down at 2:47 am and after memory DIMMs replacements was finally up and working at 1 pm. 
 
  As a consequence, there are delays in calibrations, run processing and processing to tape. 
 
 
4) DB MONITORING 
   OK. 
 
5) DH MONITORING 
   OK. 
 
7) DQM MONITORING 
   Last update: Mon Apr 11 05:00:07 2005 
   Last cron job run: Apr 11 04:30 
 
   RUN/DATASET 
    196099 : submitted but not completed 
    196085 : aexp0d finished but with processed online luminosity below 90% 
</pre>
<!-- Author: Silvia -->


<!-- Date: Mon Apr 11 16:09:50 2005 -->
<pre>Experts are working on calibrations delays. 
 
 </pre>
<!-- Author: Silvia -->


<!-- Date: Tue Apr 12 08:35:48 2005 -->
<pre> OFFLINE REPORT: Tue Apr 12 07:34:42 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196275  2005.04.07 06:38:02  (5 day(s), 0 hours ago) 
    Stream B: 196275  2005.04.07 06:38:02  (5 day(s), 0 hours ago) 
    Stream G: 196275  2005.04.07 06:38:02  (5 day(s), 0 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196232  2005.04.06 20:04:40  (5 day(s), 11 hours ago) 
    Stream B: 196229  2005.04.06 13:34:13  (5 day(s), 18 hours ago) 
    Stream G: 196153  2005.04.04 13:02:05  (7 day(s), 18 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196344         PHYSICS_3_01   2 2005.04.09   05:44:16     PROD_PHYSICS_CD 
       This is 3 day(s), 1 hours old. 
   11A is not available for: 
       196276         PHYSICS_3_01   2 2005.04.07   14:07:34     PROD_PHYSICS_CD 
       This is 4 day(s), 17 hours old. 
   17A is not available for: 
       196276         PHYSICS_3_01   2 2005.04.07   14:07:34     PROD_PHYSICS_CD 
       This is 4 day(s), 17 hours old. 
 
Delays in calibrations: experts contacted. 
 
4) DB MONITORING 
   OK. 
 
5) DH MONITORING 
   OK. 
 
7) DQM MONITORING 
   Last update: Tue Apr 12 05:00:07 2005 
   Last cron job run: Apr 12 04:30 
 
   RUN/DATASET 
    196162  finished and ok 
    196153  aexp0d and gjt10d submitted but not finished, the others finished and ok 
    196152  aexp0d could not be started, the others finished and ok 
    196100  all finished an ok, but with processed online luminosity below 90% 
    196099  all finished an ok, but with processed online luminosity below 90% 
</pre>
<!-- Author: Silvia -->


<!-- Date: Wed Apr 13 11:17:32 2005 -->
<pre> 
 
   OFFLINE REPORT: Wed Apr 13 07:47:00 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196275  2005.04.07 06:38:02  (6 day(s), 1 hours ago) 
    Stream B: 196275  2005.04.07 06:38:02  (6 day(s), 1 hours ago) 
    Stream G: 196275  2005.04.07 06:38:02  (6 day(s), 1 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196232  2005.04.06 20:04:40  (6 day(s), 11 hours ago) 
    Stream B: 196229  2005.04.06 13:34:13  (6 day(s), 18 hours ago) 
    Stream G: 196153  2005.04.04 13:02:05  (8 day(s), 18 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196344         PHYSICS_3_01   2 2005.04.09   05:44:16     PROD_PHYSICS_CD 
       This is 4 day(s), 2 hours old. 
   11A is not available for: 
       196276         PHYSICS_3_01   2 2005.04.07   14:07:34     PROD_PHYSICS_CD 
       This is 5 day(s), 17 hours old. 
   17A is not available for: 
       196276         PHYSICS_3_01   2 2005.04.07   14:07:34     PROD_PHYSICS_CD 
       This is 5 day(s), 17 hours old. 
 
 Delays in calibrations: since run 196352 PROD_PHYSICS_SVX was missing the 
 SISTRIPMASK table. Jason added by hand the SISTRIPMASK table to all the runs 
 which missed it. We expect to recover soon the delays. 
 
4) DB MONITORING 
   One user has a CPU usage greater than 500 cpu-seconds for more 
   than 1 hour --&gt; notified. 
 
5) DH MONITORING 
   OK. 
 
7) DQM MONITORING 
   Last update: Wed Apr 13 05:00:08 2005 
   Last cron job run: Apr 13 04:30 
 
   RUN/DATASET 
   196183  finished but with processed online lunimosity below 90% 
   196170  finished but with processed online lunimosity below 90% 
   196169  finished and ok 
   196163  aexp0d submitted but not finished, the others finished 
           but with processed online lunimosity below 90% 
   196153  aexp0d and gjt10d finished and ok 
   196152  gjt10d finished and ok 
 
   Histograms: ok. 
</pre>
<!-- Author: Silvia -->


<!-- Date: Wed Apr 13 15:58:33 2005 -->
<pre>10A calibration delay is becoming smaller: 
 
10A is not available for: 
       196370         PHYSICS_3_01   2 2005.04.10   22:56:39     PROD_PHYSICS_CD 
       This is 2 day(s), 16 hours old. 
 
 
 
</pre>
<!-- Author: Silvia -->


<!-- Date: Thu Apr 14 08:41:52 2005 -->
<pre>  OFFLINE REPORT: Thu Apr 14 07:39:32 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196275  2005.04.07 06:38:02  (7 day(s), 1 hours ago) 
    Stream B: 196275  2005.04.07 06:38:02  (7 day(s), 1 hours ago) 
    Stream G: 196275  2005.04.07 06:38:02  (7 day(s), 1 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196232  2005.04.06 20:04:40  (7 day(s), 11 hours ago) 
    Stream B: 196229  2005.04.06 13:34:13  (7 day(s), 18 hours ago) 
    Stream G: 196153  2005.04.04 13:02:05  (9 day(s), 18 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196370         PHYSICS_3_01   2 2005.04.10   22:56:39     PROD_PHYSICS_CD 
       This is 3 day(s), 8 hours old. 
   11A is not available for: 
       196276         PHYSICS_3_01   2 2005.04.07   14:07:34     PROD_PHYSICS_CD 
       This is 6 day(s), 17 hours old. 
   17A is not available for: 
       196276         PHYSICS_3_01   2 2005.04.07   14:07:34     PROD_PHYSICS_CD 
       This is 6 day(s), 17 hours old. 
 
   Experts are working on calibrations delays. 
 
4) DB MONITORING 
   OK. 
 
5) DH MONITORING 
   OK. 
 
7) DQM MONITORING 
   Last update: Thu Apr 14 05:00:07 2005 
   Last cron job run: Apr 14 04:30 
 
   RUN/DATASET 
    196163 aexp0d finished but with processed online luminosity below 90% 
 
   Histograms: ok. 
</pre>
<!-- Author: Silvia -->


<!-- Date: Thu Apr 14 12:37:49 2005 -->
<pre>10A calibration is up to date now.</pre>
<!-- Author: Silvia -->


<!-- Date: Fri Apr 15 09:16:06 2005 -->
<pre>  OFFLINE REPORT: Fri Apr 15 08:40:11 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196369  2005.04.10 19:18:02  (4 day(s), 13 hours ago) 
    Stream B: 196369  2005.04.10 19:18:02  (4 day(s), 13 hours ago) 
    Stream G: 196369  2005.04.10 19:18:02  (4 day(s), 13 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196232  2005.04.06 20:04:40  (8 day(s), 12 hours ago) 
    Stream B: 196229  2005.04.06 13:34:13  (8 day(s), 19 hours ago) 
    Stream G: 196153  2005.04.04 13:02:05  (10 day(s), 19 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196471 PHYSICS_HIGHLUM_3_02   1 2005.04.14   11:16:15     PROD_PHYSICS_CD 
       This is 21 hours old. 
   11A is not available for: 
       196370         PHYSICS_3_01   2 2005.04.10   22:56:39     PROD_PHYSICS_CD 
       This is 4 day(s), 9 hours old. 
   17A is not available for: 
       196370         PHYSICS_3_01   2 2005.04.10   22:56:39     PROD_PHYSICS_CD 
       This is 4 day(s), 9 hours old. 
 
4) DB MONITORING 
   OK. 
 
5) DH MONITORING 
   OK. 
 
7) DQM MONITORING 
   Last update: Fri Apr 15 05:00:09 2005 
   Last cron job run: Apr 15 04:30 
 
   RUN/DATASET 
    196275   aexp0d submitted but not finished, bhmu0d finished but with processed 
             online luminosity below 90%,  in gjt10d, gmbs0d data could not be accessed 
    196232   finished but with processed online luminosity below 90%, in gmbs0d data could 
             not be accessed 
    196231   finished and ok, in gmbs0d data could not be 
             accessed 
    196230   finished and ok, in gmbs0d data could not be      accessed 
    196229   aexp0d submitted but not finished, bhmu0d and  gjt10d finished but with processed online luminosity below 90% , in  gmbs0d data could not be accessed 
    196228   aexp0d and gjt10d submitted but not finished, bhmu0d and  gmbs0d finished but with processed online luminosity below 90% 
 
    Histograms: ok. 
</pre>
<!-- Author: Mark -->


<!-- Date: Fri Apr 15 10:03:37 2005 -->
<pre>Regarding 11A and 17A calibrations, from Mircea: 
 
"Bo is working on this..." 
</pre>
<!-- Author: Mark -->


<!-- Date: Fri Apr 15 13:04:29 2005 -->
<pre>DB and DH checks OK.</pre>
<!-- Author: Mark -->


<!-- Date: Sat Apr 16 09:54:12 2005 -->
<pre>   OFFLINE REPORT: Sat Apr 16 08:16:20 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196440  2005.04.13 04:33:59  (3 day(s), 3 hours ago) 
    Stream B: 196440  2005.04.13 04:33:59  (3 day(s), 3 hours ago) 
    Stream G: 196440  2005.04.13 04:33:59  (3 day(s), 3 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196369  2005.04.10 19:18:02  (5 day(s), 12 hours ago) 
    Stream B: 196416  2005.04.12 03:44:19  (4 day(s), 4 hours ago) 
    Stream G: 196153  2005.04.04 13:02:05  (11 day(s), 19 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       196441         PHYSICS_3_02   1 2005.04.13   08:53:35     PROD_PHYSICS_CD 
       This is 2 day(s), 23 hours old. 
   17A is not available for: 
       196441         PHYSICS_3_02   1 2005.04.13   08:53:35     PROD_PHYSICS_CD 
       This is 2 day(s), 23 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
7) DQM MONITORING 
Last update: Sat Apr 16 05:00:07 2005 
Last cron job run: Apr 16 04:30 
 
 
 
 
 
Offline calibrations still stuck on run 196441 due to inconsistency in # events reported by DFC and CSL. Experts still investigating. </pre>
<!-- Author: Mark -->


<!-- Date: Sun Apr 17 09:17:19 2005 -->
<pre>cdfopr@fcdflnx3:~&gt;cat report_05_04_17.txt 
   OFFLINE REPORT: Sun Apr 17 08:35:23 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196440  2005.04.13 04:33:59  (4 day(s), 4 hours ago) 
    Stream B: 196440  2005.04.13 04:33:59  (4 day(s), 4 hours ago) 
    Stream G: 196440  2005.04.13 04:33:59  (4 day(s), 4 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196369  2005.04.10 19:18:02  (6 day(s), 13 hours ago) 
    Stream B: 196416  2005.04.12 03:44:19  (5 day(s), 4 hours ago) 
    Stream G: 196153  2005.04.04 13:02:05  (12 day(s), 19 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       196441         PHYSICS_3_02   1 2005.04.13   08:53:35     PROD_PHYSICS_CD 
       This is 3 day(s), 23 hours old. 
   17A is not available for: 
       196441         PHYSICS_3_02   1 2005.04.13   08:53:35     PROD_PHYSICS_CD 
       This is 3 day(s), 23 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
7) DQM MONITORING 
OK 
</pre>
<!-- Author: Mark -->


<!-- Date: Mon Apr 18 08:58:49 2005 -->
<pre>   OFFLINE REPORT: Mon Apr 18 07:50:00 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196483  2005.04.14 19:13:14  (3 day(s), 12 hours ago) 
    Stream B: 196473  2005.04.14 19:13:14  (3 day(s), 12 hours ago) 
    Stream G: 196473  2005.04.14 12:49:44  (3 day(s), 19 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196441  2005.04.13 08:53:35  (4 day(s), 22 hours ago) 
    Stream B: 196416  2005.04.12 03:44:19  (6 day(s), 4 hours ago) 
    Stream G: 196153  2005.04.04 13:02:05  (13 day(s), 18 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196661         PHYSICS_3_02   1 2005.04.17   07:40:17     PROD_PHYSICS_CD 
       This is 1 day(s), 0 hours old. 
   11A is not available for: 
       196489         PHYSICS_3_02   1 2005.04.14   22:22:39     PROD_PHYSICS_CD 
       This is 3 day(s), 9 hours old. 
   17A is not available for: 
       196489         PHYSICS_3_02   1 2005.04.14   22:22:39     PROD_PHYSICS_CD 
       This is 3 day(s), 9 hours old. 
 
4) DB MONITORING 
ok 
 
5) DH MONITORING 
ok 
 
7) DQM MONITORING 
 
red boxes indicating something is wrong with the following runs/datasets: 
 
196354 aexp0d 
196354 gjt10d 
196367 gmbs0d 
 
 
Also, numerous plots in ~cdfdqm/Processing_531/PlotsDay/196367_aexp0d_psfiles 
are empty (e.g. D0.ps). Email sent to DQM experts.  
 
</pre>
<!-- Author: Mark -->


<!-- Date: Mon Apr 18 09:03:11 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-219&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-219  width='300' align='top'></a></td><td>Started getting free caf cpu as of 2am this morning. Experts contacted ;)</td></tr></table>
<!-- Author: Mark -->


<!-- Date: Mon Apr 18 09:23:41 2005 -->
<pre>This is normal CAF behavior, as we partition off a fraction of the CAF nodes for Scientific Linux testing. </pre>
<!-- Author: Mark -->


<!-- Date: Tue Apr 19 08:26:35 2005 -->
<pre>   OFFLINE REPORT: Tue Apr 19 07:45:43 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196489  2005.04.14 22:22:39  (4 day(s), 9 hours ago) 
    Stream B: 196489  2005.04.14 22:22:39  (4 day(s), 9 hours ago) 
    Stream G: 196489  2005.04.14 22:22:39  (4 day(s), 9 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196483  2005.04.14 19:13:14  (4 day(s), 12 hours ago) 
    Stream B: 196440  2005.04.13 04:33:59  (6 day(s), 3 hours ago) 
    Stream G: 196153  2005.04.04 13:02:05  (14 day(s), 18 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196665         PHYSICS_3_02   1 2005.04.18   06:43:52     PROD_PHYSICS_CD 
       This is 1 day(s), 1 hours old. 
   11A is not available for: 
       196661         PHYSICS_3_02   1 2005.04.17   07:40:17     PROD_PHYSICS_CD 
       This is 2 day(s), 0 hours old. 
   17A is not available for: 
       196661         PHYSICS_3_02   1 2005.04.17   07:40:17     PROD_PHYSICS_CD 
       This is 2 day(s), 0 hours old. 
 
4) DB MONITORING 
ok 
 
5) DH MONITORING 
ok 
 
7) DQM MONITORING 
</pre>
<!-- Author: Mark -->


<!-- Date: Wed Apr 20 09:07:21 2005 -->
<pre>1) PROCESSING (CPU): latest processed run 
    Stream A: 196663  2005.04.17 08:55:19  (2 day(s), 22 hours ago) 
    Stream B: 196663  2005.04.17 08:55:19  (2 day(s), 22 hours ago) 
    Stream G: 196663  2005.04.17 08:55:19  (2 day(s), 22 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196661  2005.04.17 07:40:17  (3 day(s), 0 hours ago) 
    Stream B: 196471  2005.04.14 11:16:15  (5 day(s), 20 hours ago) 
    Stream G: 196663  2005.04.17 08:55:19  (2 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       196664         PHYSICS_3_02   1 2005.04.17   12:33:49     PROD_PHYSICS_CD 
       This is 2 day(s), 19 hours old. 
   17A is not available for: 
       196664         PHYSICS_3_02   1 2005.04.17   12:33:49     PROD_PHYSICS_CD 
       This is 2 day(s), 19 hours old. 
 
4) DB MONITORING 
ok 
 
5) DH MONITORING 
ok 
 
7) DQM MONITORING 
ok 
 
fcdfora4 (cdfoprd) downtime 9 am - noon 
raw data logging stopped at 4am 
dcache down all day</pre>
<!-- Author: Mark -->


<!-- Date: Thu Apr 21 08:33:55 2005 -->
<pre>   OFFLINE REPORT: Thu Apr 21 08:01:22 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196663  2005.04.17 08:55:19  (3 day(s), 23 hours ago) 
    Stream B: 196663  2005.04.17 08:55:19  (3 day(s), 23 hours ago) 
    Stream G: 196663  2005.04.17 08:55:19  (3 day(s), 23 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196661  2005.04.17 07:40:17  (4 day(s), 0 hours ago) 
    Stream B: 196471  2005.04.14 11:16:15  (6 day(s), 20 hours ago) 
    Stream G: 196663  2005.04.17 08:55:19  (3 day(s), 23 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       196665         PHYSICS_3_02   1 2005.04.18   06:43:52     PROD_PHYSICS_CD 
       This is 3 day(s), 1 hours old. 
   17A is not available for: 
       196665         PHYSICS_3_02   1 2005.04.18   06:43:52     PROD_PHYSICS_CD 
       This is 3 day(s), 1 hours old. 
 
4) DB MONITORING 
 
Dead web links for: 
cdfrep01 Current Usage Table: http://cdfdbb.fnal.gov:8520/cdfr2/databases?type=ora-uaoff&do=r&fsrc=f1&nsrc=np&gsrc=fp&uaofforder=-time 
 
cdfofprd Current Usage Table: 
http://cdfdbb.fnal.gov:8520/cdfr2/databases?type=ora-uaoff&do=r&fsrc=fp&nsrc=np&gsrc=fp&uaofforder=%2Buser 
 
5) DH MONITORING 
ok 
 
7) DQM MONITORING 
 
Failed for all datasets in runs 196427-196416 (7 runs, 4 datasets) 
</pre>
<!-- Author: Mark -->


<!-- Date: Fri Apr 22 09:34:42 2005 -->
<pre>   OFFLINE REPORT: Fri Apr 22 07:58:50 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196664  2005.04.17 12:33:49  (4 day(s), 19 hours ago) 
    Stream B: 196664  2005.04.17 12:33:49  (4 day(s), 19 hours ago) 
    Stream G: 196664  2005.04.17 12:33:49  (4 day(s), 19 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196661  2005.04.17 07:40:17  (5 day(s), 0 hours ago) 
    Stream B: 196471  2005.04.14 11:16:15  (7 day(s), 20 hours ago) 
    Stream G: 196663  2005.04.17 08:55:19  (4 day(s), 23 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       196665         PHYSICS_3_02   1 2005.04.18   06:43:52     PROD_PHYSICS_CD 
       This is 4 day(s), 1 hours old. 
   17A is not available for: 
       196665         PHYSICS_3_02   1 2005.04.18   06:43:52     PROD_PHYSICS_CD 
       This is 4 day(s), 1 hours old. 
 
4) DB MONITORING 
ok 
 
5) DH MONITORING 
ok 
 
6) CAF MONITORING 
ok 
 
7) DQM MONITORING 
ok 
 
Notes: 
------ 
 
The calibrations are behind more than 24 hours for 11A and 17A as shown below.  
 
  196664         PHYSICS_3_02   1 2005.04.17   12:33:49 PROD_PHYSICS_CDF        11A     1     6    UNDEFINED    1453505 Wed Apr 20 17:40:14 2005   CDFCALIB 
  196665         PHYSICS_3_02   1 2005.04.18   06:43:52 PROD_PHYSICS_CDF        11A is not available 
 
 
 
  196664         PHYSICS_3_02   1 2005.04.17   12:33:49 PROD_PHYSICS_CDF        17A     1     5    UNDEFINED    1453503 Wed Apr 20 17:10:19 2005   CDFCALIB 
  196665         PHYSICS_3_02   1 2005.04.18   06:43:52 PROD_PHYSICS_CDF        17A is not available 
 
</pre>
<!-- Author: Mark/Terry -->


<!-- Date: Fri Apr 22 09:40:39 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-221&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-221  width='300' align='top'></a></td><td>Free CPU on CondorCAF approaching limit for notifying CAF experts. Continue to monitor...</td></tr></table>
<!-- Author: Mark/Terry -->


<!-- Date: Sat Apr 23 08:48:18 2005 -->
<pre>   OFFLINE REPORT: Sat Apr 23 08:36:08 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196752  2005.04.19 18:17:22  (3 day(s), 14 hours ago) 
    Stream B: 196752  2005.04.19 18:17:22  (3 day(s), 14 hours ago) 
    Stream G: 196752  2005.04.19 18:17:22  (3 day(s), 14 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196664  2005.04.17 12:33:49  (5 day(s), 20 hours ago) 
    Stream B: 196663  2005.04.17 08:55:19  (5 day(s), 23 hours ago) 
    Stream G: 196663  2005.04.17 08:55:19  (5 day(s), 23 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       196795 PHYSICS_HIGHLUM_3_02   1 2005.04.20   10:38:10     PROD_PHYSICS_CD 
       This is 2 day(s), 21 hours old. 
   17A is not available for: 
       196795 PHYSICS_HIGHLUM_3_02   1 2005.04.20   10:38:10     PROD_PHYSICS_CD 
       This is 2 day(s), 21 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK.  Lots of white (unused VM's) on CondorCAF but probably 
     because of limit on sections per job. 
 
7) DQM MONITORING 
OK 
</pre>
<!-- Author: Offline Shifter (Terry Watts) -->


<!-- Date: Sun Apr 24 07:58:12 2005 -->
<pre>   OFFLINE REPORT: Sun Apr 24 07:29:12 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196886  2005.04.22 10:55:33  (1 day(s), 20 hours ago) 
    Stream B: 196886  2005.04.22 10:55:33  (1 day(s), 20 hours ago) 
    Stream G: 196886  2005.04.22 10:55:33  (1 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196878  2005.04.22 04:37:53  (2 day(s), 2 hours ago) 
    Stream B: 196737  2005.04.18 22:19:20  (5 day(s), 9 hours ago) 
    Stream G: 196663  2005.04.17 08:55:19  (6 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       196889         PHYSICS_3_02   1 2005.04.22   14:51:04     PROD_PHYSICS_CD 
       This is 1 day(s), 16 hours old. 
   17A is not available for: 
       196889         PHYSICS_3_02   1 2005.04.22   14:51:04     PROD_PHYSICS_CD 
       This is 1 day(s), 16 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) DQM MONITORING 
OK 
</pre>
<!-- Author: Offline Shifter (Terry Watts) -->


<!-- Date: Mon Apr 25 09:38:59 2005 -->
<pre>   OFFLINE REPORT: Mon Apr 25 07:35:03 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196893  2005.04.23 03:07:06  (2 day(s), 4 hours ago) 
    Stream B: 196893  2005.04.23 03:07:06  (2 day(s), 4 hours ago) 
    Stream G: 196893  2005.04.23 03:07:06  (2 day(s), 4 hours ago) 
 
SAM farm is paused currently because of load on fcdfcode1. 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196892  2005.04.22 19:01:35  (2 day(s), 12 hours ago) 
    Stream B: 196879  2005.04.22 05:01:19  (3 day(s), 2 hours ago) 
    Stream G: 196663  2005.04.17 08:55:19  (7 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       196901         PHYSICS_3_02   1 2005.04.23   06:29:17     PROD_PHYSICS_CD 
       This is 2 day(s), 1 hours old. 
   17A is not available for: 
       196901         PHYSICS_3_02   1 2005.04.23   06:29:17     PROD_PHYSICS_CD 
       This is 2 day(s), 1 hours old. 
 
4) DB MONITORING 
One user has more than 600 connections for more than 2 hours. 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK. 
Am enquiry sent according to instructions was rejected by Listserv. 
 
7) DQM MONITORING 
OK. 
Last run processed was 196661. 
</pre>
<!-- Author: Terry Watts (Offline Shifter) -->


<!-- Date: Tue Apr 26 08:42:25 2005 -->
<pre>   OFFLINE REPORT: Tue Apr 26 07:13:44 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196901  2005.04.23 06:29:17  (3 day(s), 0 hours ago) 
    Stream B: 196901  2005.04.23 06:29:17  (3 day(s), 0 hours ago) 
    Stream G: 196901  2005.04.23 06:29:17  (3 day(s), 0 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196893  2005.04.23 03:07:06  (3 day(s), 4 hours ago) 
    Stream B: 196886  2005.04.22 10:55:33  (3 day(s), 20 hours ago) 
    Stream G: 196663  2005.04.17 08:55:19  (8 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196946         PHYSICS_3_02   1 2005.04.24   18:39:38     PROD_PHYSICS_CD 
       This is 1 day(s), 12 hours old. 
   11A is not available for: 
       196902         PHYSICS_3_02   1 2005.04.23   20:19:46     PROD_PHYSICS_CD 
       This is 2 day(s), 10 hours old. 
   17A is not available for: 
       196902         PHYSICS_3_02   1 2005.04.23   20:19:46     PROD_PHYSICS_CD 
       This is 2 day(s), 10 hours old. 
 
4) DB MONITORING 
Large connections load on cdfrep01 from a SAM test. 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) DQM MONITORING 
OK 
Last run processed 196681 (4/18) 
</pre>
<!-- Author: Terry Watts (Offline Shifter) -->


<!-- Date: Wed Apr 27 08:52:23 2005 -->
<pre>   OFFLINE REPORT: Wed Apr 27 07:29:37 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196901  2005.04.23 06:29:17  (4 day(s), 1 hours ago) 
    Stream B: 196901  2005.04.23 06:29:17  (4 day(s), 1 hours ago) 
    Stream G: 196901  2005.04.23 06:29:17  (4 day(s), 1 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196893  2005.04.23 03:07:06  (4 day(s), 4 hours ago) 
    Stream B: 196886  2005.04.22 10:55:33  (4 day(s), 20 hours ago) 
    Stream G: 196663  2005.04.17 08:55:19  (9 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196946         PHYSICS_3_02   1 2005.04.24   18:39:38     PROD_PHYSICS_CD 
       This is 2 day(s), 12 hours old. 
   11A is not available for: 
       196902         PHYSICS_3_02   1 2005.04.23   20:19:46     PROD_PHYSICS_CD 
       This is 3 day(s), 11 hours old. 
   17A is not available for: 
       196902         PHYSICS_3_02   1 2005.04.23   20:19:46     PROD_PHYSICS_CD 
       This is 3 day(s), 11 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) DQM MONITORING 
OK, up to Run 196737. 
</pre>
<!-- Author: Terry Watts (Offline Shifter) -->


<!-- Date: Thu Apr 28 08:32:35 2005 -->
<pre>   OFFLINE REPORT: Thu Apr 28 07:38:25 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196901  2005.04.23 06:29:17  (5 day(s), 1 hours ago) 
    Stream B: 196901  2005.04.23 06:29:17  (5 day(s), 1 hours ago) 
    Stream G: 196901  2005.04.23 06:29:17  (5 day(s), 1 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196893  2005.04.23 03:07:06  (5 day(s), 4 hours ago) 
    Stream B: 196886  2005.04.22 10:55:33  (5 day(s), 20 hours ago) 
    Stream G: 196663  2005.04.17 08:55:19  (10 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196946         PHYSICS_3_02   1 2005.04.24   18:39:38     PROD_PHYSICS_CD 
       This is 3 day(s), 12 hours old. 
   11A is not available for: 
       196902         PHYSICS_3_02   1 2005.04.23   20:19:46     PROD_PHYSICS_CD 
       This is 4 day(s), 11 hours old. 
   17A is not available for: 
       196902         PHYSICS_3_02   1 2005.04.23   20:19:46     PROD_PHYSICS_CD 
       This is 4 day(s), 11 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK. Momentary problem (worker nodes?) in CAF overnight. 
 
7) DQM MONITORING 
OK. Up to Run 196737 (4/18). 
</pre>
<!-- Author: Terry Watts (Offline Shifter) -->


<!-- Date: Fri Apr 29 08:50:31 2005 -->
<pre>   OFFLINE REPORT: Fri Apr 29 07:31:44 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196901  2005.04.23 06:29:17  (6 day(s), 1 hours ago) 
    Stream B: 196901  2005.04.23 06:29:17  (6 day(s), 1 hours ago) 
    Stream G: 196901  2005.04.23 06:29:17  (6 day(s), 1 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196901  2005.04.23 06:29:17  (6 day(s), 1 hours ago) 
    Stream B: 196901  2005.04.23 06:29:17  (6 day(s), 1 hours ago) 
    Stream G: 196901  2005.04.23 06:29:17  (6 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       196946         PHYSICS_3_02   1 2005.04.24   18:39:38     PROD_PHYSICS_CD 
       This is 4 day(s), 12 hours old. 
   11A is not available for: 
       196902         PHYSICS_3_02   1 2005.04.23   20:19:46     PROD_PHYSICS_CD 
       This is 5 day(s), 11 hours old. 
   17A is not available for: 
       196902         PHYSICS_3_02   1 2005.04.23   20:19:46     PROD_PHYSICS_CD 
       This is 5 day(s), 11 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
2 tapes with "failed to catalog filesets" 
 
6) CAF MONITORING 
OK 
 
7) DQM MONITORING 
OK. Up to Run 196801 (4/20). 
 
Notes: Waiting for databases and SAM Farm to resume functionning. 
</pre>
<!-- Author: Terry/William (Offline Shifters) -->


<!-- Date: Fri Apr 29 14:35:35 2005 -->
<pre>Reported this error to cdfdh_oper and Mircea: 
 
2005-Apr-29 10:07:43 filesets CB6444.0 CB6444.1 CB6444.2 CB6444.3 CB6444.4 spooled to tape CB6444 (encp SUCCESS) 
2005-Apr-29 10:24:36 failed to catalog fileset CB6444.0 
2005-Apr-29 10:24:38 failed to catalog fileset CB6444.1 
2005-Apr-29 10:24:39 failed to catalog fileset CB6444.2 
2005-Apr-29 10:24:40 failed to catalog fileset CB6444.3 
2005-Apr-29 10:24:41 failed to catalog fileset CB6444.4 
 
Reported this to Mircea (predator status): 
Fri Apr 29 14:30:06 CDT 2005 
 
DD-HH  Day and Hour of log file, and summary : 
 
29-14  29-13  29-12  29-11  29-10  29-09  Number of data files added: 128 
29-08  Number of data files added: 0</pre>
<!-- Author: William Wester -->


<!-- Date: Sat Apr 30 08:36:15 2005 -->
<pre>   OFFLINE REPORT: Sat Apr 30 07:54:45 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A:   2005 196041  (hours ago) 
    Stream B:   2005 196041  (hours ago) 
    Stream G:   2005 196041  (hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A:   2005 196041  (hours ago) 
    Stream B:   2005 196041  (hours ago) 
    Stream G:   2005 196041  (hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       197071 PHYSICS_HIGHLUM_3_02   1 2005.04.27   18:58:24     PROD_PHYSICS_CD 
       This is 2 day(s), 12 hours old. 
   11A is not available for: 
       196902         PHYSICS_3_02   1 2005.04.23   20:19:46     PROD_PHYSICS_CD 
       This is 6 day(s), 11 hours old. 
   17A is not available for: 
       196902         PHYSICS_3_02   1 2005.04.23   20:19:46     PROD_PHYSICS_CD 
       This is 6 day(s), 11 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
email sent to cdfdh_oper and Mircea: 
Several instances of "failed to catalog filesets" 
Predator status problem 
 
6) CAF MONITORING 
OK 
 
7) DQM MONITORING 
OK. Up to Run 196889 (4/22) 
</pre>
<!-- Author: William Wester (offline shifter) -->


<!-- Date: Sun May 1 08:13:42 2005 -->
<pre>   OFFLINE REPORT: Sun May 1 07:54:13 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A:   2005 196041  (hours ago) 
    Stream B:   2005 196041  (hours ago) 
    Stream G:   2005 196041  (hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A:   2005 196041  (hours ago) 
    Stream B:   2005 196041  (hours ago) 
    Stream G:   2005 196041  (hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       197071 PHYSICS_HIGHLUM_3_02   1 2005.04.27   18:58:24     PROD_PHYSICS_CD 
       This is 3 day(s), 12 hours old. 
   11A is not available for: 
       196902         PHYSICS_3_02   1 2005.04.23   20:19:46     PROD_PHYSICS_CD 
       This is 7 day(s), 11 hours old. 
   17A is not available for: 
       196902         PHYSICS_3_02   1 2005.04.23   20:19:46     PROD_PHYSICS_CD 
       This is 7 day(s), 11 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
Predator status problem ... needs DB being back 
 
6) CAF MONITORING 
OK 
 
7) DQM MONITORING 
OK. Up to Run 19601 (4/23)</pre>
<!-- Author: William Wester (offline shifter) -->


<!-- Date: Sun May 1 12:16:22 2005 -->
<pre> 
At the request of the CDF shift captain (Barry Wicklund) ... I sent the following email to experts (Petar, Mircea, cdfdb* mailing lists, etc.). I don't know if the offline Database problems are more system wide. William 
 
-------------------------------------- 
 
Hi, I'm the offline CO. I don't know if the following is related to all the work the DB experts are doing ... but 
 
We have some consumer experts in working on the online consumer processes (XMON YMON etc) which all failed when a new run was started. The messages in some of the  windows for the consumer processes include messages like 
   "cannot initialize oracle connection" 
 
If this might be related, please let me know and/or call our shift captain at 2080. 
 
Thanks, 
 
William </pre>
<!-- Author: William Wester -->


<!-- Date: Mon May 2 09:11:06 2005 -->
<pre>   OFFLINE REPORT: Mon May 2 07:49:52 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A:   2005 196041  (hours ago) 
    Stream B:   2005 196041  (hours ago) 
    Stream G:   2005 196041  (hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A:   2005 196041  (hours ago) 
    Stream B:   2005 196041  (hours ago) 
    Stream G:   2005 196041  (hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       197071 PHYSICS_HIGHLUM_3_02   1 2005.04.27   18:58:24     PROD_PHYSICS_CD 
       This is 4 day(s), 12 hours old. 
   11A is not available for: 
       196946         PHYSICS_3_02   1 2005.04.24   18:39:38     PROD_PHYSICS_CD 
       This is 7 day(s), 13 hours old. 
   17A is not available for: 
       196946         PHYSICS_3_02   1 2005.04.24   18:39:38     PROD_PHYSICS_CD 
       This is 7 day(s), 13 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) DQM MONITORING 
OK, up to Run 196901 (4/23) 
</pre>
<!-- Author: William Wester (offline shifter) -->


<!-- Date: Mon May 2 09:13:34 2005 -->
<pre>Recovering from disk full problem on b0dap18 (Mozilla Cache). Deleted cache area and remade cdfopr Mozilla profile from scratch.</pre>
<!-- Author: William Wester (offline shifter) -->


<!-- Date: Mon May 2 15:30:45 2005 -->
<pre>Sent email to a user with more than 20 INACTIVE jobs: cdfofprd and cdfrep01</pre>
<!-- Author: William Wester (Offline shifter) -->


<!-- Date: Tue May 3 07:56:18 2005 -->
<pre>   OFFLINE REPORT: Tue May 3 07:50:20 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 196989  2005.04.26 12:45:43  (6 day(s), 19 hours ago) 
    Stream B: 196989  2005.04.26 12:45:43  (6 day(s), 19 hours ago) 
    Stream G: 196989  2005.04.26 12:45:43  (6 day(s), 19 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 196968  2005.04.25 21:03:56  (7 day(s), 10 hours ago) 
    Stream B: 196901  2005.04.23 06:29:17  (10 day(s), 1 hours ago) 
    Stream G: 196901  2005.04.23 06:29:17  (10 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       197403         PHYSICS_3_02   1 2005.05.03   05:28:21     PROD_PHYSICS_CD 
       This is 2 hours old. 
   11A is not available for: 
       197071 PHYSICS_HIGHLUM_3_02   1 2005.04.27   18:58:24     PROD_PHYSICS_CD 
       This is 5 day(s), 12 hours old. 
   17A is not available for: 
       197071 PHYSICS_HIGHLUM_3_02   1 2005.04.27   18:58:24     PROD_PHYSICS_CD 
       This is 5 day(s), 12 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) DQM MONITORING 
OK, many jobs not out of production. Still up to 196901 (4/23) 
</pre>
<!-- Author: William Wester (offline shifter) -->


<!-- Date: Tue May 3 15:35:47 2005 -->
<pre>Sent email to user with &gt;20 jobs inactive more than 100 sec on cdfrep01. 
 
</pre>
<!-- Author: William Wester (Offline shifter) -->


<!-- Date: Wed May 4 09:50:49 2005 -->
<pre>   OFFLINE REPORT: Wed May 4 07:52:39 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 197289  2005.05.01 10:06:53  (2 day(s), 21 hours ago) 
    Stream B: 197194  2005.05.01 10:06:53  (2 day(s), 21 hours ago) 
    Stream G: 197194  2005.04.29 22:36:07  (4 day(s), 9 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 197189  2005.04.29 14:51:45  (4 day(s), 17 hours ago) 
    Stream B: 196968  2005.04.25 21:03:56  (8 day(s), 10 hours ago) 
    Stream G: 196901  2005.04.23 06:29:17  (11 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       197404         PHYSICS_3_02   1 2005.05.03   06:34:24     PROD_PHYSICS_CD 
       This is 1 day(s), 1 hours old. 
   11A is not available for: 
       197317 PHYSICS_HIGHLUM_3_02   1 2005.05.02   07:32:32     PROD_PHYSICS_CD 
       This is 2 day(s), 0 hours old. 
   17A is not available for: 
       197317 PHYSICS_HIGHLUM_3_02   1 2005.05.02   07:32:32     PROD_PHYSICS_CD 
       This is 2 day(s), 0 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) DQM MONITORING 
OK, Run 196988 (4/26) not out of production. No DQM histograms to check. 
</pre>
<!-- Author: William Wester (Offline shifter) -->


<!-- Date: Wed May 4 09:51:32 2005 -->
<pre>Sent email to user regarding ~&gt;20 inactive jobs on cdfrep01 with more than 100 sec.</pre>
<!-- Author: William Wester (offline shifter) -->


<!-- Date: Wed May 4 15:23:47 2005 -->
<pre>Sent emails to separate users. 
1) more than 10000 cpu seconds cdfrep01 
2) more than 20 connections cdfofprd</pre>
<!-- Author: William Wester (Offline shifter) -->


<!-- Date: Thu May 5 08:59:56 2005 -->
<pre>   OFFLINE REPORT: Thu May 5 07:58:24 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 197321  2005.05.02 11:04:50  (2 day(s), 20 hours ago) 
    Stream B: 197321  2005.05.02 11:04:50  (2 day(s), 20 hours ago) 
    Stream G: 197321  2005.05.02 11:04:50  (2 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 197286  2005.04.30 23:09:16  (4 day(s), 8 hours ago) 
    Stream B: 197071  2005.04.27 18:58:24  (7 day(s), 12 hours ago) 
    Stream G: 196901  2005.04.23 06:29:17  (12 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       197440         PHYSICS_3_02   1 2005.05.04   08:52:06     PROD_PHYSICS_CD 
       This is 23 hours old. 
   11A is not available for: 
       197403         PHYSICS_3_02   1 2005.05.03   05:28:21     PROD_PHYSICS_CD 
       This is 2 day(s), 2 hours old. 
   17A is not available for: 
       197403         PHYSICS_3_02   1 2005.05.03   05:28:21     PROD_PHYSICS_CD 
       This is 2 day(s), 2 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) DQM MONITORING 
OK, up to Run 196989 (4/26). Many jobs not out of production. </pre>
<!-- Author: William Wester (Offline shifter) -->


<!-- Date: Fri May 6 09:00:16 2005 -->
<pre>The DQM histograms are all empty for 194536 aexp0d, 194539 aexp0d and bhmu0d, 194560 aexp0d and gjt10d, 159209 and 195267. 
Email sent to experts.</pre>
<!-- Author: Tracey & William -->


<!-- Date: Fri May 6 09:26:25 2005 -->
<pre> OFFLINE REPORT: Fri May 6 07:53:57 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 197321  2005.05.02 11:04:50  (3 day(s), 20 hours ago) 
    Stream B: 197321  2005.05.02 11:04:50  (3 day(s), 20 hours ago) 
    Stream G: 197321  2005.05.02 11:04:50  (3 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 197286  2005.04.30 23:09:16  (5 day(s), 8 hours ago) 
    Stream B: 197288  2005.05.01 09:58:32  (4 day(s), 21 hours ago) 
    Stream G: 196901  2005.04.23 06:29:17  (13 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       197441         PHYSICS_3_02   1 2005.05.04   17:05:18     PROD_PHYSICS_CD 
       This is 1 day(s), 14 hours old. 
   11A is not available for: 
       197440         PHYSICS_3_02   1 2005.05.04   08:52:06     PROD_PHYSICS_CD 
       This is 1 day(s), 23 hours old. 
   17A is not available for: 
       197440         PHYSICS_3_02   1 2005.05.04   08:52:06     PROD_PHYSICS_CD 
       This is 1 day(s), 23 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
CondorCAF Load=0 &gt; 500 VM over two hours since 6am 
 
7) DQM MONITORING 
Older runs processed 194535 (2/26) - 195267 (3/14) 
The DQM histograms are all empty for: 
194536 aexp0d 
194539 aexp0d and bhmu0d  
194560 aexp0d and gjt10d  
159209  
195267 
</pre>
<!-- Author: Tracey & William -->


<!-- Date: Fri May 6 09:31:41 2005 -->
<pre>Emailed Elliot re the CondorCAF System Info plot being above 500 VMs for over 2 hours, since 6am.</pre>
<!-- Author: Tracey & William -->


<!-- Date: Fri May 6 12:32:56 2005 -->
<pre> 
The CondorCAF has had a relatively large amount of light blue indicating a scheduler overload, this peaked at about 10:15am but has been decreasing since, so I think that the problem is sorting itself out/or has been solved. This coincided with a dip in the sections not consuming CPU (stuck) and with an increase in the number of sections consuming a small amount of CPU. 
Elliot has been emailed to inform him of this.</pre>
<!-- Author: Tracey -->


<!-- Date: Sat May 7 08:12:04 2005 -->
<pre> 
The first run for which the 10A calibrations are not available is now more than 24 hours old: 
run number 197441, is now 2 days and 14 hours old. 
The experts have been emailed. </pre>
<!-- Author: Tracey Berry -->


<!-- Date: Sat May 7 08:17:53 2005 -->
<pre> 
The first run for which 11A calibrations are not available is more than 48 hours old: 
run number 197440, is 2 days and 23 hours old. 
The experts have been emailed.</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Sat May 7 08:25:13 2005 -->
<pre> 
The first run for which the 17A calibrations are not available is now more than 48 hours old:  
run number 197440, is now 2 days and 23 hours old.  
The experts have been emailed.</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Sat May 7 08:36:53 2005 -->
<pre>   OFFLINE REPORT: Sat May 7 07:54:08 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 197405  2005.05.03 13:15:05  (3 day(s), 18 hours ago) 
    Stream B: 197405  2005.05.03 13:15:05  (3 day(s), 18 hours ago) 
    Stream G: 197405  2005.05.03 13:15:05  (3 day(s), 18 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 197321  2005.05.02 11:04:50  (4 day(s), 20 hours ago) 
    Stream B: 197289  2005.05.01 10:06:53  (5 day(s), 21 hours ago) 
    Stream G: 196901  2005.04.23 06:29:17  (14 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       197441         PHYSICS_3_02   1 2005.05.04   17:05:18     PROD_PHYSICS_CD 
       This is 2 day(s), 14 hours old. 
   11A is not available for: 
       197440         PHYSICS_3_02   1 2005.05.04   08:52:06     PROD_PHYSICS_CD 
       This is 2 day(s), 23 hours old. 
   17A is not available for: 
       197440         PHYSICS_3_02   1 2005.05.04   08:52:06     PROD_PHYSICS_CD 
       This is 2 day(s), 23 hours old. 
 
4) DB MONITORING 
Ok 
 
5) DH MONITORING 
Ok 
 
6) CAF MONITORING 
Ok 
 
7) DQM MONITORING 
Ok 
</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Sat May 7 13:04:47 2005 -->
<pre>The System Info plot for the CAF looks a bit odd - at about 12:15 the white space indicating usused resources was more than 200 VMs. This then recovered but following this there has been an increase in the light blue indicating that the scheduler has been overloaded. The situation seems to be getting better now though. The increase in light blue also conincided with an increase of sections using small amounts of CPU. 
Elliot has been informed. I shall keep monitoring this.</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Sat May 7 15:33:40 2005 -->
<pre>The CAF System Info plot looks fine.</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Sun May 8 08:17:13 2005 -->
<pre> 
The DQM status for the last 24 hours shows more than 1/3 of the boxes on the page being red, which indicates that something is wrong. The flags say "N_STARTED-&gt; Job could not be started".  The runs with red boxes range from 194536 to 195209. Run 195267 finished and is ok. 
The experts have been emailed. 
</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Sun May 8 08:26:53 2005 -->
<pre>   OFFLINE REPORT: Sun May 8 07:47:07 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 197405  2005.05.03 13:15:05  (4 day(s), 18 hours ago) 
    Stream B: 197405  2005.05.03 13:15:05  (4 day(s), 18 hours ago) 
    Stream G: 197405  2005.05.03 13:15:05  (4 day(s), 18 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 197321  2005.05.02 11:04:50  (5 day(s), 20 hours ago) 
    Stream B: 197289  2005.05.01 10:06:53  (6 day(s), 21 hours ago) 
    Stream G: 196901  2005.04.23 06:29:17  (15 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       197440         PHYSICS_3_02   1 2005.05.04   08:52:06     PROD_PHYSICS_CD 
       This is 3 day(s), 22 hours old. 
   17A is not available for: 
       197440         PHYSICS_3_02   1 2005.05.04   08:52:06     PROD_PHYSICS_CD 
       This is 3 day(s), 22 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) DQM MONITORING 
The DQM status for the last 24 hours shows more than 1/3 of the boxes on the page being red, which indicates that something is wrong. The flags say "N_STARTED-&gt; Job could not be started".  The runs/datasets with red boxes are: 
195209	gjt10d 
194590	gjt10d 
194562	gjt10d 
194560	aexp0d	gjt10d 
194539	aexp0d	bhmu0d	 
194538	gjt10d 
194536	aexp0d	gjt10d 
 
195267  gjt10d finished and is ok.  
Experts have been emailed. 
 
Histograms are ok. 
</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Sun May 8 08:35:25 2005 -->
<pre>Emailed Bo re the calibrations getting behind. 
Since the beamlines for the runs &gt; 197405 have not been made. Although BeamExe has processed all the runs.  
</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Mon May 9 08:38:17 2005 -->
<pre>  OFFLINE REPORT: Mon May 9 07:38:31 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 197640  2005.05.06 14:18:34  (2 day(s), 17 hours ago) 
    Stream B: 197640  2005.05.06 14:18:34  (2 day(s), 17 hours ago) 
    Stream G: 197640  2005.05.06 14:18:34  (2 day(s), 17 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 197405  2005.05.03 13:15:05  (5 day(s), 18 hours ago) 
    Stream B: 197289  2005.05.01 10:06:53  (7 day(s), 21 hours ago) 
    Stream G: 196901  2005.04.23 06:29:17  (16 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       197659       JET_CALIB_TEST   4 2005.05.08   11:05:19     PROD_PHYSICS_CD 
       This is 20 hours old. 
   11A is not available for: 
       197644         PHYSICS_3_02   1 2005.05.06   22:32:55     PROD_PHYSICS_CD 
       This is 2 day(s), 9 hours old. 
   17A is not available for: 
       197644         PHYSICS_3_02   1 2005.05.06   22:32:55     PROD_PHYSICS_CD 
       This is 2 day(s), 9 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) DQM MONITORING 
Only one entry in table, which is red indicating something wrong, flag says not started: 
RunNumber 194535 Dataset gjt10d  
No histograms 
</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Mon May 9 09:34:38 2005 -->
<pre>The DQM experts have been paged re: 
only one entry in the DQM last 24 hour Monitoring table, which is red indicating something wrong. The flag says not started: 
RunNumber 194535 Dataset gjt10d. 
</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Mon May 9 12:10:21 2005 -->
<pre>User gbyu has 37 connections to the cdfofprd database (over the 20 limit). This is database is for offline production and general users should not be using this database. He has been emailed and asked to correct this. 
</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Mon May 9 13:37:34 2005 -->
<pre>Sent a second email to gbyu, since his TRIGSIM processes are still excessively accessing the cdfofprd database. Now there are over 70 connections: the limit is less than 20. 
</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Mon May 9 15:35:16 2005 -->
<pre>User gybu has stopped accessing the cdfofprd database now. The system is no longer overloaded. 
</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Mon May 9 15:44:13 2005 -->
<pre>Error in writing the data to tape. All the data written before and after this was successfully completed: 
 
2005-May-09 12:11:15 encp FAILED TO COPY FILESET CB6601.3 
source directory: /cdf/b0/disk7/stager/CB6601.3 
source files: er030454.0462phys er030454.0481phys er030454.04a1phys er030454.04c1phys er030454.04e1phys er030454.0502phys er030454.0523phys er030454.0545phys 
destination dir: /pnfs/cdfen/filesets/CB/CB66/CB6601/CB6601.3 
The rest of the filesets will not be encpied: CB6601.3 CB6601.4 
 
The experts have been notified via email.</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Tue May 10 08:38:33 2005 -->
<pre>   OFFLINE REPORT: Tue May 10 07:44:50 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 197640  2005.05.06 14:18:34  (3 day(s), 17 hours ago) 
    Stream B: 197640  2005.05.06 14:18:34  (3 day(s), 17 hours ago) 
    Stream G: 197640  2005.05.06 14:18:34  (3 day(s), 17 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 197405  2005.05.03 13:15:05  (6 day(s), 18 hours ago) 
    Stream B: 197405  2005.05.03 13:15:05  (6 day(s), 18 hours ago) 
    Stream G: 196901  2005.04.23 06:29:17  (17 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       197762         PHYSICS_3_02   1 2005.05.09   20:03:19     PROD_PHYSICS_CD 
       This is 11 hours old. 
   11A is not available for: 
       197644         PHYSICS_3_02   1 2005.05.06   22:32:55     PROD_PHYSICS_CD 
       This is 3 day(s), 9 hours old. 
   17A is not available for: 
       197644         PHYSICS_3_02   1 2005.05.06   22:32:55     PROD_PHYSICS_CD 
       This is 3 day(s), 9 hours old. 
 
4) DB MONITORING 
Ok 
 
5) DH MONITORING 
Ok 
 
6) CAF MONITORING 
Ok 
 
7) DQM MONITORING 
Ok - jobs submitted but none finished so no histograms to check 
 
</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Tue May 10 09:44:17 2005 -->
<pre>Bo exmailed re the 11A calibrations getting behind. They are over 48 hours behind the 10A calibrations. 
 
</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Tue May 10 13:52:39 2005 -->
<pre>User scuri had 26 (over the limit of 20) long-term inactive database connections.  
The user has been sent the standard email for this case and the experts were cc'ed.</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Tue May 10 14:59:09 2005 -->
<pre>User daronco had 20 (limit is ~20) long-term inactive database connections.   
The user has been sent the standard email for this case and the experts were cc'ed. 
</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Tue May 10 15:08:10 2005 -->
<pre>The CAF systems info plot has shown an increase in the amount of white (unused resources) greater than 200 VM's at around 2:30pm. This has now decreased, but there is an increasing amount of light blue, indicating a scheduler overload. This seems to coincide with an increase in the section using small amounts of CPU. I shall monitor this to see if the situation continues. 
The expert has been emailed to inform him of this.</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Tue May 10 15:43:06 2005 -->
<pre>The Condor CAF Systems Info plot looks normal now.</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Wed May 11 08:17:19 2005 -->
<pre>  OFFLINE REPORT: Wed May 11 07:37:40 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 197715  2005.05.08 20:11:08  (2 day(s), 11 hours ago) 
    Stream B: 197715  2005.05.08 20:11:08  (2 day(s), 11 hours ago) 
    Stream G: 197715  2005.05.08 20:11:08  (2 day(s), 11 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 197405  2005.05.03 13:15:05  (7 day(s), 18 hours ago) 
    Stream B: 197405  2005.05.03 13:15:05  (7 day(s), 18 hours ago) 
    Stream G: 196901  2005.04.23 06:29:17  (18 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       197762         PHYSICS_3_02   1 2005.05.09   20:03:19     PROD_PHYSICS_CD 
       This is 1 day(s), 11 hours old. 
   11A is not available for: 
       197716         PHYSICS_3_02   1 2005.05.08   20:45:02     PROD_PHYSICS_CD 
       This is 2 day(s), 10 hours old. 
   17A is not available for: 
       197716         PHYSICS_3_02   1 2005.05.08   20:45:02     PROD_PHYSICS_CD 
       This is 2 day(s), 10 hours old. 
 
4) DB MONITORING 
Ok 
 
5) DH MONITORING 
Ok 
 
6) CAF MONITORING 
Ok 
 
7) DQM MONITORING 
Page updated, but nothing processed so no entries in the table. 
No histograms 
 
</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Wed May 11 11:49:24 2005 -->
<pre>Emailed the expert (bdc) re the 10A calibrations being over 24 hours old.</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Thu May 12 08:33:18 2005 -->
<pre>  OFFLINE REPORT: Thu May 12 07:38:22 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 197716  2005.05.08 20:45:02  (3 day(s), 10 hours ago) 
    Stream B: 197716  2005.05.08 20:45:02  (3 day(s), 10 hours ago) 
    Stream G: 197716  2005.05.08 20:45:02  (3 day(s), 10 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 197405  2005.05.03 13:15:05  (8 day(s), 18 hours ago) 
    Stream B: 197405  2005.05.03 13:15:05  (8 day(s), 18 hours ago) 
    Stream G: 196901  2005.04.23 06:29:17  (19 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       197762         PHYSICS_3_02   1 2005.05.09   20:03:19     PROD_PHYSICS_CD 
       This is 2 day(s), 11 hours old. 
   11A is not available for: 
       197762         PHYSICS_3_02   1 2005.05.09   20:03:19     PROD_PHYSICS_CD 
       This is 2 day(s), 11 hours old. 
   17A is not available for: 
       197762         PHYSICS_3_02   1 2005.05.09   20:03:19     PROD_PHYSICS_CD 
       This is 2 day(s), 11 hours old. 
 
4) DB MONITORING 
Ok 
 
5) DH MONITORING 
Ok 
 
6) CAF MONITORING 
Ok 
 
7) DQM MONITORING 
Page updated, but nothing processed so no entries in the table.  
No histograms  
 
</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Thu May 12 08:33:59 2005 -->
<pre>Experts emailed re the 10A calibrations being over 24 hours old.</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Thu May 12 11:15:10 2005 -->
<pre>User Scuri has 21 inactive connections (over the limit of ~20) lasting more than 100 seconds each. He has been notified using the standard email and the experts, Petar and Susana, were cc'ed.</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Thu May 12 14:24:18 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-223&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-223  width='300' align='top'></a></td><td>User Scuri has had a large number (>~600) of connections to the database for over 2 hours. The experts have been emailed.</td></tr></table>
<!-- Author: Tracey Berry -->


<!-- Date: Thu May 12 14:27:34 2005 -->
<pre>The links from the CO offline "monitoring the Oracle DB" webpage to cdfstrm1 Current Usage Table and to the cdfofprd Current Usage Table seem to have changed to link to a the "CDF Run 2 database queryer and browser page"! 
</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Thu May 12 14:29:02 2005 -->
<pre>The experts: Petar and Susana Cabrera have been emailed about the above.</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Thu May 12 15:41:17 2005 -->
<pre> 
The links from the CO offline "monitoring the Oracle DB" webpage to cdfstrm1 Current Usage 
Table and to the cdfofprd Current Usage Table are now connecting as usual. 
 
The user Scuri has now had &gt;~600 connections to the DB for over 4 hours now! The experts have been emailed again to inform them of this. 
</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Thu May 12 15:52:45 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-226&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-226  width='300' align='top'></a></td><td>User Scuri's number of connections has dropped below 600 now. 
The experts have been emailed.</td></tr></table>
<!-- Author: Tracey Berry -->


<!-- Date: Thu May 12 16:00:25 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-227&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-227  width='300' align='top'></a></td><td>User Scuri's number of connections to the DB seem to be on the increase again! The experts have been emailed.</td></tr></table>
<!-- Author: Tracey Berry -->


<!-- Date: Fri May 13 08:25:59 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-228&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-228  width='300' align='top'></a></td><td> 
Over night user dececco had more than 600 connections for a period of about 4 hour. The experts have been emailed. 
</td></tr></table>
<!-- Author: Tracey Berry -->


<!-- Date: Fri May 13 08:42:08 2005 -->
<pre>The DQM status for the last 24 hours shows that &gt; 1/3 of boxes are Red. 
The flags say not started. 
The runs and datasets with this problem are: 
195209   gjt10d 
194590   gjt10d 
194562   gjt10d 
194560   aexp0d 
194539   aexp0d   bhmu0d 
194538   gjt10d 
194536   gjt10d 
194535   gjt10d 
194318   gjt10d 
 
The experts have been emailed.</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Fri May 13 08:44:42 2005 -->
<pre>   OFFLINE REPORT: Fri May 13 07:41:27 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 197762  2005.05.09 20:03:19  (3 day(s), 11 hours ago) 
    Stream B: 197716  2005.05.08 20:45:02  (4 day(s), 10 hours ago) 
    Stream G: 197716  2005.05.08 20:45:02  (4 day(s), 10 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 197405  2005.05.03 13:15:05  (9 day(s), 18 hours ago) 
    Stream B: 197405  2005.05.03 13:15:05  (9 day(s), 18 hours ago) 
    Stream G: 196901  2005.04.23 06:29:17  (20 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       197985 PHYSICS_HIGHLUM_3_02   1 2005.05.12   17:18:48     PROD_PHYSICS_CD 
       This is 14 hours old. 
   11A is not available for: 
       197812         PHYSICS_3_02   1 2005.05.11   21:48:00     PROD_PHYSICS_CD 
       This is 1 day(s), 9 hours old. 
   17A is not available for: 
       197812         PHYSICS_3_02   1 2005.05.11   21:48:00     PROD_PHYSICS_CD 
       This is 1 day(s), 9 hours old. 
 
4) DB MONITORING 
Ok 
 
5) DH MONITORING 
Ok 
 
6) CAF MONITORING 
Ok 
 
7) DQM MONITORING 
For the last 24 hours shows that &gt; 1/3 of boxes are Red.  
The flags say not started.  
The runs and datasets with this problem are:  
195209   gjt10d  
194590   gjt10d  
194562   gjt10d  
194560   aexp0d  
194539   aexp0d   bhmu0d  
194538   gjt10d  
194536   gjt10d  
194535   gjt10d  
194318   gjt10d  
 
No histograms to check.</pre>
<!-- Author: Tracey Berry -->


<!-- Date: Sat May 14 09:03:11 2005 -->
<pre>Daily report at the beginning of the CO offiline shift 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 197808  2005.05.10 23:48:49  (3 day(s), 8 hours ago) 
    Stream B: 197716  2005.05.08 20:45:02  (5 day(s), 11 hours ago) 
    Stream G: 197716  2005.05.08 20:45:02  (5 day(s), 11 hours ago) 
 
so stream B & G are older then 5 days 
 
2) PROCESSING (TO TAPE): 
    Stream A: 197405  2005.05.03 13:15:05  (10 day(s), 19 hours ago) 
    Stream B: 197644  2005.05.06 22:32:55  (7 day(s), 9 hours ago) 
    Stream G: 197644  2005.05.06 22:32:55  (7 day(s), 9 hours ago) 
 
they are all older then 5 days 
 
3) CALIBRATIONS 
   10A is not available for: 
       198081 PHYSICS_HIGHLUM_3_02   1 2005.05.13   23:53:53     PROD_PHYSICS_CD 
       This is 8 hours old. 
   11A is not available for: 
       197812         PHYSICS_3_02   1 2005.05.11   21:48:00     PROD_PHYSICS_CD 
       This is 2 day(s), 10 hours old. 
   17A is not available for: 
       197812         PHYSICS_3_02   1 2005.05.11   21:48:00     PROD_PHYSICS_CD 
       This is 2 day(s), 10 hours old. 
 
so 11A and 17A are older then 48 hours 
 
experts have been emailed 
 
Franco Spinella CO offline</pre>
<!-- Author: franco spinella -->


<!-- Date: Sun May 15 08:43:36 2005 -->
<pre>It seems that the production is stopped and also calib 11A  
is older then 2 days. Experts have been emailed 
Here is the report: 
 
   OFFLINE REPORT: Sun May 15 08:31:14 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 197808  2005.05.10 23:48:49  (4 day(s), 8 hours ago) 
    Stream B: 197716  2005.05.08 20:45:02  (6 day(s), 11 hours ago) 
    Stream G: 197716  2005.05.08 20:45:02  (6 day(s), 11 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 197405  2005.05.03 13:15:05  (11 day(s), 19 hours ago) 
    Stream B: 197644  2005.05.06 22:32:55  (8 day(s), 9 hours ago) 
    Stream G: 197644  2005.05.06 22:32:55  (8 day(s), 9 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       197809         PHYSICS_3_02   1 2005.05.11   16:49:23     PROD_PHYSICS_CD 
       This is 3 day(s), 15 hours old. 
   17A is not available for: 
       197809         PHYSICS_3_02   1 2005.05.11   16:49:23     PROD_PHYSICS_CD 
       This is 3 day(s), 15 hours old. 
 
 
OFFLINE CO</pre>
<!-- Author: franco spinella -->


<!-- Date: Sun May 15 08:58:19 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-230&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-230  width='300' align='top'></a></td><td>user vivek has mor then 600 connections for more then 2 hours</td></tr></table>
<!-- Author: franco spinella -->


<!-- Date: Sun May 15 09:08:17 2005 -->
<pre>I do not see DQM histograms  for  highest luminosity run since yesterday 
 
</pre>
<!-- Author: franco spinella -->


<!-- Date: Sun May 15 09:09:15 2005 -->
<pre>I do not see DQM histograms  for  highest luminosity run since yesterday. 
Experts emailed</pre>
<!-- Author: franco spinella -->


<!-- Date: Mon May 16 07:53:33 2005 -->
<pre>Report for monday 16 may 2005: 
 
OFFLINE REPORT: Mon May 16 07:42:51 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198081  2005.05.13 23:53:53  (2 day(s), 7 hours ago) 
    Stream B: 197716  2005.05.08 20:45:02  (7 day(s), 10 hours ago) 
    Stream G: 197716  2005.05.08 20:45:02  (7 day(s), 10 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 197990  2005.05.12 17:32:58  (3 day(s), 14 hours ago) 
    Stream B: 197644  2005.05.06 22:32:55  (9 day(s), 9 hours ago) 
    Stream G: 197644  2005.05.06 22:32:55  (9 day(s), 9 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       198116 PHYSICS_HIGHLUM_3_02   1 2005.05.15   18:39:56     PROD_PHYSICS_CD 
       This is 13 hours old. 
   11A is not available for: 
       198082         PHYSICS_3_02   1 2005.05.14   02:13:48     PROD_PHYSICS_CD 
       This is 2 day(s), 5 hours old. 
   17A is not available for: 
       198082         PHYSICS_3_02   1 2005.05.14   02:13:48     PROD_PHYSICS_CD 
       This is 2 day(s), 5 hours old. 
 
4) DB MONITORING 
 
 
5) DH MONITORING 
 
 
6) CAF MONITORING 
</pre>
<!-- Author: franco spinella -->


<!-- Date: Tue May 17 07:56:20 2005 -->
<pre>OFFLINE CO dayly report tuesday 17-5-2005 
 
 OFFLINE REPORT: Tue May 17 07:49:47 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198081  2005.05.13 23:53:53  (3 day(s), 7 hours ago) 
    Stream B: 197716  2005.05.08 20:45:02  (8 day(s), 11 hours ago) 
    Stream G: 197716  2005.05.08 20:45:02  (8 day(s), 11 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 197990  2005.05.12 17:32:58  (4 day(s), 14 hours ago) 
    Stream B: 197644  2005.05.06 22:32:55  (10 day(s), 9 hours ago) 
    Stream G: 197644  2005.05.06 22:32:55  (10 day(s), 9 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       198152 PHYSICS_HIGHLUM_3_02   1 2005.05.16   23:42:17     PROD_PHYSICS_CD 
       This is 8 hours old. 
   11A is not available for: 
       198082         PHYSICS_3_02   1 2005.05.14   02:13:48     PROD_PHYSICS_CD 
       This is 3 day(s), 5 hours old. 
   17A is not available for: 
       198082         PHYSICS_3_02   1 2005.05.14   02:13:48     PROD_PHYSICS_CD 
       This is 3 day(s), 5 hours old. 
 
</pre>
<!-- Author: franco spinella -->


<!-- Date: Wed May 18 09:30:24 2005 -->
<pre>Few notes on today daily report: 
- processing to tape for stream A is slightly older then 5 days 
- DQM histrogram web page link is dead 
 
OFFLINE REPORT: Wed May 18 07:48:10 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198116  2005.05.15 18:39:56  (2 day(s), 13 hours ago) 
    Stream B: 197716  2005.05.08 20:45:02  (9 day(s), 11 hours ago) 
    Stream G: 197716  2005.05.08 20:45:02  (9 day(s), 11 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 197990  2005.05.12 17:32:58  (5 day(s), 14 hours ago) 
    Stream B: 197644  2005.05.06 22:32:55  (11 day(s), 9 hours ago) 
    Stream G: 197644  2005.05.06 22:32:55  (11 day(s), 9 hours ago) 
 
3) CALIBRATIONS 
  F 10A is not available for: 
       198158 PHYSICS_TEST_HIGHLUM_3_03   2 2005.05.18   00:49:17     PROD_PHYSI 
       This is 6 hours old. 
   11A is not available for: 
       198117         PHYSICS_3_02   1 2005.05.15   20:00:14     PROD_PHYSICS_CD 
       This is 2 day(s), 11 hours old. 
   17A is not available for: 
       198117         PHYSICS_3_02   1 2005.05.15   20:00:14     PROD_PHYSICS_CD 
       This is 2 day(s), 11 hours old. 
 
4) DB MONITORING 
 
 
5) DH MONITORING 
 
 
6) CAF MONITORING 
 
 
7) DQM MONITORING 
 
 
</pre>
<!-- Author: Franco Spinella -->


<!-- Date: Thu May 19 07:54:05 2005 -->
<pre>daily report offline CO, thursday 19 may 2005 
 
OFFLINE REPORT: Thu May 19 07:52:37 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198152  2005.05.16 23:42:17  (2 day(s), 8 hours ago) 
    Stream B: 197716  2005.05.08 20:45:02  (10 day(s), 11 hours ago) 
    Stream G: 197716  2005.05.08 20:45:02  (10 day(s), 11 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198116  2005.05.15 18:39:56  (3 day(s), 13 hours ago) 
    Stream B: 197644  2005.05.06 22:32:55  (12 day(s), 9 hours ago) 
    Stream G: 197644  2005.05.06 22:32:55  (12 day(s), 9 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       198154         PHYSICS_3_02   1 2005.05.17   01:31:25     PROD_PHYSICS_CD 
       This is 2 day(s), 6 hours old. 
   17A is not available for: 
       198154         PHYSICS_3_02   1 2005.05.17   01:31:25     PROD_PHYSICS_CD 
       This is 2 day(s), 6 hours old. 
 
4) DB MONITORING 
 
 
5) DH MONITORING 
 
 
6) CAF MONITORING 
 
 
7) DQM MONITORING 
</pre>
<!-- Author: franco spinella -->


<!-- Date: Fri May 20 08:21:34 2005 -->
<pre>report for the 8 am meeting 20-05-2005 
 
everything is ok 
 
OFFLINE REPORT: Fri May 20 08:16:17 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198152  2005.05.16 23:42:17  (3 day(s), 8 hours ago) 
    Stream B: 197716  2005.05.08 20:45:02  (11 day(s), 11 hours ago) 
    Stream G: 197716  2005.05.08 20:45:02  (11 day(s), 11 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198116  2005.05.15 18:39:56  (4 day(s), 13 hours ago) 
    Stream B: 197644  2005.05.06 22:32:55  (13 day(s), 9 hours ago) 
    Stream G: 197644  2005.05.06 22:32:55  (13 day(s), 9 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       198340         PHYSICS_3_02   1 2005.05.20   02:26:45     PROD_PHYSICS_CD 
       This is 5 hours old. 
   11A is not available for: 
       198154         PHYSICS_3_02   1 2005.05.17   01:31:25     PROD_PHYSICS_CD 
       This is 3 day(s), 6 hours old. 
   17A is not available for: 
       198154         PHYSICS_3_02   1 2005.05.17   01:31:25     PROD_PHYSICS_CD 
       This is 3 day(s), 6 hours old. 
</pre>
<!-- Author: franco spinella -->


<!-- Date: Fri May 20 08:46:07 2005 -->
<pre>the beam job status says that run number 198159 created 18-05 has SAM dataset as "FILE NOT READY" 
but calibrations are available (only 5 hours old) 
 
Experts have been emaile 
offline CO</pre>
<!-- Author: franco spinella -->


<!-- Date: Fri May 20 16:02:59 2005 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Found a user with 26 inactive connections, all with more than 100 seconds idle time. Here is a sample line: 
 
CDF_READER 	wn-05-01-30-a.cr.cnaf.infn.it 	29512 	ceballos 	./exe/TRGSim++_5.3.4frontier@wn-05-01-30-a.cr.cn 	2005-05-20 15:32:19.0 	48:8 	INACTIVE 	1,240 
 
User has been notified, along with recommended recipients. 
 
</pre></td></tr></td></table>
<!-- Author: Austin Napier -->


<!-- Date: Fri May 20 16:09:19 2005 -->
<pre>The cdfstrm1 usage table for the "last period" cannot be accessed 
for the past several hours. Earlier usage tables are O.K. Can this be corrected? (E-mail sent earlier)</pre>
<!-- Author: Austin Napier -->


<!-- Date: Sat May 21 10:43:00 2005 -->
<pre>Morning report: 
   OFFLINE REPORT: Sat May 21 09:08:23 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198152  2005.05.16 23:42:17  (4 day(s), 9 hours ago) 
    Stream B: 197716  2005.05.08 20:45:02  (12 day(s), 12 hours ago) 
    Stream G: 197716  2005.05.08 20:45:02  (12 day(s), 12 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198116  2005.05.15 18:39:56  (5 day(s), 14 hours ago) 
    Stream B: 197644  2005.05.06 22:32:55  (14 day(s), 10 hours ago) 
    Stream G: 197644  2005.05.06 22:32:55  (14 day(s), 10 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       198340         PHYSICS_3_02   1 2005.05.20   02:26:45     PROD_PHYSICS_CD 
       This is 1 day(s), 6 hours old. 
   11A is not available for: 
       198154         PHYSICS_3_02   1 2005.05.17   01:31:25     PROD_PHYSICS_CD 
       This is 4 day(s), 7 hours old. 
   17A is not available for: 
       198154         PHYSICS_3_02   1 2005.05.17   01:31:25     PROD_PHYSICS_CD 
       This is 4 day(s), 7 hours old. 
 
4) DB MONITORING 
                     OK 
 
5) DH MONITORING 
                     OK 
 
6) CAF MONITORING 
 
                     OK 
7) DQM MONITORING 
 
                     OK 
 
</pre>
<!-- Author: Austin Napier -->


<!-- Date: Sat May 21 12:09:40 2005 -->
<pre>All DCache status monitors are normal.  However, when I try 
the link to get the DCA System Status, there is no response.   
Is there a problem with the cellInfo process on cdfdca?  </pre>
<!-- Author: Austin Napier -->


<!-- Date: Sat May 21 14:07:28 2005 -->
<pre>The beamline production page shows that runs 198159, 198162, 
198206, 198207, and 198230 are more than two days old and 
have 10A available.  </pre>
<!-- Author: Austin Napier -->


<!-- Date: Sun May 22 08:24:26 2005 -->
<pre>   OFFLINE REPORT: Sun May 22 08:01:02 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198152  2005.05.16 23:42:17  (5 day(s), 8 hours ago) 
    Stream B: 197716  2005.05.08 20:45:02  (13 day(s), 11 hours ago) 
    Stream G: 197716  2005.05.08 20:45:02  (13 day(s), 11 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198116  2005.05.15 18:39:56  (6 day(s), 13 hours ago) 
    Stream B: 197644  2005.05.06 22:32:55  (15 day(s), 9 hours ago) 
    Stream G: 197644  2005.05.06 22:32:55  (15 day(s), 9 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       198340         PHYSICS_3_02   1 2005.05.20   02:26:45     PROD_PHYSICS_CD 
       This is 2 day(s), 5 hours old. 
   11A is not available for: 
       198154         PHYSICS_3_02   1 2005.05.17   01:31:25     PROD_PHYSICS_CD 
       This is 5 day(s), 6 hours old. 
   17A is not available for: 
       198154         PHYSICS_3_02   1 2005.05.17   01:31:25     PROD_PHYSICS_CD 
       This is 5 day(s), 6 hours old. 
 
4) DB MONITORING 
                  OK 
 
5) DH MONITORING 
                  OK 
 
6) CAF MONITORING 
                  OK                   
 
7) DQM MONITORING 
                  Page is updated/Nothing processed 
</pre>
<!-- Author: Austin Napier -->


<!-- Date: Sun May 22 08:34:46 2005 -->
<pre>On Sat, 21 May 2005, cdfopr wrote: 
 
&gt;&gt; It looks like runs 198159, 198162, 198206, 198207, and 198230 are more than 2 days old 
&gt;&gt; and have 10A available.  All else seems O.K. 
 
On Sat, 21 May 2005, Mircea wrote: 
 
Let's wait by Monday, as we are working hard to reprocess all the 
post-shutdown data with BeamExe. This puts a heavy load on the farms. 
</pre>
<!-- Author: Austin Napier -->


<!-- Date: Mon May 23 08:28:35 2005 -->
<pre>   OFFLINE REPORT: Mon May 23 07:45:20 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198152  2005.05.16 23:42:17  (6 day(s), 8 hours ago) 
    Stream B: 197716  2005.05.08 20:45:02  (14 day(s), 11 hours ago) 
    Stream G: 197716  2005.05.08 20:45:02  (14 day(s), 11 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198116  2005.05.15 18:39:56  (7 day(s), 13 hours ago) 
    Stream B: 197644  2005.05.06 22:32:55  (16 day(s), 9 hours ago) 
    Stream G: 197644  2005.05.06 22:32:55  (16 day(s), 9 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       198340         PHYSICS_3_02   1 2005.05.20   02:26:45     PROD_PHYSICS_CD 
       This is 3 day(s), 5 hours old. 
   11A is not available for: 
       198154         PHYSICS_3_02   1 2005.05.17   01:31:25     PROD_PHYSICS_CD 
       This is 6 day(s), 6 hours old. 
   17A is not available for: 
       198154         PHYSICS_3_02   1 2005.05.17   01:31:25     PROD_PHYSICS_CD 
       This is 6 day(s), 6 hours old. 
 
4) DB MONITORING 
                       OK 
 
5) DH MONITORING 
                       OK 
 
6) CAF MONITORING 
                       OK 
 
7) DQM MONITORING 
                       Page is up to date/No Histograms 
</pre>
<!-- Author: Austin Napier -->


<!-- Date: Mon May 23 08:35:25 2005 -->
<pre>Following notes are from Mircea, reported at 8 am meeting: 
 
SVX calibrations seem to be a bit late. I sent an email to Jason. 
 
Also BeamExe jobs are a bit delayed, but this is because we are 
processing all the postshutdown data to get new beamlines. 
 
Also Stream A processing on the SAM Farm is going well, last run 
processed and on tape is 198152 (so we are within 7 days from the moment data is taken, which is very well given that we moved to SAM Farm recently and there might be failure modes which are not caught). I am still waiting for a simple web page from the SAM Farm experts, so I can link it from the operations web page. 
 
Over this weekend we have been working hard to get the final 
calibrations validated and also new beamlines, so we can start processing the new data this week. 
 
Also in the morning I am going to meet with Aafke to start 
processing the data from April till May 22nd for calibrators. 
 
</pre>
<!-- Author: Austin Napier -->


<!-- Date: Mon May 23 15:10:53 2005 -->
<pre> 
The CAF monitoring page shows a big increase in the 
number of waiting sections (light blue 
region from about 2K to 7.5K) for the past half hour. 
 
CondorCAF shows the same behavior, with an increase in the 
number of waiting sections from around 1K to 6.5K in the 
past half-hour. 
 
This may be due to scheduler overload.</pre>
<!-- Author: Austin Napier -->


<!-- Date: Tue May 24 08:28:15 2005 -->
<pre>   OFFLINE REPORT: Tue May 24 07:54:21 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198152  2005.05.16 23:42:17  (7 day(s), 8 hours ago) 
    Stream B: 197716  2005.05.08 20:45:02  (15 day(s), 11 hours ago) 
    Stream G: 197716  2005.05.08 20:45:02  (15 day(s), 11 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198116  2005.05.15 18:39:56  (8 day(s), 13 hours ago) 
    Stream B: 197644  2005.05.06 22:32:55  (17 day(s), 9 hours ago) 
    Stream G: 197644  2005.05.06 22:32:55  (17 day(s), 9 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       198340         PHYSICS_3_02   1 2005.05.20   02:26:45     PROD_PHYSICS_CD 
       This is 4 day(s), 5 hours old. 
   11A is not available for: 
       198154         PHYSICS_3_02   1 2005.05.17   01:31:25     PROD_PHYSICS_CD 
       This is 7 day(s), 6 hours old. 
   17A is not available for: 
       198154         PHYSICS_3_02   1 2005.05.17   01:31:25     PROD_PHYSICS_CD 
       This is 7 day(s), 6 hours old. 
 
4) DB MONITORING 
                    OK 
 
5) DH MONITORING 
                    OK 
 
6) CAF MONITORING 
                    OK (server repaired yesterday ~1:30 pm/ 
                        lots of held jobs released) 
7) DQM MONITORING 
                    Page up-to-date/No histograms 
 
</pre>
<!-- Author: Austin Napier -->


<!-- Date: Tue May 24 08:43:18 2005 -->
<pre>Sent E-mail to experts about calibrations: 
 
10A is not available for run 198340  (more than 4 days old) 
11A is not available for run 198154  (more than 7 days old) 
 
</pre>
<!-- Author: Austin Napier -->


<!-- Date: Tue May 24 11:21:50 2005 -->
<pre>The Oracle CPU Usage Plot has been flat since 8:30 am, but the 
Connections Plot shows continued usage.  E-mail has been sent 
to Petar and Susana to ask if this is a problem.</pre>
<!-- Author: Austin Napier -->


<!-- Date: Tue May 24 11:25:30 2005 -->
<pre>Ben reports that the calibration problem has been fixed. 
(Both 10A and 11A)</pre>
<!-- Author: Austin Napier -->


<!-- Date: Wed May 25 08:49:29 2005 -->
<pre>   OFFLINE REPORT: Wed May 25 08:21:12 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198162  2005.05.18 05:21:39  (7 day(s), 2 hours ago) 
    Stream B: 197716    (16 Day(s), 11 hours ago) 
    Stream G: 197716    (16 Day(s), 11 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198116  2005.05.15 18:39:56  (9 day(s), 13 hours ago) 
    Stream B: 197644    (18 Day(s), 9 hours ago) 
    Stream G: 197644    (18 Day(s), 9 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       198557         PHYSICS_3_02   1 2005.05.24   19:55:58     PROD_PHYSICS_CD 
       This is 12 hours old. 
   11A is not available for: 
       198206 PHYSICS_HIGHLUM_3_02   1 2005.05.18   14:33:34     PROD_PHYSICS_CD 
       This is 6 day(s), 17 hours old. 
   17A is not available for: 
       198206 PHYSICS_HIGHLUM_3_02   1 2005.05.18   14:33:34     PROD_PHYSICS_CD 
       This is 6 day(s), 17 hours old. 
 
4) DB MONITORING 
                    OK 
 
5) DH MONITORING 
                    OK                    
 
6) CAF MONITORING 
                    OK 
 
7) DQM MONITORING 
 
                    OK 
</pre>
<!-- Author: Austin Napier -->


<!-- Date: Wed May 25 12:44:25 2005 -->
<pre>In today's 8am report, the dates and times for the Stream B and G data are missing.  The files read by the makeReport script no longer contain the run numbers needed to extract this information. 
Perhaps there is a limit to the number of lines stored in these files, since the Stream B and G processing are more than 16 days behind?  E-mail sent to Mircea.</pre>
<!-- Author: Austin Napier -->


<!-- Date: Thu May 26 08:50:11 2005 -->
<pre>   OFFLINE REPORT: Thu May 26 08:01:48 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198162  2005.05.18 05:21:39  (8 day(s), 2 hours ago) 
    Stream B: 197716  2005.05.08 20:45:02  (17 day(s), 11 hours ago) 
    Stream G: 197716  2005.05.08 20:45:02  (17 day(s), 11 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198116  2005.05.15 18:39:56  (10 day(s), 13 hours ago) 
    Stream B: 197644  2005.05.06 22:32:55  (19 day(s), 9 hours ago) 
    Stream G: 197644  2005.05.06 22:32:55  (19 day(s), 9 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       198206 PHYSICS_HIGHLUM_3_02   1 2005.05.18   14:33:34     PROD_PHYSICS_CD 
       This is 7 day(s), 17 hours old. 
   17A is not available for: 
       198206 PHYSICS_HIGHLUM_3_02   1 2005.05.18   14:33:34     PROD_PHYSICS_CD 
       This is 7 day(s), 17 hours old. 
 
4) DB MONITORING 
                   OK 
 
5) DH MONITORING 
                   OK/ Predator status up-to-date, 
                       but no tapes in last two days? 
6) CAF MONITORING 
                   OK 
 
7) DQM MONITORING 
                   OK/  Histograms look O.K. for 
                        runs 198379, 198416, 198514. 
 
</pre>
<!-- Author: Austin Napier -->


<!-- Date: Fri May 27 09:46:13 2005 -->
<pre>Change of Offline CO.  New CO is Peter Renton.</pre>
<!-- Author: Austin Napier -->


<!-- Date: Fri May 27 13:54:23 2005 -->
<pre>  OFFLINE REPORT: Fri May 27 08:03:14 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198378  2005.05.20 14:22:43  (6 day(s), 17 hours ago) 
 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198116  2005.05.15 18:39:56  (11 day(s), 13 hours ago) 
  
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       198379         PHYSICS_3_02   1 2005.05.20   15:27:12     PROD_PHYSICS_CD 
       This is 6 day(s), 16 hours old. 
   17A is not available for: 
       198379         PHYSICS_3_02   1 2005.05.20   15:27:12     PROD_PHYSICS_CD 
       This is 6 day(s), 16 hours old. 
 
4) DB MONITORING 
                 OK 
 
5) DH MONITORING 
                 OK 
 
6) CAF MONITORING 
                 OK 
 
7) DQM MONITORING 
                 OK</pre>
<!-- Author: Pete Renton -->


<!-- Date: Sat May 28 08:54:06 2005 -->
<pre>   OFFLINE REPORT: Sat May 28 08:06:43 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198513  2005.05.23 12:51:53  (4 day(s), 19 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (9 day(s), 17 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       198514         PHYSICS_3_02   1 2005.05.23   13:57:03     PROD_PHYSICS_CD 
       This is 4 day(s), 18 hours old. 
   17A is not available for: 
       198514         PHYSICS_3_02   1 2005.05.23   13:57:03     PROD_PHYSICS_CD 
       This is 4 day(s), 18 hours old. 
 
4) DB MONITORING 
                  OK 
 
5) DH MONITORING 
                  OK 
 
6) CAF MONITORING 
                  OK 
 
7) DQM MONITORING 
                  no runs available to check (message sent to experts) 
</pre>
<!-- Author: Pete Renton -->


<!-- Date: Sat May 28 14:18:14 2005 -->
<pre>BeamExe monitoring not working - expert informed</pre>
<!-- Author: Pete Renton -->


<!-- Date: Sat May 28 14:54:23 2005 -->
<pre>BeamExe monitoring problem lead to discovery that SAM farm had stopped at some time after midnight. Suen investigating.</pre>
<!-- Author: Pete Renton -->


<!-- Date: Sun May 29 08:38:00 2005 -->
<pre>   OFFLINE REPORT: Sun May 29 08:04:04 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198513  2005.05.23 12:51:53  (5 day(s), 19 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (10 day(s), 17 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       198514         PHYSICS_3_02   1 2005.05.23   13:57:03     PROD_PHYSICS_CD 
       This is 5 day(s), 18 hours old. 
   17A is not available for: 
       198514         PHYSICS_3_02   1 2005.05.23   13:57:03     PROD_PHYSICS_CD 
       This is 5 day(s), 18 hours old. 
 
 BeamExe now OK - message from Suen that it is OK from run 198583 and they are trying to recover all records from SAM. 
 
4) DB MONITORING 
                user danielw has large number of connections       (peak more than 4000) since 4am. Petar and Susana informed. 
 
5) DH MONITORING 
                     OK 
 
6) CAF MONITORING 
                     OK 
 
7) DQM MONITORING 
                    no runs available 
</pre>
<!-- Author: Pete Renton -->


<!-- Date: Mon May 30 08:43:02 2005 -->
<pre>When doing the checks at 8am I noticed a lot of problems - no access to dbsmon, problems with last 2 raw data tapes written, problems with Monitor DCACHE doors & ping (not working). The SciCo informed me that there is a general network problem which has been reported. The e-mail system is also not functionning so I cannot report this to the experts directly!</pre>
<!-- Author: Pete Renton -->


<!-- Date: Mon May 30 09:35:16 2005 -->
<pre>Network connection seems to have been restored.</pre>
<!-- Author: Pete Renton -->


<!-- Date: Mon May 30 11:54:31 2005 -->
<pre>   OFFLINE REPORT: Mon May 30 08:05:26 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198513  2005.05.23 12:51:53  (6 day(s), 19 hours ago) 
 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (11 day(s), 17 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       198710         PHYSICS_3_02   1 2005.05.29   05:49:03     PROD_PHYSICS_CD 
       This is 1 day(s), 2 hours old. 
   11A is not available for: 
       198514         PHYSICS_3_02   1 2005.05.23   13:57:03     PROD_PHYSICS_CD 
       This is 6 day(s), 18 hours old. 
   17A is not available for: 
       198514         PHYSICS_3_02   1 2005.05.23   13:57:03     PROD_PHYSICS_CD 
       This is 6 day(s), 18 hours old. 
 
      
4) DB MONITORING 
                    OK 
 
5) DH MONITORING 
                 4 files between 6 and 9.30 am (when network problems) failed to copy. First is 
2005-May-30 07:13:02 encp FAILED TO COPY FILESET CB6924.4 
source directory: /cdf/b0/disk8/stager/CB6924.4 
source files: cr030837.0439phys cr030837.0454phys cr030837.0470phys cr030853.0001phys cr030853.0021phys cr030853.003aphys cr030853.0054phys cr030853.006ephys cr030853.0087phys 
destination dir: /pnfs/cdfen/filesets/CB/CB69/CB6924/CB6924.4 
The rest of the filesets will not be encpied: CB6924.4  
 
and last is  
2005-May-30 09:09:47 encp FAILED TO COPY FILESET CB6927.0 
source directory: /cdf/b0/disk9/stager/CB6927.0 
source files: hr030837.03a7phys hr030837.03b5phys hr030837.03c3phys hr030837.03d0phys hr030837.03dephys hr030837.03ecphys hr030837.03faphys hr030837.0408phys hr030837.0416phys 
destination dir: /pnfs/cdfen/filesets/CB/CB69/CB6927/CB6927.0 
The rest of the filesets will not be encpied: CB6927.0 CB6927.1 CB6927.2 CB6927.3 CB6927.4  
 
6) CAF MONITORING 
                        seems OK 
 
7) DQM MONITORING 
                        no files to inspect 
</pre>
<!-- Author: Pete Renton -->


<!-- Date: Mon May 30 15:39:43 2005 -->
<pre>Message from Krzysztof Genser that 4 failed copies are on tape and catalogued now.</pre>
<!-- Author: Pete Renton -->


<!-- Date: Tue May 31 10:55:41 2005 -->
<pre>   OFFLINE REPORT: Tue May 31 07:58:49 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198513  2005.05.23 12:51:53  (7 day(s), 19 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (12 day(s), 17 hours ago) 
 
 
3) CALIBRATIONS 
   10A is not available for: 
       198710         PHYSICS_3_02   1 2005.05.29   05:49:03     PROD_PHYSICS_CD 
       This is 2 day(s), 2 hours old. 
   11A is not available for: 
       198514         PHYSICS_3_02   1 2005.05.23   13:57:03     PROD_PHYSICS_CD 
       This is 7 day(s), 18 hours old. 
   17A is not available for: 
       198514         PHYSICS_3_02   1 2005.05.23   13:57:03     PROD_PHYSICS_CD 
       This is 7 day(s), 18 hours old. 
 
**  all these are beyond limits in instructions, so reported to experts 
 
4) DB MONITORING 
                  OK 
 
5) DH MONITORING 
                  OK 
 
6) CAF MONITORING 
                  OK 
 
7) DQM MONITORING 
                  no files to check 
</pre>
<!-- Author: Pete Renton -->


<!-- Date: Wed Jun 1 08:37:08 2005 -->
<pre>   OFFLINE REPORT: Wed Jun 1 07:58:37 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198513  2005.05.23 12:51:53  (8 day(s), 19 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (13 day(s), 17 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       198710         PHYSICS_3_02   1 2005.05.29   05:49:03     PROD_PHYSICS_CD 
       This is 3 day(s), 2 hours old. 
   11A is not available for: 
       198514         PHYSICS_3_02   1 2005.05.23   13:57:03     PROD_PHYSICS_CD 
       This is 8 day(s), 18 hours old. 
   17A is not available for: 
       198514         PHYSICS_3_02   1 2005.05.23   13:57:03     PROD_PHYSICS_CD 
       This is 8 day(s), 18 hours old. 
 
4) DB MONITORING 
        user tomura 3500-4000 connections for 5 periods upto 2hr 
 
5) DH MONITORING 
                     OK 
 
6) CAF MONITORING 
                     OK 
 
7) DQM MONITORING 
                   no files to inspect 
</pre>
<!-- Author: Pete Renton -->


<!-- Date: Thu Jun 2 08:42:57 2005 -->
<pre>   OFFLINE REPORT: Thu Jun 2 08:01:22 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198513  2005.05.23 12:51:53  (9 day(s), 19 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (14 day(s), 17 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       198710         PHYSICS_3_02   1 2005.05.29   05:49:03     PROD_PHYSICS_CD 
       This is 4 day(s), 2 hours old. 
   11A is not available for: 
       198514         PHYSICS_3_02   1 2005.05.23   13:57:03     PROD_PHYSICS_CD 
       This is 9 day(s), 18 hours old. 
   17A is not available for: 
       198514         PHYSICS_3_02   1 2005.05.23   13:57:03     PROD_PHYSICS_CD 
       This is 9 day(s), 18 hours old. 
 
4) DB MONITORING 
                   OK 
 
5) DH MONITORING 
                   OK 
 
6) CAF MONITORING 
                   OK 
 
7) DQM MONITORING 
                  no files to inspect 
 
 
</pre>
<!-- Author: Pete Renton -->


<!-- Date: Thu Jun 2 10:52:21 2005 -->
<pre>CALIBRATIONS   10A is now up to date.</pre>
<!-- Author: Pete Renton -->


<!-- Date: Fri Jun 3 08:48:03 2005 -->
<pre>   OFFLINE REPORT: Fri Jun 3 07:56:18 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198782  2005.05.31 07:21:17  (3 day(s), 0 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (15 day(s), 17 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       198843         PHYSICS_3_02   1 2005.05.31   20:59:38     PROD_PHYSICS_CD 
       This is 2 day(s), 10 hours old. 
   17A is not available for: 
       198843         PHYSICS_3_02   1 2005.05.31   20:59:38     PROD_PHYSICS_CD 
       This is 2 day(s), 10 hours old. 
 
4) DB MONITORING 
                     OK 
 
5) DH MONITORING 
                     OK 
 
6) CAF MONITORING 
                     OK 
 
7) DQM MONITORING 
                   no files to inspect</pre>
<!-- Author: Pete Renton -->


<!-- Date: Fri Jun 3 10:52:32 2005 -->
<pre>Aidan will check on status of DQM monitoring website.</pre>
<!-- Author: Alice Abulencia -->


<!-- Date: Fri Jun 3 13:58:32 2005 -->
<pre>CAF "all waiting sections" increased significantly today, but not in list of problems so not allerting CAF experts. 
 
Monitoring connections for user danielw, will email experts if still high after 2 pm.</pre>
<!-- Author: Alice Abulencia -->


<!-- Date: Fri Jun 3 16:02:52 2005 -->
<pre>Emailed experts about high connections for danielw and salaman.  All else OK for end of shift.</pre>
<!-- Author: Alice Abulencia -->


<!-- Date: Sat Jun 4 08:30:25 2005 -->
<pre>   OFFLINE REPORT: Sat Jun 4 08:11:41 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198843  2005.05.31 20:59:38  (3 day(s), 11 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (16 day(s), 17 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       198881 PHYSICS_HIGHLUM_3_03   3 2005.06.02   06:42:37     PROD_PHYSICS_CD 
       This is 2 day(s), 1 hours old. 
   17A is not available for: 
       198881 PHYSICS_HIGHLUM_3_03   3 2005.06.02   06:42:37     PROD_PHYSICS_CD 
       This is 2 day(s), 1 hours old. 
 
4) DB MONITORING 
   OK, heavy connections from yesterday have stopped 
 
5) DH MONITORING 
   Predator status strange.  0 data files added since 10pm   yesterday, emailed Mircea.   
 
6) CAF MONITORING 
   OK 
 
7) DQM MONITORING 
   Website still not working. Aidan doesn't know why.</pre>
<!-- Author: Alice Abulencia -->


<!-- Date: Sat Jun 4 09:02:21 2005 -->
<pre>DH disk space usage check shows strange behavior.  SciCo is paging the CSL expert to determine if it's really bad or not.</pre>
<!-- Author: Alice Abulencia -->


<!-- Date: Sat Jun 4 12:34:39 2005 -->
<pre>Disk usage OK, not taking data so disks-&gt;0 but all may not be exactly at 0.  Predator status 0 for same reason. 
 
Noticed fcdfdata057 disk full on SAMfarm. Experts contacted and they are working on it.</pre>
<!-- Author: Alice Abulencia -->


<!-- Date: Sun Jun 5 08:23:57 2005 -->
<pre> OFFLINE REPORT: Sun Jun 5 08:09:11 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198843  2005.05.31 20:59:38  (4 day(s), 11 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (17 day(s), 17 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       198881 PHYSICS_HIGHLUM_3_03   3 2005.06.02   06:42:37     PROD_PHYSICS_CD 
       This is 3 day(s), 1 hours old. 
   17A is not available for: 
       198881 PHYSICS_HIGHLUM_3_03   3 2005.06.02   06:42:37     PROD_PHYSICS_CD 
       This is 3 day(s), 1 hours old. 
 
       Beam Exe says file not ready, created 6/3.  Will email Aidan about this. 
 
4) DB MONITORING 
      OK. No heavy connections or usage as of morning. 
 
5) DH MONITORING 
     OK. 
    
6) CAF MONITORING 
     Full disk on SAM is taken care of. 
     SAM still has projects for ProductionExe_0f_jphysr at 2005/06/05 08:10:44 in red. 
     Will also tell Aidan about this.CAF and CondorCAF OK. 
 
7) DQM MONITORING 
     Nothing to check.     
 
</pre>
<!-- Author: Alice Abulencia -->


<!-- Date: Sun Jun 5 11:05:19 2005 -->
<pre>SAMFarm disk 57 is full again as of 10:53.  Aidan says nothing to be done and experts are working on it. 
 
Also Re: BeamExe, not worrying about that yet.</pre>
<!-- Author: Alice Abulencia -->


<!-- Date: Sun Jun 5 13:35:33 2005 -->
<pre>In looking at crashes of ProductionExe on jphysr there seem to be two common CDF code issues: 
 
1) and invalid write in TowerKey.hh line 112 
2)Problem with use of vector in structureGeometry() function defined in KalDetector.cc 
 
Also some possible memory leaks, still looking at these</pre>
<!-- Author: Alice Abulencia -->
<!-- Comment: Mon Jun  6 10;04;51 comment by...Alice Abulencia -->
<pre>Valgrind logs for these:

1)
==6267== Invalid write of size 1
==6267==    at 0x8E61408: CalorTdcTower::fill(CalorTdcTower const&) (TowerKey.hh:112)
==6267==    by 0x8E614DC: CalorTdcTower::CalorTdcTower(CalorTdcTower const&) (CalorTdcTower.cc:385)
==6267==    by 0x8636770: std::vector&lt;CalorTdcTower, std::allocator&lt;CalorTdcTower&gt; &gt;::_M_insert_aux(__gnu_cxx::__normal_iterator&lt;CalorTdcTower*, std::vector&lt;CalorTdcTower, std::allocator&lt;CalorTdcTower&gt; &gt; &gt;, CalorTdcTower const&) (new:92)
==6267==    by 0x86332D4: TimingCollMaker::event(EventRecord*) (stl_vector.h:330)
==6267==    by 0x8BC5AC7: APPSequence::event(EventRecord*) (APPSequence.cc:211)
==6267==    by 0x8BA9113: AppFramework::event(EventRecord*) (APPFramework.cc:456)
==6267==    by 0x8BA7A5A: AppFramework::continueHandler(int) (APPFramework.cc:884)
==6267==    by 0x8BD6F5B: APPEventsCommand::continueHandler(int, char**) (APPEventsCommand.cc:168)
==6267==    by 0x8BD6DEC: APPEventsCommand::handle(int, char**) (APPEventsCommand.cc:97)
==6267==    by 0x40CA20FF: TclInvokeStringCommand (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CD91B8: EvalObjv (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CD9886: Tcl_EvalEx (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CD9B74: Tcl_Eval (in /usr/lib/libtcl.so.0)
==6267==    by 0x982404D: ftclAliasExecute (ftclAlias.c:98)
==6267==    by 0x40CA20FF: TclInvokeStringCommand (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CBCB34: TclExecuteByteCode (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CA2B09: Tcl_EvalObjEx (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CA92EB: Tcl_IfObjCmd (in /usr/lib/libtcl.so.0)
==6267==    Address 0x77E27E40 is 0 bytes after a block of size 416 alloc'd
==6267==    at 0x40166CA4: __builtin_new (vg_clientfuncs.c:129)
==6267==    by 0x40166CE2: operator new(unsigned) (vg_clientfuncs.c:142)
==6267==    by 0x86365DA: std::vector&lt;CalorTdcTower, std::allocator&lt;CalorTdcTower&gt; &gt;::_M_insert_aux(__gnu_cxx::__normal_iterator&lt;CalorTdcTower*, std::vector&lt;CalorTdcTower, std::allocator&lt;CalorTdcTower&gt; &gt; &gt;, CalorTdcTower const&) (new_allocator.h:81)
==6267==    by 0x86332D4: TimingCollMaker::event(EventRecord*) (stl_vector.h:330)
==6267==    by 0x8BC5AC7: APPSequence::event(EventRecord*) (APPSequence.cc:211)
==6267==    by 0x8BA9113: AppFramework::event(EventRecord*) (APPFramework.cc:456)
==6267==    by 0x8BA7A5A: AppFramework::continueHandler(int) (APPFramework.cc:884)
==6267==    by 0x8BD6F5B: APPEventsCommand::continueHandler(int, char**) (APPEventsCommand.cc:168)
==6267==    by 0x8BD6DEC: APPEventsCommand::handle(int, char**) (APPEventsCommand.cc:97)
==6267==    by 0x40CA20FF: TclInvokeStringCommand (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CD91B8: EvalObjv (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CD9886: Tcl_EvalEx (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CD9B74: Tcl_Eval (in /usr/lib/libtcl.so.0)
==6267==    by 0x982404D: ftclAliasExecute (ftclAlias.c:98)
==6267==    by 0x40CA20FF: TclInvokeStringCommand (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CBCB34: TclExecuteByteCode (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CA2B09: Tcl_EvalObjEx (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CA92EB: Tcl_IfObjCmd (in /usr/lib/libtcl.so.0)

and an invalid read later:
==6267== Invalid read of size 1
==6267==    at 0x8E613FA: CalorTdcTower::fill(CalorTdcTower const&) (TowerKey.hh:112)
==6267==    by 0x8E614DC: CalorTdcTower::CalorTdcTower(CalorTdcTower const&) (CalorTdcTower.cc:385)
==6267==    by 0x86366EB: std::vector&lt;CalorTdcTower, std::allocator&lt;CalorTdcTower&gt; &gt;::_M_insert_aux(__gnu_cxx::__normal_iterator&lt;CalorTdcTower*, std::vector&lt;CalorTdcTower, std::allocator&lt;CalorTdcTower&gt; &gt; &gt;, CalorTdcTower const&) (new:92)
==6267==    by 0x86332D4: TimingCollMaker::event(EventRecord*) (stl_vector.h:330)
==6267==    by 0x8BC5AC7: APPSequence::event(EventRecord*) (APPSequence.cc:211)
==6267==    by 0x8BA9113: AppFramework::event(EventRecord*) (APPFramework.cc:456)
==6267==    by 0x8BA7A5A: AppFramework::continueHandler(int) (APPFramework.cc:884)
==6267==    by 0x8BD6F5B: APPEventsCommand::continueHandler(int, char**) (APPEventsCommand.cc:168)
==6267==    by 0x8BD6DEC: APPEventsCommand::handle(int, char**) (APPEventsCommand.cc:97)
==6267==    by 0x40CA20FF: TclInvokeStringCommand (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CD91B8: EvalObjv (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CD9886: Tcl_EvalEx (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CD9B74: Tcl_Eval (in /usr/lib/libtcl.so.0)
==6267==    by 0x982404D: ftclAliasExecute (ftclAlias.c:98)
==6267==    by 0x40CA20FF: TclInvokeStringCommand (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CBCB34: TclExecuteByteCode (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CA2B09: Tcl_EvalObjEx (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CA92EB: Tcl_IfObjCmd (in /usr/lib/libtcl.so.0)
==6267==    Address 0x77E27E40 is 0 bytes after a block of size 416 alloc'd
==6267==    at 0x40166CA4: __builtin_new (vg_clientfuncs.c:129)
==6267==    by 0x40166CE2: operator new(unsigned) (vg_clientfuncs.c:142)
==6267==    by 0x86365DA: std::vector&lt;CalorTdcTower, std::allocator&lt;CalorTdcTower&gt; &gt;::_M_insert_aux(__gnu_cxx::__normal_iterator&lt;CalorTdcTower*, std::vector&lt;CalorTdcTower, std::allocator&lt;CalorTdcTower&gt; &gt; &gt;, CalorTdcTower const&) (new_allocator.h:81)
==6267==    by 0x86332D4: TimingCollMaker::event(EventRecord*) (stl_vector.h:330)
==6267==    by 0x8BC5AC7: APPSequence::event(EventRecord*) (APPSequence.cc:211)
==6267==    by 0x8BA9113: AppFramework::event(EventRecord*) (APPFramework.cc:456)
==6267==    by 0x8BA7A5A: AppFramework::continueHandler(int) (APPFramework.cc:884)
==6267==    by 0x8BD6F5B: APPEventsCommand::continueHandler(int, char**) (APPEventsCommand.cc:168)
==6267==    by 0x8BD6DEC: APPEventsCommand::handle(int, char**) (APPEventsCommand.cc:97)
==6267==    by 0x40CA20FF: TclInvokeStringCommand (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CD91B8: EvalObjv (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CD9886: Tcl_EvalEx (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CD9B74: Tcl_Eval (in /usr/lib/libtcl.so.0)
==6267==    by 0x982404D: ftclAliasExecute (ftclAlias.c:98)
==6267==    by 0x40CA20FF: TclInvokeStringCommand (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CBCB34: TclExecuteByteCode (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CA2B09: Tcl_EvalObjEx (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CA92EB: Tcl_IfObjCmd (in /usr/lib/libtcl.so.0)

2)
==6267== Use of uninitialised value of size 8
==6267==    at 0x962D4B6: KalDetector::structureGeometry() (stl_construct.h:81)
==6267==    by 0x962DCD5: KalDetector::prepareGeometry() (KalDetector.cc:729)
==6267==    by 0x8AC914C: KalConverter::get(KalDetector&, bool) (KalConverter_Detector.cc:1468)
==6267==    by 0x8AC3559: KalStrategyBase::KalStrategyBase() (KalStrategyBase.cc:42)
==6267==    by 0x8A8B9DA: PVKalFinder::instance() (PVKalFinder.cc:31)
==6267==    by 0x869D304: PVFinder::beginJob(EventRecord*) (PVFinder.cc:175)
==6267==    by 0x8BADF88: AppFramework::beginJob(EventRecord*) (APPFramework.cc:325)
==6267==    by 0x8BA7BC2: AppFramework::continueHandler(int) (APPFramework.icc:44)
==6267==    by 0x8BD6F5B: APPEventsCommand::continueHandler(int, char**) (APPEventsCommand.cc:168)
==6267==    by 0x8BD6DEC: APPEventsCommand::handle(int, char**) (APPEventsCommand.cc:97)
==6267==    by 0x40CA20FF: TclInvokeStringCommand (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CD91B8: EvalObjv (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CD9886: Tcl_EvalEx (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CD9B74: Tcl_Eval (in /usr/lib/libtcl.so.0)
==6267==    by 0x982404D: ftclAliasExecute (ftclAlias.c:98)
==6267==    by 0x40CA20FF: TclInvokeStringCommand (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CBCB34: TclExecuteByteCode (in /usr/lib/libtcl.so.0)
==6267==    by 0x40CA2B09: Tcl_EvalObjEx (in /usr/lib/libtcl.so.0)
KalConverter::useSiliMap() WARNING: The KalDetector is already read. SiliMap usage flag unchanged.
SiPatternRecModule: using SimpleSiliconIntegrator (default).
KalOIStrategy around
</pre>
<!-- Comment: Mon Jun  6 10;10;23 comment by...Alice Abulencia -->
<pre>If you want to look at logs yourself are here:

cdfopr@fcdflnx3:/cdf/scratch/cdfopr/testRel/6.1.0a/results/ \
ProductionExe-alice-singleevent.00X/ \
ProductionExe-alice-singleevent.00X.log
where X goes from 01-11 (2 was a test saved as ProductionExe-aidian-singleevent.0002)</pre>
<!-- Date: Sun Jun 5 15:55:28 2005 -->
<pre>Emailed user danielw about excessive cpu usage on oracle DB.  Still no word on SAMFarm. 
 
All else OK.</pre>
<!-- Author: Alice Abulencia -->


<!-- Date: Mon Jun 6 08:16:54 2005 -->
<pre>  OFFLINE REPORT: Mon Jun 6 07:47:39 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198882  2005.06.02 07:27:23  (4 day(s), 0 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (18 day(s), 17 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       199028         PHYSICS_3_02   1 2005.06.05   20:54:49     PROD_PHYSICS_CD 
       This is 10 hours old. 
   11A is not available for: 
       198888         PHYSICS_3_02   1 2005.06.03   07:10:34     PROD_PHYSICS_CD 
       This is 3 day(s), 0 hours old. 
   17A is not available for: 
       198888         PHYSICS_3_02   1 2005.06.03   07:10:34     PROD_PHYSICS_CD 
       This is 3 day(s), 0 hours old. 
       
There was an email sent yesterday about the calibrations not being up to date. 
 
4) DB MONITORING 
   OK 
 
5) DH MONITORING 
   OK 
 
6) CAF MONITORING 
    Disk 57 on SAMFarm OK as of 7:50. CAF OK. 
 
7) DQM MONITORING 
   Still no files to check. 
 
</pre>
<!-- Author: Alice Abulencia -->




<!-- Date: Mon Jun 6 10:37:45 2005 -->
<pre>SAMFarm monitoring: 
 
Both of these went red for perhaps 20-30 minutes and then went away, no word yet on what this means: 
 
DBServer at farm1_prd at 2005/06/06  
 
DBSDBServer at farm1_prd3 at 2005/06/06 </pre>
<!-- Author: Alice Abulencia -->
<!-- Comment: Mon Jun  6 12;09;58 comment by...Alice Abulencia -->
<pre>DBServer in red again</pre>

<!-- Comment: Mon Jun  6 13;28;26 comment by...Alice Abulencia -->
<pre>Suen writes:

Hi SAM experts,

Could someone help restart

SAMDbServer.farm1_prd:SAMDbServer  fcdfcaf548.fnal.gov:52380    unknown

It is "RED" on
http://cdfsam-prd.fnal.gov/sam_local/SamAtAGlance/

Production stops while this DB server goes down.
I am sending the request to
CDFSAM-ADMIN &lt;CDFSAM-ADMIN@listserv.fnal.gov&gt;


Thanks, suen
</pre>

<!-- Comment: Mon Jun  6 13;44;21 comment by...Alice Abulencia -->
<pre>Now we also have red:

Optimizer_prd at 2005/06/06 13:35:24

Emailed experts</pre>

<!-- Comment: Mon Jun  6 14;49;58 comment by...Alice Abulencia -->
<pre>Regarding Optimizer_prd:

Hi Alice and all,

I saw Optimizer turned green later.

Farm checking script is less patient and does a time out.
When these service are busy, we see red.
The checking is every 10 min.

When any service is seen read, job submittion is prohibited.

To confirm SAM status, you may go to "SAM at a glance"
and look for them.
http://cdfsam-prd.fnal.gov/sam_local/SamAtAGlance/

Thanks, suen
</pre>


<!-- Comment: Tue Jun  7 08;12;21 comment by...Alice Abulencia -->
<pre>If you get a warning about DBServer again do this:

Hi,
It meant it was not responding. The action to take is to send an e-mail tocdfsam-admin to that effect. The sam shifter would then investigate it andprobably restart the db server as we did later in the day.

Krzysztof
</pre>
<!-- Date: Mon Jun 6 15:42:43 2005 -->
<pre>End of shift note: 
 
There is a really suprising number of high connections on the DB.  I cannot get a hold of Susana or Petar by phone and Petar's officemate says he is out of country.  I emailed cdfdb-support and got a reply from Eric agreeing this was strange.  I have emailed the users directly. 
 
</pre>
<!-- Author: Alice Abulencia -->



<!-- Date: Tue Jun 7 09:13:54 2005 -->
<pre>   OFFLINE REPORT: Tue Jun 7 07:51:49 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 198888  2005.06.03 07:10:34  (4 day(s), 0 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (19 day(s), 17 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199025         PHYSICS_3_02   1 2005.06.04   17:36:34     PROD_PHYSICS_CD 
       This is 2 day(s), 14 hours old. 
   17A is not available for: 
       199025         PHYSICS_3_02   1 2005.06.04   17:36:34     PROD_PHYSICS_CD 
       This is 2 day(s), 14 hours old. 
 
   BeamExe OK 
 
4) DB MONITORING 
   There are still very high connections from robine, nielsonj, ukyang and danielw.   
   I have emailed all of these users in the past few days, but to little effect. 
   I going to keep trying to talk to petar or susana... 
 
5) DH MONITORING 
   OK 
 
6) CAF MONITORING 
   OK 
 
7) DQM MONITORING 
   Site is back up!   
   Monitoring OK 
</pre>
<!-- Author: Alice Abulencia -->


<!-- Date: Tue Jun 7 15:34:58 2005 -->
<pre>I added SAMFarm to the list of daily jobs on webpage.</pre>
<!-- Author: Alice Abulencia -->


<!-- Date: Wed Jun 8 08:34:05 2005 -->
<pre> OFFLINE REPORT: Wed Jun 8 07:52:14 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199028  2005.06.05 20:54:49  (2 day(s), 10 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (20 day(s), 17 hours ago) 
 
   Note: Apparently this number is no longer valid as we are using SAM, don't report it! 
 
3) CALIBRATIONS 
   10A is not available for: 
       199076         PHYSICS_3_02   1 2005.06.07   22:06:25     PROD_PHYSICS_CD 
       This is 9 hours old. 
   11A is not available for: 
       199061         PHYSICS_3_02   1 2005.06.07   06:18:44     PROD_PHYSICS_CD 
       This is 1 day(s), 1 hours old. 
   17A is not available for: 
       199061         PHYSICS_3_02   1 2005.06.07   06:18:44     PROD_PHYSICS_CD 
       This is 1 day(s), 1 hours old. 
 
   BeamExe is OK 
 
4) DB MONITORING 
   OK 
 
5) DH MONITORING 
   OK 
 
6) CAF MONITORING 
   OK 
 
7) DQM MONITORING 
   OK</pre>
<!-- Author: Alice Abulenica -->


<!-- Date: Thu Jun 9 08:46:23 2005 -->
<pre>OFFLINE REPORT: Thu Jun 9 07:46:04 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199042  2005.06.06 11:22:04  (2 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (21 day(s), 17 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199061         PHYSICS_3_02   1 2005.06.07   06:18:44     PROD_PHYSICS_CD 
       This is 2 day(s), 1 hours old. 
   17A is not available for: 
       199061         PHYSICS_3_02   1 2005.06.07   06:18:44     PROD_PHYSICS_CD 
       This is 2 day(s), 1 hours old. 
     
   BeamExe status is close to calling the experts (Aidan), will need to keep an eye on this. 
 
4) DB MONITORING 
   OK 
 
5) DH MONITORING 
   OK 
 
6) CAF MONITORING 
   CAFs OK.  For SAM, there appears to be a memory leak again, I told Aidan and he sent the email. 
 
7) DQM MONITORING 
   There was an empty histogram (Inst. Lum. vs.NVtx Qual 12 in file ZVertex_aexp.ps) for run 198888, which was marked bad.  I emailed the experts about it.</pre>
<!-- Author: Alice Abulencia -->


<!-- Date: Thu Jun 9 10:44:29 2005 -->
<pre>There is now a separte website for SAM monitoring instructions, which is linked off of the daily jobs site.  If you need to edit this file, it is called sam_instructions.html.  As we are still working out the bugs with SAM monitoring, I suspect that this will be needed.  I have not yet figured out how to add it to the tabs on the left side of the screen.  If someone knows how to do this, please do so!</pre>
<!-- Author: Alice Abulencia -->
<!-- Comment: Thu Jun  9 11;49;26 comment by...Alice Abulencia -->
<pre>To add folders, etc. to the left side modify this file:

cdfopr@fcdflnx3:~/www/cdfopr/html/cdfopr_menu_items.js</pre>
<!-- Date: Thu Jun 9 11:06:30 2005 -->
<pre>SAM urgent error: 
 
Message sent to experts at 10:51 
 
Hello, 
 
A recent check shows DBServer at farm1_prd and DBServer at farm1_prd3 red and Sam at a Glance not updated for &gt;37 minutes.  Is a restart necessary? 
____________________________________________________ 
 
Got a reply, they are working on it. 
 
</pre>
<!-- Author: Alice Abulencia -->
<!-- Comment: Thu Jun  9 11;59;09 comment by...Alice Abulencia -->
<pre>Reply:

Hello,

thanks for checking this webpage. I have restarted the naming server and the problem is gone.

Cheers,

Valeria
</pre>

<!-- Comment: Thu Jun  9 12;11;08 comment by...Alice Abulencia -->
<pre>More SAM problems:

Hello,

The following are red:

Station at cdf-fncdfsrv0
fss at cdf-fncdfsrv0
Station at cdf-farm1
fss at cdf-farm1

Sam Admins are aware of database and optimizer problems

Alice, Offline CO
_______________________
Steaan is working on it.</pre>


<!-- Comment: Thu Jun  9 14;58;28 comment by...Alice Abulencia -->
<pre>SAM appears to be healthy again now.</pre>
<!-- Date: Thu Jun 9 14:05:28 2005 -->
<pre>Now there is a CAF problem: large amount of white space on system info plot, caf-admin contacted.</pre>
<!-- Author: Alice Abulencia -->
<!-- Comment: Thu Jun  9 14;24;02 comment by...Alice Abulencia -->
<pre>I talked to Elliot (you can't email caf-admin). He thinks it is because people are submitting really small jobs.  He will change to page to reflect someone that we should email.</pre>



<!-- Date: Fri Jun 10 08:46:18 2005 -->
<pre> OFFLINE REPORT: Fri Jun 10 07:50:24 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199061  2005.06.07 06:18:44  (3 day(s), 1 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (22 day(s), 17 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199064         PHYSICS_3_02   1 2005.06.07   14:48:03     PROD_PHYSICS_CD 
       This is 2 day(s), 17 hours old. 
   17A is not available for: 
       199064         PHYSICS_3_02   1 2005.06.07   14:48:03     PROD_PHYSICS_CD 
       This is 2 day(s), 17 hours old. 
 
4) DB MONITORING 
   OK 
 
5) DH MONITORING 
   OK 
 
6) CAF MONITORING 
   OK 
 
7) DQM MONITORING 
   OK 
 
--My last day of shift. 
 
</pre>
<!-- Author: Alice Abulencia -->
<!-- Comment: Fri Jun 10 12;18;36 comment by...Mike Houlden -->
<pre>Reported that no 11A calibrations seem to have been done for more
than 48 hours. See previous offline report.</pre>


<!-- Date: Fri Jun 10 15:28:29 2005 -->
<pre>Trying to look at the cdfstrm1 Integrated CPU useage Tables, 
but they don't seem to load. Can't see the Connecting or 
current usage Table either</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Fri Jun 10 16:06:02 2005 -->
<pre>End of shift. All seems OK except I still can't check the 
datbase usage. 
</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Sat Jun 11 07:51:28 2005 -->
<pre>Start of Shift. 
 
Reported that SAM FCP server at fncdfsrv1 was showing RED on the 
monitoring. Sent email to 6302189840@archwireless.net. 
Tried pager but not sure it worked. 
 
Also reported to cdfdb-support@fnal.gov that I am still unable 
to load the DB usage tables to monitor the DB activity</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Sat Jun 11 12:03:31 2005 -->
<pre>In e-mail conversation with Eric Wicklund trying to pin down 
why Offline can't see the DB Tables. Tried a few things, but at 
present it appears that a connection to the tables at 
http://dbsmon.fnal.gov/CDF/updates/CdfOfRead_CpuUsers.html 
can't be made as the connection is refused.</pre>
<!-- Author: Mike Houlden -->
<!-- Comment: Mon Jun 13 09;13;38 comment by...Mike Houlden -->
<pre>Eric has tried this and can access it from all the machines
he has tried both on and off site. Asks me to try it from
a machine in the trailers....</pre>
<!-- Comment: Mon Jun 13 09;28;15 comment by...Mike Houlden -->
<pre>Have asked remote colleagues to see if they can access the page.
It can be seen from Liverpool and the CDF Liverpool open area.</pre>
<!-- Comment: Mon Jun 13 10;56;25 comment by...Mike Houlden -->
<pre>Discover that we cannot even see http://www.fnal.gov from b0dap18 !</pre>
<!-- Comment: Mon Jun 13 13;41;36 comment by...Mike Houlden -->
<pre>Eric dropped by and said that the problem is very localized to
b0dap18. He opened a window on the next machine on the desk, 
b0dap06 and was able to access the DB tables without problem.
Will ask Aidan how to proceed.</pre>
<!-- Date: Sat Jun 11 12:52:32 2005 -->
<pre> OFFLINE REPORT: Sat Jun 11 12:47:20 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199061  2005.06.07 06:18:44  (4 day(s), 6 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (23 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199064         PHYSICS_3_02   1 2005.06.07   14:48:03     PROD_PHYSICS_CD 
       This is 3 day(s), 21 hours old. 
   17A is not available for: 
       199064         PHYSICS_3_02   1 2005.06.07   14:48:03     PROD_PHYSICS_CD 
       This is 3 day(s), 21 hours old. 
 
4) DB MONITORING 
   Still unable to monitor this, see previous entries today 
 
5) DH MONITORING 
   OK 
 
6) CAF MONITORING 
   OK 
 
7) DQM MONITORING 
 
   no data to view</pre>
<!-- Author: Mike Houlden -->
<!-- Comment: Sat Jun 11 14;19;37 comment by...Mike Houlden -->
<pre>Ask Aidan Robson why the production had stopped and he said he had found a 10A calibration job with a file not ready which seemed to
be holding everything up.</pre>


<!-- Comment: Sun Jun 12 13;02;22 comment by...Aidan -->
<pre>11/17A, not 10A.</pre>
<!-- Date: Sat Jun 11 16:04:12 2005 -->
<pre>End of Shift. 
Have not been able to monitor DB usage this shift, and the 11A 
and 17A calibrations are now more than 4 days behind. 
Otherwise everything else looks OK.</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Sun Jun 12 08:00:00 2005 -->
<pre> OFFLINE REPORT: Sun Jun 12 07:54:56 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199186  2005.06.10 10:19:58  (1 day(s), 21 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (24 day(s), 17 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199187         PHYSICS_3_02   1 2005.06.10   10:46:14     PROD_PHYSICS_CD 
       This is 1 day(s), 21 hours old. 
   17A is not available for: 
       199187         PHYSICS_3_02   1 2005.06.10   10:46:14     PROD_PHYSICS_CD 
       This is 1 day(s), 21 hours old. 
 
Backlog of 11A and 17A calibrations have been done over night.  
 
4) DB MONITORING 
 
  still unable to access the DB tables : connection refused. 
5) DH MONITORING 
   OK 
 
6) CAF MONITORING 
   OK 
 
7) DQM MONITORING 
 
   OK. In ElectroWeak.ps plot Trklso4CEMA11 has spike at zero, mean 0.1841 
       is this normal ??</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Sun Jun 12 08:09:12 2005 -->
<pre>Asked experts by e-mail if the large spike at zero in the ElectroWeak.ps plot Trklso4CEMA11 was normal. It has same stats as other plots.</pre>
<!-- Author: Mike Houlden -->
<!-- Comment: Sun Jun 12 09;04;34 comment by...Mike Houlden -->
<pre>Plots are in http://fcdfwww.fnal.gov/~cdfdqm/Processing_610/PlotsDay/199025_aexp0b_psfiles/</pre>


<!-- Comment: Tue Jun 14 08;55;05 comment by...Mike Houlden -->
<pre>No reply by Tuesday so I guess this isn't a problem.</pre>
<!-- Date: Sun Jun 12 09:52:45 2005 -->
<pre>SAM FCP Server at fncdfsrv4 at 2005/06/12 09:42:54 is showing RED 
on monitoring. Sent e-mail 
</pre>
<!-- Author: Mike Houlden -->



<!-- Date: Sun Jun 12 14:22:08 2005 -->
<pre>Several users report difficulty in accessing data and scratch 
disk on fcdflnx2. It was suggested that a Help Desk Ticket be opened on this problem, and Aidan Robson said he would liaise with  
them to progress the problem.</pre>
<!-- Author: Mike Houlden -->
<!-- Comment: Sun Jun 12 14;44;50 comment by...Mike Houlden -->
<pre>Aidan reports that this machine is not on 24/7 support. The 
Help Desk Ticket has been opened.</pre>


<!-- Date: Sun Jun 12 16:03:28 2005 -->
<pre>End of Shift 
 
I have not been able to monitor the DB tables since Friday. 
Everything else looks OK. Data taking is stopped. 
Data taking may be stopped for a long time due to the power 
failure of the CHL Transformer 
</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Mon Jun 13 07:22:22 2005 -->
<pre>Start of shift. Data collection still stopped. 
 
OFFLINE REPORT: Mon Jun 13 07:18:35 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199216  2005.06.11 14:15:07  (1 day(s), 17 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (25 day(s), 16 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199217         PHYSICS_3_02   1 2005.06.11   15:12:46     PROD_PHYSICS_CD 
       This is 1 day(s), 16 hours old. 
   17A is not available for: 
       199217         PHYSICS_3_02   1 2005.06.11   15:12:46     PROD_PHYSICS_CD 
       This is 1 day(s), 16 hours old. 
 
4) DB MONITORING 
  still unable to access the DB tables: connection to host machine refused 
 
5) DH MONITORING 
 
   OK 
6) CAF MONITORING 
   OK 
 
7) DQM MONITORING 
 
   OK. Low statistics in plots 
</pre>
<!-- Author: Mike Houlden -->





<!-- Date: Mon Jun 13 13:26:17 2005 -->
<pre>Sysadmins will turn off the CDF online production database 
(node bzora1) from 15:00 to 17:00 today.  This will prevent most DAQ operations, hardware database activity, run/store summary queries etc.,and possibly some other operations such as entering calibration constants. It should not effect any farms, caf, or offsite processing. 
</pre>
<!-- Author: Mike Houlden -->



<!-- Date: Mon Jun 13 13:58:17 2005 -->
<pre>Erik Wicklund has diagnosed that the problem  about accessing the DB tables is very local to b0dap18. By opening a window on the next machine on the desk b0dap06 I can access the offline-shifts/oracle_usage_monitoring.html and check the DB usage. Will do this as a temporary measure until the problem can be sorted out properly.</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Mon Jun 13 15:26:23 2005 -->
<pre>User vivek apperas to have 26 jobs inactive connections lasting more than few hundred seconds. E-mail sent</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Mon Jun 13 16:05:53 2005 -->
<pre>End of Shift 
No data taken, could be off for two more days. 
User vivek has 26 jobs with inactive DB connections, average about 3000s each. 
Everything else looks OK.</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Tue Jun 14 07:42:17 2005 -->
<pre>Start of shift.  No datataking in progress. 
 
OFFLINE REPORT: Tue Jun 14 07:35:18 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199216  2005.06.11 14:15:07  (2 day(s), 17 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (26 day(s), 17 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199217         PHYSICS_3_02   1 2005.06.11   15:12:46     PROD_PHYSICS_CD 
       This is 2 day(s), 16 hours old. 
   17A is not available for: 
       199217         PHYSICS_3_02   1 2005.06.11   15:12:46     PROD_PHYSICS_CD 
       This is 2 day(s), 16 hours old. 
 
4) DB MONITORING 
OK. Can see the DB tables from another machine but not from b0dap18 
 
5) DH MONITORING 
   Seeing "bad door 24 " in log and operation is taking factor 10 longer 
    than usual. E-mail sent to Aidan R. 
 
6) CAF MONITORING 
   OK 
 
7) DQM MONITORING 
 
   No plots to view 
</pre>
<!-- Author: Mike Houlden -->



<!-- Date: Tue Jun 14 12:00:41 2005 -->
<pre>Sent e-mail to run2-sys-cdfonline@fnal.gov asking why b0dap18 
couldn't see dbsmon.fnal.gov to access the DB tables, whereas 
this is possible from b0dap06.  
Sent traceroute info from both machines.</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Tue Jun 14 12:07:12 2005 -->
<pre>Sent e-mail to cdfdh_oper@fnal.gov reporting "bad door 24" error 
message on Dcache monitor. They are working on it. 
</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Tue Jun 14 15:26:05 2005 -->
<pre>Jason Hedden called by to explain why DB tables could not be accessed. 
"The Offline shifter had `xhost +` in a profile 
on a remote system that caused the X server on b0dap18 to open up. The open X server was picked up by the Nessus security scan and disabled from the network." 
He has contacted the security team to have the block removed. 
 
I have asked my system manager back home to remove the xhost 
command from the login script. 
 
Jason also asked that b0dap18 should be re-booted. This was done. 
</pre>
<!-- Author: Mike Houlden -->
<!-- Comment: Tue Jun 14 15;28;47 comment by...Mike Houlden -->
<pre>After the reboot find that a user id and password is needed to access the CAF monitoring information</pre>


<!-- Date: Tue Jun 14 16:07:20 2005 -->
<pre>End of Shift.No data taken today. 
I notice that the SAM monitoring shows RED intermittently for some 
minutes then returns to normal. Did not report this. 
 
Everything else appears OK. 
  
Much learnt this shift but little achieved! 
</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Wed Jun 15 07:44:07 2005 -->
<pre>Start of shift. No Data taking. 
 
OFFLINE REPORT: Wed Jun 15 07:21:40 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199218  2005.06.11 17:44:43  (3 day(s), 13 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (27 day(s), 16 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199223         PHYSICS_3_02   1 2005.06.12   10:05:27     PROD_PHYSICS_CD 
       This is 2 day(s), 21 hours old. 
   17A is not available for: 
       199223         PHYSICS_3_02   1 2005.06.12   10:05:27     PROD_PHYSICS_CD 
       This is 2 day(s), 21 hours old. 
 
4) DB MONITORING 
 
    Still cannot see the DB tables: b0dap18 still not un-blocked? 
 
5) DH MONITORING 
   OK. Bad Door 24 problem cured overnight 
 
6) CAF MONITORING 
 
   Currently OK. Large perturbation in plots around midnight 
 
7) DQM MONITORING 
 
   No plots to view 
</pre>
<!-- Author: Mike Houlden -->
<!-- Comment: Wed Jun 15 08;48;51 comment by...Mike Houlden -->
<pre>Jason confirms that we still await the computer security team
to unblock b0dap18.</pre>


<!-- Date: Wed Jun 15 11:17:02 2005 -->
<pre>Find that after the Wednesday meeting b0dap18 has been reconnected to the network and I can see the DB tables. They look OK with no limits exceeded.</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Wed Jun 15 11:26:39 2005 -->
<pre>DB monitoring shows users safonov and sforrest have &gt;20 jobs with 
long term inactive connections to the DB.  
Sent the appropriate e-mail.</pre>
<!-- Author: Mike Houlden -->
<!-- Comment: Wed Jun 15 12;08;29 comment by...Mike Houlden -->
<pre>Jobs for these users have been killed.</pre>
<!-- Date: Wed Jun 15 11:57:47 2005 -->
<pre>There are large spikes in the DB connections that have been going on since early today for users safonov and sforrest. They don't breech the warning guidelines, being less than about an hour in duration, but look so abnormal that I sent an e-mail to cdfdb-support@fnal.gov</pre>
<!-- Author: Mike Houlden -->
<!-- Comment: Wed Jun 15 15;43;41 comment by...Mike Houlden -->
<pre>Still there ar 15:45, see below</pre>
<!-- Date: Wed Jun 15 12:17:13 2005 -->
<pre>User vivek again has many jobs with long term inactive DB connections. Sent email.</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Wed Jun 15 13:17:52 2005 -->
<pre>Beam Exe monitoring showing File Not Ready status for runs 225 and 223 for more than 2 days now. E-mailed Aidan R.</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Wed Jun 15 15:01:55 2005 -->
<pre>preditor monitoring could have stopped. At 15:00 the last 
log entry was 
 
Wed Jun 15 11:30:01 CDT 2005 
 
DD-HH  Day and Hour of log file, and summary : 
 
15-11  Number of data files added: 63 
15-10  Number of data files added: 33 
15-09  Number of data files added: 39 
 
Sent e-mail to Aidan.</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Wed Jun 15 15:41:37 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-231&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-231  width='300' align='top'></a></td><td>Large spikes still present</td></tr></table>
<!-- Author: Mike Houlden -->
<!-- Comment: Thu Jun 16 13;44;12 comment by...Mike Houlden -->
<pre>Aidan say this is typical behaviour for large batches of MC jobs</pre>
<!-- Date: Wed Jun 15 16:02:42 2005 -->
<pre>End of Shift. 
 
No data taken. 
 
DB connections looks strange      (reported). 
pretator monitoring has stopped?  (reported).</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Thu Jun 16 07:37:47 2005 -->
<pre>Start of shift. Still no collisions. 
 
OFFLINE REPORT: Thu Jun 16 07:32:52 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199218  2005.06.11 17:44:43  (4 day(s), 13 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (28 day(s), 16 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199223         PHYSICS_3_02   1 2005.06.12   10:05:27     PROD_PHYSICS_CD 
       This is 3 day(s), 21 hours old. 
   17A is not available for: 
       199223         PHYSICS_3_02   1 2005.06.12   10:05:27     PROD_PHYSICS_CD 
       This is 3 day(s), 21 hours old. 
 
4) DB MONITORING 
   OK 
 
5) DH MONITORING 
   OK. Predator monitoring now working. 
 
6) CAF MONITORING 
   OK. A Problem occurred and was fixed overnight. 
 
7) DQM MONITORING 
   No plots to view, no data taken since the weekend. 
</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Thu Jun 16 12:00:08 2005 -->
<pre>Use paulini has &gt;25 long term inactive DB connections. E-mail sent</pre>
<!-- Author: Mike Houlden -->



<!-- Date: Thu Jun 16 16:01:08 2005 -->
<pre>End of a very quiet shift. Still waiting for beam. 
Everything else looks OK.</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Fri Jun 17 07:21:31 2005 -->
<pre>Start of shift. Still no beam 
 
OFFLINE REPORT: Fri Jun 17 07:18:48 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199218  2005.06.11 17:44:43  (5 day(s), 13 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (29 day(s), 16 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199223         PHYSICS_3_02   1 2005.06.12   10:05:27     PROD_PHYSICS_CD 
       This is 4 day(s), 21 hours old. 
   17A is not available for: 
       199223         PHYSICS_3_02   1 2005.06.12   10:05:27     PROD_PHYSICS_CD 
       This is 4 day(s), 21 hours old. 
 
4) DB MONITORING 
    OK 
 
5) DH MONITORING 
   OK 
 
6) CAF MONITORING 
   OK 
 
7) DQM MONITORING 
   No plots to view 
</pre>
<!-- Author: Mike Houlden -->


<!-- Date: Fri Jun 17 08:33:50 2005 -->
<pre>Start training the new Offline shifter, Pinghan Chu</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Fri Jun 17 10:45:33 2005 -->
<pre>Sent email. to paulini for large number inactive connections lasting more than 100 seconds each in cdfstrm1 Current Usage Table. </pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Sat Jun 18 07:55:40 2005 -->
<pre>   OFFLINE REPORT: Sat Jun 18 07:47:41 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199218  2005.06.11 17:44:43  (6 day(s), 14 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206  2005.05.18 14:33:34  (30 day(s), 17 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199223         PHYSICS_3_02   1 2005.06.12   10:05:27     PROD_PHYSICS_CD 
       This is 5 day(s), 21 hours old. 
   17A is not available for: 
       199223         PHYSICS_3_02   1 2005.06.12   10:05:27     PROD_PHYSICS_CD 
       This is 5 day(s), 21 hours old. 
 
4) DB MONITORING 
Ok. 
 
5) DH MONITORING 
Ok. 
 
6) CAF MONITORING 
Ok. 
 
7) DQM MONITORING 
No plot to view, no data taken.</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Sat Jun 18 08:28:28 2005 -->
<pre>The unavailable files in 11A and 17A are increasing. Already sent email to the experts.</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Sat Jun 18 12:00:00 2005 -->
<pre>Red Line &gt; 500 VMs in CondorCAF System Info plots over 2 hours. Already tell schsu. 
</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Sat Jun 18 14:04:07 2005 -->
<pre>SAM Form Resources Monitor 
 
fncdfsrv3 has been red over 2 hours. However, it is still green in "SAM At A Glance". 
 
Just sent an email to cdfsam-admin@fnal.gov. 
 
Hard to identify the red will be fatal or would go away.</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Sat Jun 18 14:07:36 2005 -->
<pre>fncdfsrv3 becomes normal again.  
 
</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Sat Jun 18 15:59:29 2005 -->
<pre>The node fcdfcaf548 is down fow a while.</pre>
<!-- Author: PInghan Chu -->


<!-- Date: Sun Jun 19 07:56:31 2005 -->
<pre>   OFFLINE REPORT: Sun Jun 19 07:45:22 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199218  2005.06.11 17:44:43  (7 day(s), 14 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206    (hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       199595         PHYSICS_3_02   1 2005.06.18   21:54:52     PROD_PHYSICS_CD 
       This is 9 hours old. 
   11A is not available for: 
       199223         PHYSICS_3_02   1 2005.06.12   10:05:27     PROD_PHYSICS_CD 
       This is 6 day(s), 21 hours old. 
   17A is not available for: 
       199223         PHYSICS_3_02   1 2005.06.12   10:05:27     PROD_PHYSICS_CD 
       This is 6 day(s), 21 hours old. 
 
"not available" files continue to increase. 
 
4) DB MONITORING 
OK. 
 
5) DH MONITORING 
OK. 
 
6) CAF MONITORING 
On Jun 18, the Red Line of CondorCAF once exceeded 500 VMS over 2 hours. 
The problem was fixed. 
 
7) DQM MONITORING 
No plot, no data. 
 
8)SAMFarm MONITORING 
The node fcdfcaf548 has been down since Jun 18.</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Sun Jun 19 08:27:49 2005 -->
<pre>The Monitor of SAMFarm node fcdfcaf548 should be moved from  
Fermilab CDF CAF Grid &gt; CDF CAF Worker3 &gt; fcdfcaf548.fnal.gov to  
Fermilab CDF CAF Grid &gt; CDF SAM &gt; fcdfcaf548.fnal.gov 
</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Sun Jun 19 08:31:59 2005 -->
<pre>Suen's response: 
 
1) 548 monitoring was switched to another link, 
   it is corrected 
 
2) fcp "red' coincide with the file-server CPU being busy, 
   you may ignore 
 
3) our major concern is the harddisk getting too full. 
   free disk space on these 4 servers is less than 100GB, 
   please page me  2189840  (instruction at 
   http://fcdfdata055.fnal.gov/~cdfprd/doc/resourceman.html#red)   
 
   e.g. fncdfsrv4 is now 183G free 
   show on http://fcdfdata055.fnal.gov/~cdfprd/ 
   fncdfsrv4 2.0T 1.7T 183G 91% /export 
 
   in the corner below the pink 548 block 
 
4) we are about to be out of disk space, 
   don't be surprised that CPU/projects go down. 
 
thanks, suen</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Mon Jun 20 07:55:02 2005 -->
<pre>   OFFLINE REPORT: Mon Jun 20 07:41:51 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199218  2005.06.11 17:44:43  (8 day(s), 13 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 198206    (hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199223         PHYSICS_3_02   1 2005.06.12   10:05:27     PROD_PHYSICS_CD 
       This is 7 day(s), 21 hours old. 
   17A is not available for: 
       199223         PHYSICS_3_02   1 2005.06.12   10:05:27     PROD_PHYSICS_CD 
       This is 7 day(s), 21 hours old. 
 
"not available" files still increase. 
 
4) DB MONITORING 
OK. 
 
5) DH MONITORING 
OK. 
 
6) CAF MONITORING 
OK. 
 
7) SAMFarm MONITORING 
OK. 
The link of the node to fcdfcaf548 has been updated on Jun 19. 
 
8) DQM MONITORING 
Run 199619 in aexp0b is red now.</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Mon Jun 20 14:51:47 2005 -->
<pre>A user, soha, seems having more than 700 connections for a period of more than 2 hours. Sent email to Peter ans Susana.</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Tue Jun 21 08:15:06 2005 -->
<pre>   OFFLINE REPORT: Tue Jun 21 08:03:18 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199618  2005.06.19 09:26:20  (1 day(s), 22 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 199618  2005.06.19 09:26:20  (1 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199619 PHYSICS_HIGHLUM_3_03   3 2005.06.19   10:11:45     PROD_PHYSICS_CD 
       This is 1 day(s), 21 hours old. 
   17A is not available for: 
       199619 PHYSICS_HIGHLUM_3_03   3 2005.06.19   10:11:45     PROD_PHYSICS_CD 
       This is 1 day(s), 21 hours old. 
 
4) DB MONITORING 
OK. 
 
5) DH MONITORING 
OK. 
 
6) CAF MONITORING 
White space of CondorCAF is more than 200 VMS for a while this morning. Email sent. 
 
7) SAMFarm MONITORING 
OK. 
 
8) DQM MONITORING 
No data, no plot.</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Tue Jun 21 10:59:53 2005 -->
<pre>About white space of CondorCAF: 
 
Shih-chieh's reply: 
 
The empty slot(white sapce) in offline monitor is due to the rate of started jobs can't catch up that of finished jobs. You can read from first picture of following link. 
http://cdfcaf.fnal.gov/condorcaf/history/index.html 
</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Tue Jun 21 14:37:42 2005 -->
<pre>The following links seem not useful for a while. 
Email sent. 
---------------------------- 
cdfstrm1 CPU Usage Tables 
cdfstrm1 Current Usage Table 
cdfstrm1 CPU Usage Plot 
cdfofprd Current Usage Table 
</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Tue Jun 21 14:38:36 2005 -->
<pre>predator status of 21-03 was missed. Email sent.</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Tue Jun 21 15:31:19 2005 -->
<pre>User Sarkar used over 20 connections of cdfofprd. Email sent.</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Wed Jun 22 07:58:06 2005 -->
<pre>Correction: It's Snihur, not Sarkar.</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Wed Jun 22 08:08:02 2005 -->
<pre>   OFFLINE REPORT: Wed Jun 22 07:58:43 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199618  2005.06.19 09:26:20  (2 day(s), 22 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 199618  2005.06.19 09:26:20  (2 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199619 PHYSICS_HIGHLUM_3_03   3 2005.06.19   10:11:45     PROD_PHYSICS_CD 
       This is 2 day(s), 21 hours old. 
   17A is not available for: 
       199619 PHYSICS_HIGHLUM_3_03   3 2005.06.19   10:11:45     PROD_PHYSICS_CD 
       This is 2 day(s), 21 hours old. 
 
4) DB MONITORING 
The cdfofprd database is still occupied by user, snihur, over 20 connections. Fixing. 
 
5) DH MONITORING 
Ok. 
 
6) CAF MONITORING 
Ok. 
 
7) SAMFarm MONITORING 
Ok. 
 
8) DQM MONITORING 
No data, no plot.</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Wed Jun 22 11:33:49 2005 -->
<pre>Randolph J. Herber's response: 
 
One report missing from the predator log. 
 
In this case, it was not a problem.  The predator run the previous hour ran over time and that suppress the following predator execution.</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Wed Jun 22 13:20:37 2005 -->
<pre>fcdfdata064 Autodest Server of SAMFarm seems dead. Email sent.</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Wed Jun 22 16:08:14 2005 -->
<pre>User Vivek has a large number inactive connections in cdfstrm1 lasting more than 100 seconds each. Email sent.</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Thu Jun 23 08:19:09 2005 -->
<pre>   OFFLINE REPORT: Thu Jun 23 08:11:29 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199653  2005.06.20 13:03:41  (2 day(s), 19 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 199618  2005.06.19 09:26:20  (3 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199655         PHYSICS_3_02   1 2005.06.20   13:27:49     PROD_PHYSICS_CD 
       This is 2 day(s), 18 hours old. 
   17A is not available for: 
       199655         PHYSICS_3_02   1 2005.06.20   13:27:49     PROD_PHYSICS_CD 
       This is 2 day(s), 18 hours old. 
 
4) DB MONITORING 
User snihur still has over 20 connections to cdfofprd database. 
 
5) DH MONITORING 
Ok. 
 
6) CAF MONITORING 
 White space of CAF is more than 200 VMs. Email sent. 
 
7) SAMFarm MONITORING 
Ok. 
 
8) DQM MONITORING 
No data and no plot.</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Thu Jun 23 13:13:59 2005 -->
<pre>Regis Lefevre : 
 
The DQM processing is up to date, up to run 199142 which is the last one fully available in SAM Store for dataset aexp0b. The delay is due to Production, I already talk to Suen about that. 
 
Cheers, 
 
    Regis 
</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Thu Jun 23 15:30:36 2005 -->
<pre>User paulini has a large number inactive connections lasting more than 100 seconds each. Email sent.</pre>
<!-- Author: Pinghan Chu -->


<!-- Date: Fri Jun 24 08:55:42 2005 -->
<pre>   OFFLINE REPORT: Fri Jun 24 07:24:47 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199657  2005.06.20 18:35:17  (3 day(s), 12 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 199618  2005.06.19 09:26:20  (4 day(s), 21 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199725         PHYSICS_3_02   1 2005.06.21   06:48:41     PROD_PHYSICS_CD 
       This is 3 day(s), 0 hours old. 
   17A is not available for: 
       199725         PHYSICS_3_02   1 2005.06.21   06:48:41     PROD_PHYSICS_CD 
       This is 3 day(s), 0 hours old. 
 
4) DB MONITORING 
OK. 
 
5) DH MONITORING 
OK. 
 
6) CAF MONITORING 
OK. 
 
7) SAMFarm MONITORING 
OK. 
 
8) DQM MONITORING 
No data and no plot. 
 
</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Sat Jun 25 07:58:41 2005 -->
<pre>   OFFLINE REPORT: Sat Jun 25 06:50:14 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199657  2005.06.20 18:35:17  (4 day(s), 12 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 199618  2005.06.19 09:26:20  (5 day(s), 21 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199725         PHYSICS_3_02   1 2005.06.21   06:48:41     PROD_PHYSICS_CD 
       This is 4 day(s), 0 hours old. 
   17A is not available for: 
       199725         PHYSICS_3_02   1 2005.06.21   06:48:41     PROD_PHYSICS_CD 
       This is 4 day(s), 0 hours old. 
 
 
   * Check BeamExe job status 
 
FILE NOT READY" is more than 2 days and 10A is available 
 
199852  2005/06/23 04:11:29  	 55236  	 10A  	 FILE NOT READY  	 2005/02/14 00:00:00  	 0(0) / 0  	 0(0) / 9460  	 .  	 0  	 . 
199727 	2005/06/22 10:11:42 	122234 	10A 	FILE NOT READY 	2005/02/14 00:00:00 	0(0) / 0 	0(0) / 14773 	. 	0 	. 
199725 	2005/06/22 08:11:30 	6006854 	10A 	FILE NOT READY 	2005/02/14 00:00:00 	0(0) / 0 	0(0) / 649978 	. 	0 	. 
 
Sent e-mail to Aidan, the production coordinator.  
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
White space (more than 200 "VMs") persists for more than 1/2 hour 
 
Sent e-mail to  the CAF Administrators 
 
7) SAMFarm MONITORING 
OK 
 
8) DQM MONITORING 
No data and no plot.</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Sat Jun 25 09:10:07 2005 -->
<pre>Subject: Re: CAF System Problems 
 
Hi! 
 
CAF is fine. 
The free VMs(white space) are due to too few jobs submitted to CAF. 
At time 8:16Am, There are 343 running jobs and 112 waiting jobs. 
The 112 waiting jobs all belong to group_uk.manca who only has QUOTA 30. 
 
Shih-Chieh 
</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Sat Jun 25 11:41:22 2005 -->
<pre>The internet connection outside of FNAL appears to be down. 
  
 
</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Sat Jun 25 14:51:15 2005 -->
<pre>* CDF Run2 Offline System Oracle Usage 
 
Dececco and Paulini have a large number inactive connections lasting more than 100 seconds each. 
 
Sent e-mail to the users, Petar and Susana</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Sun Jun 26 08:02:51 2005 -->
<pre>   OFFLINE REPORT: Sun Jun 26 07:11:19 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199657  2005.06.20 18:35:17  (5 day(s), 12 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 199618  2005.06.19 09:26:20  (6 day(s), 21 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199725         PHYSICS_3_02   1 2005.06.21   06:48:41     PROD_PHYSICS_CD 
       This is 5 day(s), 0 hours old. 
   17A is not available for: 
       199725         PHYSICS_3_02   1 2005.06.21   06:48:41     PROD_PHYSICS_CD 
       This is 5 day(s), 0 hours old. 
 
* The calibration is delayed for more than 5 days. 
 
Sent e-mail to boj, bdc and cdfoom. 
 
* BeamExe job status 
 
FILE NOT READY" is now more than 4 days and 10A is available.  
 
199852  2005/06/23 04:11:29  	 55236  	 10A  	 FILE NOT READY  	 2005/02/14 00:00:00  	 0(0) / 0  	 
0(0) / 9460  	 .  	 0  	 .  
 
199727 	2005/06/22 10:11:42 	122234 	10A 	FILE NOT READY 	2005/02/14 00:00:00 	0(0) / 0 	0(0) / 
14773 	. 	0 	.  
 
199725 	2005/06/22 08:11:30 	6006854 	10A 	FILE NOT READY 	2005/02/14 00:00:00 	0(0) / 0 	0(0) / 
649978 	. 	0 	.  
 
Sent e-mail to the production coordinator.   
 
4) DB MONITORING 
 
# cdfstrm1 Connections Plot:  
 
vivek has more than 700 connections for a period of more than 2 hours. 
 
# cdfstrm1 Current Usage Table:  
 
vivek has a large number (~26) inactive connections lasting more than 100 seconds each. 
 
Sent e-mail the user, Petar and Susana.  
 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) SAMFarm MONITORING 
OK 
 
8) DQM MONITORING 
No data and no plot. 
</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Sun Jun 26 09:47:40 2005 -->
<pre># cdfstrm1 Connections Plot:   
 
vivek has now more than 1200 connections for a period of more than 3 hours.  
 
Sent e-mail to Petar and Susana</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Sun Jun 26 10:44:51 2005 -->
<pre> CAF System Problems: 
 
No plots in the CAF system Info plot or Blank sections. 
 
Sent e-mail to the CAF Administrators.</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Sun Jun 26 10:52:06 2005 -->
<pre>Subject: Re: CAF System Problem 
 
I've noticed, we are trying to contact the hardware people for help. 
 
Thanks, 
Elliot 
</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Sun Jun 26 10:54:01 2005 -->
<pre># cdfstrm1 Connections Plot is now OK</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Sun Jun 26 13:26:19 2005 -->
<pre>* BeamExe job status: ok</pre>
<!-- Author: JUn Suhk Suh -->


<!-- Date: Sun Jun 26 14:54:56 2005 -->
<pre>* CAF System Problem 
 
In the System Info plot: 
Still Blank sections.</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Mon Jun 27 08:16:18 2005 -->
<pre>   OFFLINE REPORT: Mon Jun 27 06:57:28 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199657  2005.06.20 18:35:17  (6 day(s), 12 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 199618  2005.06.19 09:26:20  (7 day(s), 21 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199725         PHYSICS_3_02   1 2005.06.21   06:48:41     PROD_PHYSICS_CD 
       This is 6 day(s), 0 hours old. 
   17A is not available for: 
       199725         PHYSICS_3_02   1 2005.06.21   06:48:41     PROD_PHYSICS_CD 
       This is 6 day(s), 0 hours old. 
 
* BeamExe job status: 
 
 199979  	 2005/06/24 14:11:42  	 246623  	 10A  	 FILE NOT READY  	 2005/02/14 00:00:00  	 0(0) / 0  	 0(0) / 47720  	 .  	 0 
 
Sent e-mail to the production coordinator. 
 
 
4) DB MONITORING 
User bolshov and dececco have a large number inactive connections lasting more than 100 seconds each. Sent e-mail to the users, Petar and Susana 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK. CAF system problem was fixed. 
 
7) SAMFarm MONITORING 
OK 
 
8) DQM MONITORING 
No data and no plot.  
</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Tue Jun 28 08:13:29 2005 -->
<pre>   OFFLINE REPORT: Tue Jun 28 07:17:53 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 199853  2005.06.23 03:27:28  (5 day(s), 3 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 199618  2005.06.19 09:26:20  (8 day(s), 21 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       199979 PHYSICS_HIGHLUM_3_03   3 2005.06.24   12:58:18     PROD_PHYSICS_CD 
       This is 3 day(s), 18 hours old. 
   17A is not available for: 
       199979 PHYSICS_HIGHLUM_3_03   3 2005.06.24   12:58:18     PROD_PHYSICS_CD 
       This is 3 day(s), 18 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) SAMFarm MONITORING 
OK 
 
8) DQM MONITORING 
No data and no plot.   
</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Tue Jun 28 09:22:47 2005 -->
<pre>*SAM Farm Monitoring: 
DBServer at farm1_prd is RED. 
 
Sent e-mail to cdfsam-admin@fnal.gov 
 
</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Tue Jun 28 10:43:51 2005 -->
<pre>Subject: Re: DBServer at farm1_prd is RED 
 
Hi, 
 
those db servers are being restarted every four hours for the last several weeks. 
There was no change recently. The dbserver is working, but is overloaded by some 
long queries. sam locate file takes ~400 seconds to return; so frequent restarts 
here will not help and only probably make things worse. sam-at-a-glance will show 
red or yellow since it has short timeouts, but the farm processing should work if 
the retries are not exceeded (I know it is not a desired response). 
 
We need to see what the queries are really doing and why. 
 
Krzysztof 
 
bartsch@fnal.gov wrote: 
&gt; Hi, 
&gt; 
&gt; tried to restart the dbserver three times. Showed no effect. Krzysztof, may be you could have a look? Did something change during the 
movement of the dbservers last week? 
&gt; 
&gt; Cheers, 
&gt; 
&gt; Valeria 
&gt;</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Tue Jun 28 10:53:38 2005 -->
<pre>SAMOptimizers: SAMOptimizer_prd is RED but "green" at Sam-At-A-Glance. 
 
SAMDbServers: SAMDbServer.farm1_prd is still RED. 
 
Sent email to cdfsam-admin 
 
 
 
</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Tue Jun 28 11:34:40 2005 -->
<pre> SAM Farm Monitoring: ok 
</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Tue Jun 28 11:48:17 2005 -->
<pre>SAM Farm Monitoring:  
 
SAMOptimizers: SAMOptimizer_prd is RED but "green" at Sam-At-A-Glance.  
 
SAMDbServers: SAMDbServer.farm1_prd is RED but "green" at Sam-At-A-Glance.  
</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Tue Jun 28 13:54:11 2005 -->
<pre>Station at cdf-farm1 was red. 
 
Sent e-mail to 6302189840@archwireless.net and fixed now.</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Tue Jun 28 14:34:26 2005 -->
<pre>Everything appears OK</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Tue Jun 28 15:16:40 2005 -->
<pre>SAM Farm Monitoring:   
 
SAMOptimizers: SAMOptimizer_prd is again RED but "green" at Sam-At-A-Glance.   
</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Tue Jun 28 15:24:15 2005 -->
<pre>SAMDbServer.farm1_prd is RED but "yellow" at Sam-At-A-Glance. </pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Tue Jun 28 15:27:34 2005 -->
<pre> SAM Farm Monitoring is ok. 
</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Wed Jun 29 08:25:30 2005 -->
<pre>   OFFLINE REPORT: Wed Jun 29 07:00:40 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200004  2005.06.25 14:34:19  (3 day(s), 16 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 199618  2005.06.19 09:26:20  (9 day(s), 21 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       200196         PHYSICS_3_02   1 2005.06.29   02:00:47     PROD_PHYSICS_CD 
       This is 4 hours old. 
   11A is not available for: 
       200028         PHYSICS_3_02   1 2005.06.26   07:58:56     PROD_PHYSICS_CD 
       This is 2 day(s), 23 hours old. 
   17A is not available for: 
       200028         PHYSICS_3_02   1 2005.06.26   07:58:56     PROD_PHYSICS_CD 
       This is 2 day(s), 23 hours old. 
 
4) DB MONITORING 
Ok 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) SAMFarm MONITORING 
OK 
 
8) DQM MONITORING 
OK. 
</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Wed Jun 29 09:18:28 2005 -->
<pre>User jakraus has more than 700 connections for a period of more than 2 hours. 
 
Sent e-mail to Petar and Susana.</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Wed Jun 29 13:23:03 2005 -->
<pre>User stiegler has more than 700 connections for a period of more than 2 hours.  
 
Sent e-mail to Petar and Susana.</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Wed Jun 29 15:18:34 2005 -->
<pre>Everything appears ok</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Thu Jun 30 07:31:23 2005 -->
<pre>*DH System Monitoring 
 
New tape write error: 
 
2005-Jun-30 06:41:55: failed to catalog fileset CB7339.1  
 
Sent e-mail to cdfdh_oper@fnal.gov</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Thu Jun 30 10:28:56 2005 -->
<pre>Problem with Monitor DCACHE doors (not working) 
 
Sent e-mail to production coordinator.</pre>
<!-- Author: Jun Suhk Suh -->
<!-- Comment: Thu Jun 30 11;01;27 comment by...Aidan -->
<pre>fixed by shift-reload</pre>
<!-- Date: Thu Jun 30 10:29:41 2005 -->
<pre>   OFFLINE REPORT: Thu Jun 30 07:02:35 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200059  2005.06.28 04:07:44  (2 day(s), 2 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 199618    (hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       200204         PHYSICS_3_02   1 2005.06.29   12:08:31     PROD_PHYSICS_CD 
       This is 18 hours old. 
   11A is not available for: 
       200197         PHYSICS_3_02   1 2005.06.29   02:33:03     PROD_PHYSICS_CD 
       This is 1 day(s), 4 hours old. 
   17A is not available for: 
       200197         PHYSICS_3_02   1 2005.06.29   02:33:03     PROD_PHYSICS_CD 
       This is 1 day(s), 4 hours old. 
 
4) DB MONITORING 
OK. 
 
5) DH MONITORING 
 
New tape write error:  
2005-Jun-30 06:41:55: failed to catalog fileset CB7339.1  
 
6) CAF MONITORING 
OK. 
 
7) SAMFarm MONITORING 
OK. 
 
8) DQM MONITORING 
No data and no plot.    
 
</pre>
<!-- Author: Jun Suhk Suh -->



<!-- Date: Thu Jun 30 13:45:36 2005 -->
<pre>User vivek has a large number (26) inactive connections lasting more than 100 seconds each. 
 
Sent e-mail to Petar and Susana. 
 
</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Fri Jul 1 08:25:39 2005 -->
<pre>   OFFLINE REPORT: Fri Jul 1 07:35:34 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200196  2005.06.29 02:00:47  (2 day(s), 5 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 199618    (hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       200204         PHYSICS_3_02   1 2005.06.29   12:08:31     PROD_PHYSICS_CD 
       This is 1 day(s), 19 hours old. 
   11A is not available for: 
       200197         PHYSICS_3_02   1 2005.06.29   02:33:03     PROD_PHYSICS_CD 
       This is 2 day(s), 5 hours old. 
   17A is not available for: 
       200197         PHYSICS_3_02   1 2005.06.29   02:33:03     PROD_PHYSICS_CD 
       This is 2 day(s), 5 hours old. 
 
4) DB MONITORING 
OK. 
 
5) DH MONITORING 
OK. 
 
6) CAF MONITORING 
OK. 
 
7) SAMFarm MONITORING 
DBServer: farm1_prd is red but "green" at Sam-At-A-Glance. 
 
8) DQM MONITORING 
No data and no plot. 
 
</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Fri Jul 1 08:37:54 2005 -->
<pre>User vivek has a large number (26) inactive connections lasting more than 100 seconds each.  
 
Sent e-mail to Petar and Susana.  
 
 
</pre>
<!-- Author: Jun Suhk Suh -->


<!-- Date: Fri Jul 1 13:54:47 2005 -->
<pre>CAF  system info - more than 200 VMs unused between 3am and almost noon.</pre>
<!-- Author: D. Errede -->


<!-- Date: Sat Jul 2 08:01:51 2005 -->
<pre>load &gt; or = 500 VMs on CondorCAF from about midnight to 4am.</pre>
<!-- Author: D. Errede -->


<!-- Date: Sat Jul 2 08:34:13 2005 -->
<pre>   OFFLINE REPORT: Sat Jul 2 07:37:13 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200200  2005.06.29 11:42:11  (2 day(s), 19 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 199618    (hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       200204         PHYSICS_3_02   1 2005.06.29   12:08:31     PROD_PHYSICS_CD 
       This is 2 day(s), 19 hours old. 
   17A is not available for: 
       200204         PHYSICS_3_02   1 2005.06.29   12:08:31     PROD_PHYSICS_CD 
       This is 2 day(s), 19 hours old. 
 
4) DB MONITORING 
   OK. 
 
5) DH MONITORING 
   OK. 
 
6) CAF MONITORING 
   &gt; 200 VMs unused between 07/01 - 3am and almost noon. 
   &gt; or = 500 VMs on CondorCAF from about midnight to 4am. 
 
7) SAMFarm MONITORING 
   OK. 
 
8) DQM MONITORING 
   no data, no plots. 
 
</pre>
<!-- Author: D. Errede -->


<!-- Date: Sat Jul 2 14:54:13 2005 -->
<pre>calibrations 11A and 17A unavailable &gt; 3 days.  </pre>
<!-- Author: D. Errede -->


<!-- Date: Sun Jul 3 08:39:38 2005 -->
<pre>   OFFLINE REPORT: Sun Jul 3 08:26:34 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200215  2005.06.29 17:34:14  (3 day(s), 14 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 199618    (hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       200231         PHYSICS_3_04   1 2005.06.29   18:35:25     PROD_PHYSICS_CD 
       This is 3 day(s), 13 hours old. 
   17A is not available for: 
       200231         PHYSICS_3_04   1 2005.06.29   18:35:25     PROD_PHYSICS_CD 
       This is 3 day(s), 13 hours old. 
 
4) DB MONITORING 
   OK. 
 
5) DH MONITORING 
   OK. 
 
6) CAF MONITORING 
   OK. 
 
7) SAMFarm MONITORING 
   OK. 
 
8) DQM MONITORING 
   no data and no plots. 
</pre>
<!-- Author: Debbie Errede -->


<!-- Date: Sun Jul 3 08:43:07 2005 -->
<pre>11A and 17A calibrations not available &gt; 3 days 
 
</pre>
<!-- Author: D. Errede -->


<!-- Date: Mon Jul 4 08:22:47 2005 -->
<pre>   OFFLINE REPORT: Mon Jul 4 07:37:58 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200284  2005.06.30 16:41:27  (3 day(s), 14 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 200271  2005.06.30 05:34:18  (4 day(s), 2 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       200309         PHYSICS_3_04   1 2005.07.01   13:18:20     PROD_PHYSICS_CD 
       This is 2 day(s), 18 hours old. 
   17A is not available for: 
       200309         PHYSICS_3_04   1 2005.07.01   13:18:20     PROD_PHYSICS_CD 
       This is 2 day(s), 18 hours old. 
 
4) DB MONITORING 
   OK. 
 
5) DH MONITORING 
   OK. 
 
6) CAF MONITORING 
   OK. 
 
7) SAMFarm MONITORING 
   OK. 
 
8) DQM MONITORING 
   no data, no plots. 
</pre>
<!-- Author: D. Errede -->


<!-- Date: Tue Jul 5 08:11:32 2005 -->
<pre>   OFFLINE REPORT: Tue Jul 5 07:35:36 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200309  2005.07.01 13:18:20  (3 day(s), 18 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 200284  2005.06.30 16:41:27  (4 day(s), 14 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       200533         PHYSICS_3_04   1 2005.07.05   06:12:53     PROD_PHYSICS_CD 
       This is 1 hours old. 
   11A is not available for: 
       200324         PHYSICS_3_04   1 2005.07.02   14:55:02     PROD_PHYSICS_CD 
       This is 2 day(s), 16 hours old. 
   17A is not available for: 
       200324         PHYSICS_3_04   1 2005.07.02   14:55:02     PROD_PHYSICS_CD 
       This is 2 day(s), 16 hours old. 
 
4) DB MONITORING 
   OK. 
 
5) DH MONITORING 
   OK. 
 
6) CAF MONITORING 
   unused resources &gt; 200 VMs on CAF 
 
   CondorCAF looks normal. 
 
7) SAMFarm MONITORING 
   OK. 
 
8) DQM MONITORING 
   no data, no plots. 
</pre>
<!-- Author: Debbie Errede -->


<!-- Date: Tue Jul 5 08:16:08 2005 -->
<pre>CAF monitoring -&gt; unused resouces &gt; 200 VMs since ~6:30am today.</pre>
<!-- Author: Debbie Errede -->


<!-- Date: Tue Jul 5 10:16:30 2005 -->
<pre>The unused resources on CAF come from the sl3test queue being empty. 
</pre>
<!-- Author: Debbie Errede  -->


<!-- Date: Tue Jul 5 13:55:50 2005 -->
<pre>fron Joseph Syu: 
 
XFS upgrade to fncdfsrv4 was done, also upgraded 3ware driver. Please let me 
&gt; know if any problems, thanks, 
&gt; 
&gt; /joseph  
 
</pre>
<!-- Author: Debbie Errede -->


<!-- Date: Tue Jul 5 15:26:56 2005 -->
<pre>CAF unused resources &lt; 200 VMs since ~3:00pm.  (normal running conditions)</pre>
<!-- Author: Debbie Errede -->


<!-- Date: Wed Jul 6 07:57:59 2005 -->
<pre>3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       200324         PHYSICS_3_04   1 2005.07.02   14:55:02     PROD_PHYSICS_CD 
       This is 3 day(s), 16 hours old. 
   17A is not available for: 
       200324         PHYSICS_3_04   1 2005.07.02   14:55:02     PROD_PHYSICS_CD 
       This is 3 day(s), 16 hours old. 
 
 
11A and 17A are over the warning threshold...please check.</pre>
<!-- Author: Debbie Errede -->


<!-- Date: Wed Jul 6 08:28:30 2005 -->
<pre>   OFFLINE REPORT: Wed Jul 6 07:45:06 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200309  2005.07.01 13:18:20  (4 day(s), 18 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 200284  2005.06.30 16:41:27  (5 day(s), 15 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       200324         PHYSICS_3_04   1 2005.07.02   14:55:02     PROD_PHYSICS_CD 
       This is 3 day(s), 16 hours old. 
   17A is not available for: 
       200324         PHYSICS_3_04   1 2005.07.02   14:55:02     PROD_PHYSICS_CD 
       This is 3 day(s), 16 hours old. 
 
4) DB MONITORING 
   OK. 
 
5) DH MONITORING 
   OK. 
 
6) CAF MONITORING 
   OK.(CAF monitoring -&gt; unused resouces &gt; 200 VMs while XFS upgrade to fncdfsrv4 was done, also upgraded 3ware drive; J. Syu) 
 
7) SAMFarm MONITORING 
   OK. 
 
8) DQM MONITORING 
   no data, no plots 
</pre>
<!-- Author: Debbie Errede -->


<!-- Date: Wed Jul 6 09:03:34 2005 -->
<pre>CORRECTED REPORT FOR JULY 6, 2005 
 
   OFFLINE REPORT: Wed Jul 6 07:45:06 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200309  2005.07.01 13:18:20  (4 day(s), 18 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 200284  2005.06.30 16:41:27  (5 day(s), 15 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       200324         PHYSICS_3_04   1 2005.07.02   14:55:02     PROD_PHYSICS_CD 
       This is 3 day(s), 16 hours old. 
   17A is not available for: 
       200324         PHYSICS_3_04   1 2005.07.02   14:55:02     PROD_PHYSICS_CD 
       This is 3 day(s), 16 hours old. 
 
4) DB MONITORING 
   OK. 
 
5) DH MONITORING 
   OK. 
 
6) CAF MONITORING 
   OK.(CAF monitoring -&gt; unused resouces &gt; 200 VMs because sl3test had low usage.) 
 
7) SAMFarm MONITORING 
   OK. fncdfsrv4 was down part of 7/5 (while XFS upgrade to fncdfsrv4 was done, also upgraded 3ware drive; J. Syu) 
 
8) DQM MONITORING 
   no data, no plots 
</pre>
<!-- Author: Debbie Errede -->


<!-- Date: Thu Jul 7 08:22:38 2005 -->
<pre>   OFFLINE REPORT: Thu Jul 7 07:37:40 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200535  2005.07.05 07:05:14  (2 day(s), 0 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 200533  2005.07.05 06:12:53  (2 day(s), 1 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       200570         PHYSICS_3_04   1 2005.07.05   21:59:14     PROD_PHYSICS_CD 
       This is 1 day(s), 9 hours old. 
   17A is not available for: 
       200570         PHYSICS_3_04   1 2005.07.05   21:59:14     PROD_PHYSICS_CD 
       This is 1 day(s), 9 hours old. 
 
4) DB MONITORING 
   OK. 
 
5) DH MONITORING 
   OK. 
 
6) CAF MONITORING 
   OK. 
 
7) SAMFarm MONITORING 
   OK. 
 
8) DQM MONITORING 
   no data, no plots. 
 
</pre>
<!-- Author: Debbie Errede -->


<!-- Date: Fri Jul 8 10:32:32 2005 -->
<pre>   OFFLINE REPORT: Fri Jul 8 07:54:26 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200635  2005.07.06 17:05:36  (1 day(s), 14 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 200535  2005.07.05 07:05:14  (3 day(s), 0 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       200641         PHYSICS_3_04   1 2005.07.06   18:19:43     PROD_PHYSICS_CD 
       This is 1 day(s), 13 hours old. 
   17A is not available for: 
       200641         PHYSICS_3_04   1 2005.07.06   18:19:43     PROD_PHYSICS_CD 
       This is 1 day(s), 13 hours old. 
 
4) DB MONITORING 
   OK. 
 
5) DH MONITORING 
   OK. 
 
6) CAF MONITORING 
   OK. 
 
7) SAMFarm MONITORING 
   OK. 
 
8) DQM MONITORING 
   no data, no plots. 
</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Fri Jul 8 10:52:15 2005 -->
<pre>User pmf uses more than 2000 connections to cdfstrm1. The user is  
using more than 700 connections since 8am. Petar and Susana will 
be notified.</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Fri Jul 8 11:05:19 2005 -->
<pre>SAM Farm showed a problem on a fcp at fncdfsrv3, the Ganglia plots of this machine showed that the load started to increase very much. Also the mem_buffers go up on the Ganglia page. Will contact Suen.</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Fri Jul 8 13:51:09 2005 -->
<pre>fncdfsrv3 had a hardware problem. It is solved by now.</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Sat Jul 9 08:06:16 2005 -->
<pre>   OFFLINE REPORT: Sat Jul 9 08:04:29 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200641  2005.07.06 18:19:43  (2 day(s), 13 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 200635  2005.07.06 17:05:36  (2 day(s), 14 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       200659         PHYSICS_3_04   1 2005.07.07   05:07:50     PROD_PHYSICS_CD 
       This is 2 day(s), 2 hours old. 
   17A is not available for: 
       200659         PHYSICS_3_04   1 2005.07.07   05:07:50     PROD_PHYSICS_CD 
       This is 2 day(s), 2 hours old. 
 
4) DB MONITORING 
user pmf used more than 700 connections from 10am on, the number of connections finally decreased around 2pm.Status back to normal 
 
 
5) DH MONITORING 
o.k. 
 
6) CAF MONITORING 
o.k. 
 
7) SAMFarm MONITORING 
fcp problem caused by a hardware problem, problem solved by Suen. 
 
8) DQM MONITORING 
not available 
 
</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Sat Jul 9 10:13:38 2005 -->
<pre>SAM Farm: fcp transfer from fncdfsrv3 failed again, sent email to Suen. 
 
CAF: user Larry Nodulman complained that a CAF disk is full, forwarded the email to Elliot and Shih-Chieh</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Sat Jul 9 14:07:35 2005 -->
<pre>(1)SAM Farm: no further problem with fcp occured. Suen wants to improve the documentation: 
 
If the network and load on ganglia are reasonale, load&lt;5, traffic heavy. Then it is a time-out to ResourceMan for being too busy. To be certain that fcp is normal, causing no farm, I would add this on the web for expert checking. The fcps response should be instant. 
 
( tcsh) 
$ source ~cdfsoft/cdf2.cshrc 
$ setup fcp 
$ fcps fncdfsrv3 
 RID                  user.pid@host TS Free/Avbl(MB)   Size File 
---- ------------------------------ -- ------------- ------ ---------- 
 215          cdfprd.11764@fncdf187 U* 1367514/1366086   1063 
/export/stage/ProductionExe_6.1.0a_maxopt_0f/hbhd0f/ 
 222          cdfprd.24972@fncdf315 U* 1367514/1366086    364 
/export/stage/ProductionExe_6.1.0a_maxopt_0f/xbhd0f/ 
 225          cdfprd.11836@fncdf187 U* 1367514/1366086      1 
/export/stage/ProductionExe_6.1.0a_maxopt_0f/meta/ 
 226          cdfprd.11010@fncdf322 Uw 1367514/1366086      1 
 
 
(2)CAF: disk on condorcaf full, Shih-Chieh has removed the jobs which were responsible for filling up the disk.</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Sun Jul 10 08:08:11 2005 -->
<pre>   OFFLINE REPORT: Sun Jul 10 08:05:55 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200661  2005.07.07 17:23:17  (2 day(s), 14 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 200641  2005.07.06 18:19:43  (3 day(s), 13 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       200663         PHYSICS_3_04   1 2005.07.07   18:28:49     PROD_PHYSICS_CD 
       This is 2 day(s), 13 hours old. 
   17A is not available for: 
       200663         PHYSICS_3_04   1 2005.07.07   18:28:49     PROD_PHYSICS_CD 
       This is 2 day(s), 13 hours old. 
 
4) DB MONITORING 
o.k. 
 
5) DH MONITORING 
o.k. 
 
6) CAF MONITORING 
o.k., but user complained that disks at CondorCaf full, Shih-Chieh stopped jobs which were responsible for this behaviour around 2pm. 
 
7) SAMFarm MONITORING 
at 10am one fcp transfer from fncdfsrv3 red. Contacted Suen, he suggested to add to the documentation:  
 
If the network and load on ganglia are reasonale, load&lt;5, traffic heavy. Then it is a time-out to ResourceMan for being too busy. To be certain that fcp is normal, causing no farm, I would add this on the web for expert checking. the fcps response should be instant. 
 
( tcsh) 
$ source ~cdfsoft/cdf2.cshrc 
$ setup fcp 
$ fcps fncdfsrv3 
 RID                  user.pid@host TS Free/Avbl(MB)   Size File 
---- ------------------------------ -- ------------- ------ ---------- 
 215          cdfprd.11764@fncdf187 U* 1367514/1366086   1063 
/export/stage/ProductionExe_6.1.0a_maxopt_0f/hbhd0f/ 
 222          cdfprd.24972@fncdf315 U* 1367514/1366086    364 
/export/stage/ProductionExe_6.1.0a_maxopt_0f/xbhd0f/ 
 225          cdfprd.11836@fncdf187 U* 1367514/1366086      1 
/export/stage/ProductionExe_6.1.0a_maxopt_0f/meta/ 
 226          cdfprd.11010@fncdf322 Uw 1367514/1366086      1 
/export/stage/ProductionExe_6.1.0a_maxopt_0f/meta/ 
 227          cdfprd.28180@fncdf197 Uw 1367514/1366086     14 
/export/stage/ProductionExe_6.1.0a_maxopt_0f/xbpp0f/ 
 228          cdfprd.29485@fncdf323 Uw 1367514/1366086      6 
/export/stage/ProductionExe_6.1.0a_maxopt_0f/xbpp0f/ 
 229          cdfprd.13406@fncdf346 Uw 1367514/1366086      1 
/export/stage/ProductionExe_6.1.0a_maxopt_0f/meta/ 
 230           cdfprd.6470@fncdf344 Uw 1367514/1366086      3 
 
Further information about the SAM Farm in a more recent email from Suen.  
 
8) DQM MONITORING 
not available. 
 
 
</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Mon Jul 11 08:21:38 2005 -->
<pre>Calibration for 11A and 17A out of date. Contacted Aidan and Bernd.</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Mon Jul 11 08:22:40 2005 -->
<pre>   OFFLINE REPORT: Mon Jul 11 07:49:53 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200661  2005.07.07 17:23:17  (3 day(s), 14 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 200641  2005.07.06 18:19:43  (4 day(s), 13 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       200663         PHYSICS_3_04   1 2005.07.07   18:28:49     PROD_PHYSICS_CD 
       This is 3 day(s), 13 hours old. 
   17A is not available for: 
       200663         PHYSICS_3_04   1 2005.07.07   18:28:49     PROD_PHYSICS_CD 
       This is 3 day(s), 13 hours old. 
 
4) DB MONITORING 
o.k. 
 
5) DH MONITORING 
o.k. 
 
6) CAF MONITORING 
o.k. 
 
7) SAMFarm MONITORING 
o.k., more information see Tsan's email. 
 
8) DQM MONITORING 
not available 
 
</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Mon Jul 11 10:16:05 2005 -->
<pre>Lots of plots show a gap on the SAM Farm homepage from 2am to 6am: 
 
- SamProjects show a gap 
- running processes show a gap 
- CAF monitoring: no running sections from 2am - 6am 
- load and CPU on farms during that time low 
- low number of connections at SAM dbserver farm1station_prd 
- dip in SamStore@Work 
 
I wonder what had happened during that time. Right now things seem to be normal again. Sent email to Tsan and Suen.</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Mon Jul 11 11:36:05 2005 -->
<pre>The SAM Farm homepage at ResourceMan@Work showed a red box at fcp to fncdfsrv2. I checked with: 
 
[fcdflnx2:~] cdfopr&gt; source ~cdfsoft/cdf2.cshrc 
[fcdflnx2:~] cdfopr&gt; setup fcp 
[fcdflnx2:~] cdfopr&gt; fcps fncdfsrv2 
 RID                  user.pid@host TS Free/Avbl(MB)   Size File 
---- ------------------------------ -- ------------- ------ ---------- 
Can not connect to fcpd 
 
Contacted Suen and Tsan about this.  
</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Mon Jul 11 13:13:47 2005 -->
<pre>SAM Farm: srv2 fcp deamon restarted by Suen. The gaps in the plots are o.k. because *0f production close to the end. Therefore spikes will occur in the next days more often.</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Mon Jul 11 14:37:12 2005 -->
<pre>CondorCaf: It looks again as if there is a headnode problem. There are no more sections anywhere. Sent email to Shih-Chieh and Elliot.</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Tue Jul 12 07:50:56 2005 -->
<pre>The CAF has many free VMs although there are still many jobs waiting. The "wasted line" is also above 500 since 10pm yesterday and increasing. Right now it is at about 1000 VMs. At around 11pm yesterday there is again a totally blank section at the CAF. But that is o.k. now. 
 
Sent email to Elliot and Shih-Chieh and also asked if this is connected with the downtime at the 14.th although the ramp down for this downtime is scheduled on the 13.th. 
</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Tue Jul 12 10:02:59 2005 -->
<pre>CAF: hardware problem now solved.</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Tue Jul 12 13:00:29 2005 -->
<pre>   OFFLINE REPORT: Tue Jul 12 07:54:03 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200752  2005.07.09 16:24:14  (2 day(s), 15 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 200641  2005.07.06 18:19:43  (5 day(s), 13 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       200753         PHYSICS_3_04   1 2005.07.09   19:28:41     PROD_PHYSICS_CD 
       This is 2 day(s), 12 hours old. 
   17A is not available for: 
       200753         PHYSICS_3_04   1 2005.07.09   19:28:41     PROD_PHYSICS_CD 
       This is 2 day(s), 12 hours old. 
 
4) DB MONITORING 
o.k. 
 
5) DH MONITORING 
o.k. 
 
6) CAF MONITORING 
many free VMs although there are lots of waiting sections, hardware problem, now solved 
 
7) SAMFarm MONITORING 
o.k. now, in between fcp on srv2 went down, because Joseph power cycled this node 
 
8) DQM MONITORING 
not available 
 
</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Wed Jul 13 15:40:57 2005 -->
<pre>WEDNESDAY report: 
 
   OFFLINE REPORT: Tue Jul 12 12:51:46 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200752  2005.07.09 16:24:14  (2 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 200669  2005.07.07 22:26:48  (4 day(s), 14 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       200753         PHYSICS_3_04   1 2005.07.09   19:28:41     PROD_PHYSICS_CD 
       This is 2 day(s), 17 hours old. 
   17A is not available for: 
       200753         PHYSICS_3_04   1 2005.07.09   19:28:41     PROD_PHYSICS_CD 
       This is 2 day(s), 17 hours old. 
 
The calibrations 11A and 17A were on Monday older than 3 days. 
 
4) DB MONITORING 
Friday: User pmf uses more than 2000 connections to cdfstrm1. Number of connections decreased during the day. 
 
5) DH MONITORING 
 
 
6) CAF MONITORING 
Saturday: jobs were filling disks at CondorCaf, solved by now 
Monday: Elliot's search of log-file caused the monitoring not to update 
Tuesday: hardware problems 
 
7) SAMFarm MONITORING 
fncdfsrv3 and fncdfsrv2 had fcp problems, on Friday and Monday hardware problem in between no hardware problem. Suen suggested to check in these cases: 
 
( tcsh) 
$ source ~cdfsoft/cdf2.cshrc 
$ setup fcp 
$ fcps fncdfsrv3 
RID                  user.pid@host TS Free/Avbl(MB)   Size File 
---- ------------------------------ -- ------------- ------ ---------- 
215          cdfprd.11764@fncdf187 U* 1367514/1366086   1063 /export/stage/ProductionExe_6.1.0a_maxopt_0f/hbhd0f/ 
222          cdfprd.24972@fncdf315 U* 1367514/1366086    364 /export/stage/ProductionExe_6.1.0a_maxopt_0f/xbhd0f/ 
225          cdfprd.11836@fncdf187 U* 1367514/1366086      1 
$ 
 
Documentation at SAM Farm webpages has changed.  
 
 
8) DQM MONITORING 
since Tueday available again. 
 
</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Wed Jul 13 15:41:18 2005 -->
<pre>  OFFLINE REPORT: Wed Jul 13 15:35:44 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200755  2005.07.09 20:30:36  (3 day(s), 19 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 200669  2005.07.07 22:26:48  (5 day(s), 17 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       200887         PHYSICS_3_04   1 2005.07.12   23:21:56     PROD_PHYSICS_CD 
       This is 16 hours old. 
   11A is not available for: 
       200756         PHYSICS_3_04   1 2005.07.09   21:32:10     PROD_PHYSICS_CD 
       This is 3 day(s), 18 hours old. 
   17A is not available for: 
       200756         PHYSICS_3_04   1 2005.07.09   21:32:10     PROD_PHYSICS_CD 
       This is 3 day(s), 18 hours old. 
 
4) DB MONITORING 
o.k. 
 
5) DH MONITORING 
o.k. 
 
6) CAF MONITORING 
o.k. 
 
7) SAMFarm MONITORING 
o.k. 
 
8) DQM MONITORING 
o.k. 
 
Today the home area of fcdflnxX was not mounted, therefore the plots could only be checked after 3pm and the report is coming really late. 
 
</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Thu Jul 14 08:20:39 2005 -->
<pre>   OFFLINE REPORT: Thu Jul 14 07:55:21 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200755  2005.07.09 20:30:36  (4 day(s), 11 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 200669  2005.07.07 22:26:48  (6 day(s), 9 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       200887         PHYSICS_3_04   1 2005.07.12   23:21:56     PROD_PHYSICS_CD 
       This is 1 day(s), 8 hours old. 
   11A is not available for: 
       200887         PHYSICS_3_04   1 2005.07.12   23:21:56     PROD_PHYSICS_CD 
       This is 1 day(s), 8 hours old. 
   17A is not available for: 
       200887         PHYSICS_3_04   1 2005.07.12   23:21:56     PROD_PHYSICS_CD 
       This is 1 day(s), 8 hours old. 
I wonder if this is a combination of yesterday's downtime of the home areas and today's downtime of the CAF? 
 
4) DB MONITORING 
o.k. 
 
5) DH MONITORING 
o.k. 
 
6) CAF MONITORING 
o.k., CAF stopped the queues yesterday evening and now is stopped due to the data handling downtime. 
 
7) SAMFarm MONITORING 
o.k., SAMFarm nearly finished the processing and will be also affected by today's data handling downtime. 
 
8) DQM MONITORING 
o.k. 
 
</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Fri Jul 15 08:26:39 2005 -->
<pre>   OFFLINE REPORT: Fri Jul 15 07:53:18 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200883  2005.07.12 13:51:52  (2 day(s), 18 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 200756  2005.07.09 21:32:10  (5 day(s), 10 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       200887         PHYSICS_3_04   1 2005.07.12   23:21:56     PROD_PHYSICS_CD 
       This is 2 day(s), 8 hours old. 
   11A is not available for: 
       200887         PHYSICS_3_04   1 2005.07.12   23:21:56     PROD_PHYSICS_CD 
       This is 2 day(s), 8 hours old. 
   17A is not available for: 
       200887         PHYSICS_3_04   1 2005.07.12   23:21:56     PROD_PHYSICS_CD 
       This is 2 day(s), 8 hours old. 
 
4) DB MONITORING 
o.k. 
 
5) DH MONITORING 
o.k. 
 
6) CAF MONITORING 
not o.k., probably connected with yesterday's downtime, will ask Elliot and Shih-Chieh 
 
7) SAMFarm MONITORING 
o.k. 
 
8) DQM MONITORING 
o.k. 
</pre>
<!-- Author: Valeria Bartsch -->


<!-- Date: Fri Jul 15 14:01:24 2005 -->
<pre>condor caf pathological 9.45-10.15am - then jobs started running agin</pre>
<!-- Author: Aidan -->


<!-- Date: Sat Jul 16 08:12:39 2005 -->
<pre>   OFFLINE REPORT: Sat Jul 16 07:58:42 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 200970  2005.07.13 22:43:18  (2 day(s), 9 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 200756  2005.07.09 21:32:10  (6 day(s), 10 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       201040         PHYSICS_3_04   1 2005.07.15   10:59:11     PROD_PHYSICS_CD 
       This is 20 hours old. 
   17A is not available for: 
       201040         PHYSICS_3_04   1 2005.07.15   10:59:11     PROD_PHYSICS_CD 
       This is 20 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) SAMFarm MONITORING 
OK (idle) 
 
8) DQM MONITORING 
OK 
</pre>
<!-- Author: Aidan -->


<!-- Date: Sun Jul 17 08:12:50 2005 -->
<pre>   OFFLINE REPORT: Sun Jul 17 08:08:03 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 201129  2005.07.16 04:24:18  (1 day(s), 3 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 201083  2005.07.15 17:25:39  (1 day(s), 14 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       201130         PHYSICS_3_04   1 2005.07.16   05:59:36     PROD_PHYSICS_CD 
       This is 1 day(s), 2 hours old. 
   17A is not available for: 
       201130         PHYSICS_3_04   1 2005.07.16   05:59:36     PROD_PHYSICS_CD 
       This is 1 day(s), 2 hours old. 
 
4) DB MONITORING 
harper has 25 connections to cdfofprd from caf jobs - have enquired 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) SAMFarm MONITORING 
OK (idle) 
 
8) DQM MONITORING 
OK 
 
</pre>
<!-- Author: Aidan -->


<!-- Date: Mon Jul 18 08:24:58 2005 -->
<pre>   OFFLINE REPORT: Mon Jul 18 07:16:56 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 201133  2005.07.16 14:31:25  (1 day(s), 16 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 201083  2005.07.15 17:25:39  (2 day(s), 13 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       201152 PHYSICS_HIGHLUM_3_04   1 2005.07.17   08:05:12     PROD_PHYSICS_CD 
       This is 23 hours old. 
   17A is not available for: 
       201152 PHYSICS_HIGHLUM_3_04   1 2005.07.17   08:05:12     PROD_PHYSICS_CD 
       This is 23 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
condor caf has 60% sections stuck but no evident associated DH problem 
 
7) SAMFarm MONITORING 
OK (idle) 
 
8) DQM MONITORING 
OK 
</pre>
<!-- Author: Aidan -->
<!-- Comment: Mon Jul 18 16;54;39 comment by...aidan -->
Elliot killed many jobs at 1340.


<!-- Date: Tue Jul 19 06:56:28 2005 -->
<pre>   OFFLINE REPORT: Tue Jul 19 06:48:27 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 201133  2005.07.16 14:31:25  (2 day(s), 16 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 201132  2005.07.16 09:38:57  (2 day(s), 21 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       201152 PHYSICS_HIGHLUM_3_04   1 2005.07.17   08:05:12     PROD_PHYSICS_CD 
       This is 1 day(s), 22 hours old. 
   17A is not available for: 
       201152 PHYSICS_HIGHLUM_3_04   1 2005.07.17   08:05:12     PROD_PHYSICS_CD 
       This is 1 day(s), 22 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) SAMFarm MONITORING 
OK (idle) 
 
8) DQM MONITORING 
OK 
</pre>
<!-- Author: Aidan -->


<!-- Date: Wed Jul 20 08:31:49 2005 -->
<pre>   OFFLINE REPORT: Wed Jul 20 07:05:29 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 201155  2005.07.17 14:27:50  (2 day(s), 16 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 201154  2005.07.17 09:03:00  (2 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       201349         PHYSICS_3_05   1 2005.07.19   23:50:16     PROD_PHYSICS_CD 
       This is 7 hours old. 
   11A is not available for: 
       201212         PHYSICS_3_05   1 2005.07.18   15:13:20     PROD_PHYSICS_CD 
       This is 1 day(s), 15 hours old. 
   17A is not available for: 
       201212         PHYSICS_3_05   1 2005.07.18   15:13:20     PROD_PHYSICS_CD 
       This is 1 day(s), 15 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
known problem between 8pm and 11pm last night; OK now 
 
7) SAMFarm MONITORING 
200 test jobs successful overnight; ramping up on 0g since 7am  
 
8) DQM MONITORING 
OK 
 
</pre>
<!-- Author: Aidan -->


<!-- Date: Thu Jul 21 08:21:33 2005 -->
<pre>   OFFLINE REPORT: Thu Jul 21 06:57:37 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 201155  2005.07.17 14:27:50  (3 day(s), 16 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 201154  2005.07.17 09:03:00  (3 day(s), 21 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       201362         PHYSICS_3_05   1 2005.07.20   08:37:33     PROD_PHYSICS_CD 
       This is 22 hours old. 
   11A is not available for: 
       201212         PHYSICS_3_05   1 2005.07.18   15:13:20     PROD_PHYSICS_CD 
       This is 2 day(s), 15 hours old. 
   17A is not available for: 
       201212         PHYSICS_3_05   1 2005.07.18   15:13:20     PROD_PHYSICS_CD 
       This is 2 day(s), 15 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) SAMFarm MONITORING 
OK 
 
8) DQM MONITORING 
OK 
</pre>
<!-- Author: Aidan -->


<!-- Date: Thu Jul 21 15:40:17 2005 -->
<pre> 
Farm CPU usage began to fall off from 7am, became clear about noon. 
CAF jobs showing 100% all the time. 
Suen suspected network problems; around 2pm Jason diagnosed: 
1) ftp to cdfkits is not working, he is on it. 
2) CDF enstore server not responding. 
</pre>
<!-- Author: Aidan -->
<!-- Comment: Fri Jul 22 08;49;00 comment by...Aidan -->
<pre>cdfkits restored at 10pm</pre>
<!-- Date: Fri Jul 22 08:48:29 2005 -->
<pre>dcache monitoring not updated since yesterday 4pm - we suspect processes were not restarted on cdfkits.  Sent mail to cdfdh_oper and Chuck.</pre>
<!-- Author: ByeongRok/Aidan -->



<!-- Date: Fri Jul 22 09:14:26 2005 -->
<pre>   OFFLINE REPORT: Fri Jul 22 07:20:03 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 201155  2005.07.17 14:27:50  (4 day(s), 16 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 201154  2005.07.17 09:03:00  (4 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       201375    PHYSICS_TEST_3_05   7 2005.07.20   15:16:08     PROD_PHYSICS_CD 
       This is 1 day(s), 16 hours old. 
   11A is not available for: 
       201212         PHYSICS_3_05   1 2005.07.18   15:13:20     PROD_PHYSICS_CD 
       This is 3 day(s), 16 hours old. 
   17A is not available for: 
       201212         PHYSICS_3_05   1 2005.07.18   15:13:20     PROD_PHYSICS_CD 
       This is 3 day(s), 16 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
Dcache monitoring not updated - probably owing to cdfkits reboot 
 
6) CAF MONITORING 
OK 
 
7) SAMFarm MONITORING 
Looked like job sections plot dipping - keep watching it 
CPU usage not as high as before yesterday's downtime? 
 
8) DQM MONITORING 
OK 
</pre>
<!-- Author: Aidan/ByeongRok -->


<!-- Date: Fri Jul 22 10:27:34 2005 -->
<pre>Warning from BeamExe status. 
Sent an email to Aidan and Bernd. 
 
WARNING: This page has not updated in over 5101 minutes! 
Is the Nameserver running? Is the cronjob running? Investigation Required!</pre>
<!-- Author: ByeongRok -->
<!-- Comment: Fri Jul 22 12;04;00 comment by...Aidan -->
<pre>Try opening:
http://www-cdf.fnal.gov/~cdfopr/cdfopr/html/menu.html
force an update by holding shift and clicking "reload"
then try clicking "beamline production" again. The page
you should be looking at is:
http://fncdfsrv0.fnal.gov/~cdfprd/metalia/BeamExe_1_gphysr.html

Regards
Aidan
</pre>


<!-- Comment: Sun Jul 24 08;16;15 comment by...ByeongRok -->
<pre>forward to cdf_offline_operations</pre>
<!-- Date: Fri Jul 22 15:47:12 2005 -->
<pre>cdfstrm1 Connections Plot and 
cdfstrm1 CPU Usage Plot don't show up. 
Sent an email to petar and cdfdb-support. 
</pre>
<!-- Author: ByeongRok -->


<!-- Date: Sat Jul 23 07:59:16 2005 -->
<pre>Tape write error 
 
2005-Jul-23 07:26:20 encp FAILED TO COPY FILESET CB7647.0 
source directory: /cdf/b0/disk6/stager/CB7647.0 
source files: gr031387.0066phys gr031387.007cphys gr031387.0093phys 
gr031387.00abphys gr031387.00c4phys gr031387.00ddphys gr031387.00f7phys 
gr031387.0112phys 
destination dir: /pnfs/cdfen/filesets/CB/CB76/CB7647/CB7647.0 
The rest of the filesets will not be encpied: CB7647.0 CB7647.1 CB7647.2 
CB7647.3 CB7647.4 
 
Sent an email to cdfdh_oper</pre>
<!-- Author: ByeongRok -->
<!-- Comment: Sun Jul 24 08;18;15 comment by...ByeongRok -->
<pre>forward to cdf_offline_operations</pre>
<!-- Date: Sat Jul 23 08:20:47 2005 -->
<pre># DQM MONITORING 
194862_jpmm0f_psfiles/Jpsi.ps 
No Bottom three plots. 
 
Sent an email to mmp@fnal.gov, regis@fnal.gov, portell@fnal.gov 
</pre>
<!-- Author: ByeongRok -->
<!-- Comment: Sat Jul 23 10;35;09 comment by...ByeongRok -->
<pre>They (mmp@fnal.gov, regis@fnal.gov, portell@fnal.gov)
know about this run having a problem with Silicon
calibration constants and sent an email to Tracking Experts.
</pre>
<!-- Date: Sat Jul 23 10:21:17 2005 -->
<pre>smwang is having more than 20 connection to cdfofprd database. 
email is sent to him 
</pre>
<!-- Author: ByeongRok -->


<!-- Date: Sat Jul 23 10:26:33 2005 -->
<pre>sam cpu usage is very small. 
sent an email to Aidan and Bernd. 
</pre>
<!-- Author: ByeongRok -->


<!-- Date: Sat Jul 23 10:33:04 2005 -->
<pre>   OFFLINE REPORT: Sat Jul 23 07:50:12 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 201371  2005.07.20 11:10:20  (2 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 201154  2005.07.17 09:03:00  (5 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       201373    PHYSICS_TEST_3_05   7 2005.07.20   14:36:23     PROD_PHYSICS_CD 
       This is 2 day(s), 17 hours old. 
   17A is not available for: 
       201373    PHYSICS_TEST_3_05   7 2005.07.20   14:36:23     PROD_PHYSICS_CD 
       This is 2 day(s), 17 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
 
Tape write error 
 
2005-Jul-23 07:26:20 encp FAILED TO COPY FILESET CB7647.0 
source directory: /cdf/b0/disk6/stager/CB7647.0 
source files: gr031387.0066phys gr031387.007cphys gr031387.0093phys 
gr031387.00abphys gr031387.00c4phys gr031387.00ddphys gr031387.00f7phys 
gr031387.0112phys 
destination dir: /pnfs/cdfen/filesets/CB/CB76/CB7647/CB7647.0 
The rest of the filesets will not be encpied: CB7647.0 CB7647.1 CB7647.2 
CB7647.3 CB7647.4 
 
 
6) CAF MONITORING 
OK 
 
7) SAMFarm MONITORING 
OK 
 
8) DQM MONITORING 
194862_jpmm0f_psfiles/Jpsi.ps 
No Bottom three plots. 
</pre>
<!-- Author: ByeongRok -->



<!-- Date: Sun Jul 24 08:15:09 2005 -->
<pre>   OFFLINE REPORT: Sun Jul 24 07:43:11 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 201371  2005.07.20 11:10:20  (3 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 201154  2005.07.17 09:03:00  (6 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       201625         PHYSICS_3_06   1 2005.07.23   17:34:34     PROD_PHYSICS_CD 
       This is 14 hours old. 
   11A is not available for: 
       201373    PHYSICS_TEST_3_05   7 2005.07.20   14:36:23     PROD_PHYSICS_CD 
       This is 3 day(s), 17 hours old. 
   17A is not available for: 
       201373    PHYSICS_TEST_3_05   7 2005.07.20   14:36:23     PROD_PHYSICS_CD 
       This is 3 day(s), 17 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
DCache is back 
 
6) CAF MONITORING 
OK 
 
7) SAMFarm MONITORING 
OK 
 
8) DQM MONITORING 
OK 
</pre>
<!-- Author: ByeongRok -->




<!-- Date: Mon Jul 25 07:42:51 2005 -->
<pre>   OFFLINE REPORT: Mon Jul 25 07:34:38 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 201371  2005.07.20 11:10:20  (4 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 201154  2005.07.17 09:03:00  (7 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       201657         PHYSICS_3_06   1 2005.07.24   19:12:52     PROD_PHYSICS_CD 
       This is 12 hours old. 
   11A is not available for: 
       201607         PHYSICS_3_06   1 2005.07.22   19:20:11     PROD_PHYSICS_CD 
       This is 2 day(s), 12 hours old. 
   17A is not available for: 
       201607         PHYSICS_3_06   1 2005.07.22   19:20:11     PROD_PHYSICS_CD 
       This is 2 day(s), 12 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) SAMFarm MONITORING 
OK 
 
8) DQM MONITORING 
no DQM histograms 
</pre>
<!-- Author: ByeongRok -->


<!-- Date: Mon Jul 25 15:45:43 2005 -->
<pre>Tape write error. 
email sent to cdfdh_oper. 
 
2005-Jul-25 14:45:54 encp FAILED TO COPY FILESET CB7691.2 
source directory: /cdf/b0/disk7/stager/CB7691.2 
source files: er0313c9.009dphys er0313c9.00a7phys er0313c9.00b1phys 
er0313c9.00bbphys er0313ca.0001phys er0313ca.0011phys er0313ca.001cphys 
er0313ca.0027phys er0313ca.0032phys 
destination dir: /pnfs/cdfen/filesets/CB/CB76/CB7691/CB7691.2 
The rest of the filesets will not be encpied: CB7691.2 CB7691.3 CB7691.4 
</pre>
<!-- Author: ByeongRok -->
<!-- Comment: Tue Jul 26 07;50;27 comment by...ByeongRok -->
<pre>forward to cdf_offline_operations</pre>
<!-- Date: Tue Jul 26 07:38:24 2005 -->
<pre>   OFFLINE REPORT: Tue Jul 26 07:34:18 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 201371  2005.07.20 11:10:20  (5 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 201154  2005.07.17 09:03:00  (8 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       201657         PHYSICS_3_06   1 2005.07.24   19:12:52     PROD_PHYSICS_CD 
       This is 1 day(s), 12 hours old. 
   11A is not available for: 
       201644         PHYSICS_3_06   1 2005.07.24   01:10:34     PROD_PHYSICS_CD 
       This is 2 day(s), 6 hours old. 
   17A is not available for: 
       201644         PHYSICS_3_06   1 2005.07.24   01:10:34     PROD_PHYSICS_CD 
       This is 2 day(s), 6 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) SAMFarm MONITORING 
OK 
 
8) DQM MONITORING 
OK 
</pre>
<!-- Author: ByeongRok -->



<!-- Date: Tue Jul 26 08:44:52 2005 -->
<pre>data logger disks 
plots are out of date. 
 
sent an email to cdf_csl 
ByeongRok(offline CO) 
</pre>
<!-- Author: ByeongRok -->
<!-- Comment: Tue Jul 26 14;05;24 comment by...ByeongRok -->
<pre>fixed</pre>
<!-- Date: Tue Jul 26 11:41:49 2005 -->
<pre>One of UIUC member, 
ID is cawilsn2 is having long-term inactive database connections. 
Notifying one of UIUC member (Heather) 
</pre>
<!-- Author: ByeongRok -->
<!-- Comment: Tue Jul 26 13;56;42 comment by...ByeongRok -->
<pre>fixed.</pre>



<!-- Date: Wed Jul 27 07:51:11 2005 -->
<pre>   OFFLINE REPORT: Wed Jul 27 07:37:26 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 201371  2005.07.20 11:10:20  (6 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 201154  2005.07.17 09:03:00  (9 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       201673         PHYSICS_3_06   1 2005.07.25   09:45:23     PROD_PHYSICS_CD 
       This is 1 day(s), 21 hours old. 
   11A is not available for: 
       201658         PHYSICS_3_06   1 2005.07.24   19:33:26     PROD_PHYSICS_CD 
       This is 2 day(s), 12 hours old. 
   17A is not available for: 
       201658         PHYSICS_3_06   1 2005.07.24   19:33:26     PROD_PHYSICS_CD 
       This is 2 day(s), 12 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) SAMFarm MONITORING 
OK 
 
8) DQM MONITORING 
no D0.ps in 198881_aexp0b_psfiles 
</pre>
<!-- Author: ByeongRok -->
<!-- Comment: Wed Jul 27 13;34;35 comment by...ByeongRok -->
<pre>For the DQM monitoring,
they said that's normal
</pre>
<!-- Date: Wed Jul 27 11:45:36 2005 -->
<pre>fncdfsrv2 are not responding. 
sent an email to 
suen, aidan and bernd.</pre>
<!-- Author: ByeongRok -->
<!-- Comment: Wed Jul 27 13;33;49 comment by...ByeongRok -->
<pre>fixed</pre>



<!-- Date: Wed Jul 27 13:40:04 2005 -->
<pre>fcdfdata064 is dead. 
sent an email to cdfsam-admin 
</pre>
<!-- Author: ByeongRok -->
<!-- Comment: Wed Jul 27 14;42;48 comment by...ByeongRok -->
<pre>fixed</pre>


<!-- Date: Thu Jul 28 07:38:21 2005 -->
<pre>   OFFLINE REPORT: Thu Jul 28 07:24:20 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 201371  2005.07.20 11:10:20  (7 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 201154  2005.07.17 09:03:00  (10 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       201795         PHYSICS_3_06   1 2005.07.27   08:02:31     PROD_PHYSICS_CD 
       This is 23 hours old. 
   11A is not available for: 
       201673         PHYSICS_3_06   1 2005.07.25   09:45:23     PROD_PHYSICS_CD 
       This is 2 day(s), 21 hours old. 
   17A is not available for: 
       201673         PHYSICS_3_06   1 2005.07.25   09:45:23     PROD_PHYSICS_CD 
       This is 2 day(s), 21 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) SAMFarm MONITORING 
OK 
 
8) DQM MONITORING 
SiCt.ps in 200883_aexp0b_psfiles is empty 
</pre>
<!-- Author: ByeongRok -->


<!-- Date: Thu Jul 28 15:53:31 2005 -->
<pre>express production  
more than 8 hours ago. 
 
Jul 28 06:23 aa03145e.0001phys  run number 201822 
 
Sent an email to cwolfe Aidan 
</pre>
<!-- Author: ByeongRok -->


<!-- Date: Fri Jul 29 07:41:04 2005 -->
<pre>   OFFLINE REPORT: Fri Jul 29 07:32:22 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 201371  2005.07.20 11:10:20  (8 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 201154  2005.07.17 09:03:00  (11 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       201795         PHYSICS_3_06   1 2005.07.27   08:02:31     PROD_PHYSICS_CD 
       This is 1 day(s), 23 hours old. 
   17A is not available for: 
       201795         PHYSICS_3_06   1 2005.07.27   08:02:31     PROD_PHYSICS_CD 
       This is 1 day(s), 23 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) SAMFarm MONITORING 
OK 
 
8) DQM MONITORING 
OK 
</pre>
<!-- Author: ByeongRok -->


<!-- Date: Fri Jul 29 07:43:41 2005 -->
<pre>two usesr(ggiurgiu, necula) are having more than 20 connection to cdfofprd database.  
email is sent to them and cdfdb-support  
 
</pre>
<!-- Author: ByeongRok -->


<!-- Date: Fri Jul 29 08:44:30 2005 -->
<pre>Control Room is freezing cold  
 
Brrrr ... </pre>
<!-- Author: Arkadiy -->


<!-- Date: Fri Jul 29 10:42:06 2005 -->
<pre>DCache doors information is not updating  
 
sent an email to Aidan </pre>
<!-- Author: Arkadiy -->
<!-- Comment: Fri Jul 29 11;23;35 comment by...Aidan -->
<pre>door 5 is in trouble - Dehong has contacted dcache_admin</pre>


<!-- Date: Fri Jul 29 11:33:39 2005 -->
<pre>These calibrations are not available: 
 
 
11A: 
 
 201822         PHYSICS_3_06   1 2005.07.27   23:22:59     PROD_PHYSICS_CDF        11A is not available 
  201883         PHYSICS_3_06   1 2005.07.28   12:02:34     PROD_PHYSICS_CDF        11A is not available 
  201947         PHYSICS_3_06   1 2005.07.29   02:22:21     PROD_PHYSICS_CDF        11A is not available 
 
 
17A: 
  201822         PHYSICS_3_06   1 2005.07.27   23:22:59     PROD_PHYSICS_CDF        17A is not available 
  201883         PHYSICS_3_06   1 2005.07.28   12:02:34     PROD_PHYSICS_CDF        17A is not available 
  201947         PHYSICS_3_06   1 2005.07.29   02:22:21     PROD_PHYSICS_CDF        17A is not available 
 
sent an email to the experts 
 
</pre>
<!-- Author: Arkadiy -->


<!-- Date: Fri Jul 29 12:43:15 2005 -->
<pre>user ggiurgiu has a large number of open inactive  
connections to the cdfofprd database  
 
email sent </pre>
<!-- Author: Arkadiy -->


<!-- Date: Fri Jul 29 13:13:49 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-232&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-232  width='300' align='top'></a></td><td>this plot looks suspicious</td></tr></table>
<!-- Author: Arkadiy -->


<!-- Date: Fri Jul 29 16:11:49 2005 -->
<pre>working on a new version of makeReport script </pre>
<!-- Author: Arkadiy -->


<!-- Date: Sat Jul 30 08:05:20 2005 -->
<pre>I'm having trouble starting a new shift  
Where is the button :))</pre>
<!-- Author: Arkadiy -->


<!-- Date: Sat Jul 30 08:08:42 2005 -->
<pre>  OFFLINE REPORT: Sat Jul 30 07:53:13 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 201371  2005.07.20 11:10:20  (9 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 201154  2005.07.17 09:03:00  (12 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is not available for: 
       201948         PHYSICS_3_06   1 2005.07.29   04:18:18     PROD_PHYSICS_CD 
       This is 1 day(s), 3 hours old. 
   11A is not available for: 
       201883         PHYSICS_3_06   1 2005.07.28   12:02:34     PROD_PHYSICS_CD 
       This is 1 day(s), 19 hours old. 
   17A is not available for: 
       201883         PHYSICS_3_06   1 2005.07.28   12:02:34     PROD_PHYSICS_CD 
       This is 1 day(s), 19 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) SAMFarm MONITORING 
OK 
 
8) DQM MONITORING 
OK 
 
 
This is done using the old version of makeReport script  
 
 
</pre>
<!-- Author: Arkadiy -->


<!-- Date: Sat Jul 30 09:09:14 2005 -->
<pre>DCache doors information is not updating  
 
 
Sat Jul 30 05:44:50 CDT 2005 30 doors in 689 sec 
 OOPS, bad door 05  
Sat Jul 30 06:38:22 CDT 2005 30 doors in 301 sec 
 OOPS, bad door 05  
 OOPS, bad door 05  
mv: cannot stat `valdoors.tmp': No such file or directory 
 
This is the last log </pre>
<!-- Author: Arkadiy -->


<!-- Date: Sat Jul 30 09:12:01 2005 -->
<pre>DCache ping times out from time to time  
 
Sat Jul 30 07:09:45 CDT 2005 0:00.55 
Sat Jul 30 07:19:47 CDT 2005 1:22:09 
Sat Jul 30 08:52:00 CDT 2005 8:04.13 
Sat Jul 30 09:10:06 CDT 2005 0:01.05 
 
 
</pre>
<!-- Author: Arkadiy -->


<!-- Date: Sat Jul 30 12:03:04 2005 -->
<pre>the new version of makeReport script is working  
this is how it looks  
 
 OFFLINE REPORT: Sat Jul 30 12:01:28 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 201371  2005.07.20 11:10:20  (10 day(s), 0 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 201154  2005.07.17 09:03:00  (13 day(s), 2 hours ago) 
 
The latest consumed by cdf-fncdfsrv0 aphysr file's run: 
Run Date  2005-07-30 
Run number  201707 
Run time  00:29:33 
 
3) CALIBRATIONS 
   10A is not available for: 
       201948         PHYSICS_3_06   1 2005.07.29   04:18:18     PROD_PHYSICS_CD 
       This is 1 day(s), 7 hours old. 
   11A is not available for: 
       201883         PHYSICS_3_06   1 2005.07.28   12:02:34     PROD_PHYSICS_CD 
       This is 1 day(s), 23 hours old. 
   17A is not available for: 
       201883         PHYSICS_3_06   1 2005.07.28   12:02:34     PROD_PHYSICS_CD 
       This is 1 day(s), 23 hours old. 
 
4) DB MONITORING 
 
 
5) DH MONITORING 
 
 
6) CAF MONITORING 
 
 
7) SAMFarm MONITORING 
 
 
8) DQM MONITORING 
</pre>
<!-- Author: Arkadiy -->


<!-- Date: Sat Jul 30 12:14:44 2005 -->
<pre>DCashe door 5 is still bad  
 
 OOPS, bad door 05  
Sat Jul 30 09:37:51 CDT 2005 30 doors in 269 sec 
 OOPS, bad door 05  
Sat Jul 30 10:37:23 CDT 2005 30 doors in 241 sec 
 OOPS, bad door 05  
Sat Jul 30 11:37:22 CDT 2005 30 doors in 240 sec 
 
</pre>
<!-- Author: Arkadiy -->


<!-- Date: Sat Jul 30 14:16:41 2005 -->
<pre>User necula has more than 20 inactive connections to cdfofprd  
database. 
 
Sent an email to him and to cdfdb-support </pre>
<!-- Author: Arkadiy -->


<!-- Date: Sun Jul 31 07:51:56 2005 -->
<pre>  OFFLINE REPORT: Sun Jul 31 07:43:31 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR: 
   RUN DATE  2005-07-30 
   RUN NUMBER  201672 
   RUN TIME  21:54:36 
 
2) CALIBRATIONS 
   10A is not available for: 
       201948         PHYSICS_3_06   1 2005.07.29   04:18:18     PROD_PHYSICS_CD 
       This is 2 day(s), 3 hours old. 
   11A is not available for: 
       201883         PHYSICS_3_06   1 2005.07.28   12:02:34     PROD_PHYSICS_CD 
       This is 2 day(s), 19 hours old. 
   17A is not available for: 
       201883         PHYSICS_3_06   1 2005.07.28   12:02:34     PROD_PHYSICS_CD 
       This is 2 day(s), 19 hours old. 
 
3) DB MONITORING 
 
OK 
 
4) DH MONITORING 
 
Yesterday DCACHE door 5 was not working  
it seems to be fine right now  
5) CAF MONITORING 
 
OK 
6) SAMFarm MONITORING 
 
OK 
7) DQM MONITORING 
OK</pre>
<!-- Author: Arkadiy -->


<!-- Date: Sun Jul 31 09:00:28 2005 -->
<pre>fcdfdata064 appears to be red on the ResourceMan monitor  
 
sent an email to cdfsam-admin</pre>
<!-- Author: Arkadiy -->


<!-- Date: Mon Aug 1 08:03:23 2005 -->
<pre> OFFLINE REPORT: Mon Aug 1 08:01:00 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR: 
   RUN DATE  2005-07-31 
   RUN NUMBER  201947 
   RUN TIME  15:07:00 
 
2) CALIBRATIONS 
   10A is not available for: 
       201948         PHYSICS_3_06   1 2005.07.29   04:18:18     PROD_PHYSICS_CD 
       This is 3 day(s), 3 hours old. 
   11A is not available for: 
       201948         PHYSICS_3_06   1 2005.07.29   04:18:18     PROD_PHYSICS_CD 
       This is 3 day(s), 3 hours old. 
   17A is not available for: 
       201948         PHYSICS_3_06   1 2005.07.29   04:18:18     PROD_PHYSICS_CD 
       This is 3 day(s), 3 hours old. 
 
3) DB MONITORING 
 
OK 
4) DH MONITORING 
 
OK 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
fcdfdata064 was giving trouble yesterday  
now is fine  
 
7) DQM MONITORING 
 
OK</pre>
<!-- Author: Arkadiy -->


<!-- Date: Mon Aug 1 08:21:52 2005 -->
<pre>New tape write error: 
 
  
 
2005-Aug-01 00:40:16 encp FAILED TO COPY FILESET CB7782.1 
source directory: /cdf/b0/disk8/stager/CB7782.1 
source files: dr031539.01bfphys dr031539.01e0phys dr031539.0204phys dr031539.0221phys dr031539.0243phys dr031539.0260phys dr031539.0284phys dr031539.029fphys 
destination dir: /pnfs/cdfen/filesets/CB/CB77/CB7782/CB7782.1 
The rest of the filesets will not be encpied: CB7782.1 CB7782.2 CB7782.3 CB7782.4  
 
email sent to: cdfdh_oper@fnal.gov 
</pre>
<!-- Author: Arkadiy -->


<!-- Date: Mon Aug 1 08:32:55 2005 -->
<pre>DCHACHE problems  
 
an email sent to Aidan Robson  
 
. 
The last update on the status of DCHACHE doors 
 
was at Mon Aug  1 02:43:48 CDT 2005 30 doors in 21 sec 
 
about 6 hours ago 
 
2. DCACHE load 
is 
 
Waiting for 
     149 Door 
     238 Pnfs 
       0 Pool 
       5 Limbo 
 
waiting for pnfs is 238 &gt; 50 
 
3. DCACHE ping 
 
the last happened at Mon Aug  1 03:34:05 CDT 2005 0:00.54 
 
</pre>
<!-- Author: Arkadiy -->


<!-- Date: Mon Aug 1 08:39:40 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-233&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-233  width='300' align='top'></a></td><td>Opps  
what is this ?</td></tr></table>
<!-- Author: Arkadiy Bolshov -->


<!-- Date: Mon Aug 1 08:40:41 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-234&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-234  width='300' align='top'></a></td><td>And this </td></tr></table>
<!-- Author: Arkadiy -->


<!-- Date: Tue Aug 2 07:48:32 2005 -->
<pre>  OFFLINE REPORT: Tue Aug 2 07:41:47 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR: 
   RUN DATE  2005-07-31 
   RUN NUMBER  201947 
   RUN TIME  15:07:00 
 
2) CALIBRATIONS 
   10A is not available for: 
       202041         PHYSICS_3_06   1 2005.07.31   10:00:33     PROD_PHYSICS_CD 
       This is 1 day(s), 21 hours old. 
   11A is not available for: 
       201948         PHYSICS_3_06   1 2005.07.29   04:18:18     PROD_PHYSICS_CD 
       This is 4 day(s), 3 hours old. 
   17A is not available for: 
       201948         PHYSICS_3_06   1 2005.07.29   04:18:18     PROD_PHYSICS_CD 
       This is 4 day(s), 3 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
 
OK 
7) DQM MONITORING 
OK</pre>
<!-- Author: Arkadiy -->


<!-- Date: Tue Aug 2 13:48:08 2005 -->
<pre>User makagan has a lot of inactive connections to the  
 
to the cdfstrm1 database  
 
am email sent to him and to cdfdb-support</pre>
<!-- Author: Arkadiy -->


<!-- Date: Tue Aug 2 13:51:00 2005 -->
<pre>a new tape write error  
 
005-Aug-02 12:51:49 encp FAILED TO COPY FILESET CB7810.4 
source directory: /cdf/b0/disk9/stager/CB7810.4 
source files: hr031568.02a0phys hr031568.02adphys hr031568.02baphys hr031568.02c7phys hr031568.02d4phys hr031568.02e1phys hr031568.02eephys hr031568.02fcphys hr031568.0309phys 
destination dir: /pnfs/cdfen/filesets/CB/CB78/CB7810/CB7810.4 
The rest of the filesets will not be encpied: CB7810.4  
 
sent an email to cdfdh_oper </pre>
<!-- Author: Arkadiy -->


<!-- Date: Tue Aug 2 15:41:38 2005 -->
<pre>User herndon has a number of stfstrm1 connections to the data base 
above the limit  
 
sent an eamil to Petar Maksimovic  
</pre>
<!-- Author: Arkadiy -->


<!-- Date: Wed Aug 3 08:19:23 2005 -->
<pre>  OFFLINE REPORT: Wed Aug 3 08:03:08 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-03 
   RUN NUMBER  202003 
   RUN TIME  22:13:26 
 
2) CALIBRATIONS 
   10A is not available for: 
       202070 PHYSICS_HIGHLUM_3_06   1 2005.08.01   13:39:15     PROD_PHYSICS_CD 
       This is 1 day(s), 18 hours old. 
   11A is not available for: 
       202041         PHYSICS_3_06   1 2005.07.31   10:00:33     PROD_PHYSICS_CD 
       This is 2 day(s), 22 hours old. 
   17A is not available for: 
       202041         PHYSICS_3_06   1 2005.07.31   10:00:33     PROD_PHYSICS_CD 
       This is 2 day(s), 22 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
one tape writing error happened yesterday  
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
OK 
 
</pre>
<!-- Author: Arkadiy -->


<!-- Date: Wed Aug 3 08:25:50 2005 -->
<pre>Number of jobs not consuming cpu is abnormally high on both caf and cafcondor  
 
an email sent to the experts</pre>
<!-- Author: Arkadiy -->


<!-- Date: Wed Aug 3 15:45:04 2005 -->
<pre>Yet another tape write error 
email sent to cdfdh_oper 
 
2005-Aug-03 15:13:59 encp FAILED TO COPY FILESET CB7827.1 
source directory: /cdf/b0/disk7/stager/CB7827.1 
source files: er031597.0338phys er031597.0349phys er031597.035bphys er031598.0001phys er031598.001aphys er031598.002ephys er031598.0042phys er031598.0057phys er031598.006bphys 
destination dir: /pnfs/cdfen/filesets/CB/CB78/CB7827/CB7827.1 
The rest of the filesets will not be encpied: CB7827.1 CB7827.2 CB7827.3 CB7827.4</pre>
<!-- Author: Arkadiy -->


<!-- Date: Wed Aug 3 15:47:23 2005 -->
<pre>Waiting for pnfs is too high  
an email sent to Aidan  
 
Waiting for 
     341 Door 
     336 Pnfs 
      93 Pool 
     661 Limbo 
 
 
</pre>
<!-- Author: Arkadiy -->


<!-- Date: Wed Aug 3 15:48:25 2005 -->
<pre>DCACHE ping is not updating for a long time </pre>
<!-- Author: Arkadiy  -->


<!-- Date: Wed Aug 3 15:49:42 2005 -->
<pre>both CAF and CAFCONDOR are bad all day long  
with red line (number of stuck sections)  
well over 0.5 k </pre>
<!-- Author: Arkadiy -->


<!-- Date: Thu Aug 4 08:26:10 2005 -->
<pre>  OFFLINE REPORT: Thu Aug 4 07:59:34 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-03 
   RUN NUMBER  202041 
   RUN TIME  11:30:47 
 
2) CALIBRATIONS 
   10A is not available for: 
       202081         PHYSICS_3_06   1 2005.08.01   17:22:29     PROD_PHYSICS_CD 
       This is 2 day(s), 14 hours old. 
   11A is not available for: 
       202054         PHYSICS_3_06   1 2005.08.01   10:05:04     PROD_PHYSICS_CD 
       This is 2 day(s), 21 hours old. 
   17A is not available for: 
       202054         PHYSICS_3_06   1 2005.08.01   10:05:04     PROD_PHYSICS_CD 
       This is 2 day(s), 21 hours old. 
 
3) DB MONITORING 
OK 
 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
OK 
</pre>
<!-- Author: Arkadiy -->


<!-- Date: Thu Aug 4 09:51:56 2005 -->
<pre>tape write error  
 
an email sent to dfdh_oper@fnal.gov 
 
 
2005-Aug-03 18:45:10 encp FAILED TO COPY FILESET CB7829.3 
source directory: /cdf/b0/disk7/stager/CB7829.3 
source files: jr031598.0153phys jr031598.0175phys jr031598.0198phys jr031598.01bbphys jr031598.01e0phys jr031598.0205phys jr031598.022bphys jr031598.0252phys 
destination dir: /pnfs/cdfen/filesets/CB/CB78/CB7829/CB7829.3 
The rest of the filesets will not be encpied: CB7829.3 CB7829.4 </pre>
<!-- Author: Arkadiy -->


<!-- Date: Thu Aug 4 15:16:49 2005 -->
<pre>user tsuno has a large number of inactive database connections  
 
an email sent to him and to Petar</pre>
<!-- Author: Arkadiy -->


<!-- Date: Fri Aug 5 07:38:00 2005 -->
<pre>   OFFLINE REPORT: Fri Aug 5 07:30:52 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-03 
   RUN NUMBER  202041 
   RUN TIME  11:30:47 
 
2) CALIBRATIONS 
   10A is not available for: 
       202174         PHYSICS_3_06   1 2005.08.03   20:56:11     PROD_PHYSICS_CD 
       This is 1 day(s), 10 hours old. 
   11A is not available for: 
       202054         PHYSICS_3_06   1 2005.08.01   10:05:04     PROD_PHYSICS_CD 
       This is 3 day(s), 21 hours old. 
   17A is not available for: 
       202054         PHYSICS_3_06   1 2005.08.01   10:05:04     PROD_PHYSICS_CD 
       This is 3 day(s), 21 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
DCACHE is in trouble right now  
an email sent to Aidan  
 
5) CAF MONITORING 
the number of idle sections went up  
probably due to DCACHE problems 
 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
OK 
</pre>
<!-- Author: Arkadiy -->


<!-- Date: Fri Aug 5 08:29:04 2005 -->
<pre>uimoiiookmm[k[mp</pre>
<!-- Author: jkmlm;ok;lkm -->


<!-- Date: Sat Aug 6 10:19:28 2005 -->
<pre>   OFFLINE REPORT: Fri Aug 6 07:32:22 CDT 2005 
 
1) PROCESSING (CPU): latest processed run 
    Stream A: 201371  2005.07.20 11:10:20  (8 day(s), 20 hours ago) 
 
2) PROCESSING (TO TAPE): 
    Stream A: 201154  2005.07.17 09:03:00  (11 day(s), 22 hours ago) 
 
3) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       201795         PHYSICS_3_06   1 2005.07.27   08:02:31     PROD_PHYSICS_CD 
       This is 2 day(s), 23 hours old. 
   17A is not available for: 
       201795         PHYSICS_3_06   1 2005.07.27   08:02:31     PROD_PHYSICS_CD 
       This is 2 day(s), 23 hours old. 
 
4) DB MONITORING 
OK 
 
5) DH MONITORING 
OK 
 
6) CAF MONITORING 
OK 
 
7) SAMFarm MONITORING 
OK 
 
8) DQM MONITORING 
OK 
</pre>
<!-- Author: menzione -->


<!-- Date: Sun Aug 7 09:10:24 2005 -->
<pre>   OFFLINE REPORT: Sun Aug 7 08:01:20 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-06 
   RUN NUMBER  202135 
   RUN TIME  17:29:08 
 
2) CALIBRATIONS 
   10A is not available for: 
       202174         PHYSICS_3_06   1 2005.08.03   20:56:11     PROD_PHYSICS_CD 
       This is 3 day(s), 12 hours old. 
   11A is not available for: 
       202174         PHYSICS_3_06   1 2005.08.03   20:56:11     PROD_PHYSICS_CD 
       This is 3 day(s), 12 hours old. 
   17A is not available for: 
       202174         PHYSICS_3_06   1 2005.08.03   20:56:11     PROD_PHYSICS_CD 
       This is 3 day(s), 12 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
OK 
 
</pre>
<!-- Author: menzione -->


<!-- Date: Mon Aug 8 09:06:27 2005 -->
<pre>  OFFLINE REPORT: Mon Aug 8 07:57:44 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-06 
   RUN NUMBER  202135 
   RUN TIME  17:29:08 
 
2) CALIBRATIONS 
   10A is not available for: 
       202174         PHYSICS_3_06   1 2005.08.03   20:56:11     PROD_PHYSICS_CD 
       This is 4 day(s), 12 hours old. 
   11A is not available for: 
       202174         PHYSICS_3_06   1 2005.08.03   20:56:11     PROD_PHYSICS_CD 
       This is 4 day(s), 12 hours old. 
   17A is not available for: 
       202174         PHYSICS_3_06   1 2005.08.03   20:56:11     PROD_PHYSICS_CD 
       This is 4 day(s), 12 hours old. 
This problem persists since early friday. 
Is due to silicon calibrations too old. 
Apparently was cured by hands, Bo saturday morning. 
Meanwhile sunday evening new calib were taken. 
Still, from this side of the screen the flow looks to 
me rather clogged. 
 
3) DB MONITORING 
 
 
4) DH MONITORING 
 
 
5) CAF MONITORING 
 
 
6) SAMFarm MONITORING 
 
 
7) DQM MONITORING 
 
 
</pre>
<!-- Author: Aldo -->


<!-- Date: Tue Aug 9 09:28:39 2005 -->
<pre>  OFFLINE REPORT: Tue Aug 9 08:25:11 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-06 
   RUN NUMBER  202135 
   RUN TIME  17:29:08 
 
2) CALIBRATIONS 
   10A is not available for: 
       202504         PHYSICS_3_06   1 2005.08.09   04:59:05     PROD_PHYSICS_CD 
       This is 4 hours old. 
       This problem is cured 
        
   11A is not available for: 
       202174         PHYSICS_3_06   1 2005.08.03   20:56:11     PROD_PHYSICS_CD 
       This is 5 day(s), 12 hours old. 
       Expected flow for later today 
        
   17A is not available for: 
       202174         PHYSICS_3_06   1 2005.08.03   20:56:11     PROD_PHYSICS_CD 
       This is 5 day(s), 12 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
??</pre>
<!-- Author: Aldo -->


<!-- Date: Wed Aug 10 09:36:09 2005 -->
<pre> OFFLINE REPORT: Wed Aug 10 09:32:28 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-09 
   RUN NUMBER  202435 
   RUN TIME  20:06:39 
 
2) CALIBRATIONS 
   10A is not available for: 
       202509         PHYSICS_3_06   1 2005.08.09   10:25:16     PROD_PHYSICS_CD 
       This is 23 hours old. 
   11A is not available for: 
       202452         PHYSICS_3_06   1 2005.08.08   16:59:48     PROD_PHYSICS_CD 
       This is 1 day(s), 16 hours old. 
   17A is not available for: 
       202452         PHYSICS_3_06   1 2005.08.08   16:59:48     PROD_PHYSICS_CD 
       This is 1 day(s), 16 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
Ok 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
OK 
 
</pre>
<!-- Author: aldo -->


<!-- Date: Thu Aug 11 08:31:43 2005 -->
<pre> OFFLINE REPORT: Thu Aug 11 08:06:27 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-10 
   RUN NUMBER  202509 
   RUN TIME  18:06:38 
 
2) CALIBRATIONS 
   10A is not available for: 
       202526         PHYSICS_3_06   1 2005.08.09   20:08:31     PROD_PHYSICS_CD 
       This is 1 day(s), 12 hours old. 
   11A is not available for: 
       202513         PHYSICS_3_06   1 2005.08.09   11:19:01     PROD_PHYSICS_CD 
       This is 1 day(s), 21 hours old. 
   17A is not available for: 
       202513         PHYSICS_3_06   1 2005.08.09   11:19:01     PROD_PHYSICS_CD 
       This is 1 day(s), 21 hours old. 
        
       all those processes start to get delayed limit 48 hours 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
Start working 
 
There is been a few messages of misfunctionig 
All had been answered 
 
</pre>
<!-- Author: aldo -->


<!-- Date: Fri Aug 12 08:38:22 2005 -->
<pre>   OFFLINE REPORT: Fri Aug 12 08:32:38 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-11 
   RUN NUMBER  202541 
   RUN TIME  19:07:06 
 
2) CALIBRATIONS 
   10A is not available for: 
       202695         PHYSICS_3_06   1 2005.08.12   01:35:19     PROD_PHYSICS_CD 
       This is 6 hours old. 
   11A is not available for: 
       202542         PHYSICS_3_06   1 2005.08.10   07:58:51     PROD_PHYSICS_CD 
       This is 2 day(s), 0 hours old. 
   17A is not available for: 
       202542         PHYSICS_3_06   1 2005.08.10   07:58:51     PROD_PHYSICS_CD 
       This is 2 day(s), 0 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
 
</pre>
<!-- Author: aldo -->


<!-- Date: Sat Aug 13 08:48:01 2005 -->
<pre>Everything seems to be OK, except that no automated report could be produced: an attempt to produce report led to the following: 
 
cdfopr@fcdflnx3:~&gt;./makeReport 
cut: write error: Disk quota exceeded 
cut: write error: Disk quota exceeded 
cut: write error: Disk quota exceeded 
NEdit: Can't open display 
</pre>
<!-- Author: Guenakh -->


<!-- Date: Sat Aug 13 10:03:04 2005 -->
<pre>Paolo Giovacchini helped to clean the disk from ald reports. 
Now new report can be produced, but one problem remains: 
cdfopr@fcdflnx3:~&gt;./makeReport 
NEdit: Can't open display 
 
On the other hand, is the command ./makeReport.sh is used, I can get the report in the same window, here it is: 
 
 OFFLINE REPORT: Sat Aug 13 09:49:16 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR: 
   RUN DATE  2005-08-12 
   RUN NUMBER  202591 
   RUN TIME  18:03:56 
 
2) CALIBRATIONS 
   10A is not available for: 
       202697         PHYSICS_3_06   1 2005.08.12   10:16:29     PROD_PHYSICS_CD 
       This is 23 hours old. 
   11A is not available for: 
       202592         PHYSICS_3_06   1 2005.08.11   00:23:38     PROD_PHYSICS_CD 
       This is 2 day(s), 9 hours old. 
   17A is not available for: 
       202592         PHYSICS_3_06   1 2005.08.11   00:23:38     PROD_PHYSICS_CD 
       This is 2 day(s), 9 hours old. 
 
3) DB MONITORING 
 
ok 
 
4) DH MONITORING 
 
ok 
5) CAF MONITORING 
 
ok 
6) SAMFarm MONITORING 
 
ok, CPU usage low as expected 
 
7) DQM MONITORING 
 
not operational (as expected) 
</pre>
<!-- Author: Guenakh -->


<!-- Date: Sun Aug 14 08:28:15 2005 -->
<pre>  OFFLINE REPORT: Sun Aug 14 08:24:35 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR: 
   RUN DATE  2005-08-13 
   RUN NUMBER  202695 
   RUN TIME  18:09:06 
 
2) CALIBRATIONS 
   10A is not available for: 
       202735         PHYSICS_3_06   1 2005.08.13   07:32:37     PROD_PHYSICS_CD 
       This is 1 day(s), 0 hours old. 
   11A is not available for: 
       202697         PHYSICS_3_06   1 2005.08.12   10:16:29     PROD_PHYSICS_CD 
       This is 1 day(s), 22 hours old. 
   17A is not available for: 
       202697         PHYSICS_3_06   1 2005.08.12   10:16:29     PROD_PHYSICS_CD 
       This is 1 day(s), 22 hours old. 
 
3) DB MONITORING 
ok 
 
4) DH MONITORING 
ok 
 
5) CAF MONITORING 
ok 
 
6) SAMFarm MONITORING 
ok, not much activity (as expected) 
 
7) DQM MONITORING 
not working 
</pre>
<!-- Author: Guenakh -->


<!-- Date: Mon Aug 15 08:15:53 2005 -->
<pre>  OFFLINE REPORT: Mon Aug 15 08:13:37 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR: 
   RUN DATE  2005-08-13 
   RUN NUMBER  202695 
   RUN TIME  18:09:06 
 
2) CALIBRATIONS 
   10A is not available for: 
       202741         PHYSICS_3_06   1 2005.08.14   07:17:45     PROD_PHYSICS_CD 
       This is 1 day(s), 0 hours old. 
   11A is not available for: 
       202697         PHYSICS_3_06   1 2005.08.12   10:16:29     PROD_PHYSICS_CD 
       This is 2 day(s), 21 hours old. 
   17A is not available for: 
       202697         PHYSICS_3_06   1 2005.08.12   10:16:29     PROD_PHYSICS_CD 
       This is 2 day(s), 21 hours old. 
 
3) DB MONITORING 
ok 
 
4) DH MONITORING 
ok 
 
5) CAF MONITORING 
ok 
 
6) SAMFarm MONITORING 
ok 
 
7) DQM MONITORING 
not working 
</pre>
<!-- Author: Guenakh -->


<!-- Date: Mon Aug 15 10:33:19 2005 -->
<pre>Calibrations 11A and 17A are 3 days old (see below) 
Sent a message  
 
 CALIBRATIONS 
   10A is not available for: 
       202770 PHYSICS_HIGHLUM_3_06   1 2005.08.14   14:57:19     PROD_PHYSICS_CD 
       This is 19 hours old. 
   11A is not available for: 
       202697         PHYSICS_3_06   1 2005.08.12   10:16:29     PROD_PHYSICS_CD 
       This is 3 day(s), 0 hours old. 
   17A is not available for: 
       202697         PHYSICS_3_06   1 2005.08.12   10:16:29     PROD_PHYSICS_CD 
       This is 3 day(s), 0 hours old. 
 
</pre>
<!-- Author: Guenakh -->


<!-- Date: Tue Aug 16 08:17:30 2005 -->
<pre>   OFFLINE REPORT: Tue Aug 16 08:14:54 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR: 
   RUN DATE  2005-08-15 
   RUN NUMBER  202738 
   RUN TIME  12:04:35 
 
2) CALIBRATIONS 
   10A is not available for: 
       202817         PHYSICS_3_06   1 2005.08.15   17:19:24     PROD_PHYSICS_CD 
       This is 14 hours old. 
   11A is not available for: 
       202741         PHYSICS_3_06   1 2005.08.14   07:17:45     PROD_PHYSICS_CD 
       This is 2 day(s), 0 hours old. 
   17A is not available for: 
       202741         PHYSICS_3_06   1 2005.08.14   07:17:45     PROD_PHYSICS_CD 
       This is 2 day(s), 0 hours old. 
 
3) DB MONITORING 
ok 
 
4) DH MONITORING 
ok 
 
5) CAF MONITORING 
ok 
 
6) SAMFarm MONITORING 
underloaded (as expected) 
 
7) DQM MONITORING 
not working (as expected)</pre>
<!-- Author: Guenakh -->


<!-- Date: Wed Aug 17 08:39:11 2005 -->
<pre>   OFFLINE REPORT: Wed Aug 17 08:37:56 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-16 
   RUN NUMBER  202770 
   RUN TIME  10:09:06 
 
2) CALIBRATIONS 
   10A is not available for: 
       202817         PHYSICS_3_06   1 2005.08.15   17:19:24     PROD_PHYSICS_CD 
       This is 1 day(s), 15 hours old. 
   11A is not available for: 
       202771         PHYSICS_3_06   1 2005.08.14   17:45:32     PROD_PHYSICS_CD 
       This is 2 day(s), 14 hours old. 
   17A is not available for: 
       202771         PHYSICS_3_06   1 2005.08.14   17:45:32     PROD_PHYSICS_CD 
       This is 2 day(s), 14 hours old. 
 
3) DB MONITORING 
ok 
 
4) DH MONITORING 
ok 
 
5) CAF MONITORING 
ok 
 
6) SAMFarm MONITORING 
ok 
 
7) DQM MONITORING 
ok 
 
</pre>
<!-- Author: guenakh -->


<!-- Date: Thu Aug 18 08:00:34 2005 -->
<pre>   OFFLINE REPORT: Thu Aug 18 07:58:46 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-16 
   RUN NUMBER  202770 
   RUN TIME  10:09:06 
 
2) CALIBRATIONS 
   10A is not available for: 
       202817         PHYSICS_3_06   1 2005.08.15   17:19:24     PROD_PHYSICS_CD 
       This is 2 day(s), 14 hours old. 
   11A is not available for: 
       202771         PHYSICS_3_06   1 2005.08.14   17:45:32      
 
PROD_PHYSICS_CD 
       This is 3 day(s), 14 hours old. 
   17A is not available for: 
       202771         PHYSICS_3_06   1 2005.08.14   17:45:32     PROD_PHYSICS_CD 
       This is 3 day(s), 14 hours old. 
 
 
Calibrations are old. Sent a message 
 
3) DB MONITORING 
ok 
 
4) DH MONITORING 
ok 
 
5) CAF MONITORING 
ok 
 
6) SAMFarm MONITORING 
ok 
 
7) DQM MONITORING 
ok 
 
</pre>
<!-- Author: Guenakh -->


<!-- Date: Thu Aug 18 08:20:33 2005 -->
<pre>SAM Farm is not working for the last 14 hours. Is it the way it should be at the moment? 
 
 
</pre>
<!-- Author: Guenakh -->


<!-- Date: Fri Aug 19 08:23:10 2005 -->
<pre>   OFFLINE REPORT: Fri Aug 19 07:55:01 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-16 
   RUN NUMBER  202770 
   RUN TIME  10:09:06 
 
2) CALIBRATIONS 
   10A is not available for: 
       202817         PHYSICS_3_06   1 2005.08.15   17:19:24     PROD_PHYSICS_CD 
       This is 3 day(s), 14 hours old. 
   11A is not available for: 
       202771         PHYSICS_3_06   1 2005.08.14   17:45:32     PROD_PHYSICS_CD 
       This is 4 day(s), 14 hours old. 
   17A is not available for: 
       202771         PHYSICS_3_06   1 2005.08.14   17:45:32     PROD_PHYSICS_CD 
       This is 4 day(s), 14 hours old. 
 
3) DB MONITORING 
ok 
 
 
4) DH MONITORING 
ok 
 
5) CAF MONITORING 
ok 
 
6) SAMFarm MONITORING 
still not working 
 
7) DQM MONITORING 
ok 
 
</pre>
<!-- Author: Guenakh -->


<!-- Date: Fri Aug 19 15:46:48 2005 -->
<pre>Farms have started processing data again at 12pm</pre>
<!-- Author: ceballos -->


<!-- Date: Sat Aug 20 09:07:01 2005 -->
<pre>   OFFLINE REPORT: Sat Aug 20 09:05:32 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-16 
   RUN NUMBER  202770 
   RUN TIME  10:09:06 
 
2) CALIBRATIONS 
   10A is not available for: 
       203055         PHYSICS_3_07   1 2005.08.18   22:11:08     PROD_PHYSICS_CD 
       This is 1 day(s), 10 hours old. 
   11A is not available for: 
       202771         PHYSICS_3_06   1 2005.08.14   17:45:32     PROD_PHYSICS_CD 
       This is 5 day(s), 15 hours old. 
   17A is not available for: 
       202817         PHYSICS_3_06   1 2005.08.15   17:19:24     PROD_PHYSICS_CD 
       This is 4 day(s), 15 hours old. 
 
3) DB MONITORING 
ok 
 
4) DH MONITORING 
ok 
 
5) CAF MONITORING 
ok 
 
6) SAMFarm MONITORING 
It was wroking a few hours yesterday, no data processing right now. 
 
7) DQM MONITORING 
Some jobs could not start, e-mail has been sent to experts 
</pre>
<!-- Author: Guillelmo -->


<!-- Date: Sun Aug 21 08:17:57 2005 -->
<pre>   OFFLINE REPORT: Sun Aug 21 08:15:43 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-16 
   RUN NUMBER  202770 
   RUN TIME  10:09:06 
 
2) CALIBRATIONS 
   10A is up to date. 
   11A is not available for: 
       202771         PHYSICS_3_06   1 2005.08.14   17:45:32     PROD_PHYSICS_CD 
       This is 6 day(s), 14 hours old. 
   17A is not available for: 
       202817         PHYSICS_3_06   1 2005.08.15   17:19:24     PROD_PHYSICS_CD 
       This is 5 day(s), 14 hours old. 
 
3) DB MONITORING 
ok, just the typical DQM spike at ~1-2am 
 
4) DH MONITORING 
ok 
 
5) CAF MONITORING 
ok 
 
6) SAMFarm MONITORING 
Not working 
 
7) DQM MONITORING 
2 out of 3 jobs couldn't start, e-mail will be sent to experts</pre>
<!-- Author: Guillelmo -->


<!-- Date: Mon Aug 22 08:45:14 2005 -->
<pre>   OFFLINE REPORT: Mon Aug 22 08:43:23 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-16 
   RUN NUMBER  202770 
   RUN TIME  10:09:06 
 
2) CALIBRATIONS 
   10A is not available for: 
       203150         PHYSICS_3_07   1 2005.08.22   06:36:01     PROD_PHYSICS_CD 
       This is 2 hours old. 
   11A is not available for: 
       202771         PHYSICS_3_06   1 2005.08.14   17:45:32     PROD_PHYSICS_CD 
       This is 7 day(s), 14 hours old. 
   17A is not available for: 
       202817         PHYSICS_3_06   1 2005.08.15   17:19:24     PROD_PHYSICS_CD 
       This is 6 day(s), 15 hours old. 
 
3) DB MONITORING 
ok 
 
4) DH MONITORING 
ok 
 
5) CAF MONITORING 
ok 
 
6) SAMFarm MONITORING 
still not working, until beamline problem is fixed 
 
7) DQM MONITORING 
ok, nothing yet 
</pre>
<!-- Author: Guillelmo/Enrique -->


<!-- Date: Tue Aug 23 11:12:33 2005 -->
<pre>   OFFLINE REPORT: Tue Aug 23 11:04:16 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-22 
   RUN NUMBER  200719 
   RUN TIME  13:33:52 
 
2) CALIBRATIONS 
   10A is not available for: 
       203153         PHYSICS_3_07   1 2005.08.22   09:55:52     PROD_PHYSICS_CD 
       This is 1 day(s), 1 hours old. 
   11A is not available for: 
       202771         PHYSICS_3_06   1 2005.08.14   17:45:32     PROD_PHYSICS_CD 
       This is 8 day(s), 17 hours old. 
   17A is not available for: 
       203141         PHYSICS_3_07   1 2005.08.21   16:24:02     PROD_PHYSICS_CD 
       This is 1 day(s), 18 hours old. 
 
Calibrations pass 10A is not available for run 203153, which is 1day, 1 hour old. Mail has been sent. 
 
3) DB MONITORING 
ok 
 
4) DH MONITORING 
ok 
 
5) CAF MONITORING 
ok 
 
6) SAMFarm MONITORING 
Farm started working around midnight 
 
7) DQM MONITORING 
Jobs corresponding to runs 202817 and 202771 are mark as "job could not be started". And there are no plots. Mail was sent to experts and they will check it. 
 
</pre>
<!-- Author: Enrique -->


<!-- Date: Wed Aug 24 08:31:03 2005 -->
<pre>   OFFLINE REPORT: Wed Aug 24 08:24:17 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-22 
   RUN NUMBER  200719 
   RUN TIME  13:33:52 
 
2) CALIBRATIONS 
   10A is not available for: 
       203153         PHYSICS_3_07   1 2005.08.22   09:55:52     PROD_PHYSICS_CD 
       This is 1 day(s), 22 hours old. 
   11A is not available for: 
       202771         PHYSICS_3_06   1 2005.08.14   17:45:32     PROD_PHYSICS_CD 
       This is 9 day(s), 14 hours old. 
   17A is not available for: 
       203141         PHYSICS_3_07   1 2005.08.21   16:24:02     PROD_PHYSICS_CD 
       This is 2 day(s), 16 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
Working fine since yesterday 
 
7) DQM MONITORING 
Job corresponding to run 202972 could not be started. And there are no plots. Mail was sent to experts.  
 
</pre>
<!-- Author: Enrique -->


<!-- Date: Thu Aug 25 08:34:13 2005 -->
<pre>   OFFLINE REPORT: Thu Aug 25 08:29:21 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-22 
   RUN NUMBER  200719 
   RUN TIME  13:33:52 
 
2) CALIBRATIONS 
   10A is not available for: 
       203190         PHYSICS_3_07   1 2005.08.23   13:10:39     PROD_PHYSICS_CD 
       This is 1 day(s), 19 hours old. 
   11A is not available for: 
       202771         PHYSICS_3_06   1 2005.08.14   17:45:32     PROD_PHYSICS_CD 
       This is 10 day(s), 14 hours old. 
   17A is not available for: 
       203153         PHYSICS_3_07   1 2005.08.22   09:55:52     PROD_PHYSICS_CD 
       This is 2 day(s), 22 hours old. 
 
Latest runs (203205, 203206, 203212, 203213 and 203215) have calibrations available in pass 10A but there are 3 runs that are 1 day 19 hours old (203190, 203192 and 203202) that are not available yet. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
4 out of 5 jobs could not be started. 
 
</pre>
<!-- Author: Enrique -->


<!-- Date: Fri Aug 26 08:30:42 2005 -->
<pre>   OFFLINE REPORT: Fri Aug 26 08:28:01 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-22 
   RUN NUMBER  200719 
   RUN TIME  13:33:52 
 
2) CALIBRATIONS 
   10A is not available for: 
       203190         PHYSICS_3_07   1 2005.08.23   13:10:39     PROD_PHYSICS_CD 
       This is 2 day(s), 19 hours old. 
   11A is not available for: 
       202771         PHYSICS_3_06   1 2005.08.14   17:45:32     PROD_PHYSICS_CD 
       This is 11 day(s), 14 hours old. 
   17A is not available for: 
       203190         PHYSICS_3_07   1 2005.08.23   13:10:39     PROD_PHYSICS_CD 
       This is 2 day(s), 19 hours old. 
 
There are 4 runs 2 days, 19 hours old with no calibrations available 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
No jobs 
 
</pre>
<!-- Author: Enrique/Andrew -->


<!-- Date: Sat Aug 27 08:17:55 2005 -->
<pre>   OFFLINE REPORT: Sat Aug 27 08:00:35 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-22 
   RUN NUMBER  200719 
   RUN TIME  13:33:52 
 
2) CALIBRATIONS 
   10A is not available for: 
       203303         PHYSICS_3_08   1 2005.08.26   05:23:53     PROD_PHYSICS_CD 
       This is 1 day(s), 2 hours old. 
       also 203304 is the same 
   11A is not available for: 
       202771         PHYSICS_3_06   1 2005.08.14   17:45:32     PROD_PHYSICS_CD 
       This is 12 day(s), 14 hours old. 
   17A is not available for: 
       203190         PHYSICS_3_07   1 2005.08.23   13:10:39     PROD_PHYSICS_CD 
       This is 3 day(s), 18 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
OK 
 
</pre>
<!-- Author: Andrew Martin -->


<!-- Date: Sun Aug 28 08:09:02 2005 -->
<pre>   OFFLINE REPORT: Sun Aug 28 08:06:11 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-27 
   RUN NUMBER  203015 
   RUN TIME  21:17:01 
 
2) CALIBRATIONS 
   10A is not available for: 
       203303         PHYSICS_3_08   1 2005.08.26   05:23:53     PROD_PHYSICS_CD 
       This is 2 day(s), 2 hours old. 
   11A is not available for: 
       202771         PHYSICS_3_06   1 2005.08.14   17:45:32     PROD_PHYSICS_CD 
       This is 13 day(s), 14 hours old. 
   17A is not available for: 
       203190         PHYSICS_3_07   1 2005.08.23   13:10:39     PROD_PHYSICS_CD 
       This is 4 day(s), 18 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
no jobs</pre>
<!-- Author: Andrew Martin -->


<!-- Date: Mon Aug 29 08:21:42 2005 -->
<pre>   OFFLINE REPORT: Mon Aug 29 07:51:40 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-27 
   RUN NUMBER  203015 
   RUN TIME  21:17:01 
 
2) CALIBRATIONS 
   10A is not available for: 
       203303         PHYSICS_3_08   1 2005.08.26   05:23:53     PROD_PHYSICS_CD 
       This is 3 day(s), 2 hours old. 
   11A is not available for: 
       202771         PHYSICS_3_06   1 2005.08.14   17:45:32     PROD_PHYSICS_CD 
       This is 14 day(s), 14 hours old. 
   17A is not available for: 
       203190         PHYSICS_3_07   1 2005.08.23   13:10:39     PROD_PHYSICS_CD 
       This is 5 day(s), 18 hours old. 
 
3) DB MONITORING 
OK since oracle restart yesterday 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
202998 could not be started 
5 others not submitted yet 
</pre>
<!-- Author: Andrew Martin -->


<!-- Date: Tue Aug 30 08:19:55 2005 -->
<pre>   OFFLINE REPORT: Tue Aug 30 07:45:36 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-30 
   RUN NUMBER  203206 
   RUN TIME  03:43:42 
 
2) CALIBRATIONS 
   10A is not available for: 
         203303         PHYSICS_3_08   1 2005.08.26   05:23:53     PROD_PHYSICS_CDF        10A is not available 
       This is 4 days 2 hours old. 
   11A is up to date 
   17A is not available for: 
         203303         PHYSICS_3_08   1 2005.08.26   05:23:53     PROD_PHYSICS_CDF        17A is not available 
       This is 4 days 2 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
OK 
 
</pre>
<!-- Author: Andrew -->


<!-- Date: Wed Aug 31 07:54:23 2005 -->
<pre>   OFFLINE REPORT: Wed Aug 31 07:39:05 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-30 
   RUN NUMBER  203206 
   RUN TIME  03:43:42 
 
2) CALIBRATIONS 
   10A is up to date. 
   11A is up to date 
   17A is not available for: 
       203383         PHYSICS_3_07   1 2005.08.27   21:58:09     PROD_PHYSICS_CD 
       This is 3 days 9 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
many jobs run or running 
plots look OK 
 
</pre>
<!-- Author: Andrew -->


<!-- Date: Thu Sep 1 08:32:27 2005 -->
<pre>   OFFLINE REPORT: Thu Sep 1 07:40:54 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-08-31 
   RUN NUMBER  203439 
   RUN TIME  20:33:50 
 
2) CALIBRATIONS 
   10A is up to date. 
   11A is up to date 
   17A is not available for: 
       203488         PHYSICS_3_08   2 2005.08.30   11:46:25     PROD_PHYSICS_CD 
       This is 1 day(s), 19 hours old. 
 
3) DB MONITORING 
user malde had more than 700 connections to cdfstrm1, reported this to Petar. 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
OK 
 
</pre>
<!-- Author: Andrew -->


<!-- Date: Fri Sep 2 08:58:02 2005 -->
<pre>   OFFLINE REPORT: Fri Sep 2 08:19:02 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-09-02 
   RUN NUMBER  203454 
   RUN TIME  05:53:20 
 
2) CALIBRATIONS 
   10A is not available for: 
       203625         PHYSICS_3_07   1 2005.09.01   07:22:15     PROD_PHYSICS_CD 
	This is 1 day 1 hours old. 
   17A is not available for: 
       203625         PHYSICS_3_07   1 2005.09.01   07:22:15     PROD_PHYSICS_CD 
	This is 1 day 1 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
Condor caf went down about 11 pm, was back up and running about 2 AM 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
OK 
</pre>
<!-- Author: Andrew/Ivan -->


<!-- Date: Sat Sep 3 08:31:12 2005 -->
<pre>mail send to SAM farm experts 
to: suen@fnal.gov, cc: robson@fnal.gov, bernd@fnal.gov 
 
Yesterday night between ~ 2am  and 7am a significant drop of the 
cpu usage of the SAM farm has been monitored, the cpu usage minimum 
(10%) was around 5am 9/3/05 
 
</pre>
<!-- Author: Ivan  -->


<!-- Date: Sat Sep 3 08:37:45 2005 -->
<pre>Sam Farm Optimizer_prd in fatal error, I notice it around 8:25; recover green status around 8.30. Optimizer status in sam-at-glance was yellow,after Optimizer_prd becomes green, the status in sam-at-glance is still yellow. </pre>
<!-- Author: Ivan  -->


<!-- Date: Sat Sep 3 09:19:00 2005 -->
<pre>   OFFLINE REPORT: Sat Sep 3 09:08:53 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-09-02 
   RUN NUMBER  203509 
   RUN TIME  06:04:17 
 
2) CALIBRATIONS 
    
17A is not available since   203686         PHYSICS_3_07   1 2005.09.02   08:35:59     PROD_PHYSICS_CDF  
 
 
17A is not available since   203634         PHYSICS_3_07   1 2005.09.01   11:38:12     PROD_PHYSICS_CDF         
    
    
3) DB MONITORING 
 
OK 
 
4) DH MONITORING 
 
OK 
 
5) CAF MONITORING 
 
OK 
 
6) SAMFarm MONITORING 
 
see Sat Sep 3 08:31:12  and Sat Sep 3 08:37:45 entries in the logbook 
 
 
7) DQM MONITORING 
 
OK, as a remark most of 203432 tracking related histos are 
empty, this was a bad run, no magnet and tracking was off. </pre>
<!-- Author: Ivan  -->


<!-- Date: Sat Sep 3 10:33:48 2005 -->
<pre>fcp: fncdfsrv3 at 2005/09/03 10:12:12(-1)  is in error 
after scps it I got this answer, i does not seem to be hang 
 
cdfopr@fcdflnx3:~&gt;fcps fncdfsrv3 
 
 
 RID                  user.pid@host TS Free/Avbl(MB)   Size File 
---- ------------------------------ -- ------------- ------ ---------- 
9127           cdfprd.3920@fncdf196 U* 1277212/1276797     27 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbck0h/ 
9130          cdfprd.31107@fncdf176 U* 1277212/1276797     27 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbck0h/ 
9131          cdfprd.21393@fncdf186 U* 1277212/1276797    361 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbhd0h/ 
9132          cdfprd.25243@fncdf270 Uw 1277212/1276797     18 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbpp0h/ 
9133          cdfprd.11918@fncdf226 Uw 1277212/1276797      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9134          cdfprd.28629@fncdf303 Uw 1277212/1276797   1057 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbhd0h/ 
9135          cdfprd.10777@fncdf274 Uw 1277211/1276796      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9136          cdfprd.26200@fncdf182 Uw 1277211/1276796      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/logs/ 
9137          cdfprd.21373@fncdf214 Uw 1277211/1276796    381 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbhd0h/ 
9138           cdfprd.3947@fncdf196 Uw 1277211/1276796     42 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbpp0h/ 
9139           cdfprd.2715@fncdf236 Uw 1277211/1276796    360 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbhd0h/ 
9140          cdfprd.18574@fncdf323 Uw 1277211/1276796      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9141          cdfprd.14422@fncdf181 Uw 1277211/1276796      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9142          cdfprd.18167@fncdf277 Uw 1277211/1276796     20 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbpp0h/ 
9143          cdfprd.14429@fncdf181 Uw 1277211/1276796      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9144          cdfprd.10776@fncdf260 Uw 1277211/1276796    365 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbhd0h/ 
9145          cdfprd.18542@fncdf318 Uw 1277211/1276796      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9146           cdfprd.9322@fncdf309 Uw 1277211/1276796      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9147            cdfprd.682@fncdf198 Uw 1277211/1276796     27 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbck0h/ 
9148           cdfprd.2190@fncdf259 Uw 1277211/1276796      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9149          cdfprd.12389@fncdf177 Uw 1277211/1276796     55 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbck0h/ 
9150           cdfprd.1840@fncdf193 Uw 1277211/1276796     26 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbck0h/ 
9151          cdfprd.13790@fncdf293 Uw 1277211/1276796     21 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbck0h/ 
9152          cdfprd.11987@fncdf320 Uw 1277211/1276796     21 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbpp0h/ 
9153           cdfprd.7089@fncdf315 Uw 1277211/1276796   1102 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbhd0h/ 
9154          cdfprd.25102@fncdf266 Uw 1277211/1276796    374 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbhd0h/ 
9155           cdfprd.8160@fncdf333 Uw 1277211/1276796      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9156          cdfprd.25797@fncdf328 Uw 1277114/1276699     18 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbpp0h/ 
9157          cdfprd.26866@fncdf175 Uw 1277114/1276699   1074 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbhd0h/ 
9158          cdfprd.13232@fncdf243 Uw 1277114/1276699      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/crashed/ 
9159          cdfprd.23628@fncdf288 Uw 1277114/1276699      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9160           cdfprd.2005@fncdf193 Uw 1277114/1276699   1076 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbhd0h/ 
9161          cdfprd.15224@fncdf289 Uw 1277114/1276699     20 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbck0h/ 
9162           cdfprd.2047@fncdf237 Uw 1277114/1276699      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9163            cdfprd.603@fncdf233 Uw 1277114/1276699      8 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbck0h/ 
9164           cdfprd.2495@fncdf229 Uw 1277114/1276699   1157 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbhd0h/ 
9165           cdfprd.9502@fncdf287 Uw 1277114/1276699     20 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbck0h/ 
9166          cdfprd.12403@fncdf285 Uw 1277114/1276699      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9167          cdfprd.17087@fncdf207 Uw 1277114/1276699      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9168          cdfprd.30608@fncdf201 Uw 1277114/1276699      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9169           cdfprd.1086@fncdf198 Uw 1277114/1276699     28 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbck0h/ 
9170           cdfprd.7698@fncdf261 Uw 1277114/1276699      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9171           cdfprd.7335@fncdf329 Uw 1277114/1276699     20 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbck0h/ 
9172          cdfprd.26591@fncdf182 Uw 1277114/1276699     25 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbck0h/ 
9173          cdfprd.31871@fncdf189 Uw 1277114/1276699     19 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbck0h/ 
9174          cdfprd.12248@fncdf226 Uw 1277114/1276699      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9175          cdfprd.20835@fncdf232 Uw 1277114/1276699      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9176          cdfprd.12002@fncdf219 Uw 1277114/1276699      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9177          cdfprd.10772@fncdf294 Uw 1277114/1276699      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9178           cdfprd.1786@fncdf247 Uw 1277114/1276699     30 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbck0h/ 
9179          cdfprd.29194@fncdf199 Uw 1277114/1276699      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/crashed/ 
9180           cdfprd.8977@fncdf239 Uw 1277113/1276698     10 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbpp0h/ 
9181          cdfprd.31784@fncdf317 Uw 1277113/1276698   1112 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbhd0h/ 
9182          cdfprd.32516@fncdf218 Uw 1277113/1276698      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/crashed/ 
9183          cdfprd.31544@fncdf338 Uw 1277113/1276698      8 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbck0h/ 
9184          cdfprd.16072@fncdf297 Uw 1277113/1276698      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/crashed/ 
9185          cdfprd.19940@fncdf200 Uw 1277113/1276698      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9186          cdfprd.21758@fncdf271 Uw 1277113/1276698     22 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbpp0h/ 
9187          cdfprd.32631@fncdf265 Uw 1277113/1276698      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9188           cdfprd.9144@fncdf304 Uw 1277113/1276698      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9189          cdfprd.14060@fncdf293 Uw 1277113/1276698     20 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbpp0h/ 
9190          cdfprd.15755@fncdf212 Uw 1277113/1276698   1122 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbhd0h/ 
9191          cdfprd.18046@fncdf231 Uw 1277113/1276698      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9192           cdfprd.5380@fncdf208 Uw 1277113/1276698     16 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbpp0h/ 
9193           cdfprd.1084@fncdf344 Uw 1277113/1276698      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9194          cdfprd.29208@fncdf173 Uw 1277113/1276698      7 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbpp0h/ 
9195          cdfprd.23597@fncdf197 Uw 1277113/1276698    375 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbhd0h/ 
9196          cdfprd.23848@fncdf241 Uw 1277113/1276698   1139 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbhd0h/ 
9197          cdfprd.30401@fncdf183 Uw 1277113/1276698   1080 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbhd0h/ 
9198           cdfprd.9890@fncdf281 Uw 1277113/1276698      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9199          cdfprd.12737@fncdf263 Uw 1277113/1276698   1108 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbhd0h/ 
9200          cdfprd.22412@fncdf191 Uw 1277113/1276698      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9201          cdfprd.18200@fncdf235 Uw 1277113/1276698    375 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbhd0h/ 
9202          cdfprd.20098@fncdf200 Uw 1277113/1276698     58 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbck0h/ 
9203           cdfprd.9106@fncdf324 Uw 1277113/1276698      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9204          cdfprd.31841@fncdf317 Uw 1277113/1276698      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9205          cdfprd.17796@fncdf251 Uw 1277113/1276698    366 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbhd0h/ 
9206          cdfprd.27263@fncdf175 Uw 1277113/1276698     22 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbck0h/ 
9207          cdfprd.16179@fncdf178 Uw 1277113/1276698      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
9208          cdfprd.11041@fncdf291 Uw 1277113/1276698     19 /export/stage/ProductionExe_6.1.1a_maxopt_0h/hbpp0h/ 
</pre>
<!-- Author: Ivan -->


<!-- Date: Sat Sep 3 10:36:03 2005 -->
<pre>mail to sam farm expert after new drop in cpu usage 
now it is below 50%, after a drop of mor than 20%</pre>
<!-- Author: ivan  -->


<!-- Date: Sat Sep 3 12:46:50 2005 -->
<pre>fcp: fncdfsrv3 at 2005/09/03 12:42:24(-16) in error  
expert paged</pre>
<!-- Author: Ivan -->


<!-- Date: Sun Sep 4 08:42:47 2005 -->
<pre>   OFFLINE REPORT: Sun Sep 4 08:30:16 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR: 
   RUN DATE  2005-09-02 
   RUN NUMBER  203509 
   RUN TIME  06:04:17 
 
2) CALIBRATIONS 
   10A is up to date. 
   17A is not available for: 
 
203764 PHYSICS_3_07   1 2005.09.03   00:23:45   PROD_PHYSICS_CDF 
 
3) DB MONITORING 
 
OK 
 
4) DH MONITORING 
 
OK 
 
5) CAF MONITORING 
 
OK 
 
6) SAMFarm MONITORING 
 
OK, yesterday there were two cpu usage drop, around 
2am and again around 4pm; the first one was triggered 
by enstore problems. 
 
7) DQM MONITORING 
 
203452 DO and Silicon based monitoring histos are empty, 
reason: this run did not include the silicon. 
</pre>
<!-- Author: Ivan -->


<!-- Date: Sun Sep 4 11:00:34 2005 -->
<pre>Sam farm problems: ResourceMan got time out  
fcp: fncdfsrv2 at 2005/09/04 10:52:10(-9) 
 
"fcps" check up shows heavy load of this node 
 
cdfopr@fcdflnx3:~&gt;fcps fncdfsrv2 
 RID                  user.pid@host TS Free/Avbl(MB)   Size File 
---- ------------------------------ -- ------------- ------ ---------- 
8180           cdfprd.5514@fncdf176 U* 698722/697987    282 /export/stage/ProductionExe_6.1.1a_maxopt_0h/jbmu0h/ 
8181          cdfprd.21988@fncdf248 U* 698722/697987    452 /export/stage/ProductionExe_6.1.1a_maxopt_0h/jrar0h/ 
8192          cdfprd.16964@fncdf178 U* 698722/697987      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/crashed/ 
8193          cdfprd.13132@fncdf193 Uw 698722/697987    177 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xrar0h/ 
8194          cdfprd.22380@fncdf254 Uw 698722/697987      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8195          cdfprd.20030@fncdf252 Uw 698722/697987    253 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xrar0h/ 
8196          cdfprd.22176@fncdf184 Uw 698722/697987      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/crashed/ 
8197          cdfprd.20137@fncdf233 Uw 698722/697987      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8198           cdfprd.8439@fncdf172 Uw 698722/697987     98 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbmu0h/ 
8199          cdfprd.29091@fncdf239 Uw 698722/697987      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8200           cdfprd.6754@fncdf196 Uw 698722/697987    184 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xrar0h/ 
8201          cdfprd.12504@fncdf192 Uw 698722/697987      3 /export/stage/ProductionExe_6.1.1a_maxopt_0h/jpee0h/ 
8202          cdfprd.20959@fncdf237 Uw 698722/697987      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8203           cdfprd.7741@fncdf214 Uw 698722/697987      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/crashed/ 
8204           cdfprd.9799@fncdf224 Uw 698722/697987    445 /export/stage/ProductionExe_6.1.1a_maxopt_0h/jrar0h/ 
8205          cdfprd.30461@fncdf223 Uw 698722/697987      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8206          cdfprd.10701@fncdf241 Uw 698722/697987      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8207           cdfprd.9855@fncdf224 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8208          cdfprd.17972@fncdf238 Uw 698721/697986      7 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbmm0h/ 
8209           cdfprd.5816@fncdf176 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8210          cdfprd.23808@fncdf242 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8211          cdfprd.14335@fncdf213 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8212           cdfprd.1599@fncdf230 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8213          cdfprd.26996@fncdf244 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/crashed/ 
8214          cdfprd.31217@fncdf200 Uw 698721/697986    285 /export/stage/ProductionExe_6.1.1a_maxopt_0h/jbmu0h/ 
8215           cdfprd.2821@fncdf251 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8216          cdfprd.21910@fncdf262 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8217           cdfprd.6677@fncdf190 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8218          cdfprd.20145@fncdf233 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/crashed/ 
8219           cdfprd.7909@fncdf205 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/crashed/ 
8220          cdfprd.16324@fncdf257 Uw 698721/697986      8 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbmm0h/ 
8221          cdfprd.23797@fncdf256 Uw 698721/697986     93 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbmu0h/ 
8222           cdfprd.1222@fncdf240 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8223          cdfprd.22687@fncdf254 Uw 698721/697986      7 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xbmm0h/ 
8224          cdfprd.27032@fncdf244 Uw 698721/697986    258 /export/stage/ProductionExe_6.1.1a_maxopt_0h/jpmm0h/ 
8225          cdfprd.11022@fncdf206 Uw 698721/697986    261 /export/stage/ProductionExe_6.1.1a_maxopt_0h/jpmm0h/ 
8226           cdfprd.4698@fncdf204 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8227            cdfprd.623@fncdf211 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8228          cdfprd.32258@fncdf189 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/logs/ 
8229          cdfprd.31796@fncdf197 Uw 698721/697986    188 /export/stage/ProductionExe_6.1.1a_maxopt_0h/xrar0h/ 
8230          cdfprd.18036@fncdf194 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8231          cdfprd.22391@fncdf179 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/logs/ 
8232           cdfprd.8061@fncdf214 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
8233          cdfprd.16381@fncdf257 Uw 698721/697986      1 /export/stage/ProductionExe_6.1.1a_maxopt_0h/meta/ 
</pre>
<!-- Author: Ivan -->


<!-- Date: Mon Sep 5 08:17:29 2005 -->
<pre>   OFFLINE REPORT: Mon Sep 5 08:03:20 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-09-02 
   RUN NUMBER  203509 
   RUN TIME  06:04:17 
 
2) CALIBRATIONS 
   10A is not available for: 
   203799         PHYSICS_3_07   1 2005.09.04   15:38:49     PROD_PHYSICS_CDF      
 
   17A is not available for: 
 
   203797         PHYSICS_3_07   1 2005.09.04   02:46:30     PROD_PHYSICS_CDF  
 
 
 
3) DB MONITORING 
 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
 
No histograms for un number 203488 although  
I do not see any problem with it, all status 
bits are set. Mail to experts sent.</pre>
<!-- Author: Ivan -->


<!-- Date: Tue Sep 6 07:58:58 2005 -->
<pre>   OFFLINE REPORT: Tue Sep 6 07:47:40 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-09-05 
   RUN NUMBER  203764 
   RUN TIME  07:43:34 
 
2) CALIBRATIONS 
 
   10A is not available for: 
   203824         PHYSICS_3_07   1 2005.09.05   16:10:58     PROD_PHYSICS_CDF 
   17A is not available for: 
   203799         PHYSICS_3_07   1 2005.09.04   15:38:49     PROD_PHYSICS_CDF  
 
3) DB MONITORING 
 
   OK 
 
4) DH MONITORING 
 
   OK  
 
5) CAF MONITORING 
 
   OK 
 
6) SAMFarm MONITORING 
 
    OK, BUT again the cpu is dropping slowly, the concatenator  
    server fncdfsrv2 is extremely busy getting fcp'ed by the  
    worker nodes; however the other servers are completely free,  
    I do not understand this. 		 
 
cdfopr@fcdflnx3:~&gt;fcps fncdfsrv3 |wc 
      2      12     136     
cdfopr@fcdflnx3:~&gt;fcps fncdfsrv1 |wc 
      2      12     136 
cdfopr@fcdflnx3:~&gt;fcps fncdfsrv4 |wc 
      3      18     256 
cdfopr@fcdflnx3:~&gt;fcps fncdfsrv2 |wc 
     95     570   10567 
    	 
 
7) DQM MONITORING 
 
OK</pre>
<!-- Author: Ivan -->


<!-- Date: Wed Sep 7 08:23:26 2005 -->
<pre>   OFFLINE REPORT: Wed Sep 7 07:59:51 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-09-05 
   RUN NUMBER  203764 
   RUN TIME  07:43:34 
 
2) CALIBRATIONS 
   10A is not available for: 
    
   203869         PHYSICS_3_07   1 2005.09.06   12:07:42     PROD_PHYSICS_CDF 
    
   17A is not available for: 
    
   203819         PHYSICS_3_07   1 2005.09.05   05:21:49     PROD_PHYSICS_CDF 
 
3) DB MONITORING 
 
OK		 
 
4) DH MONITORING 
 
OK 
 
5) CAF MONITORING 
 
OK 
 
6) SAMFarm MONITORING 
 
OK 
 
7) DQM MONITORING 
 
OK 
</pre>
<!-- Author: Ivan -->


<!-- Date: Thu Sep 8 07:57:26 2005 -->
<pre>   OFFLINE REPORT: Thu Sep 8 07:48:30 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-09-08 
   RUN NUMBER  203819 
   RUN TIME  04:33:45 
 
2) CALIBRATIONS 
 
   10A is not available for: 
      203894         PHYSICS_3_07   1 2005.09.07   12:22:09     PROD_PHYSICS_CDF       
    
   17A is not available for: 
      203881         PHYSICS_3_07   1 2005.09.06   16:56:12     PROD_PHYSICS_CDF    
 
3) DB MONITORING 
 
OK  
 
4) DH MONITORING 
 
 
OK 
 
5) CAF MONITORING 
 
OK 
 
6) SAMFarm MONITORING 
 
OK 
 
7) DQM MONITORING 
 
No run processed</pre>
<!-- Author: Ivan -->


<!-- Date: Fri Sep 9 08:34:32 2005 -->
<pre>   OFFLINE REPORT: Fri Sep 9 07:56:34 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-09-08 
   RUN NUMBER  203824 
   RUN TIME  04:33:28 
 
2) CALIBRATIONS 
   10A is not available for: 
   204076         PHYSICS_3_07   1 2005.09.08   14:28:38     PROD_PHYSICS_CDF 
   17A is not available for: 
   203874         PHYSICS_3_07   1 2005.09.06   14:58:01     PROD_PHYSICS_CDF  
    
3) DB MONITORING 
 
One user holds about 1500 connection for more than 8 hours,  
experts informed. 
 
4) DH MONITORING 
 
OK 
 
5) CAF MONITORING 
 
OK 
 
6) SAMFarm MONITORING 
 
OK 
 
 
7) DQM MONITORING 
 
OK</pre>
<!-- Author: Matthew and Ivan -->


<!-- Date: Fri Sep 9 11:28:42 2005 -->
<pre>Email sent to experts and user pmf regarding large number of 
inactive connections to the database.</pre>
<!-- Author: Matthew Norman -->


<!-- Date: Fri Sep 9 17:44:47 2005 -->
<pre>CPU Usage for SAM-Farms dropped below 30% after experiencing 
gradual decline all day.  The most dramatic loss was only noticed after the end of shift, hence delayed response.  Email sent to  
experts.</pre>
<!-- Author: Matthew Norman -->


<!-- Date: Sat Sep 10 08:25:13 2005 -->
<pre>   OFFLINE REPORT: Sat Sep 10 08:16:50 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-09-08 
   RUN NUMBER  203824 
   RUN TIME  04:33:28 
 
2) CALIBRATIONS 
   10A is not available for: 
       204105         PHYSICS_3_07   1 2005.09.09   10:07:35     PROD_PHYSICS_CD 
       This is 22 hours old. 
   17A is not available for: 
       203894         PHYSICS_3_07   1 2005.09.07   12:22:09     PROD_PHYSICS_CD 
       This is 2 day(s), 19 hours old. 
 
3) DB MONITORING 
	OK 
 
4) DH MONITORING 
	OK 
 
5) CAF MONITORING 
	OK 
 
6) SAMFarm MONITORING 
	Very busy load on fncdfsrv5 and fncdfsrv6.  Instructions are unclear.  It shows  
	red on the SAM@Work page, but the Ganglia plots don't look at all different from 
	fncdfsrv3, which is all I had to look at.  According to Ganglia, these machines 
	just went active yesterday at about 11:00pm.  fcps returns the following: 
	 
	bash-2.05a$ fcps fncdfsrv6 | wc 
	    244    1464   29187 
	bash-2.05a$ fcps fncdfsrv5 | wc 
	    109     654   12988 
	     
	Although they seem to be functional, I will send an email to 
	cdf_offline_operations, warning them of the change in status. 
 
	 
	 
 
7) DQM MONITORING 
	OK</pre>
<!-- Author: Matthew Norman -->


<!-- Date: Sat Sep 10 13:15:21 2005 -->
<pre>  203894         PHYSICS_3_07   1 2005.09.07   12:22:09     PROD_PHYSICS_CDF        17A is not available 
 
This is the earliest 17A calibration not available, and it is now 
over 3 days old.  As requested, I have emailed the experts.</pre>
<!-- Author: Matthew Norman -->


<!-- Date: Sun Sep 11 08:10:50 2005 -->
<pre>   OFFLINE REPORT: Sun Sep 11 07:59:46 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-09-08 
   RUN NUMBER  203824 
   RUN TIME  04:33:28 
 
2) CALIBRATIONS 
   10A is not available for: 
       204105         PHYSICS_3_07   1 2005.09.09   10:07:35     PROD_PHYSICS_CD 
       This is 1 day(s), 21 hours old. 
   17A is not available for: 
       204105         PHYSICS_3_07   1 2005.09.09   10:07:35     PROD_PHYSICS_CD 
       This is 1 day(s), 21 hours old. 
 
3) DB MONITORING 
	OK 
 
 
4) DH MONITORING 
	OK 
	 
 
5) CAF MONITORING 
	OK 
 
 
6) SAMFarm MONITORING 
	SAM CPU usage dropped to zero this morning.  fcdfdata064 has also 
	stopped.  I presume these two are related to server and ftp problems 
	reported through the cdfopr email account, but I will send an email to 
	confirm.  fcdfcaf548 is now mostly idle. 
 
 
7) DQM MONITORING 
	OK</pre>
<!-- Author: Matthew Norman -->


<!-- Date: Sun Sep 11 08:58:44 2005 -->
<pre>Standard email sent to user oglez concerning ~20 inactive database 
connections, each with inactive time greater than 300 seconds. 
Message cc'ed to petar, wicklund and cdfdb-support.</pre>
<!-- Author: Matthew Norman -->


<!-- Date: Sun Sep 11 09:50:43 2005 -->
<pre>At this point I noticed that cdfkits.fnal.gov was not responding 
to requests from the web browser, and hence I was unable to monitor 
the dCache Load or Ping logs.  Email sent to experts warning them 
of the change in the situation.</pre>
<!-- Author: Matthew Norman -->


<!-- Date: Sun Sep 11 10:26:19 2005 -->
<pre>10A calibrations for run 204105 are now not available after more than 24 hours: 
 
204105         PHYSICS_3_07   1 2005.09.09   10:07:35     PROD_PHYSICS_CDF        10A is not available 
 
Email sent to experts.</pre>
<!-- Author: Matthew Norman -->


<!-- Date: Sun Sep 11 12:42:59 2005 -->
<pre>cdfkits came back online at this point, restoring full dcache monitoring.  However 
dcache ping and dcache load were not updating regularly.  Email was sent to the experts.</pre>
<!-- Author: Matthew Norman -->


<!-- Date: Sun Sep 11 15:56:53 2005 -->
<pre>dCache door monitoring came back with cdfkits.  However, after updating once, it did 
not update again for one and a half hours, crossing that mark at 15:53.  I have sent email 
on this subject to Aidan.</pre>
<!-- Author: Matthew Norman -->


<!-- Date: Mon Sep 12 08:14:46 2005 -->
<pre>   OFFLINE REPORT: Mon Sep 12 08:12:02 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-09-08 
   RUN NUMBER  203824 
   RUN TIME  04:33:28 
 
2) CALIBRATIONS 
   10A is not available for: 
       204105         PHYSICS_3_07   1 2005.09.09   10:07:35     PROD_PHYSICS_CD 
       This is 2 day(s), 22 hours old. 
   17A is not available for: 
       204105         PHYSICS_3_07   1 2005.09.09   10:07:35     PROD_PHYSICS_CD 
       This is 2 day(s), 22 hours old. 
        
    This is the same anomalous run I sent email about yesterday.  We have calibrations for runs 
    204107 and 204109 
 
3) DB MONITORING 
	OK 
 
 
4) DH MONITORING 
	Monitor dCache doors has not been updated since 9/11 14:23:21.  Another email will be sent. 
 
 
5) CAF MONITORING 
	OK 
 
 
6) SAMFarm MONITORING 
	OK 
 
 
7) DQM MONITORING 
	OK</pre>
<!-- Author: Matthew Norman -->


<!-- Date: Mon Sep 12 11:16:32 2005 -->
<pre>  204107         PHYSICS_3_07   1 2005.09.09   11:06:59     PROD_PHYSICS_CDF        17A is not available 
  204105         PHYSICS_3_07   1 2005.09.09   10:07:35     PROD_PHYSICS_CDF        17A is not available 
 
17A calibrations are now over 3 days old.  Email sent to the experts.</pre>
<!-- Author: Matthew Norman -->


<!-- Date: Mon Sep 12 14:41:37 2005 -->
<pre>Tape write error noted: 
 
2005-Sep-12 13:46:01 failed to catalog fileset CB8433.1 
 
Email sent to cdfdh_oper@fnal.gov as instructed.</pre>
<!-- Author: Matthew Norman -->


<!-- Date: Mon Sep 12 15:08:57 2005 -->
<pre>CPU Usage on SAM-Farms dropped 20% over the course of an hour.   
Email sent to the experts to notify them.</pre>
<!-- Author: Matthew Norman -->


<!-- Date: Tue Sep 13 08:17:09 2005 -->
<pre>   OFFLINE REPORT: Tue Sep 13 08:13:46 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-09-08 
   RUN NUMBER  203824 
   RUN TIME  04:33:28 
 
2) CALIBRATIONS 
   10A is up to date. 
   17A is not available for: 
       204105         PHYSICS_3_07   1 2005.09.09   10:07:35     PROD_PHYSICS_CD 
       This is 3 day(s), 22 hours old. 
        
   This problem was mentioned to the experts yesterday, who had to update the 
   10A runs as well.  They claimed that 17A would straighten itself out. 
 
3) DB MONITORING 
	OK 
 
 
4) DH MONITORING 
	dCache Door monitoring is still non-functional.  Both Aidan and the 
	appropriate computing people have been informed. 
	 
 
5) CAF MONITORING 
	OK 
 
 
6) SAMFarm MONITORING 
	OK 
 
 
7) DQM MONITORING 
	No runs submitted in last 24 hours.</pre>
<!-- Author: Matthew Norman -->


<!-- Date: Wed Sep 14 08:13:34 2005 -->
<pre>   OFFLINE REPORT: Wed Sep 14 08:11:52 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-09-13 
   RUN NUMBER  204185 
   RUN TIME  14:37:05 
 
2) CALIBRATIONS 
   10A is up to date. 
   17A is not available for: 
       204211         PHYSICS_3_07   2 2005.09.12   19:20:22     PROD_PHYSICS_CD 
       This is 1 day(s), 12 hours old. 
 
3) DB MONITORING 
	OK 
	 
 
4) DH MONITORING 
	dCache monitoring came back up for dCache doors with some initial 
	errors.  Also some errors as we switched away from cdfosft2 6.1.0.  
	Email send to inform experts, but everything running normally now. 
 
 
5) CAF MONITORING 
	OK 
 
 
6) SAMFarm MONITORING 
	OK 
 
 
7) DQM MONITORING 
	OK</pre>
<!-- Author: Matthew Norman -->


<!-- Date: Thu Sep 15 08:00:39 2005 -->
<pre>   OFFLINE REPORT: Thu Sep 15 07:59:02 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-09-13 
   RUN NUMBER  204185 
   RUN TIME  14:37:05 
 
2) CALIBRATIONS 
   10A is not available for: 
       204262         PHYSICS_3_07   2 2005.09.14   13:26:10     PROD_PHYSICS_CD 
       This is 18 hours old. 
   17A is not available for: 
       204221         PHYSICS_3_07   2 2005.09.13   05:11:43     PROD_PHYSICS_CD 
       This is 2 day(s), 2 hours old. 
 
3) DB MONITORING 
	OK 
 
 
4) DH MONITORING 
	OK 
	 
 
5) CAF MONITORING 
	CAF shutting down in preparation for downtime: 
	OK 
 
 
6) SAMFarm MONITORING 
	Production Farm shutting down  
	OK 
 
 
7) DQM MONITORING 
	No highest luminosity runs available for checking. 
</pre>
<!-- Author: Matthew Norman -->


<!-- Date: Fri Sep 16 08:17:25 2005 -->
<pre>   OFFLINE REPORT: Fri Sep 16 08:15:28 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-09-16 
   RUN NUMBER  204265 
   RUN TIME  02:51:55 
 
2) CALIBRATIONS 
   10A is not available for: 
       204340         PHYSICS_3_07   2 2005.09.15   08:45:43     PROD_PHYSICS_CD 
       This is 23 hours old. 
   17A is not available for: 
       204266         PHYSICS_3_07   2 2005.09.14   16:29:51     PROD_PHYSICS_CD 
       This is 1 day(s), 15 hours old. 
 
3) DB MONITORING 
	Heavy connections over the night by users sforrest and baroiant.  Email 
	sent to Petar as specified. 
 
 
4) DH MONITORING 
	OK 
 
 
5) CAF MONITORING 
	OK.  Restarted nicely. 
 
 
6) SAMFarm MONITORING 
	OK.  Restarted nicely. 
 
 
7) DQM MONITORING 
	No high luminosity runs available at this time.</pre>
<!-- Author: Matthew Norman -->
<!-- Comment: Fri Sep 16 09;06;09 comment by...m mattson -->
<pre>&lt;br&gt;There is no Offline CO this coming week.  I've been asked to do tasks 2 (Calibrations) and 4 (DH Monitoring) in between my normal Online CO duties.</pre>


<!-- Date: Sat Sep 17 08:09:14 2005 -->
<pre>OFFLINE REPORT: Sat Sep 17 08:09:48 CDT 2005 
 
2) CALIBRATIONS  OK 
10A is not available for: 
 204376 PHYSICS_TEST_3_06_L3L2   2 2005.09.15   18:11:02 
This is 1 day, 14 hours old. 
17A is not available for: 
204266         PHYSICS_3_07   2 2005.09.14   16:29:51 
This is 2 days, 8 hours old 
 
4) DH MONITORING OK </pre>
<!-- Author: m mattson -->


<!-- Date: Sun Sep 18 08:12:24 2005 -->
<pre>OFFLINE REPORT: Sun Sep 18 08:12:34 CDT 2005  
 
2) CALIBRATIONS  OK  
   10A is not available for:  
       204444         PHYSICS_3_07   2 2005.09.17   15:14:17     PROD_PHYSICS_CDF 
This is 17 hours old.  
   17A is not available for:  
       204413         PHYSICS_3_07   2 2005.09.16   12:36:20     PROD_PHYSICS_CDF 
   This is 1 days, 16 hours old  
 
4) DH MONITORING OK </pre>
<!-- Author: m mattson -->


<!-- Date: Mon Sep 19 08:07:36 2005 -->
<pre>OFFLINE REPORT: Mon Sep 19 08:07:45 CDT 2005   
 
2) CALIBRATIONS  OK   
   10A is available for the last run:   
     204447         PHYSICS_3_07   2 2005.09.18   02:44:35     PROD_PHYSICS_CDF 
   17A is not available for:   
     204413         PHYSICS_3_07   2 2005.09.16   12:36:20     PROD_PHYSICS_CDF 
   This is 2 days, 16 hours old   
 
   BeamExe run 9/18  16:27 
 
4) DH MONITORING OK </pre>
<!-- Author: m mattson -->


<!-- Date: Tue Sep 20 08:04:07 2005 -->
<pre>OFFLINE REPORT: Tue Sep 20 08:04:12 CDT 2005   
 
2) CALIBRATIONS  OK   
   10A is not available for:   
        204471         PHYSICS_3_07   2 2005.09.19   17:19:40 
        This is 14 hours old. 
 
   17A is not available for:   
        204469         PHYSICS_3_07   2 2005.09.18   23:35:54 
        This is 1 day, 8 hours old. 
    BeamExe is current. 
 
4) DH MONITORING OK </pre>
<!-- Author: m mattson -->


<!-- Date: Wed Sep 21 08:03:29 2005 -->
<pre>OFFLINE REPORT: Tue Sep 20 08:04:12 CDT 2005    
 
2) CALIBRATIONS  OK    
   10A is not available for:    
204471         PHYSICS_3_07   2 2005.09.19   17:19:40 
       This is 1 day, 14 hours old.  
 
   17A is not available for:    
204471         PHYSICS_3_07   2 2005.09.19   17:19:40 
        This is 1 day, 14 hours old.  
    BeamExe is current.  
 
4) DH MONITORING OK</pre>
<!-- Author: m mattson -->
<!-- Comment: Wed Sep 21 08;04;22 comment by...m mattson -->
<pre>Forgot to change the time stamp: Wed Sep 21 08:04:23 CDT 2005</pre>


<!-- Date: Thu Sep 22 08:04:50 2005 -->
<pre>OFFLINE REPORT: Thu Sep 22 08:04:32 CDT 2005     
 
2) CALIBRATIONS  *BAD*     
   10A is not available for:     
204471         PHYSICS_3_07   2 2005.09.19   17:19:40 
       This is 2 days, 15 hours old.  (Should be no runs over 48 hours) 
       Sending an e-mail to the appropriate experts. 
   17A is not available for:     
204471         PHYSICS_3_07   2 2005.09.19   17:19:40  
        This is 2 days, 15 hours old. 
 
    BeamExe is current.   
 
4) DH MONITORING OK</pre>
<!-- Author: m mattson -->


<!-- Date: Fri Sep 23 08:06:14 2005 -->
<pre>OFFLINE REPORT: Fri Sep 23 08:06:57 CDT 2005      
 
Last day as a scab CO... 
Vine and Robson replied to the e-mail about 10A Calibration 
 
2) CALIBRATIONS  *BAD*      
   10A is not available for:      
204471         PHYSICS_3_07   2 2005.09.19   17:19:40  
       This is 3 days, 15 hours old.  (Should be no runs over 48 hours) 
       There are more recent runs that have been processed, 
     and others have been skipped over. 
       Sending an e-mail to the appropriate experts.  
   17A is not available for:      
204471         PHYSICS_3_07   2 2005.09.19   17:19:40   
        This is 3 days, 15 hours old. 
        This is now over the 3 day limit. 
 
    BeamExe is current.    
 
4) DH MONITORING OK</pre>
<!-- Author: m mattson -->


<!-- Date: Fri Sep 23 08:55:03 2005 -->
<pre>   OFFLINE REPORT: Fri Sep 23 08:53:36 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  
   RUN NUMBER  
   RUN TIME  
 
2) CALIBRATIONS 
   10A is not available for: 
       204471         PHYSICS_3_07   2 2005.09.19   17:19:40     PROD_PHYSICS_CD 
       This is 3 day(s), 15 hours old. 
   17A is not available for: 
       204471         PHYSICS_3_07   2 2005.09.19   17:19:40     PROD_PHYSICS_CD 
       This is 3 day(s), 15 hours old. 
 
3) DB MONITORING 
user pmf &gt; 700 cpu-s 
 
4) DH MONITORING 
two bad doors early am but OK now 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
idle 
 
7) DQM MONITORING 
ok</pre>
<!-- Author: Sarah/Aidan -->


<!-- Date: Fri Sep 23 15:53:01 2005 -->
<pre> OFFLINE REPORT: Fri Sep 23 15:46:32 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  
   RUN NUMBER  
   RUN TIME  
 
2) CALIBRATIONS 
   10A is not available for: 
       204639         PHYSICS_3_07   2 2005.09.22   18:50:42     PROD_PHYSICS_CD 
       This is 20 hours old. 
   17A is not available for: 
       204471         PHYSICS_3_07   2 2005.09.19   17:19:40     PROD_PHYSICS_CD 
       This is 3 day(s), 22 hours old. 
 
3) DB MONITORING 
cdfstrm1 Connections Plot 
 
user pmf had &gt; 700 connections for &gt; 2 hrs. The user has had &gt; 700 
&gt; connections since 3am this morning. 
emailed petar and cdfdb-support. 
4) DH MONITORING 
 
Disk space usage : total storage in disk 6 seems a little high but probably not 
higher than 75%. 
 
5) CAF MONITORING 
OK 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
OK 
 
</pre>
<!-- Author: sarah -->


<!-- Date: Sun Sep 25 08:13:53 2005 -->
<pre>   OFFLINE REPORT: Sun Sep 25 08:05:21 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  
   RUN NUMBER  
   RUN TIME  
 
2) CALIBRATIONS 
   10A is not available for: 
       204680         PHYSICS_3_07   2 2005.09.24   08:19:29     PROD_PHYSICS_CD 
       This is 23 hours old. 
   17A is not available for: 
       204650         PHYSICS_3_07   2 2005.09.23   11:20:05     PROD_PHYSICS_CD 
       This is 1 day(s), 20 hours old. 
 
3) DB MONITORING 
 
OK 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
a sudden drop in overall CPU usage. 30% drop in 2 hours. Climbing back up to 80% 
mark. 
 
7) DQM MONITORING 
2 empty histograms in 3 runs, maybe runs without silicon, notified relevant 
people. 
 
 
195574_jpmm0h_psfiles/Jpsi.ps 
195585_jpmm0h_psfiles/Jpsi.ps 
195590_jpmm0h_psfiles/Jpsi.ps 
 
</pre>
<!-- Author: sarah -->


<!-- Date: Thu Sep 29 07:57:46 2005 -->
<pre> OFFLINE REPORT: Wed Sep 28 15:37:33 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  
   RUN NUMBER  
   RUN TIME  
 
2) CALIBRATIONS 
   10A is not available for: 
       204750         PHYSICS_3_07   2 2005.09.27   11:45:06     PROD_PHYSICS_CD 
       This is 1 day(s), 3 hours old. 
   17A is not available for: 
       204722         PHYSICS_3_07   2 2005.09.25   22:38:26     PROD_PHYSICS_CD 
       This is 2 day(s), 16 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
a 10% drop earlier, notified relevant people but it picked up again. 
OK now. 
7) DQM MONITORING 
OK 
</pre>
<!-- Author: sarah -->


<!-- Date: Fri Sep 30 08:36:43 2005 -->
<pre>   OFFLINE REPORT: Fri Sep 30 08:00:11 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  
   RUN NUMBER  
   RUN TIME  
 
2) CALIBRATIONS 
   10A is not available for: 
       204750         PHYSICS_3_07   2 2005.09.27   11:45:06     PROD_PHYSICS_CD 
       This is 2 day(s), 20 hours old. 
   17A is not available for: 
       204750         PHYSICS_3_07   2 2005.09.27   11:45:06     PROD_PHYSICS_CD 
       This is 2 day(s), 20 hours old. 
 
3) DB MONITORING 
 
 
4) DH MONITORING 
 
 
5) CAF MONITORING 
&gt; 200VMs of white space in the system info plot and a blank section , notified relevant people. 
 
6) SAMFarm MONITORING 
 
 
7) DQM MONITORING 
 
</pre>
<!-- Author: sarah -->


<!-- Date: Sat Oct 1 08:05:04 2005 -->
<pre>   OFFLINE REPORT: Sat Oct 1 07:52:45 CDT 2005 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  
   RUN NUMBER  
   RUN TIME  
 
2) CALIBRATIONS 
   10A is not available for: 
       204855         PHYSICS_3_07   2 2005.09.29   19:30:08     PROD_PHYSICS_CD 
       This is 1 day(s), 12 hours old. 
   17A is not available for: 
       204750         PHYSICS_3_07   2 2005.09.27   11:45:06     PROD_PHYSICS_CD 
       This is 3 day(s), 20 hours old. 
 
3) DB MONITORING 
   OK 
 
4) DH MONITORING 
   OK 
 
5) CAF MONITORING 
   OK 
 
6) SAMFarm MONITORING 
   OK 
 
7) DQM MONITORING 
   OK</pre>
<!-- Author: Gabriele -->


<!-- Date: Sun Oct 2 09:23:56 2005 -->
<pre>   OFFLINE REPORT: Sun Oct 2 09:20:16 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  
   RUN NUMBER  
   RUN TIME  
 
2) CALIBRATIONS 
   10A is not available for: 
       204855         PHYSICS_3_07   2 2005.09.29   19:30:08     PROD_PHYSICS_CD 
       This is 2 day(s), 13 hours old. 
   17A is not available for: 
       204750         PHYSICS_3_07   2 2005.09.27   11:45:06     PROD_PHYSICS_CD 
       This is 4 day(s), 20 hours old. 
 
3) DB MONITORING 
   Lots of repeated db connections by the same user in the last hours, reported to experts. 
 
4) DH MONITORING 
   OK 
 
5) CAF MONITORING 
   'System Info' graph shows unused job resources in the last hour, reported to 
   experts. 
 
6) SAMFarm MONITORING 
   OK 
 
7) DQM MONITORING 
   some empty graphs for run 204681, due to absence of silicon. 
 
Gabriele</pre>
<!-- Author: Gabriele -->


<!-- Date: Mon Oct 3 08:40:43 2005 -->
<pre>   OFFLINE REPORT: Mon Oct 3 08:30:54 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  
   RUN NUMBER  
   RUN TIME  
 
2) CALIBRATIONS 
   10A is not available for: 
       205007         PHYSICS_3_07   2 2005.10.02   22:40:14     PROD_PHYSICS_CD 
       This is 9 hours old. 
   17A is not available for: 
       204929         PHYSICS_3_07   2 2005.10.01   09:11:08     PROD_PHYSICS_CD 
       This is 1 day(s), 23 hours old. 
 
3) DB MONITORING 
   Ok, peaks of Connections (&gt;2200) and of Cpu usage (&gt;700) by cdfdqm between 0:00 and 
   1:00 10/03/2005, reported to experts. 
 
4) DH MONITORING 
   Ok. 
 
5) CAF MONITORING 
   Graphs show a lot of unused resources by Caf System, starting from 22:00 10/02/2005, 
   reported to experts. CondorCAF Ok. 
 
6) SAMFarm MONITORING 
   Graph shows 0% Cpu usage, starting from 6:00 10/03/2005, reported to experts. 
 
7) DQM MONITORING 
   Ok.</pre>
<!-- Author: Gabriele -->


<!-- Date: Tue Oct 4 08:31:30 2005 -->
<pre>   OFFLINE REPORT: Tue Oct 4 08:30:50 CDT 2005 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  
   RUN NUMBER  
   RUN TIME  
 
2) CALIBRATIONS 
   10A is not available for: 
       205018         PHYSICS_3_07   2 2005.10.03   10:20:12     PROD_PHYSICS_CD 
       This is 21 hours old. 
   17A is not available for: 
       205007         PHYSICS_3_07   2 2005.10.02   22:40:14     PROD_PHYSICS_CD 
       This is 1 day(s), 9 hours old. 
 
3) DB MONITORING 
   Ok	 
 
4) DH MONITORING 
   Ok 
 
5) CAF MONITORING 
   Caf monitoring could be down, there are no graphs on the monitoring page. 
   Reported to Experts. 
   CondorCaf Ok. 
 
6) SAMFarm MONITORING 
   Cpu usage at 0% since yesterday: this is due to the fact that the FARM  
   finished the 0h reprocessing for the 7 Dec 04 - 18 March 05. 
 
7) DQM MONITORING 
   Some empty graphs in  
   /~cdfdqm/Processing_610/PlotsDay/204828_aexp0b_psfiles 
   /~cdfdqm/Processing_610/PlotsDay/204831_aexp0b_psfiles 
   due to absence of silicon in these runs.</pre>
<!-- Author: Gabriele -->


<!-- Date: Thu Oct 6 08:45:44 2005 -->
<pre>   OFFLINE REPORT: Thu Oct 6 08:39:03 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  
   RUN NUMBER  
   RUN TIME  
 
2) CALIBRATIONS 
   10A is not available for: 
       205076         PHYSICS_3_07   2 2005.10.05   09:01:43     PROD_PHYSICS_CD 
       This is 23 hours old. 
   17A is not available for: 
       205073         PHYSICS_3_07   2 2005.10.04   17:14:08     PROD_PHYSICS_CD 
       This is 1 day(s), 15 hours old. 
 
3) DB MONITORING 
   Ok. 
 
4) DH MONITORING 
   Ok. 
 
5) CAF MONITORING 
   CondorCAF System Info plot shows a lot of jobs with load=0 and very few jobs consuming CPU, reported to experts. 
 
6) SAMFarm MONITORING 
   Ok. 
 
7) DQM MONITORING 
   in /~cdfdqm/Processing_610/PlotsDay/192364_bhmu0h_psfiles the file Jpsi.ps is empty, its size is 0 Kb, reported to experts.</pre>
<!-- Author: Gabriele -->


<!-- Date: Fri Oct 7 09:32:22 2005 -->
<pre>   OFFLINE REPORT: Fri Oct 7 09:21:03 CDT 2005  
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:   
   RUN DATE   
   RUN NUMBER   
   RUN TIME   
 
2) CALIBRATIONS  
   10A is not available for:  
   205076         PHYSICS_3_07   2 2005.10.05   09:01:43 
   This is 48 hours old.  
   17A is not available for:  
   205076         PHYSICS_3_07   2 2005.10.05   09:01:43 
   This is 48 hours old.  
 
3) DB MONITORING  
   Too much CPU used by user 'menzemer' (10798 Cpu secs), reported to experts and to user. 
 
4) DH MONITORING  
   Problems in Dcache Doors, Dcache Ping and Dcache Load. Reported to experts. 
 
5) CAF MONITORING  
   Ok 
 
6) SAMFarm MONITORING  
   Ok.  
 
7) DQM MONITORING  
   No graphs available today, reported to experts. 
   The DQM processing was offline yesterday because there was no     
   dhaccess in the morning and no database access in the evening. 
</pre>
<!-- Author: Gabriele & Paolo -->


<!-- Date: Fri Oct 7 15:40:54 2005 -->
<pre>   OFFLINE REPORT: Fri Oct 7 15:24:32 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  
   RUN NUMBER  
   RUN TIME  
 
2) CALIBRATIONS 
   10A is not available for: 
       205076         PHYSICS_3_07   2 2005.10.05   09:01:43     PROD_PHYSICS_CD 
       This is 2 day(s), 6 hours old. 
   17A is not available for: 
       205076         PHYSICS_3_07   2 2005.10.05   09:01:43     PROD_PHYSICS_CD 
       This is 2 day(s), 6 hours old. 
 
3) DB MONITORING 
   Ok. 
 
4) DH MONITORING 
   Problems in DCACHE Doors, DCACHE Load and DCACHE Ping. 
   Reported to the experts. 
   DCACHE Load and DCACHE Ping now available but not updated. 
 
5) CAF MONITORING 
   Ok. 
 
6) SAMFarm MONITORING 
   Ok. 
 
7) DQM MONITORING 
   No graphs available today, reported to experts.  
   The DQM processing was offline yesterday because there was no      
   dhaccess in the morning and no database access in the evening.  
 
</pre>
<!-- Author: Paolo -->


<!-- Date: Sat Oct 8 16:07:29 2005 -->
<pre>   OFFLINE REPORT: Sat Oct 8 15:46:50 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-04 
   RUN NUMBER  205018 
   RUN TIME  18:33:19 
 
2) CALIBRATIONS 
   10A is not available for: 
       205076         PHYSICS_3_07   2 2005.10.05   09:01:43     PROD_PHYSICS_CD 
       This is 3 day(s), 6 hours old. 
   17A is not available for: 
       205076         PHYSICS_3_07   2 2005.10.05   09:01:43     PROD_PHYSICS_CD 
       This is 3 day(s), 6 hours old. 
   (email to experts) 
 
3) DB MONITORING 
   Ok. 
 
4) DH MONITORING 
   Ok. 
 
5) CAF MONITORING 
   Ok. 
 
6) SAMFarm MONITORING 
   Ok. 
 
7) DQM MONITORING 
   Ok. 
</pre>
<!-- Author: Paolo -->


<!-- Date: Sun Oct 9 15:59:44 2005 -->
<pre>   OFFLINE REPORT: Sun Oct 9 15:49:25 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-04 
   RUN NUMBER  205018 
   RUN TIME  18:33:19 
 
2) CALIBRATIONS 
   10A is not available for: 
       205076         PHYSICS_3_07   2 2005.10.05   09:01:43     PROD_PHYSICS_CD 
       This is 4 day(s), 6 hours old. 
   17A is not available for: 
       205076         PHYSICS_3_07   2 2005.10.05   09:01:43     PROD_PHYSICS_CD 
       This is 4 day(s), 6 hours old. 
 
3) DB MONITORING 
   Ok. 
 
4) DH MONITORING 
   The dchace doors monitor script is not functioning. 
   (production coordinator notified) 
 
5) CAF MONITORING 
   Some problem with the caf-sam station, since saturday evening to sunday 
   morning. 
   Caf administrators notified. 
   They fixed the problem restarting the caf-sam station. 
 
6) SAMFarm MONITORING 
   Ok. 
 
7) DQM MONITORING 
   Ok.</pre>
<!-- Author: Paolo -->


<!-- Date: Mon Oct 10 15:49:39 2005 -->
<pre>   OFFLINE REPORT: Mon Oct 10 15:40:34 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-04 
   RUN NUMBER  205018 
   RUN TIME  18:33:19 
 
2) CALIBRATIONS 
   Problems with run 205076: 
    both 10A and 17A calibration not available (5 day(s), 6 hours old). 
 
   17A is not available for: 
       205079         PHYSICS_3_07   2 2005.10.05    11:11:33    PROD_PHYSICS_CD 
       This is 5 day(s), 5 hours old. 
 
3) DB MONITORING 
   Ok. 
 
4) DH MONITORING 
   The Dcache doors monitoring script is not working since Friday. 
 
5) CAF MONITORING 
   Ok. 
 
6) SAMFarm MONITORING 
   Ok. 
 
7) DQM MONITORING 
   Ok. 
</pre>
<!-- Author: Paolo -->


<!-- Date: Tue Oct 11 15:21:19 2005 -->
<pre>   OFFLINE REPORT: Tue Oct 11 14:51:05 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-04 
   RUN NUMBER  205018 
   RUN TIME  18:33:19 
 
2) CALIBRATIONS 
   Problems with run 205076: 
    both 10A and 17A calibration not available (more than 6 days old). 
 
   10A is not available for: 
       205399         PHYSICS_3_07   2 2005.10.11   09:31:45     PROD_PHYSICS_CDF 
       This is 5 hours old. 
   17A is not available for: 
       205321         PHYSICS_3_07   2 2005.10.09   00:58:49     PROD_PHYSICS_CDF 
       This is 2 days, 15 hours old. 
 
3) DB MONITORING 
   2005.10.11, 15:00 user "pmf" has more than 700 connections to cdfstrm1 since 7:30am.  
 
4) DH MONITORING 
   Disk space monitoring reports a blank section (16:30-18:30) at the same time of a 
   work on the main CSL machine. 
 
   The Dcache doors monitoring is not available since Friday (last update 3:14 am). 
 
5) CAF MONITORING 
   Ok. 
 
6) SAMFarm MONITORING 
   Ok. 
 
7) DQM MONITORING 
   Ok. 
</pre>
<!-- Author: Paolo -->
<!-- Comment: Tue Oct 11 15;24;31 comment by...Paolo -->
<pre>Concerning the point 4 see the online e-log &lt;a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&action=view&page=849&frame=2&anchor=16:36:48.842'&gt;
user link text &lt;/a&gt;


</pre>

<!-- Comment: Tue Oct 11 15;25;04 comment by...Paolo -->
<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&action=view&page=849&frame=2&anchor=16:36:48.842'>
user link text </a>




<!-- Comment: Tue Oct 11 15;27;23 comment by...Paolo -->
sorry, wrong format for the first comment...
concerning the point 4 see the online e-log using this link <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&action=view&page=849&frame=2&anchor=16:36:48.842'>
user link text </a>





<!-- Date: Wed Oct 12 16:06:24 2005 -->
<pre>   OFFLINE REPORT: Wed Oct 12 15:59:27 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-11 
   RUN NUMBER  205078 
   RUN TIME  18:37:26 
 
2) CALIBRATIONS 
   10A is not available for: 
       205399         PHYSICS_3_07   2 2005.10.11   09:31:45     PROD_PHYSICS_CD 
       This is 1 day(s), 6 hours old. 
   17A is not available for: 
       205325         PHYSICS_3_07   2 2005.10.09   09:16:34     PROD_PHYSICS_CD 
       This is 3 day(s), 6 hours old. 
 
3) DB MONITORING 
   User "baroinat" has more than 25 connections to cdfstrm1 
   lasting inactive more than 280 s: experts and user notified. 
 
4) DH MONITORING 
   The Dcache doors monitoring is not available since Friday (last update 3:14 am). 
 
5) CAF MONITORING 
   Load=0 line in the CAF and CondorCAF System Info plots near the limits: CAF 
   administrators notified. 
 
6) SAMFarm MONITORING 
   Ok. 
 
7) DQM MONITORING 
   No histos available today - nothing was processed. 
</pre>
<!-- Author: Paolo -->


<!-- Date: Thu Oct 13 15:29:46 2005 -->
<pre>   OFFLINE REPORT: Thu Oct 13 15:10:19 CDT 2005 
 
   Today dCache, SAM, and Offline Production Data Base systems downtime since 
   8:00am to 8:00pm. 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-11 
   RUN NUMBER  205078 
   RUN TIME  18:37:26 
 
2) CALIBRATIONS 
   10A is not available for: 
       205400         PHYSICS_3_07   2 2005.10.11   10:17:46     PROD_PHYSICS_CD 
       This is 2 day(s), 4 hours old. 
   17A is not available for: 
       205325         PHYSICS_3_07   2 2005.10.09   09:16:34     PROD_PHYSICS_CD 
       This is 4 day(s), 5 hours old. 
 
3) DB MONITORING 
   Ok. 
 
4) DH MONITORING 
   Ok. 
   (Predator monitor out of order since 7:00am to 9:00am due to downtime - see 
   above). 
 
5) CAF MONITORING 
   Ok. 
 
6) SAMFarm MONITORING 
   Ok. 
 
7) DQM MONITORING 
   Ok. 
</pre>
<!-- Author: Paolo -->


<!-- Date: Fri Oct 14 16:13:19 2005 -->
<pre>   OFFLINE REPORT: Fri Oct 14 15:46:48 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-11 
   RUN NUMBER  205078 
   RUN TIME  18:37:26 
 
2) CALIBRATIONS 
   10A is not available for: 
       205400         PHYSICS_3_07   2 2005.10.11   10:17:46     PROD_PHYSICS_CDF 
       This is 3 day(s), 5 hours old. 
   17A is not available for: 
       205325         PHYSICS_3_07   2 2005.10.09   09:16:34     PROD_PHYSICS_CDF 
       This is 5 day(s), 6 hours old. 
 
   Experts noticed. 
 
3) DB MONITORING 
   OK 
 
4) DH MONITORING 
   OK 
 
5) CAF MONITORING 
   OK 
 
6) SAMFarm MONITORING 
   OK 
 
7) DQM MONITORING 
   OK 
 
 
Others: 
 
cdfopr@fcdflnx3 receives the following email from Cron job every hour (past 10 min)    
 
Cron &lt;cdfopr@fcdflnx3&gt; * /cdf/scratch/cdfopr/cdfcalib/cdfopr/cdfcalib/cron_scripts/fill_passtable/fill_passtable_10A.tcsh &gt;&gt; /cdf/scratch/cdfopr/cdfcalib/cdfopr/log_files/fill_passtable_10A.log 2&gt;&1 
From: root@fcdflnx3.fnal.gov (Cron Daemon) 
To: cdfopr@fcdflnx3.fnal.gov 
Date: Today 10:10:00 am 
/bin/sh: fg: no job control 
 
Bernd contacted Troy about it.</pre>
<!-- Author: Koji -->


<!-- Date: Sat Oct 15 16:03:38 2005 -->
<pre>   OFFLINE REPORT: Sat Oct 15 15:42:34 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-11 
   RUN NUMBER  205078 
   RUN TIME  18:37:26 
 
2) CALIBRATIONS 
   10A is not available for: 
       205400         PHYSICS_3_07   2 2005.10.11   10:17:46     PROD_PHYSICS_CDF 
       This is 4 day(s), 5 hours old. 
   17A is not available for: 
       205399         PHYSICS_3_07   2 2005.10.11   09:31:45     PROD_PHYSICS_CDF 
       This is 4 day(s), 6 hours old. 
 
   Experts noticed by email. 
 
3) DB MONITORING 
   A user baroiant had &gt;700 connections for a period of almost 2 hours from 12pm to 2pm  
   in cdfstrm1 Connections Plot. It was below 700 and kept falling after 2pm, so it may not 
   be a problem. Sent email to experts anyway. 
 
 
4) DH MONITORING 
   OK 
 
5) CAF MONITORING 
   OK 
 
6) SAMFarm MONITORING 
   OK 
 
7) DQM MONITORING 
   Last cron jon run was Oct 13 08:01 (&gt;24 hours old). Sent message to DQM people.  
   Regis will check it. 
</pre>
<!-- Author: Koji -->


<!-- Date: Sun Oct 16 16:03:24 2005 -->
<pre>   OFFLINE REPORT: Sun Oct 16 15:47:32 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-11 
   RUN NUMBER  205078 
   RUN TIME  18:37:26 
 
2) CALIBRATIONS 
   10A is not available for: 
       205400         PHYSICS_3_07   2 2005.10.11   10:17:46     PROD_PHYSICS_CDF 
       This is 5 day(s), 5 hours old. 
   17A is not available for: 
       205399         PHYSICS_3_07   2 2005.10.11   09:31:45     PROD_PHYSICS_CDF 
       This is 5 day(s), 6 hours old. 
 
   Experts are working on these. 
 
3) DB MONITORING 
   A user pmf (Pedro A M Fernandez) has been having &gt;700 connections since ~5am this morning.  
   Now (at the end of the shift) it is arount 1800. Sent messages to DB experts at the beginning  
   and end of the shift. 
 
 
4) DH MONITORING 
   OK 
    
   Sent email to robson, regarding the following line in "Monitor Predator (DFC to SAM metadata)" 
 
      16-05  Number of data files added: 0 (skipped) 
 
   "(skipped)" is not usually seen, though it doesn't look like a problem. 
 
 
5) CAF MONITORING 
   At the beginning of the shift, there was a large amount of light blue (assigned but not  
   running resources) in CAF system info plot. It increased to about 200 VMs around ~6:45am  
   this morning and stayed for about 2 hours. Noticed CAF Administrators. One of them responded  
   and seems to have fixed.  
 
 
6) SAMFarm MONITORING 
   OK 
 
 
7) DQM MONITORING 
   Cron job is not running since Oct 13th. Regis will check and restart the cron job  
   in Monday morning.  
 
</pre>
<!-- Author: Koji -->


<!-- Date: Mon Oct 17 16:14:24 2005 -->
<pre>   OFFLINE REPORT: Mon Oct 17 16:01:45 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-11 
   RUN NUMBER  205078 
   RUN TIME  18:37:26 
 
2) CALIBRATIONS 
   10A is up to date. 
       available up to  
       205641         PHYSICS_3_07   2 2005.10.17   08:28:28     PROD_PHYSICS_CDF         
 
   17A is not available for: 
       205577         PHYSICS_3_07   2 2005.10.15   06:00:45     PROD_PHYSICS_CDF 
       This is 2 day(s), 10 hours old. 
 
3) DB MONITORING 
   A user pmf had &gt;700 connections to cdfstrm1 for more than 2 hours this morning.  
   Sent message. Now his connection is very small. 
    
   A user jakraus had &gt;20 inactive connections to cdfstrm1 lasting &gt;100 seconds each.  
   Sent email. As of 4pm, his inactive jobs seem to be killed... 
   
4) DH MONITORING 
   OK 
 
5) CAF MONITORING 
   OK 
 
6) SAMFarm MONITORING 
   OK 
 
7) DQM MONITORING 
   Cron job has been restarted, and the monitoring seems to be working.  
   Checked all the histograms and they look fine. 
 
</pre>
<!-- Author: Koji -->


<!-- Date: Tue Oct 18 16:18:41 2005 -->
<pre>   OFFLINE REPORT: Tue Oct 18 16:04:09 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-18 
   RUN NUMBER  205628 
   RUN TIME  03:58:43 
 
 
2) CALIBRATIONS 
   10A is not available for: 
       205647         PHYSICS_3_07   2 2005.10.17   18:34:32     PROD_PHYSICS_CDF 
       This is 21 hours old. 
   17A is not available for: 
       205641         PHYSICS_3_07   2 2005.10.17   08:28:28     PROD_PHYSICS_CDF 
       This is 1 day(s), 7 hours old. 
 
   For 10A, 205647 is only a run whose calibration is not available. 
 
   BeamExe has been updated during the day, and now (at the end of the shift)  
   FILE NOT READY is only run 205413 which was skipped because of solenoid being off 
 
  
3) DB MONITORING 
   A user pmf had &gt;700 connections to cdfstrm1 again from ~6:30am until ~10am.  
   Experts notified. 
 
 
4) DH MONITORING 
   A red entry found in fcdfsgi2 stage log. Expert notified, and the job was 
   re-submitted by an expert. 
 
 
5) CAF MONITORING 
   OK 
 
 
6) SAMFarm MONITORING 
   A big drop in CDF-Farms CPU plot, starting around 6pm on Oct. 17, was due to  
   Enstore tape robots problems. Experts fixed it already.  
 
 
7) DQM MONITORING 
   Many histograms in 205399_aexp0b_psfiles in DQM offline monitoring were empty 
   and it is due to solenoid being off in that run. Confirmed by DQM experts. 
 
</pre>
<!-- Author: Koji -->


<!-- Date: Wed Oct 19 16:01:02 2005 -->
<pre>   OFFLINE REPORT: Wed Oct 19 15:40:59 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-18 
   RUN NUMBER  205628 
   RUN TIME  03:48:01 
 
2) CALIBRATIONS 
   10A is up to date. 
   17A is not available for: 
       205647         PHYSICS_3_07   2 2005.10.17   18:34:32     PROD_PHYSICS_CDF 
       This is 1 day(s), 21 hours old. 
 
 
3) DB MONITORING 
   OK 
 
 
4) DH MONITORING 
   OK 
 
 
5) CAF MONITORING 
   OK 
 
 
6) SAMFarm MONITORING 
   OK 
 
 
7) DQM MONITORING 
   OK 
</pre>
<!-- Author: Koji -->


<!-- Date: Thu Oct 20 15:58:55 2005 -->
<pre>   OFFLINE REPORT: Thu Oct 20 15:51:18 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-20 
   RUN NUMBER  205716 
   RUN TIME  02:16:48 
 
 
2) CALIBRATIONS 
   10A is not available for: 
       205889         PHYSICS_3_07   2 2005.10.19   21:36:58     PROD_PHYSICS_CDF 
       This is 18 hours old. 
   17A is not available for: 
       205720         PHYSICS_3_07   2 2005.10.18   09:38:27     PROD_PHYSICS_CDF 
       This is 2 day(s), 6 hours old. 
 
 
3) DB MONITORING 
   A user "cafuser4" had about 15 inactive connections for more than 100 seconds  
   each to cdfstrm1. Since I dit not know who he/she was, I notified just experts.  
   cafuser4's connections are gone after that... 
 
 
4) DH MONITORING 
   OK 
 
 
5) CAF MONITORING 
   OK 
 
 
6) SAMFarm MONITORING 
   OK 
 
 
7) DQM MONITORING 
   OK 
 
</pre>
<!-- Author: Koji -->


<!-- Date: Fri Oct 21 13:01:41 2005 -->
<pre>Lots of red boxes for DQM status. 
Reported to Regis who is looking into it.</pre>
<!-- Author: Emily -->


<!-- Date: Fri Oct 21 13:03:08 2005 -->
<pre>data logger disk 6 was 67% full. 
Sent mail to CSL list. Jeedong said not to worry (probably a long cosmic run).</pre>
<!-- Author: Emily -->


<!-- Date: Fri Oct 21 15:55:25 2005 -->
<pre>  OFFLINE REPORT: Fri Oct 21 08:39:33 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-20 
   RUN NUMBER  205716 
   RUN TIME  02:15:05 
 
2) CALIBRATIONS 
   10A is not available for: 
       205892         PHYSICS_3_07   2 2005.10.20   07:01:47     PROD_PHYSICS_CD 
       This is 1 day(s), 1 hours old. 
   17A is not available for: 
       205889         PHYSICS_3_07   2 2005.10.19   21:36:58     PROD_PHYSICS_CD 
       This is 1 day(s), 11 hours old.</pre>
<!-- Author: Emily -->


<!-- Date: Sat Oct 22 08:06:45 2005 -->
<pre>1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-21 
   RUN NUMBER  205891 
   RUN TIME  12:51:23 
 
2) CALIBRATIONS 
   10A is not available for: 
       205925         PHYSICS_3_07   2 2005.10.21   09:51:29     PROD_PHYSICS_CD 
       This is 22 hours old. 
   17A is not available for: 
       205893         PHYSICS_3_07   2 2005.10.20   09:11:24     PROD_PHYSICS_CD 
       This is 1 day(s), 22 hours old. 
</pre>
<!-- Author: Emily -->


<!-- Date: Sat Oct 22 15:55:31 2005 -->
<pre>Everything looks ok today.  
</pre>
<!-- Author: Emily -->


<!-- Date: Sun Oct 23 09:39:24 2005 -->
<pre>OFFLINE REPORT: Sun Oct 23 07:59:14 CDT 2005 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR: 
   RUN DATE  2005-10-21 
   RUN NUMBER  205891 
   RUN TIME  12:51:23 
 
2) CALIBRATIONS 
   10A is not available for: 
       205960 PHYSICS_HIGHLUM_3_06   1 2005.10.22   10:59:39     PROD_PHYSICS_CD 
       This is 20 hours old. 
   17A is not available for: 
       205925         PHYSICS_3_07   2 2005.10.21   09:51:29     PROD_PHYSICS_CD 
       This is 1 day(s), 22 hours old. 
 
DH MONITORING 
disk 4 reports 1 MB out of 0. Not sure what this means so sent email to 
CSL experts who are looking into it. 
DQM MONITORING 
D0.ps has no entries for run  205716 even though run was taken with 
silicon and other plots have stats. Have sent email to mmp@fnal.gov, 
regis@fnal.gov, portell@fnal.gov 
 
 
</pre>
<!-- Author: Emily Nurse. -->


<!-- Date: Sun Oct 23 12:41:15 2005 -->
<pre>DQM MONITORING 
D0.ps having no entries for run  205716 is fine as it was taken with a high lumi trigger. 
</pre>
<!-- Author: Emily Nurse -->


<!-- Date: Sun Oct 23 12:44:14 2005 -->
<pre>DH: the following tape write errors are showing up for the record of raw data tapes written: 
 
2005-Oct-23 10:15:31 encp FAILED TO COPY FILESET CB9035.0 
source directory: /cdf/b0/disk7/stager/CB9035.0 
source files: er03246a.017bphys er03246a.0191phys er03246a.01a8phys 
er03246a.01bfphys er03246a.01d8phys er03246a.01f1phys er03246a.020aphys er03246a.0224phys er03246a.023fphys 
destination dir: /pnfs/cdfen/filesets/CB/CB90/CB9035/CB9035.0 
The rest of the filesets will not be encpied: CB9035.0 CB9035.1 CB9035.2 CB9035.3 CB9035.4 
2005-Oct-23 11:02:51 encp FAILED TO COPY FILESET CB9039.0 
source directory: /cdf/b0/disk7/stager/CB9039.0 
source files: er03248c.000aphys er03248c.0013phys er03248c.001bphys 
er03248c.0022phys er03248c.0029phys er03248c.0030phys er03248c.0037phys er03248c.003ephys er03248c.0045phys 
destination dir: /pnfs/cdfen/filesets/CB/CB90/CB9039/CB9039.0 
The rest of the filesets will not be encpied: CB9039.0 CB9039.1 CB9039.2 CB9039.3 CB9039.4 
 
Have sent e-mail to cdfdh_oper@fnal.gov 
 
 
</pre>
<!-- Author: Emily Nurse -->


<!-- Date: Sun Oct 23 16:52:04 2005 -->
<pre>shift summary- 
DB: 
CSL expert checked disk4 and said it seems normal (55% usage). 
But the plot still looks strange. 
 
DH: 
There is one more file copy error 
005-Oct-23 11:02:51 encp FAILED TO COPY FILESET CB9039.0 
source directory: /cdf/b0/disk7/stager/CB9039.0 
source files: er03248c.000aphys er03248c.0013phys er03248c.001bphys 
er03248c.0022phys er03248c.0029phys er03248c.0030phys er03248c.0037phys 
er03248c.003ephys er03248c.0045phys 
destination dir: /pnfs/cdfen/filesets/CB/CB90/CB9039/CB9039.0 
The rest of the filesets will not be encpied: CB9039.0 CB9039.1 CB9039.2 CB9039.3 CB9039.4 
 
Valeria replied to my earlier email stating that Dehong is looking into it. 
 
</pre>
<!-- Author: Emily Nurse. -->


<!-- Date: Mon Oct 24 08:49:00 2005 -->
<pre>  OFFLINE REPORT: Mon Oct 24 08:01:57 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR: 
   RUN DATE  2005-10-21 
   RUN NUMBER  205891 
   RUN TIME  12:51:23 
 
2) CALIBRATIONS 
   10A is not available for: 
       205968         PHYSICS_3_07   2 2005.10.23   13:28:47     PROD_PHYSICS_CD 
       This is 18 hours old. 
   17A is not available for: 
       205925         PHYSICS_3_07   2 2005.10.21   09:51:29     PROD_PHYSICS_CD 
       This is 2 day(s), 22 hours old. 
</pre>
<!-- Author: Emily -->


<!-- Date: Mon Oct 24 08:50:25 2005 -->
<pre>data logger disk4 now states that it is using 150 GB, but it still states that 100% = 0 GB. 
Have informed the CSL experts.</pre>
<!-- Author: Emily -->


<!-- Date: Mon Oct 24 15:58:17 2005 -->
<pre>All looks ok at end of shift</pre>
<!-- Author: Emily -->


<!-- Date: Tue Oct 25 08:44:39 2005 -->
<pre>1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR: 
   RUN DATE  2005-10-24 
   RUN NUMBER  205964 
   RUN TIME  11:09:38 
 
2) CALIBRATIONS 
   10A is not available for: 
       206029 PHYSICS_HIGHLUM_3_06   1 2005.10.25   03:41:05     PROD_PHYSICS_CD 
       This is 4 hours old. 
   17A is not available for: 
       205968         PHYSICS_3_07   2 2005.10.23   13:28:47     PROD_PHYSICS_CD 
       This is 1 day(s), 18 hours old. 
 
 
Everything else looks good. 
The disk 4 data logger plot is back to normal.</pre>
<!-- Author: Emily -->


<!-- Date: Wed Oct 26 08:55:18 2005 -->
<pre>   RUN DATE  2005-10-18 
   RUN NUMBER  205628 
   RUN TIME  03:48:01 
CALIBRATIONS: 
  10A is not available for: 
       206033         PHYSICS_3_07   2 2005.10.25   08:12:57     PROD_PHYSICS_CD 
       This is 23 hours old. 
   17A is not available for: 
       206029 PHYSICS_HIGHLUM_3_06   1 2005.10.25   03:41:05     PROD_PHYSICS_CD 
       This is 1 day(s), 4 hours old 
 
Everything else fine</pre>
<!-- Author: Emily -->


<!-- Date: Thu Oct 27 09:00:35 2005 -->
<pre>User rforrest had  &gt; 700 connections for 4 hours last night. He is now back to below 600. I have informed Petar, Piedra and cdfdb-support.</pre>
<!-- Author: Emily -->


<!-- Date: Thu Oct 27 09:02:07 2005 -->
<pre>I am seeing the following data handling write tape errors: 
 
2005-Oct-26 16:16:06 encp FAILED TO COPY FILESET CB9114.0 
source directory: /cdf/b0/disk8/stager/CB9114.0 
source files: dr0324d1.01acphys dr0324d1.01cfphys dr0324d1.01f0phys 
dr0324d1.0214phys dr0324d1.0236phys dr0324d1.0255phys dr0324d1.0273phys 
dr0324d1.0292phys 
destination dir: /pnfs/cdfen/filesets/CB/CB91/CB9114/CB9114.0 
The rest of the filesets will not be encpied: CB9114.0 CB9114.1 
2005-Oct-26 16:51:05 encp FAILED TO COPY FILESET CB9114.0 
source directory: /cdf/b0/disk8/stager/CB9114.0 
source files: dr0324d1.01acphys dr0324d1.01cfphys dr0324d1.01f0phys 
dr0324d1.0214phys dr0324d1.0236phys dr0324d1.0255phys dr0324d1.0273phys 
dr0324d1.0292phys 
destination dir: /pnfs/cdfen/filesets/CB/CB91/CB9114/CB9114.0 
The rest of the filesets will not be encpied: CB9114.0 CB9114.1 
2005-Oct-26 19:26:41 encp FAILED TO COPY FILESET CB9116.0 
source directory: /cdf/b0/disk5/stager/CB9116.0 
source files: br0324d1.0331phys br0324d1.0346phys br0324d1.035bphys 
br0324d1.0370phys br0324d1.0386phys 
destination dir: /pnfs/cdfen/filesets/CB/CB91/CB9116/CB9116.0 
The rest of the filesets will not be encpied: CB9116.0 
2005-Oct-26 19:59:37 encp FAILED TO COPY FILESET CB9117.0 
source directory: /cdf/b0/disk7/stager/CB9117.0 
source files: jr0324d1.02c9phys jr0324d1.02e0phys jr0324d1.02f7phys 
jr0324d1.030ephys jr0324d1.0326phys jr0324d1.033ephys jr0324d1.0355phys 
jr0324d1.036ephys jr0324d1.0386phys 
destination dir: /pnfs/cdfen/filesets/CB/CB91/CB9117/CB9117.0 
The rest of the filesets will not be encpied: CB9117.0 
 
Have informed cdfdh_oper@fnal.gov 
</pre>
<!-- Author: Emily -->


<!-- Date: Thu Oct 27 09:47:52 2005 -->
<pre>Dehong has informed that the write tape errors have been delt with and the files stored to tape.</pre>
<!-- Author: Emily -->


<!-- Date: Thu Oct 27 16:13:00 2005 -->
<pre>Now user soha has had &gt; 700 database connections for more than 2 hours. Have reported to Petar and database experts. 
</pre>
<!-- Author: Emily -->


<!-- Date: Fri Oct 28 08:38:34 2005 -->
<pre>User Soha used cpus greater than 6,000 cpu-seconds for the database. I have e-mailed to the data-base experts and the user. 
 
User jrsmith had more then 700 connections last night. 
I have e-mailed to the data-base experts. 
</pre>
<!-- Author: Kazuhiro  -->


<!-- Date: Fri Oct 28 14:07:47 2005 -->
<pre>User Kreps had 25 inactive connections lasting more than 100 seconds each, I sent e-mails to the user, Petar, and cdfdb-support. </pre>
<!-- Author: Kazuhiro -->


<!-- Date: Fri Oct 28 15:03:18 2005 -->
<pre>Now the user kreps' inactive connections lasting more than 100 seconds are now dowm to around 10.</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Fri Oct 28 15:47:31 2005 -->
<pre>Since I found one RED region at "SAM@work page", I have e-mailed to the CDF Offline Operation group.  
The symptom is "fcp: fncdfsrv4 at 2005/10/28 15:32:13(-11)" which 
appears at "ResourceMan@Work". 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Fri Oct 28 16:06:39 2005 -->
<pre>All except the previous issue look OK at the end of the shift.</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Sat Oct 29 08:07:22 2005 -->
<pre>   OFFLINE REPORT: Sat Oct 29 07:59:32 CDT 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-28 
   RUN NUMBER  206169 
   RUN TIME  22:13:17 
 
2) CALIBRATIONS 
   10A is not available for: 
       206248         PHYSICS_3_07   2 2005.10.28   21:20:08     PROD_PHYSICS_CD 
       This is 10 hours old. 
   17A is not available for: 
       206174         PHYSICS_3_07   2 2005.10.27   04:52:59     PROD_PHYSICS_CD 
       This is 2 day(s), 3 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
OK 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Sat Oct 29 08:30:13 2005 -->
<pre>Found the following errors in fcdfsgi1 stage log. 
 
2005-Oct-29 06:11:05 encp FAILED TO COPY FILESET CB9151.1 
source directory: /cdf/b0/disk5/stager/CB9151.1 
source files: br0325ab.0039phys br0325ab.0044phys br0325ab.0050phys 
br0325ab.005dphys br0325ab.006aphys br0325ab.0078phys br0325ab.0087phys 
br0325ab.0095phys br0325ab.00a3phys 
destination dir: /pnfs/cdfen/filesets/CB/CB91/CB9151/CB9151.1 
The rest of the filesets will not be encpied: CB9151.1 CB9151.2 CB9151.3 
CB9151.4 
 
I have sent an e-mail to Data-Base experts. 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Sat Oct 29 09:02:07 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-239&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-240  width='300' align='top'></a></td><td>Found an empty histogram in silicon hits in 201656_cph10h_psfiles/Photon.ps. 
But this was a no-silicon run. 
</td></tr></table>
<!-- Author: Kazuhiro -->


<!-- Date: Sat Oct 29 11:49:45 2005 -->
<pre>Found new tape write errors in the fcdfsgi1 stage log. 
 
2005-Oct-29 10:23:48 encp FAILED TO COPY FILESET CB9126.1 
source directory: /cdf/b0/disk8/stager/CB9126.1 
source files: dr03255e.00b3phys dr03255e.00cbphys dr03255e.00e1phys dr03255e.00f7phys dr03255e.010cphys dr03255e.0125phys dr03255e.013ephys dr03255e.0156phys 
destination dir: /pnfs/cdfen/filesets/CB/CB91/CB9126/CB9126.1 
The rest of the filesets will not be encpied: CB9126.1 CB9126.2 CB9126.3 CB9126.4  
 
E-mailed to data-base experts. 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Sat Oct 29 14:38:37 2005 -->
<pre>User sforrest has 26 inactive connections lasting more than 100 seconds each.  
Have e-mailed to the user, Petar and cdfdb-support. 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Sat Oct 29 16:04:53 2005 -->
<pre>Found new tape write errors as follows: 
 
2005-Oct-29 15:19:21 encp FAILED TO COPY FILESET CB9126.0 
source directory: /cdf/b0/disk8/stager/CB9126.0 
source files: dr032559.000fphys dr03255e.0001phys dr03255e.0019phys dr03255e.0030phys dr03255e.0047phys dr03255e.005aphys dr03255e.0072phys dr03255e.0086phys dr03255e.009dphys 
destination dir: /pnfs/cdfen/filesets/CB/CB91/CB9126/CB9126.0 
The rest of the filesets will not be encpied: CB9126.0 CB9126.1 CB9126.2 CB9126.3 CB9126.4  
 
Have send an e-mail to the data-base experts. 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Sun Oct 30 07:51:45 2005 -->
<pre>   OFFLINE REPORT: Sun Oct 30 07:45:46 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-28 
   RUN NUMBER  206169 
   RUN TIME  22:13:17 
 
2) CALIBRATIONS 
   10A is not available for: 
       206255         PHYSICS_3_07   2 2005.10.29   01:29:19     PROD_PHYSICS_CD 
       This is 1 day(s), 7 hours old. 
   17A is not available for: 
       206174         PHYSICS_3_07   2 2005.10.27   04:52:59     PROD_PHYSICS_CD 
       This is 3 day(s), 3 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
OK 
 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Sun Oct 30 08:05:24 2005 -->
<pre>There was a tape error last evening. 
 
2005-Oct-29 18:17:02 encp FAILED TO COPY FILESET CB9140.0 
source directory: /cdf/b0/disk8/stager/CB9140.0 
source files: dr03256a.007cphys dr03256a.009ephys dr03256a.00bephys dr03256a.00dcphys dr03256a.00fbphys dr03256a.0118phys dr03256a.013aphys dr03256a.015bphys 
destination dir: /pnfs/cdfen/filesets/CB/CB91/CB9140/CB9140.0 
The rest of the filesets will not be encpied: CB9140.0 CB9140.1 CB9140.2 CB9140.3 CB9140.4  
 
Have e-mailed to the cdfdh_oper. 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Sun Oct 30 08:26:34 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-241&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-242  width='300' align='top'></a></td><td>I found a blank cdfstrm1 Connections Plot. 
Have e-mailed to the data-base experts.</td></tr></table>
<!-- Author: Kazuhiro -->


<!-- Date: Sun Oct 30 09:55:53 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-243&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-244  width='300' align='top'></a></td><td>Now it's back to normal. Thanks. 
</td></tr></table>
<!-- Author: Kazuhiro -->


<!-- Date: Sun Oct 30 15:20:11 2005 -->
<pre>Another message on tape write errors appeared. 
 
2005-Oct-30 13:49:58 encp FAILED TO COPY FILESET CB9174.0 
source directory: /cdf/b0/disk8/stager/CB9174.0 
source files: dr0325b3.0326phys dr0325b3.0350phys dr0325b3.0377phys dr0325b3.03a1phys dr0325b3.03cbphys dr0325b3.03f5phys dr0325b3.0420phys dr0325b3.0448phys 
destination dir: /pnfs/cdfen/filesets/CB/CB91/CB9174/CB9174.0 
The rest of the filesets will not be encpied: CB9174.0 CB9174.1 CB9174.2 CB9174.3 CB9174.4 
 
Have sent an e-mail to cdfdh_oper. 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Sun Oct 30 16:00:29 2005 -->
<pre>Everything except the above looks fine at the end of the shift. 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Mon Oct 31 07:51:53 2005 -->
<pre>   OFFLINE REPORT: Mon Oct 31 07:48:10 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-31 
   RUN NUMBER  206251 
   RUN TIME  02:22:39 
 
2) CALIBRATIONS 
   10A is not available for: 
       206255         PHYSICS_3_07   2 2005.10.29   01:29:19     PROD_PHYSICS_CD 
       This is 2 day(s), 7 hours old. 
   17A is not available for: 
       206255         PHYSICS_3_07   2 2005.10.29   01:29:19     PROD_PHYSICS_CD 
       This is 2 day(s), 7 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
OK 
 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Mon Oct 31 08:39:07 2005 -->
<pre>As shown above, one of the 10A calibrations have been not available for more than 48 hours, I have report the experts of it. 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Mon Oct 31 09:39:47 2005 -->
<pre>I found the today's first tape write error: 
 
2005-Oct-31 09:04:37 encp FAILED TO COPY FILESET CB9186.0 
source directory: /cdf/b0/disk8/stager/CB9186.0 
source files: dr0325cb.003fphys dr0325cb.005cphys dr0325cb.0081phys dr0325cb.009fphys dr0325cb.00bfphys dr0325cb.00dephys dr0325cd.0001phys dr0325cd.0024phys dr0325cd.0048phys 
destination dir: /pnfs/cdfen/filesets/CB/CB91/CB9186/CB9186.0 
The rest of the filesets will not be encpied: CB9186.0 CB9186.1 CB9186.2 CB9186.3 CB9186.4 
 
Have reported it to the experts. 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Mon Oct 31 11:28:15 2005 -->
<pre>The 10A calibration of which unavailable status lasted over 2 days, which I have reported above, has passed the process. 
 
206255         PHYSICS_3_07   2 2005.10.29   01:29:19     PROD_PHYSICS_CDF        10A     1     4    UNDEFINED    1515976 Sun Oct 30 01:10:17 2005     CDFOPR  
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Mon Oct 31 16:06:09 2005 -->
<pre>All look OK at the end of the shift. 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Tue Nov 1 08:37:09 2005 -->
<pre>   OFFLINE REPORT: Tue Nov 1 07:51:12 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-10-31 
   RUN NUMBER  206281 
   RUN TIME  17:23:15 
 
2) CALIBRATIONS 
   10A is not available for: 
       206288         PHYSICS_3_07   2 2005.10.31   06:39:18     PROD_PHYSICS_CD 
       This is 1 day(s), 1 hours old. 
   17A is not available for: 
       206283         PHYSICS_3_07   2 2005.10.30   08:58:00     PROD_PHYSICS_CD 
       This is 1 day(s), 22 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
Also I made sure that all runs which failed to write to tapes over the weekend and yesterday had been reprocessed successfully. 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
OK 
 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Tue Nov 1 11:51:09 2005 -->
<pre>Found the quota of cdfopr at fcdflnx3 is almost full. 
Sometimes even a tiny text file cannot be saved.  
Please increase the quota size.</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Tue Nov 1 12:20:52 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-245&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-246  width='300' align='top'></a></td><td>Found a dip in the CondorCAF plot.  
Although seems like it's back to normal now, I have e-mailed to the CAF experts.</td></tr></table>
<!-- Author: Kazuhiro -->


<!-- Date: Tue Nov 1 15:24:08 2005 -->
<pre>Bernd, Aidan and Bo delt with the disk quota issue reported above. 
It's fine now. Thanks. 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Tue Nov 1 15:42:19 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-249&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-250  width='300' align='top'></a></td><td>Vacant DQM pages. 
Maybe due to the failed cron jobs this morning on which lots of wanining emails were delivered to cdfopr@fnal.gov. 
</td></tr></table>
<!-- Author: Kazuhiro -->


<!-- Date: Tue Nov 1 15:44:37 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-251&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-252  width='300' align='top'></a></td><td>I wanted to add this image on the previous log, too. 
</td></tr></table>
<!-- Author: Kazuhiro -->


<!-- Date: Tue Nov 1 16:10:11 2005 -->
<pre>Bernd and Bo told that the above my guess was wrong. 
Have sent an e-mail to the DQM experts. 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Tue Nov 1 17:49:48 2005 -->
<pre>Regis, the DQM expert, gave me the answer about the previous issue, saying that's OK, just no new histogram made available in the last 24h. 
Seems no problem. 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Wed Nov 2 07:54:16 2005 -->
<pre>   OFFLINE REPORT: Wed Nov 2 07:49:59 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-02 
   RUN NUMBER  206300 
   RUN TIME  02:16:50 
 
2) CALIBRATIONS 
   10A is not available for: 
       206414 PHYSICS_HIGHLUM_3_06   1 2005.11.02   03:20:14     PROD_PHYSICS_CD 
       This is 4 hours old. 
   17A is not available for: 
       206326         PHYSICS_3_07   2 2005.10.31   14:22:52     PROD_PHYSICS_CD 
       This is 1 day(s), 17 hours old. 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
OK 
 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Wed Nov 2 15:41:42 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-253&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-254  width='300' align='top'></a></td><td>SAMFarm CPU monitors. 
Is this suspicious ? 
Have sent an e-mail to the experts.</td></tr></table>
<!-- Author: Kazuhiro -->


<!-- Date: Wed Nov 2 16:35:03 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-255&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-256  width='300' align='top'></a></td><td>SAMFarm overall CPU usage. Still dropping.</td></tr></table>
<!-- Author: Kazuhiro -->


<!-- Date: Wed Nov 2 16:50:17 2005 -->
<pre>Everything else is fine at the end of the shift. 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Thu Nov 3 08:21:03 2005 -->
<pre>   OFFLINE REPORT: Thu Nov 3 07:50:36 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-03 
   RUN NUMBER  206326 
   RUN TIME  02:10:02 
 
2) CALIBRATIONS 
   10A is up to date. 
   17A is up to date 
 
3) DB MONITORING 
OK 
 
4) DH MONITORING 
OK 
 
5) CAF MONITORING 
OK 
 
6) SAMFarm MONITORING 
OK 
 
7) DQM MONITORING 
OK 
 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Thu Nov 3 08:36:48 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-257&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-258  width='300' align='top'></a></td><td>CondorCAF last night. 
It this OK ?  Seems back to normal now. 
Have send an e-mail to CAF exprerts. 
</td></tr></table>
<!-- Author: Kazuhiro -->


<!-- Date: Thu Nov 3 10:15:51 2005 -->
<pre>The CAF expert came in and told that it is OK if the status is continuous between before and after the gap. The gap happens due to 
the web service down. So, The previous case is fine. 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Thu Nov 3 16:05:01 2005 -->
<pre>Everything is fine at the end of the shift. 
</pre>
<!-- Author: Kazuhiro -->


<!-- Date: Fri Nov 4 08:37:22 2005 -->
<pre>While checking the  Offline Oracle DB Usage Monitoring in the cdfstrm1 Current Usage Table one user (fernand) was found to have 25 inactive connections lasting more than 100 seconds each.  An email was sent to the user, Petar and cdfdb-support  according to the procedure.</pre>
<!-- Author: salvador -->


<!-- Date: Fri Nov 4 09:03:50 2005 -->
<pre>While checking the DH System Monitoring it was found the following tape write error: 
------------------------------------- 
2005-Nov-04 00:02:54 encp FAILED TO COPY FILESET CB9252.3 
source directory: /cdf/b0/disk6/stager/CB9252.3 
source files: gr0326c9.032bphys gr0326c9.034aphys gr0326c9.036aphys gr0326c9.038bphys gr0326c9.03acphys gr0326c9.03cephys gr0326c9.03f0phys gr0326c9.0413phys 
destination dir: /pnfs/cdfen/filesets/CB/CB92/CB9252/CB9252.3 
The rest of the filesets will not be encpied: CB9252.3 CB9252.4  
------------------------------------- 
 
The DH expert was notified</pre>
<!-- Author: salvador -->


<!-- Date: Fri Nov 4 15:59:42 2005 -->
<pre>The Offline Oracle DB Usage Monitoring in the cdfstrm1 Current Usage Table one 
user (bello) was found to have 18 inactive connections lasting more than 100 seconds each.  An email was sent to the user, Petar and cdfdb-support  according to the procedure. 
</pre>
<!-- Author: salvador -->


<!-- Date: Fri Nov 4 16:17:39 2005 -->
End of Shift Report:<br> 
 
<b>DB)</b> <br> 
- One user found to have inactive connections and was notified, the expert got in concatc with him (fernand), and the problem was fix.<br> 
 
- Another user (bello) at the end of the shift was found with the similar problem and was notified.<br> 
 
<b>DH)</b><br> 
- A Tape write error was found, and the expert was contact (Dehong Zhang) and he resubmitted the tape job.<P> 
 
No other problems to report.<br>
<!-- Author: Salvador Carrillo -->


<!-- Date: Sat Nov 5 12:10:07 2005 -->
<b>DH)</b><br> 
 The Monitor DCache Load Seems OK (The Pnfs=0), as seen below<br> 
<pre> 
--------------------------------------------------------- 
Run Sat Nov  5 11:49:38 CST 2005 Data from Sat Nov  5 11:45:06  2005 
    1029 Logins 
    1025 Active 
       0 GridFTP 
       4 Killed 
 
Waiting for 
     899 Door 
       0 Pnfs 
     103 Pool 
      23 Limbo 
. 
. 
. 
--------------------------------------------------------- 
</pre> 
But There are many "TAPE Errors, no access" in the Enstore history (~200 
Total), that continue repeat every 10 to 15 minutes, the nodes seems ok an are 
responding with less than 1 sec in  DCA System Status/Cell Services<p> 
 
The experts were contacted to check if this is a normal process.<P>  
<!-- Author: Salvador Carrillo -->


<!-- Date: Sat Nov 5 16:12:18 2005 -->
<br><b>End of Shift Report</b><br> 
   OFFLINE REPORT: Sat Nov 5 15:59:31 CST 2005<p> 
 
 
<b>1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:</b> <br> 
   RUN DATE  2005-11-05<br> 
   RUN NUMBER  206531<br> 
   RUN TIME  03:10:54<p> 
 
<b>2) CALIBRATIONS</b><br> 
- 10A is not available for:<br> 
206668         PHYSICS_3_07   2 2005.11.04   00:00:33     PROD_PHYSICS_CD<br> 
This is 1 day(s), 15 hours old.<br> 
- 17A is not available for:<br> 
206537         PHYSICS_3_07   2 2005.11.03   05:46:29     PROD_PHYSICS_CD<br> 
This is 2 day(s), 10 hours old.<p> 
 
<b>3) DH MONITORING</b><br> 
In Enstore history some error at the Mover Interface were found like<br> 
- USER ERROR, No Such File<br> 
- TAPER ERROR, No access<p> 
The Nodes are responding and this not seems to be a mayor problem, an email was sent to the experts to consult this feature.<P> 
 
Nothing more to report at the end of the shift.<br> 
 
 
  
<!-- Author: Salvador Carrillo -->


<!-- Date: Sun Nov 6 09:17:15 2005 -->
<br> 
<b>DB)</b><br> 
While checking the usage fo the Oracle DB user fernand was found to have more than 20 inactive connections lasting more than 2,000 seconds each! Some of them even reach to 3,000 seconds and then the "Active (code)" changed from INACTIVE to KILLED.<p> 
 
An email was sent to the user and to the experts to keep looking for a solution to this problem.<br>  
<!-- Author: salvador -->


<!-- Date: Sun Nov 6 09:38:04 2005 -->
<br> 
<b>Offline Web Page updated</b><br> 
In the web page of the Offline Shift: <i>"Report offline status at 8am meeting (M-F only)"</i>,  two links were found not working and they were updated according to the <i>"Monitor offline calibrations"</i> web page:<p> 
 
The links were those pointing to the Availability of 10A & 17A for the calibration<p> 
 
--> links were changed form<br> 
http://www-cdf.fnal.gov/~cdfopr/calibrations/prod_physics_cdf_XX.html 
<br> 
--> to <br> 
http://www-cdf.fnal.gov/offline/calibrations/prod_physics_cdf_XX.html 
<p> 
where the XX stands for 10A or 17A<p>  
<!-- Author: Salvador Carrillo -->


<!-- Date: Sun Nov 6 09:59:56 2005 -->
<br> 
<b>CALIBRATIONS RUN</b><br> 
--> 10A is not available for more than 48hrs for: <br> 
206668 PHYSICS_3_07 2 2005.11.04 00:00:33 PROD_PHYSICS_CD<p> 
This is already 2 days 9 hours old<p> 
An email was sent to the experts<br>  
<!-- Author: Salvador Carrillo -->


<!-- Date: Sun Nov 6 11:56:02 2005 -->
<br> 
<b>CALIBRATION RUNS.- </b>  PROBLEM: 10A not available<p> 
Jason Nielsen reply to the email: <br> 
------------------------------------------ 
<pre> 
Hi CO Offline, 
 
Thank you for catching this problem.  The runs 206668 and 206669 could not be assigned a silicon calibration because we have not taken a silicon calibration in more than a month.  In this case, the used sets have to be assigned by hand.  I've assigned calibrations to runs 206668 and 206669.  You may notice some runs over the next day or two which will have the same problem. 
 
The operations team is aware of this problem, and they will request a silicon calibration be completed in the next few days.  Until that happens, I'll be assigning the calibrations manually. 
 
Best regards, 
Jason Nielsen 
</pre><br> 
-----------------------------------------------------<br> 
 
This means the calibrations will be asigned manually by Jason for the next one or two days, and we expect to see this behavior of the 10A Calibration runs in the next few days.<br> 
-----------------------------------------------------<br>  
<!-- Author: Salvador Carrillo -->


<!-- Date: Sun Nov 6 12:45:02 2005 -->
<br> 
<b> DH)</b><br> 
 
Problem:  In the web page that checks that raw data is being logged to tape the plot for disk-3 in the web page for monitoring diskspace usage is not been ploted properly.<p> 
  
--> disk-3 usage is been ploted from (-1 to 1) at the "Y" axis, 
instead to been ploted from 0 to its maximum value of 237 GBytes (100%).<P> 
 
An email was sent to the CSL group.<br>  
<!-- Author: Salvador Carrillo -->


<!-- Date: Sun Nov 6 14:52:51 2005 -->
<br> 
<b>DH)</b><br> 
PROBLEM: Data logger disk3 not been ploted properly<p> 
 
Jedong says since the disk is small it has been disable.  See message below<br> 
---------------------------------------------------------<br> 
<pre>  
Hello, 
 
disk3 is our oldest 237 GB disk array which is about 6 % of our total disk buffer size. I disabled the monitoring of the disk for a couple of reasons. 
 
The disk is not being used now, and will be decommissioned soon, when our new disk (1.4 TB) is installed. I will see if I can make a chenge on the monitoring web page to prevent further confusions. 
 
Thank you, 
Jedong 
</pre> 
---------------------------------------------------------<p> 
 
The web page that is been use in the CO Offline Shift has been updated so people will not look into that plot until the disk is been replaced.<p>  
<!-- Author: Salvador Carrillo -->


<!-- Date: Sun Nov 6 16:09:18 2005 -->
<br> 
<b>End of CO Offline Shift Report</b><br> 
OFFLINE REPORT: Sun Nov 6 15:55:49 CST 2005<br><br> 
 
 
<b>1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:</b><br>  
   RUN DATE  2005-11-05<br> 
   RUN NUMBER  206537<br> 
   RUN TIME  18:09:34<p> 
 
<b>2) CALIBRATIONS</b><br> 
 -  10A is not available for:<br> 
206671 PHYSICS_3_07   2 2005.11.05   13:56:22 PROD_PHYSICS_CD<br> 
        This is 1 day(s), 1 hours old.<p> 
 -   17A is not available for:<br> 
206668 PHYSICS_3_07 2 2005.11.04 00:00:33 PROD_PHYSICS_CD<br> 
       This is 2 day(s), 15 hours old.<p> 
 
Since there are no silicon calibration Jason Nielsen is submitting the calibrations 10A manually for one or two days<P> 
 
 
<b>3) DB MONITORING</b><br> 
User fernand keeps with more than 20 inactive connections with more than 2,000 seconds.  Another message was sent to him in order to contact the expert and fix the problem.<p> 
 
 
<b>4) DH MONITORING</b><br> 
Jedong disable the use of disk3 for raw data to be logger to tape.   He says that disk3 is only 6% of the total disk bufer size, so there is no need to keep monitoring it until it has been replaced (a warning was place in the Monitoring CO Offline page so people do not check this disk until is replaced).<p> 
 
Nothing more to report.<br>  
<!-- Author: Salvador Carrillo -->


<!-- Date: Mon Nov 7 13:17:06 2005 -->
<br> 
<b>Oracle DB)</b><br> 
The service will be down for one hour: applying a patch, see following email:<br> 
----------------------------------------------------------------------<br> 
<pre> 
Date: Mon, 07 Nov 2005 11:20:04 -0600 
From: Anil Kumar <akumar@fnal.gov> 
To: cdfdb-support@fnal.gov, cdf-ops-managers@fnal.gov, cdf_offline_operations@fnal.gov, cdfdb-users@fnal.gov 
Subject: cdf online dev/int db will be down for Oracle's patch at 1:00PM Today 
Parts/Attachments: 
   1   OK     ~8 lines  Text 
   2 Shown   ~15 lines  Text 
---------------------------------------- 
 
 
cdf online dev/int db cdfondev/cdfonint will be down for Oracle's patch at 1:00PM Today  
 
Interruption is expected to last for 1 hour. 
 
Thanks, 
 
Anil. 
</pre><br> 
---------------------------------------------------------------------------------<br>  
<!-- Author: Salvador Carrillo -->


<!-- Date: Mon Nov 7 15:06:39 2005 -->
<br> 
<b>Oracle Data Base</b><br> 
While monitoring cdfofprd database a general user dececco was found with more than 20 connections.  An email was sent to the user in order to contact cdfdb-support group.<p>  
<!-- Author: Salvador Carrillo -->


<!-- Date: Mon Nov 7 16:06:34 2005 -->
<br> 
<b>End of Shift Report</b><p> 
OFFLINE REPORT: Mon Nov 7 15:51:46 CST 2005<br> 
 
 
<b>1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:</b> <br> 
   RUN DATE  2005-11-05<br> 
   RUN NUMBER  206537<br> 
   RUN TIME  18:09:34<p> 
 
<b>2) CALIBRATIONS</b><br> 
-   10A is not available for:<br> 
206717 PHYSICS_HIGHLUM_3_06 1 2005.11.06 22:44:43 PROD_PHYSICS_CD 
<br> 
       This is 17 hours old.<br> 
-   17A is not available for:<br> 
206669 PHYSICS_3_07 2 2005.11.05 00:22:36 PROD_PHYSICS_CD<br> 
       This is 2 day(s), 15 hours old.<p> 
 
<b>3) DB MONITORING</b><br> 
-for one hour starting from 1:00 pm cdf online dev/int db was down for Oracle's patch.  Also a drive was lost on fcdfora6~cdfstrm1.<br> 
-one user was found using cdfofprd database with more than 20 connections.  The user and experts were notified.<p> 
 
<b>5) CAF MONITORING</b><br> 
In the CAF System Info some increase above 500 VMs was found in CondorCAF (& also in CAF with values below 500 VMs), during one hour when the fcdfora6-cdfstrm1 drive was lost and Oracle's patch was being apply.  Now is going back to normal.<p> 
  
<!-- Author: Salvador Carrillo -->


<!-- Date: Tue Nov 8 08:40:32 2005 -->
<br> 
<b>Starting of the CO Offline Shift for Tue Nov 08 2005</b><p> 
 
 
 
<b>DB Monitoring</b><br> 
- Some of the DB Monitoring weg pages are not working, the experts where notified, probably cdfstrm1 is not yet back this morning.<p> 
 
<b>CondorCAF</b><br> 
- An email was received from Subir Sarkar saying that <b>Condof Caf will not be working this morning</b>, only dedicated resources. (no schedule especification was send).<p> 
 
  
<!-- Author: Salvador Carrillo -->


<!-- Date: Tue Nov 8 08:56:43 2005 -->
<pre>Monitoring of database connections today shows that user fernand is  using cdfofprd database with more than 20 connections, this is only for offline production.  
 
The user and experts were notified.</pre>
<!-- Author: Salvador Carrillo -->


<!-- Date: Tue Nov 8 09:12:48 2005 -->
<b>DB Monitoring</b><br> 
 
Since cdfstrm1 is down some db connections will fail over cdfofprd, as the following email from Eric Wicklund says:<br> 
------------------------------------------------------------<br> 
<pre> 
Date: Tue, 08 Nov 2005 08:58:53 -0600 
From: wicklund@fnal.gov 
To: Aidan Robson <a.robson@physics.gla.ac.uk> 
Cc: cdfdb-support@fnal.gov, Sinead Farrington <sineadf@fnal.gov>, Bernd Stelzer <bernd@fnal.gov>, cdfopr@fnal.gov 
Subject: Re: DB CPU Usage Tables not working. (fwd) 
 
Hi Aidan, 
 
One other thing, while cdfstrm1 is down, database connections 
will fail over to cdfofprd. 
 
So, it is expected to see ordinary user connections 
on cdfofprd until cdfstrm1 comes back. 
 
Best Regards, 
</pre> 
------------------------------------------------------------<br>  
<!-- Author: Salvador Carrillo -->


<!-- Date: Tue Nov 8 09:53:52 2005 -->
<p> 
<b>NOTE: Condor CAF will be donw tomorroy Nov 9 2005, not today!</b><br> See next email from Igor<br> 
-----------------------------------------------------------<br> 
<pre> 
Date: Tue, 08 Nov 2005 09:42:38 -0600 
From: Sfiligoi Igor <sfiligoi@fnal.gov> 
To: cdf_caf@fnal.gov 
Cc: CDF DCAF Administrators <cdf-dcaf-admin@fnal.gov> 
Subject: FermiGlideCAF, CNAFCAF, LyonCAF and SDSCCAF reduced 
tomorrow, Nov 9th 
 
Hi. 
 
Due to maintenance on the CDF VOMS server, FermiGlideCAF, CNAFCAF, LyonCAF and SDSCCAF will experience temporary loss of resources tommorrow, Nov 9th. 
 
Running jobs will not be affected, but the amount of available 
resources could be very low for most of the day. (i.e. no new VMs will be added during the day) 
 
Sorry for the inconvenience, 
  Igor 
</pre><br> 
---------------------------------------------------------<br>  
<!-- Author: Salvador Carrillo -->


<!-- Date: Tue Nov 8 13:15:27 2005 -->
<br> 
<b>Calibrations</b><br> 
 
- 17A is not available for:<br> 
 206669 PHYSICS_3_07 2 2005.11.05 00:22:36 PROD_PHYSICS_CD<br> 
This is already 3 days 12 hours old.<p> 
Since this calibration for 10A was submitted manualy, an email was sent to the experts and to Jason Nielsen to respond for the problem.<br>  
<!-- Author: Salvador Carrillo -->


<!-- Date: Tue Nov 8 16:07:41 2005 -->
<br> 
<b> End of CO Offline Report: Tue Nov 8 15:49:06 CST 2005</b><p> 
 
 
<b>1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR: </b><br> 
   RUN DATE  2005-11-05<br> 
   RUN NUMBER  206537<br> 
   RUN TIME  18:09:34<p> 
 
<b>2) CALIBRATIONS</b><br> 
-   10A is not available for:<br> 
 206723 PHYSICS_3_07 2 2005.11.07 00:37:23 PROD_PHYSICS_CD<br> 
       This is 1 day(s), 15 hours old.<br> 
-   17A is not available for:<br> 
 206669 PHYSICS_3_07 2 2005.11.05 00:22:36 PROD_PHYSICS_CD<br> 
       This is 3 day(s), 15 hours old.<p> 
 
- A broken script failed, now is fix and this will be ready this evening: 17A calib/jobs must be started by then.<p> 
 
 
<b>3) DB MONITORING</b><br> 
cdfstrm1 has not come back, this means that many monitoring pages are not working.<p>   
 
 
<b>4) CAF MONITORING</b><br> 
CondorCAF <b>will be down tomorrow morning</b>  
Due to maintenance on the CDF VOMS server, 
FermiGlideCAF, CNAFCAF, LyonCAF and SDSCCAF 
will experience temporary loss of resources 
tommorrow, Nov 9th.<p> 
 
Running jobs will not be affected, but the amount of available 
resources could be very low for most of the day. 
(i.e. no new VMs will be added during the day)<p> 
 
 
<b>5) SAMFarm MONITORING</b><br> 
Started running late last night and with no problems so far.<br> 
 
 
  
<!-- Author: Salvador Carrillo -->


<!-- Date: Wed Nov 9 08:42:50 2005 -->
<pre> CALIBRATIONS 
   10A is not available for: 
       206723         PHYSICS_3_07   2 2005.11.07   00:37:23     PROD_PHYSICS_CD 
       This is 2 day(s), 7 hours old. 
 
This is 7 hours already late, the experts were notified by email.</pre>
<!-- Author: Salvador Carrillo -->


<!-- Date: Wed Nov 9 16:05:18 2005 -->
<br> 
<b>End of CO Offline Report:</b> Wed Nov 9 15:57:09 CST 2005<p> 
 
 
<b>1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR: </b><br> 
   RUN DATE  2005-11-08<br> 
   RUN NUMBER  206694<br> 
   RUN TIME  18:14:53<p> 
 
<b>2) CALIBRATIONS</b><br> 
- 10A is not available for:<br> 
 206733 PHYSICS_3_07 2 2005.11.07 14:32:47 PROD_PHYSICS_CD<br> 
       This is 2 day(s), 1 hours old.  
 ==> (Experts already notified).<p> 
-   17A is not available for: 
 206723 PHYSICS_3_07 2 2005.11.07 00:37:23 PROD_PHYSICS_CD<br> 
       This is 2 day(s), 15 hours old.<p> 
 
<b>3) DB:</b><br> 
-  Oracle cdfstrm1 Current Usage Table is now responding with only root and oracle users.<br> 
-  Not all other monitoring pages are back yet.<p> 
 
No more to report<br>
<!-- Author: Salvador Carrillo -->


<!-- Date: Thu Nov 10 08:20:13 2005 -->
<pre>2) CALIBRATIONS 
   10A is not available for: 
       206733         PHYSICS_3_07   2 2005.11.07   14:32:47     PROD_PHYSICS_CD 
       This is 2 day(s), 17 hours old. 
   17A is not available for: 
       206723         PHYSICS_3_07   2 2005.11.07   00:37:23     PROD_PHYSICS_CD 
       This is 3 day(s), 7 hours old. 
 
Experts are still waiting for table PROD_PHYSICS_SVX.</pre>
<!-- Author: Salvador Carrillo -->


<!-- Date: Thu Nov 10 08:31:42 2005 -->
<br> 
<b>DB)</b><p> 
Monitoring web pages are back, just one missing: cdfstrm1 CPU Usage Plot, it will probably come back during the day. <p> 
 
Aidan and Bernd were notified.<br>  
<!-- Author: Salvador Carrillo -->


<!-- Date: Thu Nov 10 09:12:08 2005 -->
<br> 
<b>DH)</b><br> 
  "Failed to catalog errors" were found for fileset CB9358.0 to CB9358.4<p> 
The experts were notified<br>  
<!-- Author: Salvador Carrillo -->


<!-- Date: Thu Nov 10 10:57:41 2005 -->
<pre>CondorCAF  & CAF 
 
Some strange behavior of CondofCAF where the red Line &gt; 500 VMs at 
the System Info plots.  Also CAF is showing this behavior with but below the 500K. 
 
Experts were notified 
</pre>
<!-- Author: Salvador Carrillo -->


<!-- Date: Thu Nov 10 16:04:46 2005 -->
<br> 
<b>End of Shift Report</b><p> 
 
<b>1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:</b><br>  
   RUN DATE  2005-11-08<br> 
   RUN NUMBER  206694<br> 
   RUN TIME  18:14:53<p> 
 
 
<b>2) CALIBRATIONS</b><br> 
-   10A is not available for:<br> 
 206951 PHYSICS_3_07 2 2005.11.09 14:02:58 PROD_PHYSICS_CD<br> 
       This is 1 day(s), 1 hours old.<br> 
-   17A is not available for:<br> 
 206733 PHYSICS_3_07 2 2005.11.07 14:32:47 PROD_PHYSICS_CD<br> 
       This is 3 day(s), 1 hours old.<p> 
 
==> 206733 still waiting for silicon calibration<p> 
 
No more to report<br>  
<!-- Author: Salvador Carrillo -->


<!-- Date: Fri Nov 11 09:10:45 2005 -->
<pre>DB)  
An email was sent to Petar because there is few activity in the  
the "cdfstrm1 Current Usage Table" that is being use to monitor for the Offline Shift 
</pre>
<!-- Author: Salvador & Yen-Chu -->


<!-- Date: Fri Nov 11 09:57:39 2005 -->
<pre>Responce from Nelly stated that there is continuing DB acitivity  
with cdfstrm1. 
 
------------------------------------------------------------- 
 
This is the dba's re-establishing streams from cdfofpr2 
to cdfstrm1.    We will continue these efforts today 
and finished off stream'g on all the onapps as well 
as filecatalog, frontier, cdfdbb, cdf_views_prd 
later today.   On Monday we will 
proceed with stream'g  the books as well. 
 
-Nelly 
</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Fri Nov 11 11:23:17 2005 -->
<pre>Noticed that cdf-farm2, fcdfdata055 and fcdfdata057 are all on RED. 
Could not connect to fcdfdata055 and fcdfdata057 via cdfopr account. 
 
Notify the farmers.</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Fri Nov 11 13:22:35 2005 -->
<pre>Just for documentary purpose. 
 
There was a high CPU load on the cdfstrm1. The total period  
was less then 30 minutes. Don't qualify a warning.  
 
The relative account was oracle. It appears to be DB experts  
updating, testing the system.</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Fri Nov 11 14:29:03 2005 -->
<pre>There were errors regarding reading from tape IA1348. 
 
Alarmed Aidon and rest. 
 
A responce from Dehong. 
 
--------------------------------------------------------------- 
 
Date: Fri, 11 Nov 2005 14:00:14 -0600 (CST) 
From: Dehong_Zhang &lt;dhzhang@fnal.gov&gt; 
To: Aidan Robson &lt;a.robson@physics.gla.ac.uk&gt; 
Cc: CDF &lt;cdfopr@fnal.gov&gt;, cdfdh_oper@fnal.gov, cdfoom@fnal.gov 
Subject: Re: what is done for tape errors? 
 
Hi Aidan, 
 
This tape has been "NOACCESS" since 11:45 this morning.  Usually enstore 
people will fix this kind of problems within a few hours.  It should be 
released today. 
 
Best regards, 
Dehong 
</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Sat Nov 12 08:21:04 2005 -->
<pre>   OFFLINE REPORT: Sat Nov 12 08:08:55 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-11 
   RUN NUMBER  206981 
   RUN TIME  18:14:52 
 
2) CALIBRATIONS 
   10A is not available for: 
       206989         PHYSICS_3_07   2 2005.11.09   19:41:17     PROD_PHYSICS_CD 
       This is 2 day(s), 12 hours old. 
   17A is not available for: 
       206989         PHYSICS_3_07   2 2005.11.09   19:41:17     PROD_PHYSICS_CD 
       This is 2 day(s), 12 hours old. 
</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Sat Nov 12 10:06:24 2005 -->
<pre>Thanks to Jason. The 10A calibration caught up to within 2 two days. 
 
================================================================ 
 
   OFFLINE REPORT: Sat Nov 12 10:04:01 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-11 
   RUN NUMBER  206981 
   RUN TIME  18:14:52 
 
2) CALIBRATIONS 
   10A is not available for: 
       207079         PHYSICS_3_07   2 2005.11.10   21:55:42     PROD_PHYSICS_CD 
       This is 1 day(s), 12 hours old. 
   17A is not available for: 
       206989         PHYSICS_3_07   2 2005.11.09   19:41:17     PROD_PHYSICS_CD 
       This is 2 day(s), 14 hours old. 
</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Sat Nov 12 16:04:53 2005 -->
<pre>End of shift report: 
 
* 10A calibration caught up 
 
* /data1 disk on fcdfdata125 was found full. User notified and  
  action was taken cleaning up the disk.  
 
* The rest of the offline systems work properly.</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Sun Nov 13 08:16:32 2005 -->
<pre>   OFFLINE REPORT: Sun Nov 13 07:59:50 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-11 
   RUN NUMBER  206981 
   RUN TIME  18:14:52 
 
2) CALIBRATIONS 
   10A is not available for: 
       207079         PHYSICS_3_07   2 2005.11.10   21:55:42     PROD_PHYSICS_CD 
       This is 2 day(s), 10 hours old. 
   17A is not available for: 
       207078 PHYSICS_HIGHLUM_3_06   1 2005.11.10   20:11:34     PROD_PHYSICS_CD 
       This is 2 day(s), 11 hours old. 
 
 
Calibration 10A is more than two days old. Experts notified!</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Sun Nov 13 08:32:22 2005 -->
<pre>DB, DH, CAF, SAMFarm, Offline DQM all checked OK.</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Sun Nov 13 16:04:52 2005 -->
<pre>End of shift report: 
 
   Everything look fine! 
 
   The calibration is very up to date except that there is a  
run, 207079, which is not ready and is causing the report to  
say more than two days old. 
 
   The COT beamline is not available for run 207079. 
 
   The expert is notified about this run. 
 
============================================================== 
 
   OFFLINE REPORT: Sun Nov 13 15:48:54 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-11 
   RUN NUMBER  206981 
   RUN TIME  18:14:52 
 
2) CALIBRATIONS 
   10A is not available for: 
       207079         PHYSICS_3_07   2 2005.11.10   21:55:42     PROD_PHYSICS_CD 
       This is 2 day(s), 17 hours old. 
   17A is not available for: 
       207078 PHYSICS_HIGHLUM_3_06   1 2005.11.10   20:11:34     PROD_PHYSICS_CD 
       This is 2 day(s), 19 hours old. 
 
================================================================ 
 
  207079         PHYSICS_3_07   2 2005.11.10   21:55:42  PAD_PHYSICS_COTBEAM            is not available</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Mon Nov 14 08:16:26 2005 -->
<pre>   OFFLINE REPORT: Mon Nov 14 08:05:23 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-14 
   RUN NUMBER  207114 
   RUN TIME  03:12:37 
 
2) CALIBRATIONS 
   10A is up to date. 
   17A is not available for: 
       207115         PHYSICS_3_07   2 2005.11.12   16:15:33     PROD_PHYSICS_CD 
       This is 1 day(s), 15 hours old.</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Mon Nov 14 09:24:05 2005 -->
<pre>DB, CAF, SamFarm are fine. 
 
Regarding DH, there is a cdfensrv3 server that has error  
transferring data. Expert notifed for the error. </pre>
<!-- Author: Yen-Chu -->


<!-- Date: Mon Nov 14 12:28:49 2005 -->
<pre>Stager error on fcdfsgi1-eg1 
 
2005-11-14 11:12:42     fcdfsgi1-eg1    stager  ERROR (1) 
 
expert notified.</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Mon Nov 14 16:08:04 2005 -->
<pre>End of shift report: 
 
The calibration is very up to date! Cann't be better. 
 
Everything else in the offline is fine except that there  
were two incidences in the taping activity. One happened  
at cdfensrv3 and the other one on the fcdfsgi1. Experts  
were notified. 
 
=========================================================== 
 
   OFFLINE REPORT: Mon Nov 14 16:07:25 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-14 
   RUN NUMBER  207114 
   RUN TIME  03:12:37 
 
2) CALIBRATIONS 
   10A is up to date. 
   17A is not available for: 
       207151 PHYSICS_HIGHLUM_3_06   1 2005.11.13   22:44:57     PROD_PHYSICS_CD 
       This is 17 hours old. 
</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Tue Nov 15 08:34:02 2005 -->
<pre>Start of shift report: 
 
   Calibration is up to date. 
 
   DB, DH, CAF, SamFarm and DQM all checked fine! 
 
=============================================================== 
 
   OFFLINE REPORT: Tue Nov 15 08:13:42 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-14 
   RUN NUMBER  207115 
   RUN TIME  10:09:32 
 
2) CALIBRATIONS 
   10A is not available for: 
       207210         PHYSICS_3_07   2 2005.11.15   06:50:17     PROD_PHYSICS_CD 
       This is 1 hours old. 
   17A is not available for: 
       207155         PHYSICS_3_07   2 2005.11.14   10:51:23     PROD_PHYSICS_CD 
       This is 21 hours old.</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Wed Nov 16 09:40:39 2005 -->
<pre>   OFFLINE REPORT: Wed Nov 16 08:05:37 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-15 
   RUN NUMBER  207156 
   RUN TIME  18:12:05 
 
2) CALIBRATIONS 
   10A is up to date. 
   17A is not available for: 
       207214         PHYSICS_3_07   2 2005.11.15   08:43:28     PROD_PHYSICS_CD 
       This is 23 hours old. 
 
 
Good activity in cdfstrm1!   
 
All system fine!</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Wed Nov 16 13:26:05 2005 -->
<pre>Tape error in reading. 
 
Volume IA9272 is marked NOACCESS 
 
DH experts notified.</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Wed Nov 16 15:49:23 2005 -->
<pre>End of shift report: 
 
The calibration is up to date. 
 
Today is scheduled down time.  
 
There were activities on the CAF to test dCache. 
 
There was an error on tape reading. Tape volume is IA9272. Experts notified. </pre>
<!-- Author: Yen-Chu  -->


<!-- Date: Thu Nov 17 08:13:10 2005 -->
<pre>Start of shift report: 
 
   OFFLINE REPORT: Thu Nov 17 08:05:05 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-15 
   RUN NUMBER  207156 
   RUN TIME  18:12:05 
 
2) CALIBRATIONS 
   10A is up to date. 
   17A is not available for: 
       207216         PHYSICS_3_07   2 2005.11.15   20:26:37     PROD_PHYSICS_CD 
       This is 1 day(s), 11 hours old. 
 
The CAF was back online since 1 AM today, Nov. 17. 
 
DH, DB SamFarm are all fine.  
 
Nothing for DQM.</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Thu Nov 17 16:09:39 2005 -->
<pre>End of shift report: 
 
The calibration is OK.  
 
The CondorCAF has high amount of CPU being idle for the whole  
day.  
 
There is a tape readind error on IA9272.  
 
There is possibility that the above two are correlated.  
 
Asked the DH experts to send request to the Enstore admin  
for checking/releasing this tape in question.  
 
As of 15:42 there is still tape reading error related to this  
tape.  
 
All the other systems are fine.  
 
=========================================================== 
 
   OFFLINE REPORT: Thu Nov 17 16:03:08 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-15 
   RUN NUMBER  207156 
   RUN TIME  18:12:05 
 
2) CALIBRATIONS 
   10A is up to date. 
   17A is not available for: 
       207217         PHYSICS_3_07   2 2005.11.15   20:48:55     PROD_PHYSICS_CD 
       This is 1 day(s), 19 hours old. 
 
</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Fri Nov 18 09:19:57 2005 -->
<pre>Start of shift report: 
 
   OFFLINE REPORT: Fri Nov 18 07:58:38 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-18 
   RUN NUMBER  207215 
   RUN TIME  11:53:15 
 
2) CALIBRATIONS 
   10A is up to date. 
   17A is not available for: 
       207217         PHYSICS_3_07   2 2005.11.15   20:48:55     PROD_PHYSICS_CD 
       This is 2 day(s), 11 hours old. 
 
 
DB is OK. 
 
DH still has the same tape reading error on IA9272. 
 
SAMFarm is OK. 
 
DQM is OK. 
 
CAF has some worker nodes off. 
 
CondorCAF is off due to power outage since midnight. 
</pre>
<!-- Author: Franco, Yen-Chu -->


<!-- Date: Fri Nov 18 14:33:22 2005 -->
<pre>Another Enstore problem: 
Movers in Trouble:  	    1 / 18 
9940B16.mover is offline 
</pre>
<!-- Author: Franco -->


<!-- Date: Fri Nov 18 16:06:43 2005 -->
<pre>End shift report: 
 
CAF: CondorCAF  is running now, after the power failure at GCC this night. CAF still reduced to about 2/3. 
 
DH: there is still tape reading error on IA9272. 
Two movers in trouble: 9940B16 is offline, 9940B25 is dead. 
 
All other systems are fine.</pre>
<!-- Author: Franco -->


<!-- Date: Sat Nov 19 08:23:15 2005 -->
<pre>Start shift report: 
 
   OFFLINE REPORT: Sat Nov 19 08:12:01 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-18 
   RUN NUMBER  207215 
   RUN TIME  11:53:15 
 
2) CALIBRATIONS 
   10A is not available for: 
       207405         PHYSICS_3_07   2 2005.11.18   06:56:39     PROD_PHYSICS_CD 
       This is 1 day(s), 1 hours old. 
   17A is not available for: 
       207405         PHYSICS_3_07   2 2005.11.18   06:56:39     PROD_PHYSICS_CD 
       This is 1 day(s), 1 hours old. 
 
 
3) DB MONITORING 
ok 
 
4) DH MONITORING 
still IA9272 noaccess, 9940B16 mover offline 
 
5) CAF MONITORING 
ok 
 
6) SAMFarm MONITORING 
ok 
 
7) DQM MONITORING 
ok 
 
</pre>
<!-- Author: Franco -->


<!-- Date: Sat Nov 19 14:28:23 2005 -->
<pre>User gensen has reached 1100 connections to the database, but only for a limit period of time.</pre>
<!-- Author: Franco -->


<!-- Date: Sat Nov 19 15:50:48 2005 -->
<pre>User fernand has 7 inactive connetions to the db lasting more then 100 seconds. Not notified.</pre>
<!-- Author: Franco -->


<!-- Date: Sat Nov 19 16:07:01 2005 -->
<pre>End of shift report: 
 
The CAF is raising up (about 1200 VMs). 
 
DH:  
There is still the reading error on tape IA9272 
Mover 9940B16 is still offline 
 
All other systems are fine</pre>
<!-- Author: Franco -->


<!-- Date: Sun Nov 20 08:47:11 2005 -->
<pre>Start of shift report: 
 
   OFFLINE REPORT: Sun Nov 20 07:58:18 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-18 
   RUN NUMBER  207215 
   RUN TIME  11:53:15 
 
2) CALIBRATIONS 
   10A is up to date. 
   17A is not available for: 
       207421         PHYSICS_3_07   2 2005.11.18   12:04:16     PROD_PHYSICS_CD 
       This is 1 day(s), 19 hours old. 
 
3) DB MONITORING 
User wagnp has many inactive connections lasting 44000 seconds.  
User and experts notified. 
 
4) DH MONITORING 
There is still the reading error on tape IA9272  
Mover 9940B16 is still offline  
 
5) CAF MONITORING 
CAF is restored (1600 VMs). CondorCAF is ok. 
 
6) SAMFarm MONITORING 
ok 
 
7) DQM MONITORING 
The boxes in the DQM status page for run 207214 and 207158 are red. 
Experts notified 
 
All histograms in D0.ps and all histograms related to Silicon in SiCt.ps are empty for run 207210 and 207213 because taken without silicon 
 
</pre>
<!-- Author: Franco -->


<!-- Date: Sun Nov 20 11:20:44 2005 -->
<pre>Another user, fernand, has many inactive connections lasting several thousands seconds. User and experts notified.</pre>
<!-- Author: Franco -->


<!-- Date: Sun Nov 20 14:58:30 2005 -->
<pre>Now no users has inactive connections lasting more then 100 seconds</pre>
<!-- Author: Franco -->


<!-- Date: Sun Nov 20 16:02:54 2005 -->
<pre>End of shift report: 
 
Everithing is going well (but the error on tape IA9272   
and mover 9940B16 offline) 
 
 
</pre>
<!-- Author: Franco -->


<!-- Date: Mon Nov 21 08:13:41 2005 -->
<pre>Start of shift report: 
 
   OFFLINE REPORT: Mon Nov 21 07:42:21 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-20 
   RUN NUMBER  207421 
   RUN TIME  09:06:39 
 
2) CALIBRATIONS 
   10A is up to date. 
  DF 17A is not available for: 
       207427 PHYSICS_TEST_3_07_DIFFLOWL   1 2005.11.18   21:02:11     PROD_PHYS 
       This is 2 day(s), 10 hours old. 
---- 
The CondorCAF has &gt; 500 sections not consuming CPU. Experts notified. 
 
There is still the error on tape IA9272   
Mover 9940B16 is still offline   
 
All histograms in D0.ps related to Silicon are empty for run 207216 
and 207218  because taken without silicon. 
</pre>
<!-- Author: Franco -->


<!-- Date: Mon Nov 21 11:27:24 2005 -->
<pre>User fernand has again many inactive connections to database lasting several thousands seconds. User and experts notified 
 
The CondorCAF has still &gt; 500 sections not consuming CPU.</pre>
<!-- Author: Franco. -->


<!-- Date: Mon Nov 21 13:37:48 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-261&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-262  width='300' align='top'></a></td><td>Black vertical lines in these graphs.</td></tr></table>
<!-- Author: Franco -->


<!-- Date: Mon Nov 21 15:05:07 2005 -->
<pre>Another mover offline: 9940B26</pre>
<!-- Author: Franco -->


<!-- Date: Mon Nov 21 16:07:32 2005 -->
<pre>End of shift report: 
 
At about noon there appeared a black vertical line in each of the graphs in 
the data logger disk. This is under investigation by the experts. 
 
Still error on tape IA9272 and now two movers offline: the old 9940B16  
and the new 9940B26. 
 
CondorCAF now has less the 500 stuck sections. 
 
Nobody now has many inactive connections to the db. 
</pre>
<!-- Author: Franco -->


<!-- Date: Tue Nov 22 08:29:35 2005 -->
<pre>Start of shift report: 
 
  OFFLINE REPORT: Tue Nov 22 07:48:44 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-21 
   RUN NUMBER  207440 
   RUN TIME  11:02:34 
 
2) CALIBRATIONS 
   10A is not available for: 
       207513         PHYSICS_3_07   2 2005.11.21   09:16:51     PROD_PHYSICS_CD 
       This is 22 hours old. 
   17A is not available for: 
       207513         PHYSICS_3_07   2 2005.11.21   09:16:51     PROD_PHYSICS_CD 
       This is 22 hours old. 
=== 
The caf_prd sam db server is not responding. Experts notified. 
</pre>
<!-- Author: Franco -->


<!-- Date: Tue Nov 22 11:22:18 2005 -->
<pre>User jgarcian has used 8507 cpu-seconds on database instance cdfstrm1. user and experts notified. 
</pre>
<!-- Author: Franco -->


<!-- Date: Tue Nov 22 11:24:27 2005 -->
<pre>All db servers are now responding.</pre>
<!-- Author: Franco -->


<!-- Date: Tue Nov 22 13:11:28 2005 -->
<pre>User sforrest has more then 20 inactive connections lasting thousands 
seconds each. User and experts notified. 
 
 
Problem in in the DCache monitor : 
http://www-cdf.fnal.gov/~stager/DHAtAGlance/ 
there is this line: 
Door with the most logins:          Door 00 
                                  ^^^^^^^  this is RED 
</pre>
<!-- Author: Franco -->


<!-- Date: Tue Nov 22 14:56:04 2005 -->
<pre>According to  http://www-cdf.fnal.gov/~stager/DHAtAGlance/ 
DCache has 3 cells offline 
Enstore has 2 movers offline 
</pre>
<!-- Author: Franco -->


<!-- Date: Tue Nov 22 16:04:35 2005 -->
<pre>End of shift report: 
 
DCache: 
these three cell are offline: 
r-fcdfdata110-1 
r-fcdfdata110-2 
r-fcdfdata110-3 
 
Enstore: 
movers 9940B16 and 9940B28 are offline and tape  IA9272 still noacces 
 
All the other systems are fine. 
</pre>
<!-- Author: Franco -->


<!-- Date: Wed Nov 23 08:10:38 2005 -->
<pre>Start of shift report: 
   OFFLINE REPORT: Wed Nov 23 07:46:14 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-22 
   RUN NUMBER  207488 
   RUN TIME  15:01:26 
 
2) CALIBRATIONS 
   10A is not available for: 
       207513         PHYSICS_3_07   2 2005.11.21   09:16:51     PROD_PHYSICS_CD 
       This is 1 day(s), 22 hours old. 
   17A is not available for: 
       207513         PHYSICS_3_07   2 2005.11.21   09:16:51     PROD_PHYSICS_CD 
       This is 1 day(s), 22 hours old. 
==== 
DChache has still 3 cells offline 
Enstore has still 3 movers offline 
The IA9272 problem has been solved. 
</pre>
<!-- Author: Franco -->


<!-- Date: Wed Nov 23 16:03:34 2005 -->
<pre>End of shift report: 
 
DCache: still 3 cells offline. Door 00 red 
Enstore : still 1 mover offline. 
 
CAF: there is a growing number of both idle and waiting resources in the CAF 
Experts notified. 
 
</pre>
<!-- Author: Franco -->


<!-- Date: Thu Nov 24 07:57:32 2005 -->
<pre>   OFFLINE REPORT: Thu Nov 24 07:47:10 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-11-22 
   RUN NUMBER  207488 
   RUN TIME  15:01:26 
 
2) CALIBRATIONS 
   10A is up to date. 
   17A is not available for: 
       207513         PHYSICS_3_07   2 2005.11.21   09:16:51     PROD_PHYSICS_CD 
       This is 2 day(s), 22 hours old. 
=== 
- CAF: Condor CAF is off, CAF reduced to 100 VMs (out of 1600). Another power failure? 
 
- DH: three cells and 1 mover still offline. 
</pre>
<!-- Author: Franco -->


<!-- Date: Thu Nov 24 11:13:28 2005 -->
<pre>CondorCAF seems to be to still off, even if the power is back in FCC since at least two hours. Problem to the monitoring system? 
</pre>
<!-- Author: Franco. -->


<!-- Date: Thu Nov 24 12:36:47 2005 -->
<pre>CondorCAF is actually still off. Quoting Matthew: 
"It appears that the power system restarted at some point, which flips off the system" 
</pre>
<!-- Author: Franco -->


<!-- Date: Thu Nov 24 13:09:49 2005 -->
<pre>Matthew has restarted most of the CondorCAF</pre>
<!-- Author: Franco -->


<!-- Date: Thu Nov 24 16:06:51 2005 -->
<pre>End of shift report: 
 
- There has been a power failure at GCC and CondorCAF went down at about 
3.30AM.  It has been restarted at about 12.30 and is coming up with 
1200 VMs (out of 1650) 
 
- In the DH At A Glance web page there are these new warning: 
 
DCache: 
Total number of files to restore: 523 
 
Enstore: 
9940B Library: CANNOT UPDATE STATUS 
Movers in Trouble:  17 / 18 
 
Experts notified.</pre>
<!-- Author: Franco -->


<!-- Date: Thu Dec 15 08:51:13 2005 -->
<pre>There is again a problem with cdfkits.fnal.gov as  
some of web pages are not accessible. 
 
10A calib. streanm is older than 48hrs. 
 
In both cases the experts arenotified iwth emails.</pre>
<!-- Author: Igor Gorelov. -->


<!-- Date: Thu Dec 15 11:42:44 2005 -->
<pre>CAF Monitor plots sometimes appear as having confusing gaps 
of a dark blue color. This is not a CAF fails but just  
occasional interrupts of monitoring data stream filling  
these plots. The CAF expert has been notified.</pre>
<!-- Author: Igor Gorelov. -->


<!-- Date: Thu Dec 15 11:48:10 2005 -->
<pre>Still nothing (after 3 days of data taking) 
has been processed for DQM -- no histogramms  
avail. from  corresponding web-page.</pre>
<!-- Author: Igor Gorelov. -->


<!-- Date: Fri Dec 16 11:17:29 2005 -->
<pre>The new offline shifter, Sandra,  has started her job. 
The environment for new offline shift has been setup. 
 
A number of concerns have been sent to experts 
via emails.</pre>
<!-- Author: Sandra & Igor. -->


<!-- Date: Fri Dec 16 16:03:15 2005 -->
<pre>End of shift 
 
Nothing new with respect to this morning entry.</pre>
<!-- Author: Sandra -->


<!-- Date: Sat Dec 17 08:07:22 2005 -->
<pre>   OFFLINE REPORT: Sat Dec 17 07:58:57 CST 2005 
 
2) CALIBRATIONS 
   10A is not available for: 
       208792         PHYSICS_3_07   2 2005.12.13   01:31:50     PROD_PHYSICS_CD 
       This is 4 day(s), 6 hours old. 
   17A is not available for: 
       208792         PHYSICS_3_07   2 2005.12.13   01:31:50     PROD_PHYSICS_CD 
       This is 4 day(s), 6 hours old. 
 
The same as yesterday morning 
</pre>
<!-- Author: Sandra -->


<!-- Date: Sat Dec 17 16:02:23 2005 -->
<pre>End of shift 
 
Nothing new or relevant with respect to this morning 
 
</pre>
<!-- Author: Sandra -->


<!-- Date: Sun Dec 18 08:03:36 2005 -->
<pre>Beginning of shift: 
 
  OFFLINE REPORT: Sun Dec 18 08:00:22 CST 2005 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR: 
   RUN DATE  2005-11-22 
   RUN NUMBER  207488 
   RUN TIME  15:01:26 
 
2) CALIBRATIONS 
   10A is not available for: 
       208993    PHYSICS_TEST_3_07   8 2005.12.16   16:15:33     PROD_PHYSICS_CD 
       This is 1 day(s), 15 hours old. 
    
    17A is not available for: 
       208792         PHYSICS_3_07   2 2005.12.13   01:31:50     PROD_PHYSICS_CD 
       This is 5 day(s), 6 hours old. 
</pre>
<!-- Author: Sandra -->


<!-- Date: Sun Dec 18 15:58:05 2005 -->
<pre>Summary of today shift: 
 
Monitor of usage of oracle DB: always OK 
data logger disks: OK 
record of raw data tapes written: OK 
predator: OK 
 
Monitor Dcache: Not up top date: 
DH at a glance:  Thu Dec 15 09:15:01 2005  
Monitor Dcache load:  
Run Wed Dec 14 08:36:05 CST 2005      Data from Wed Dec 14 08:30:50 2005 
Monitor Dcache ping: last entry:ERROR: No database specified on command line or in $PRODUCTS. 
/home/cdfsoft/maint/dcache/check/check: setup: command not found 
Wed Dec 14 08:29:59 CST 2005 /usr/bin/time: cannot run dccp: Input/output error 
Command exited with non-zero status 126 
0:00.00 
Experts were notified already on friday.  
 
CAF:  OK 
 
SAM: OK cdfsam22 monitoring plots restored by Bernd and now working (the machine got renamed, link was not working) 
 
DQM for last 24 hours: page up to date, but no entries in the table  
 
</pre>
<!-- Author: sandra -->


<!-- Date: Mon Dec 19 08:29:14 2005 -->
<pre>Beginning of shit: 
 
   OFFLINE REPORT: Mon Dec 19 07:51:32 CST 2005 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR: 
   RUN DATE  2005-12-18 
   RUN NUMBER  207513 
   RUN TIME  11:58:42 
 
2) CALIBRATIONS 
   10A is not available for: 
       208993    PHYSICS_TEST_3_07   8 2005.12.16   16:15:33     PROD_PHYSICS_CD 
       This is 2 day(s), 15 hours old. 
   17A is not available for: 
       208993    PHYSICS_TEST_3_07   8 2005.12.16   16:15:33     PROD_PHYSICS_CD 
       This is 2 day(s), 15 hours old. 
</pre>
<!-- Author: Sandra -->


<!-- Date: Mon Dec 19 16:04:36 2005 -->
<pre>Summary of today shift:  
 
Monitor of usage of oracle DB: 1 user was notified because had over 20 connections to cdfofprd database  
data logger disks: OK  
record of raw data tapes written: OK  
predator: cdfoom list was notified for 'strange' entry 
 
Monitor Dcache: Still not up top date (experts notified) 
  
CAF:  drop in CDPU due to scheduled outage 
 
SAM: OK  
 
DQM OK  
</pre>
<!-- Author: Sandra -->


<!-- Date: Tue Dec 20 08:38:18 2005 -->
<pre>Beginning of shift: 
 
   OFFLINE REPORT: Tue Dec 20 07:59:40 CST 2005 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR: 
   RUN DATE  2005-12-20 
   RUN NUMBER  209020 
   RUN TIME  02:01:22 
 
2) CALIBRATIONS 
   10A is not available for: 
       208993    PHYSICS_TEST_3_07   8 2005.12.16   16:15:33     PROD_PHYSICS_CD 
       This is 3 day(s), 15 hours old. 
   17A is not available for: 
       208993    PHYSICS_TEST_3_07   8 2005.12.16   16:15:33     PROD_PHYSICS_CD 
       This is 3 day(s), 15 hours old.          
 
By the way, Troy said this run used a test trigger table and is not marked as a good run, so ignoring runs using test trigger tables, the first run for which 10A is not available is  
 
209021   PHYSICS_3_07   2 2005.12.18   01:08:56     PROD_PHYSICS_CDF        10A is not available 
 
and for 17A: same run number. This is &gt; 48 h old. 
</pre>
<!-- Author: Sandra -->


<!-- Date: Tue Dec 20 08:49:15 2005 -->
<pre>"DH at a Glance" web page is finally up to date. 
 
dCache ping monitoring still stuck to Dec. 14. Experts were notified already yesterday.</pre>
<!-- Author: Sandra -->


<!-- Date: Tue Dec 20 15:58:03 2005 -->
<pre>Summary of today shift:   
 
Monitor of usage of oracle DB: OK 
 
data logger disks: OK   
Just as a reference for future shifters: 
Jedong said that the drop in the shape of the data disk usage in the last 24 hours (which gives rise to a very irregular shape in those plots) occurs when the files are deleted after the tape writing is finished. 
 
record of raw data tapes written: OK   
predator:  OK 
 
Monitor Dcache: reported "13 failed" SAM declaration in DH at a glance page (but don't know if it is a problem)  
dCache ping still stuck at dec. 14.  </pre>
<!-- Author: Sandra -->


<!-- Date: Wed Dec 21 08:24:29 2005 -->
<pre>Beginning of shift: 
 
Calibrations: 
first run for which 10A calibrations are not available is more than 48 hours old: 
209053         PHYSICS_3_07   2 2005.12.19   03:40:30     PROD_PHYSICS_CDF        10A is not available 
 
first run for which 17A calibrations are not available is more than 3 days old: 
209021         PHYSICS_3_07   2 2005.12.18   01:08:56     PROD_PHYSICS_CDF        17A is not available 
 
and COT beamlines are available for this same run.</pre>
<!-- Author: Sandra -->


<!-- Date: Wed Dec 21 15:56:03 2005 -->
<pre>End of shift. 
Nothing particular today. 
 
For calibration: 
Bo just said that Beamlines  should be ready for addition to pass 17A up to 209052.  
 
For CAF: 
Igor pointed out that  
FermiGlideCaf is up and running, at least on the GPfarms resources.</pre>
<!-- Author: Sandra -->


<!-- Date: Thu Dec 22 08:20:38 2005 -->
<pre>Beginning of shift: 
 
10A calibrations absolutely up to date! 
209191         PHYSICS_3_07   2 2005.12.21   17:49:34     PROD_PHYSICS_CDF        10A     1     4     
 
17A calibrations: 
first run for which 17A calibrations (and COT beamlines as well)  are not available is more than 3 days old: 
209053         PHYSICS_3_07   2 2005.12.19   03:40:30     PROD_PHYSICS_CDF        17A is not available 
 
</pre>
<!-- Author: Sandra -->


<!-- Date: Thu Dec 22 08:26:17 2005 -->
<pre>DH at a glance page is not updated: Wed Dec 21 11:30:00 2005  
 
Monitor dCache load page was updated (until yesterday the date was Dec14) but it looks stopped again: 
Run Wed Dec 21 17:22:19 CST 2005      Data from Wed Dec 21 17:20:10 2005 
     471 Logins 
     467 Active 
       0 GridFTP 
       4 Killed 
 
Monitor dCache ping still at Dec. 14. 
 
</pre>
<!-- Author: Sandra -->


<!-- Date: Thu Dec 22 16:01:18 2005 -->
<pre>Summary for today shift: 
 
Monitor usage of Oracle DB: always OK 
 
Monitor of data handling operations: 
DH at a glance page has been moved, and now is up to date. 
Noticed 1 door NOT responding and 1/18 movers in trouble (usually these are 0), production coordinators were notified. 
dCache load monitor now is regularly updated. 
 
Joseph notified that All the problem with nodes in GCC have been fixed, and At the present time, all CDF production farm (phase2 sam-farm) condor/caf nodes are up. 
</pre>
<!-- Author: Sandra -->


<!-- Date: Fri Dec 23 08:20:51 2005 -->
<pre>This morning calibrations were absolutely up-to-date! 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2005-12-22 
   RUN NUMBER  209187 
   RUN TIME  22:49:29 
 
The authomatic program makeReport cannot be used to check the last available run, because it signals PHYSICS_TEST_3_07 runs which don't have calibrations. 
So for PHYSICS trigger table runs, 10A are available up to: 
209264         PHYSICS_3_07   2 2005.12.23   02:04:39     PROD_PHYSICS_CDF        10A     
 
and 17A are available up to: 
209189         PHYSICS_3_07   2 2005.12.21   17:27:58     PROD_PHYSICS_CDF        17A     1     5 </pre>
<!-- Author: Sandra -->


<!-- Date: Fri Dec 23 11:23:07 2005 -->
<pre> Offline Oracle DB Usage Monitoring: everything OK 
 
DH System Monitoring: 
    In DH at a Glance appears to be 1/18 Movers in trouble. 
    offline managers notified 
    Monitor dCache ping still not working (it was never working during this past week 
Anything else looks OK 
         
CAf & ConforCAF up and working fine. 
 
Offline DQM: OK 
 
There will be no CO offline shifter replacing me for the following week, so this is the last entry in this e-log for a while. 
Before leaving today I'll remove my kerberos principal from cdffarm0 .k5login</pre>
<!-- Author: Sandra -->


<!-- Date: Fri Jan 6 10:28:33 2006 -->
<pre>Beginning of Shift..  
 
   OFFLINE REPORT: Fri Jan 6 10:21:50 CST 2006 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2006-01-04 
   RUN NUMBER  209511 
   RUN TIME  07:01:01 
 
2) CALIBRATIONS 
   10A is not available for: 
       208993    PHYSICS_TEST_3_07   8 2005.12.16   16:15:33     PROD_PHYSICS_CD 
       This is 20 day(s), 18 hours old. 
   17A is not available for: 
       208993    PHYSICS_TEST_3_07   8 2005.12.16   16:15:33     PROD_PHYSICS_CD 
       This is 20 day(s), 18 hours old. 
</pre>
<!-- Author: YeonSei Chung -->


<!-- Date: Mon Jan 9 07:51:29 2006 -->
<pre>Start of shift. 
 
   OFFLINE REPORT: Mon Jan 9 07:50:36 CST 2006 
 
 
1) THE LATEST RUN CONSUMED BY CDF-FNCDFSRV0 APHYSR:  
   RUN DATE  2006-01-08 
   RUN NUMBER  209760 
   RUN TIME  05:28:57 
 
2) CALIBRATIONS 
   10A is not available for: 
       208993    PHYSICS_TEST_3_07   8 2005.12.16   16:15:33     PROD_PHYSICS_CD 
       This is 23 day(s), 15 hours old. 
   17A is not available for: 
       208993    PHYSICS_TEST_3_07   8 2005.12.16   16:15:33     PROD_PHYSICS_CD 
       This is 23 day(s), 15 hours old. 
</pre>
<!-- Author: YeonSei -->