<!-- Title: 10:28:31  Fri May  9 2003 -->
<!-- Crewchief: Matt Herndon -->
<!-- Op1: Murat -->
<!-- Op2: Sexton -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Start of new offline shift period</pre>


<!-- Date: Fri May 9 10:30:16 2003 -->
<pre>Starting build and testing of 4.10.4c patch 
Disables PhotonValidation - frequently crashing in test 
Patches TrackingKal/src/KalTrack.cc - traps crash in KalTrack destructor</pre>
<!-- Author: herndon -->
<!-- Comment: Sat May 10 08;20;46 comment by...herndon -->
<pre>10K tests of 4.10.4c sucessful.
Tar ball build and farms tests is underway</pre>
<!-- Comment: Sat May 10 15;40;29 comment by...herndon -->
<pre>Crashes during farms testing are as expected</pre>
<!-- Date: Fri May 9 14:45:50 2003 -->
<pre>Checked physics performance. 
 
Data lost due to aliasing of ls command.  This caused some 
scritps used to parse out file names to fail. 
 
run 162481(0x27ab1)  - no data 
run 162498(0x27ac2)  - no data 
run 162519(0x27ad7)  - no data 
</pre>
<!-- Author: herndon -->



<!-- Date: Sat May 10 14:34:07 2003 -->
<pre>opr disk filled up 
space freed up by deleting older core files</pre>
<!-- Author: herndon -->


<!-- Date: Sat May 10 14:36:11 2003 -->
<pre>Looked at physics plot for 
run 162521(0x27ad9) 
run 162631(0x27b47) 
run 162663(0x27b67) 
 
Copy failures due to disk being full led to loss of one file from 
162631 and all files from 162663.  162631 look low, probably due 
to mismatch between lumi and amount of data sucessfully analyzed</pre>
<!-- Author: herndon -->


<!-- Date: Sat May 10 14:54:42 2003 -->
<pre>Calibrations are 4 days behind.  Informed experts. 
</pre>
<!-- Author: herndon -->
<!-- Comment: Sat May 10 15;25;18 comment by...herndon -->
<pre>Reply from Rob

Date: Sat, 10 May 2003 15:14:25 -0500
From: Rob Snihur &lt;snihur@fnal.gov&gt;
Thanks for the info.
This is being caught by our new error checking - yay!
Compared to our reference run, new tables have
been added to the calibration set.
To be conservative, the pass tables are not filled
when such differences are detected.

The new tables are related to the FWD CAL (miniplug, etc.),
which we now pick up on-the-fly (as intended).
The problem is that we forgot to define a new reference run!
Dustin is the right guy to make this change (he's the author
of that script), and flush thru all the outstanding runs.
 If he does not get to this by later today,
I'll try to plod my way thru it (I have to run now).

thanks,
-rob
 </pre>


<!-- Comment: Sun May 11 15;16;56 comment by...herndon -->
<pre>Rob says that the calibration should new be up to date</pre>
<!-- Date: Sat May 10 15:39:36 2003 -->
<pre>4.10.4a crashes 
 
Informed experts about farms crashes and provided debug info 
COT cadidate crashs: more than 30K cadidates - Chris Hays 
Root IO crashes                              - Liz Sexton 
SimpleReconstruced track destructor crash    - Matt Herndon 
KalTrack destructor crash                    - Previously known 
</pre>
<!-- Author: herndon -->



<!-- Date: Sun May 11 11:12:33 2003 -->
<pre>4.10.4c Crashes 
 
1) KalTrack destructor crash - Previously seen crash 
   Still cannot reproduce on single events. 
   Valgrind study in progress - Kurt Rinnert, Matt Herndon 
 
2) Four strikes and your out - Silicon variant 
   Large hit multiplicity in ISL and SVX causes large 
   number of silicon standalone track candidates to be formed. 
   Error catching can detect this problem and abort silicon 
   standalone tracking.  Program aborts with a bad memory 
   alocation before the error catching is tripped.  Probably 
   has to do with combination of density of gjet data and 
   having two such dense datasets being processed simultaniously 
   on a given farm node.  Files may be processably if rerun 
   solo. 
 
3) Found strikes and your out - COT variant 
   Large number of COT hit/segment candidates allocated causing 
   a bad memeory allocation error.  Expert informed - Chris Hays 
   </pre>
<!-- Author: herndon -->
<!-- Comment: Sun May 11 15;59;30 comment by...herndon -->
<pre>Striping off crashed event plus several event preceeding the
problem event does not help replecate any of the problems.
Tried 10 events

Also memory usage does not look that bad on events that crashed
from memery allocation errors.  These events tend to cause on
order 200 addtional megabytes to be alocated to a total of 700MB.</pre>

<!-- Comment: Mon May 12 09;22;02 comment by...herndon -->
<pre>4) Crash in Cpr clustering.  Segmentation fault do to Null pointer.
    Experts Informed Tania and Beate


</pre>


<!-- Date: Mon May 12 12:48:25 2003 -->
<pre>Recieved guidance from an expert(Kurt Rinnert) on how to run 
valgrind with the CDF code and the KAI compiler.  Run with the 
following runtime switches. 
 
/cdf/home/jbk/bin/valgrind --demangle=no \ --suppressions=/cdf/opr/cdfopr/valTest/root.supp 
 
valgrind exe and root.supp(suppresion file) like in the 
specified directories for now 
 
Result was information on memeory usage violations in the 
Kal tracking code when dealing with KalSiHits.  Forward this 
information to the relevent experts: Kurt, Stephanie and Thorsten</pre>
<!-- Author: herndon -->


<!-- Date: Mon May 12 15:42:34 2003 -->
<pre>Performed calibration check. 
 
Calibrations are 4 days behind. 
 
Noticed that one of the Jet Prereq modules still tried to use 
TL3D. 
 
Notified experts</pre>
<!-- Author: herndon -->


<!-- Date: Tue May 13 11:12:42 2003 -->
<pre>4.10.4c Crash analysis Cont. 
 
A new crash found in VxPrim.  The crash happens when trying 
to deal with particles that point outside of the beam pipe. 
Crash only happens twice in 5M event. 
 
Expert has been informed and is working on it: Wolfgang Wagner</pre>
<!-- Author: herndoin -->


<!-- Date: Tue May 13 11:14:01 2003 -->
<pre>Stntuple job from Cigdem Issever thrshes the database. 
 
We are pusuing which version of Stntuple was used and wether 
the situation can be improved</pre>
<!-- Author: herndon -->
<!-- Comment: Tue May 13 14;36;04 comment by...herndon -->
<pre>Problem traced down to poor database access in Stntuple
Pasha informed</pre>
<!-- Date: Tue May 13 11:18:37 2003 -->
<pre>4.10.4c crash analysis Cont. 
 
Memory allocation crashes in silicon turn out to be due to 
a known silicon hardware error.  When 5 level one accepts 
are sent and no reset(should normally be sent) the silicon 
readout chip go crazy.  The symptoms are that every other 
strip in the detector on the Z sides and in large chunks of 
the ISL is read out with a large amount of charge.  This leads 
to an abnormal number of 3D hit pair cadidates in the standalone 
tracking code. 
 
This hardware error was diagnosed and corrected in late September 
of 2002. 
 
Some possible solutions may cut into physics data.  The new 
clustering code has a cut that could provide a solution without 
endangering physics.  I propose leaving this for 5.0.0</pre>
<!-- Author: herndon -->
<!-- Comment: Tue May 13 14;37;11 comment by...herndon -->
<pre>Fix disigned to catch half of errors.  Bail out if 80000
clusters found.

-r 2.128.1.1 TrackingMods/src/SiPatternRecModule.cc</pre>


<!-- Date: Wed May 14 15:55:24 2003 -->
<pre>build 4.11.0pre1 Release 
 
Performing 10K test</pre>
<!-- Author: herndon -->
<!-- Comment: Thu May 15 15;43;03 comment by...herndon -->
<pre>Test resutls look good

Many error logger useage vioations - experts notified
  - Tara - SiliconMon
  - Matt - L00
  - Beate - Calor

Comparison of 4.11.0pre1 to 4.8.4c looks good no significant differences</pre>
<!-- Date: Wed May 14 15:58:47 2003 -->
<pre>Calbrations are now up to date and physics data looks good</pre>
<!-- Author: herndon -->


<!-- Date: Thu May 15 12:53:59 2003 -->
<pre>Build 4.10.4d version patch release 
 
Starting 10K test</pre>
<!-- Author: herndon -->
<!-- Comment: Fri May 16 10;56;11 comment by...herndon -->
<pre>10K test completed without crashes or error problems

Histograms look okay - except that BeamWidthModule histograms
  are missing.  Prepared first version of e patch fixing this.

Started building 4.10.4d release on cdfcode

Concatination test failed - reported to expert: Liz Sexton

</pre>

<!-- Comment: Fri May 16 14;42;55 comment by...herndon -->
<pre>Finished building 4.10.4d and made tarball
Tarball still does not properly include AC++Dump
Tarball on hold waiting for input from experts on above
issues</pre>


<!-- Date: Fri May 16 14:51:20 2003 -->
<pre>End of shift status 
 
4.10.4d tar ball built 
  includes compression software 
  e tarball including updated version of compression software 
   will be needed 
  use of tar ball waiting for expert input on failure of 
   concatination test and inclusing of AC++Dump 
  Experts informed - Liz Secton and Pasha Murat 
 
4.8.4h test release built 
  10K test in progress 
  i version including updated version of compression software 
   will be needed 
  Release and tar ball not yet built - waiting on analysis 
    of 10K results 
 
4.11.0pre1 10K test performed 
  Several error logger violations found 
  One fixed by Yimei Huang - following files should be tagged 
    for pre2 
  ElectroWeakValidation.hh/1.3/Thu May 15 19:38:24 2003 
  ElectroWeakValidation.cc/1.5/Fri May 16 16:03:19 2003 
  One fixed in L00 code - Matt Herndon will tag the fix for pre2 
  Waiting for input on other calor and silimon error logger 
    problems 
 
Somewhat unreleated note 
  4.11.0pre1 cdfSim output can be processed by 4.9.1.  This is 
  a very useful feature for people performing 4.8.4 Production 
  based analysis.  I think we should make this a goal of the 
  4.11.x release. 
 
 
</pre>
<!-- Author: herndon -->
<!-- Comment: Fri May 16 14;59;29 comment by...herndon -->
<pre>4.10.4 crashes

VxPrim Extropalator crash         - Contact Wolfgang Wagner
COT candidate memeory crash       - Contact Chris Hays
Silicon KalTrack destructor crash - Contacts Matt Herndon 
                                             Kurt Rinnert

Other issues
Jet Prereq module still uses TL3D bank
PhotonValidation need to be fixed and reintroduced
</pre>

<!-- Comment: Fri May 16 15;29;09 comment by...herndon -->
<pre>  2 of the 10K jobs for 4.8.4h crashed before the first event. 
When I ran them as standalone job they didn't crash.  All of the
jobs had error in  the trigger bit section.  Example:

 No such path with path Id = 100

</pre>


<!-- Date: Sat May 17 09:11:05 2003 -->
<pre>When checking calibration run 
there is error, 
&gt;./check_run.sh 163025 163026 
Processing run 163025... 
Input file is root://fcdfsgi2.fnal.gov//cdf/data08/look/aphysr/ar027cd1.00f2phys 
./check_run.sh: line 71: 27285 Aborted                 (core dumped) ProductionE 
xe Production/ProductionExe.tcl -i $input_file -o /tmp/junk_check_cal.jnk 1&gt;&$lo 
gfile 
Log file is results/check_run.163025.log 
Number of ERLOG-e errors          = 0 
Number of ERLOG-s severe errors   = 1 
Number of BADPASSCALIB errors     = 1     
 
&gt; grep ERLOG-s results/check_run.163025.log  
%ERLOG-s Abnormal termination: caused by signal:  6 Framework abortJob()</pre>
<!-- Author: Dae Jung -->


<!-- Date: Sat May 17 11:43:42 2003 -->
<pre>Build 4.10.4e version patch release 
 Starting 10K test</pre>
<!-- Author: Dae Jung -->


<!-- Date: Sat May 17 15:53:04 2003 -->
<pre>Database Usage : OK  
Oracle Usage   : OK  
Checking that kahuna process is running...  
    kahuna process is running OK  
10K test is now running.... 4.10.4e</pre>
<!-- Author: Dae Jung -->

<!-- Date: Sat May 17 19:08:07 2003 -->
<pre>

Roman, Yen-Chu, all,

               Updated status of the farms as of :

                      7:30 A.M.  5/17/03


              Behind by approx. 25 Million on Raw Data
              (This is taken from the graph on the farms page,
                a snapshot at about 1 A.M. each day)

               5.2 Million events processed in last 24 hours
                 (all 4.8.4g, streams B,C,G,J)


              Issues:

                1. Will watch concatenation carefully.
                2. 4.10.4e was created and is being tested by
                   offline shifter.
                   Will get to farms soon and will be tested.

              Plan for the day:

                1. Process raw data with 4.8.4g.
                2. Run streams to catch up to calibrations
                     with priority on B and G.
                3. Validate 4.10.4e if possible.


            Stream        Run        Date run was taken   Events behind*

           Stream A      162520        6-May-03           820,345
           Stream B      162937       14-May-03           899,295
           Stream C      162837       12-May-03           981,851
           Stream D      162498        5-May-03         1,251,600
           Stream E      162498        5-May-03         3,884,822
           Stream G      162686        9-May-03         2,032,306
           Stream H      162498        5-May-03        10,387,888
           Stream J      162686        9-May-03         2,562,337

            * This is more or less behind raw data on tape.  The farms
                cannot process events if the calibrations aren't
                prepared.

                Calibrations (07)  163025   17-May-03


                Current data-taking   163026

             Reprocessing:  Restarting



Steve
</pre>
<!-- Author: wolbers -->


<!-- Date: Sun May 18 07:33:08 2003 -->
<pre>
Roman, Yen-Chu, all,

              Updated status of the farms as of :

                     7:30 A.M.  5/18/03


             Behind by approx. 20 Million on Raw Data
             (This is taken from the graph on the farms page,
               a snapshot at about 1 A.M. each day)

              5.4 Million events processed in last 24 hours
                (all 4.8.4g, streams B,C,G,J)
              + Special 4.8.4g_L3 on runs 163031, 163032
                (requested by Steve Nahn)


             Issues:

               1. Will watch concatenation carefully.
               2. 4.10.4e was created and is being tested by
                  offline shifter.
                  Will get to farms soon and will be tested.

             Plan for the day:

               1. Process raw data with 4.8.4g.
               2. Run streams to catch up to calibrations
                    with priority on B and G.
               3. Validate 4.10.4e if possible.


           Stream        Run        Date run was taken   Events behind*

          Stream A      162520        6-May-03           975,385
          Stream B      163024       17-May-03           103,301
          Stream C      163011       16-May-03           228,244
          Stream D      162498        5-May-03         1,347,498
          Stream E      162498        5-May-03         3,966,303
          Stream G      162820       11-May-03         2,053,164
          Stream H      162498        5-May-03        10,625,507
          Stream J      163011       16-May-03           301,038

           * This is more or less behind raw data on tape.  The farms
               cannot process events if the calibrations aren't
               prepared.

               Calibrations (07)  163025   17-May-03


               Current data-taking   163035

            Reprocessing:  Restarting



Steve
</pre>
<!-- Author: wolbers -->



<!-- Date: Sun May 18 11:26:36 2003 -->
<pre>Database Usage : OK 
Oracle Usage   : OK 
Checking that kahuna process is running... 
    kahuna process is running OK . 
 
10K test was finished successfully.. 4.10.4e -&gt; report to Liz sexton 
Concatenate Test crash.(10 events) 
Making a tarball : OK  </pre>
<!-- Author: Dae Jung -->

<!-- Date: Mon May 19 08:11:02 2003 -->
<pre>
Roman, Yen-Chu, all,

              Updated status of the farms as of :

                     8:00 A.M.  5/19/03


             Behind by approx. 23 Million on Raw Data
 
              2.1 Million events processed in last 24 hours
  

             Issues:

               1. Output of the farm is falling behind.  Events
                  are processed but are saved in the farms buffers.
                  Will update later with new information.
               2. 4.10.4e was created and is being tested by
                  offline shifter.
                  Will get to farms soon and will be tested.

             Plan for the day:

               1. Process raw data with 4.8.4g.
               2. Run streams to catch up to calibrations
                    with priority on B and G.
               3. Validate 4.10.4e if possible.


           Stream        Run        Date run was taken   Events behind*

          Stream A      163012       16-May-03           168,642
          Stream B      163024       17-May-03           619,760
          Stream C      163011       16-May-03           506,615
          Stream D      162498        5-May-03         1,517,152
          Stream E      162498        5-May-03         4,532,222
          Stream G      163024       17-May-03           433,334
          Stream H      162498        5-May-03        11,928,424
          Stream J      163012       16-May-03           487,383

           * This is more or less behind raw data on tape.  The farms
               cannot process events if the calibrations aren't
               prepared.

               Calibrations (07)  163025   17-May-03


               Current data-taking   163064

            Reprocessing:  Restarting as soon as possible.



Steve
</pre>
<!-- Author: wolbers -->


<!-- Date: Mon May 19 12:11:53 2003 -->
<pre>
Roman, Yen-Chu, all,

              Updated status of the farms as of :

                     8:00 A.M.  5/19/03


             Behind by approx. 23 Million on Raw Data
 
              2.1 Million events processed in last 24 hours
  

             Issues:

     --->      1. Problem with 4.8.4g concatenation.  Pasha has
                  been informed.  Debugging in progress.
     --->      2. Crash rate with 4.8.4g is higher than 4.8.4a.
               3. 4.10.4e was created and is being tested by
                  offline shifter.
                  Will get to farms soon and will be tested.

             Plan for the day:

     --->      1. Get concatenator debugged.
               2. Process raw data with 4.8.4g.  
               3. Run streams to catch up to calibrations
                    with priority on B and G.
               4. Validate 4.10.4e if possible.


           Stream        Run        Date run was taken   Events behind*

          Stream A      163012       16-May-03           168,642
          Stream B      163024       17-May-03           619,760
          Stream C      163011       16-May-03           506,615
          Stream D      162498        5-May-03         1,517,152
          Stream E      162498        5-May-03         4,532,222
          Stream G      163024       17-May-03           433,334
          Stream H      162498        5-May-03        11,928,424
          Stream J      163012       16-May-03           487,383

           * This is more or less behind raw data on tape.  The farms
               cannot process events if the calibrations aren't
               prepared.

               Calibrations (07)  163025   17-May-03


               Current data-taking   163064

            Reprocessing:  Restarting as soon as possible.



Steve
</pre>
<!-- Author: wolbers -->



<!-- Date: Mon May 19 12:12:47 2003 -->
<pre>Database Usage : OK 
Oracle Usage   : OK 
Checking that kahuna process is running... 
    kahuna process is running OK . 
removed 4.10.4e.patch, there was problems - Pasha and Liz. 
Check calibrations for the most recent runs : not finished new runs. </pre>
<!-- Author: Dae Jung -->

<!-- Date: Tue May 20 08:28:21 2003 -->
<pre>
Roman, Yen-Chu, all,

              Updated status of the farms as of :

                     8:00 A.M.  5/20/03


             Behind by approx. 22 Million on Raw Data

              2.7 Million events processed in last 24 hours


             Issues:

               1. Problem with 4.8.4g executable.  Processing
                  has been halted while we figure out what to do.
               2. Will use old concatenator once we start
                  up again.
               3. Almost no 4.8.4g has been written to tape.
                  The files are in the farm output disk buffers.
               4. 4.10.4e was created and is being tested by
                  offline shifter.  
                  Will get to farms soon and will be tested.

             Plan for the day:

               1. Sort out 4.8.4g issues.
               2. Process raw data with 4.8.4h?
               3. Run streams to catch up to calibrations
                    with priority on B and G.
               4. Validate 4.10.4e if possible.


           Stream        Run        Date run was taken   Events behind*

          Stream A      163024       17-May-03           237,248
          Stream B      163024       17-May-03           809,685
          Stream C      163024       17-May-03           474,982
          Stream D      163024       17-May-03           208,149
          Stream E      162498        5-May-03         4,872,066
          Stream G      163024       17-May-03           814,487
          Stream H      162498        5-May-03        12,700,880
          Stream J      163024       17-May-03           650,442

           * This is more or less behind raw data on tape.  The farms
               cannot process events if the calibrations aren't
               prepared.

               Calibrations (07)  163025   17-May-03


               Current data-taking   163148

            Reprocessing:  Restarting as soon as possible.



Steve
</pre>
<!-- Author: wolbers -->


<!-- Date: Tue May 20 11:56:59 2003 -->
<pre>
Roman, Yen-Chu, all,

              Updated status of the farms as of :

                    12:00 P.M.  5/20/03


             Issues:

               1. 4.8.4g is being fixed.
               2. Will use the old concatenator once we start
                  up again.
               3. 4.10.4e work continues.
                  Will get to farms soon and will be tested.

             Plan for the day:

               1. Concatenate and write to tape streams A,B,C,D,G,J.
                  Only one file was not processed because of the 
                  4.8.4g problem.  
               2. Process E and H with the new executable.
               3. Validate 4.10.4e if possible.




Steve
</pre>
<!-- Author: wolbers -->



<!-- Date: Tue May 20 16:02:18 2003 -->
<pre>Database Usage : OK 
Oracle Usage   : OK 
Checking that kahuna process is running... 
    kahuna process is running OK . 
removed 4.10.4e.patch, problems was fixed -Liz 
10K test is running ... 
Check calibrations for the most recent runs : has problems -Pasha </pre>
<!-- Author: Dae Jung -->

<!-- Date: Wed May 21 08:01:16 2003 -->
<pre>
Roman, Yen-Chu, all,

              Updated status of the farms as of :

                     8:00 A.M.  5/21/03


             Behind by approx. 22 Million on Raw Data

              0.0 Million events processed in last 24 hours


             Issues:

               1. Problem with 4.8.4g executable.  4.8.4g1 is
                  being created and tested.  May rerun all
                  data from 162519 on.
               2. 4.10.4e was created and is being tested by
                  offline shifter.  
                  Will get to farms soon and will be tested.

             Plan for the day:

               1. Sort out 4.8.4g issues.  
               2. Process raw data with 4.8.4g1.
               3. Run streams to catch up to calibrations
                    with priority on B and G.
               4. Validate 4.10.4e if possible.


           Stream        Run        Date run was taken   Events behind*

          Stream A      163024       17-May-03           237,248
          Stream B      163024       17-May-03           986,823
          Stream C      163024       17-May-03           565,703
          Stream D      163024       17-May-03           354,251
          Stream E      162498        5-May-03         5,016,221
          Stream G      163024       17-May-03           914,850
          Stream H      162498        5-May-03        12,705,237
          Stream J      163024       17-May-03           728,648

           * This is more or less behind raw data on tape.  The farms
               cannot process events if the calibrations aren't
               prepared.

               Calibrations (07)  163026  17-May-03


               Current data-taking   163148

            Reprocessing:  Restarting as soon as possible.



Steve
</pre>
<!-- Author: wolbers -->


<!-- Date: Wed May 21 09:01:13 2003 -->
<pre>
Dear Farmers, 4.8.4g1 ProductionExe tarball is available from FCDFLNX3:

  /home/cdfsoft/dist/cdfpatch/4.8.4g1:
  total 71724
  drwxr-xr-x   10 cdfsoft  cdfsoft      4096 May 21 08:53 .
  drwxr-xr-x   26 cdfsoft  cdfsoft      4096 May 21 08:52 ..
  -rw-r--r--    1 cdfsoft  cdfsoft         6 May 21 08:52 .base_release
  -rw-r--r--    1 cdfsoft  cdfsoft       154 May 21 08:52 GNUmakefile
  -rw-r--r--    1 cdfsoft  cdfsoft  73316228 May 21 08:53 ProductionExe_4.8.4g1.tar.gz
  drwxr-xr-x    2 cdfsoft  cdfsoft      4096 May 21 08:52 bin
  drwxr-xr-x    2 cdfsoft  cdfsoft      4096 May 21 08:52 doc

						-best, Pasha
</pre>
<!-- Author: murat -->



<!-- Date: Wed May 21 11:47:59 2003 -->
<pre>Checking calibration : No more runs 
Database Usage : OK  
Oracle Usage   : OK  
Checking that kahuna process is running...  
    kahuna process is running OK .  
4.10.4e: Production Executable : OK 
         10K test              : OK 
         Concatenate test      : OK 
         Making Tarball        : OK --&gt; report to Liz & Pasha 
</pre>
<!-- Author: Dae Jung -->

<!-- Date: Thu May 22 07:36:55 2003 -->
<pre>
Roman, Yen-Chu, all,

              Updated status of the farms as of :

                     7:30 A.M.  5/22/03


             Behind by approx. 22 Million on Raw Data

              0.0 Million events processed in last 24 hours


             Issues:

               1. 4.8.4g1 is being tested and debugged.  Plan is 
                  to rerun all data from 162519 with 4.8.4g1.
               2. Enstore downtime Thursday morning.  No tape 
                  access during that time.
               3. fcdflnx3 downtime Thursday morning.  Log files,
                  histograms, core files will remain on the 
                  farm during that time.  Transfers will occur
                  later.
               4. 4.10.4e was created and is being tested by
                  offline shifter.  
                  Will get to farms soon and will be tested.

             Plan for the day:

               1. Sort out 4.8.4g1 issues.  
               2. Process raw data with 4.8.4g1.
               3. Run streams to catch up to calibrations
                    with priority on B and G.
               4. Validate 4.10.4e if possible.


           Stream        Run        Date run was taken   Events behind*

          Stream A      163024       17-May-03           237,248
          Stream B      163024       17-May-03           986,823
          Stream C      163024       17-May-03           565,703
          Stream D      163024       17-May-03           354,251
          Stream E      162498        5-May-03         5,016,221
          Stream G      163024       17-May-03           914,850
          Stream H      162498        5-May-03        12,705,237
          Stream J      163024       17-May-03           728,648

           * This is more or less behind raw data on tape.  The farms
               cannot process events if the calibrations aren't
               prepared.

               Calibrations (07)  163026  17-May-03


               Current data-taking   163148

            Reprocessing:  Restarting as soon as possible.



Steve
</pre>
<!-- Author: wolbers -->


<!-- Date: Fri May 23 08:01:15 2003 -->
<pre>
Roman, Yen-Chu, all,

              Updated status of the farms as of :

                     8:00 A.M.  5/23/03


             Behind by approx. 36 Million on Raw Data

              0.0 Million events processed in last 24 hours


             Issues:

               1. 4.8.4g1 is being tested and debugged.  Plan is 
                  to rerun all data from 162519 with 4.8.4g1.
               2. 4.10.4e was created and is being tested by
                  offline shifter.  
                  Will get to farms soon and will be tested.

             Plan for the day:

               1. Sort out 4.8.4g1 issues.  
               2. Process raw data with 4.8.4g1.
               3. Run streams to catch up to calibrations
                    with priority on B and G.
               4. Validate 4.10.4e if possible.

-----------------------------------------------------------------------
                 The following table has been modified to 
                 reflect the fact that runs 162519 and
                 higher that were processed with 4.8.4g 
                 need to be rerun with 4.8.4g1.  
                 The histograms, etc. from 4.8.4g are available.

           Stream        Run        Date run was taken   Events behind*

          Stream A      162498        5-May-03         1,235,266
          Stream B      162498        5-May-03         4,788,873
          Stream C      162498        5-May-03         2,745,939
          Stream D      162498        5-May-03         1,663,254
          Stream E      162498        5-May-03         5,016,221
          Stream G      162498        5-May-03         4,481,914
          Stream H      162498        5-May-03        12,705,237
          Stream J      162498        5-May-03         3,684,740

           * This is more or less behind raw data on tape.  The farms
               cannot process events if the calibrations aren't
               prepared.

               Calibrations (07)  Up to date.


               Current data-taking   163148

            Reprocessing:  Restarting as soon as possible.



Steve
</pre>
<!-- Author: wolbers -->



<!-- Date: Fri May 23 16:17:16 2003 -->
<pre>Check calibrations for the most recent runs: OK 
Database Usage : OK  
Oracle Usage   : OK  
Checking that kahuna process is running...  
    kahuna process is running OK .  
Data Logger disks: OK 
Check Express production: problems: 
last run produced 027d4c  (May 20 00:22). 
Experts (Knuteson/Wolfe) informed.   
</pre>
<!-- Author: Stefano -->

<!-- Date: Sat May 24 07:45:48 2003 -->
<pre>
Roman, Yen-Chu, all,

              Updated status of the farms as of :

                     8:00 A.M.  5/24/03


             Behind by approx. 32 Million on Raw Data

              2.8 Million events processed in last 24 hours


             Issues:

               1. 4.8.4g1 processing has started!
               2. 4.10.4e was created and is being tested by
                  offline shifter.  
                  Will get to farms soon and will be tested.

             Plan for the day:
  
               1. Process raw data with 4.8.4g1.
               2. Run streams to catch up to calibrations
                    with priority on B and G.
               3. Validate 4.10.4e if possible.

-----------------------------------------------------------------------
                 The following table has been modified to 
                 reflect the fact that runs 162519 and
                 higher that were processed with 4.8.4g 
                 need to be rerun with 4.8.4g1.  
                 The histograms, etc. from 4.8.4g are available.

           Stream        Run        Date run was taken   Events behind*

          Stream A      162498        5-May-03         1,250,506
          Stream B      162631        7-May-03         4,278,817
          Stream C      162498        5-May-03         2,800,926
          Stream D      162498        5-May-03         1,704,830
          Stream E      162838       12-May-03         2,824,523
          Stream G      162686        9-May-03         3,274,761
          Stream H      162631        7-May-03        11,257,920
          Stream J      162631        7-May-03         3,278,270

           * This is more or less behind raw data on tape.  The farms
               cannot process events if the calibrations aren't
               prepared.

               Calibrations (07)  163390  23-May-03


               Current data-taking   163430

            Reprocessing:  Restarting as soon as possible.



Steve
</pre>
<!-- Author: wolbers -->



<!-- Date: Sat May 24 14:35:41 2003 -->
<pre>Check calibrations for the most recent runs: OK  
Database Usage : OK   
Oracle Usage   : OK   
Checking that kahuna process is running...   
    kahuna process is running OK .   
Data Logger disks: OK </pre>
<!-- Author: Stefano -->


<!-- Date: Sat May 24 14:43:34 2003 -->
<pre>Looked at physics plots for: 
  
run 162989(0x27cad) 
run 163009(0x27cc1) 
run 163011(0x27cc3) 
run 163023(0x27ccf) 
run 163024(0x27cd0) 
run 163035(0x27cdb) 
 
run 162989:  
Number Wenn slightly low (0.5 VS average of 0.8) 
Number J/psi low (2.3 nb (I repeated the yield fit manually  
just to be sure) VS average yield of 6.5). 
Looking at the online e-log this run was marked bad  
cause SMX problems. Moreover this run was taken with a  
new trigger table with muon rate limit implemented.  
All the other runs (more recent) does not show any  
problem in J/psi yields. 
 
run 163035: 
Number Zee = 0. (all other histograms OK).</pre>
<!-- Author: Stefano -->

<!-- Date: Sun May 25 08:02:14 2003 -->
<pre>
Roman, Yen-Chu, all,

               Updated status of the farms as of :

                      8:00 A.M.  5/25/03


              Behind by approx. 27 Million on Raw Data

                6.9 Million events processed in last 24 hours


              Issues:

                1. 4.10.4e is ready for farms testing.

              Plan for the day:

                1. Process raw data with 4.8.4g1.
                2. Run streams to catch up to calibrations
                     with priority on B and G.
                3. Test 4.10.4e.

-----------------------------------------------------------------------
                  The following table has been modified to
                  reflect the fact that runs 162519 and
                  higher that were processed with 4.8.4g
                  need to be rerun with 4.8.4g1.
                  The histograms, etc. from 4.8.4g are available.

            Stream        Run        Date run was taken   Events behind*

           Stream A      162498        5-May-03         1,311,360
           Stream B      162825       11-May-03         3,524,104
           Stream C      162498        5-May-03         3,000,950
           Stream D      162498        5-May-03         1,864,924
           Stream E      162986       15-May-03         1,962,816
           Stream G      162857       13-May-03         2,473,847
           Stream H      162664        8-May-03        10,877,931
           Stream J      162856       13-May-03         2,242,595

            * This is more or less behind raw data on tape.  The farms
                cannot process events if the calibrations aren't
                prepared.

                Calibrations (07)  163390  23-May-03


                Current data-taking   163463

             Reprocessing:  Restarting as soon as possible.



Steve
</pre>
<!-- Author: wolbers -->



<!-- Date: Sun May 25 15:56:36 2003 -->
<pre>Check calibrations for the most recent runs: OK   
Database Usage : OK    
Oracle Usage   : OK    
Checking that kahuna process is running...    
    kahuna process is running OK .    
Data Logger disks: OK </pre>
<!-- Author: Stefano -->

<!-- Date: Mon May 26 08:41:02 2003 -->
<pre>
Roman, Yen-Chu, all,

                Updated status of the farms as of :

                       8:00 A.M.  5/26/03


               Behind by approx. 22 Million on Raw Data

                 6.8 Million events processed in last 24 hours


               Issues:

                 1. 4.10.4e is being tested on the farms.

               Plan for the day:

                 1. Process raw data with 4.8.4g1.
                 2. Run streams to catch up to calibrations
                      with priority on B and G.
                 3. Check 4.10.4e.

-----------------------------------------------------------------------
                   The following table has been modified to
                   reflect the fact that runs 162519 and
                   higher that were processed with 4.8.4g
                   need to be rerun with 4.8.4g1.
                   The histograms, etc. from 4.8.4g are available.

             Stream        Run        Date run was taken   Events behind*

            Stream A      162498        5-May-03         1,352,762
            Stream B      162937       14-May-03         2,384,123
            Stream C      162498        5-May-03         3,136,374
            Stream D      162498        5-May-03         1,964,773
            Stream E      163064       17-May-03           863,717
            Stream G      163064       17-May-03           852,612
            Stream H      162686        9-May-03         9,969,622
            Stream J      163012       16-May-03         1,104,564

             * This is more or less behind raw data on tape.  The farms
                 cannot process events if the calibrations aren't
                 prepared.

                 Calibrations (07)  163390  23-May-03


                 Current data-taking   163474

              Reprocessing:  Restarting as soon as possible.



Steve
</pre>
<!-- Author: wolbers -->



<!-- Date: Mon May 26 16:07:12 2003 -->
<pre>Check calibrations for the most recent runs: OK    
Database Usage : OK     
Oracle Usage   : OK     
Checking that kahuna process is running...     
    kahuna process is running OK .     
Data Logger disks: OK </pre>
<!-- Author: Stefano -->

<!-- Date: Tue May 27 08:26:54 2003 -->
<pre>
Roman, Yen-Chu, all,

                Updated status of the farms as of :

                       8:00 A.M.  5/27/03


               Behind by approx.  16 Million on Raw Data

                 7.2 Million events processed in last 24 hours
                 
                 0.3 Million gjet08 processed through 4.10.4e


               Issues:

                 1. 4.10.4e testing/running.  

               Plan for the day:

                 1. Process raw data with 4.8.4g1.
                 2. Run streams to catch up to calibrations
                      with priority on B and G.
                 3. Run 4.10.4e reprocessing.

-----------------------------------------------------------------------
                   The following table has been modified to
                   reflect the fact that runs 162519 and
                   higher that were processed with 4.8.4g
                   need to be rerun with 4.8.4g1.
                   The histograms, etc. from 4.8.4g are available.

             Stream        Run        Date run was taken   Events
behind*

            Stream A      162498        5-May-03         1,352,762
            Stream B      163064       17-May-03         1,222,890
            Stream C      163385       23-May-03           436,963
            Stream D      162498        5-May-03         1,964,773
            Stream E      163385       23-May-03           703,694
            Stream G      163385       23-May-03           757,069
            Stream H      162836       12-May-03         8,935,597
            Stream J      163385       23-May-03           573,587

             * This is more or less behind raw data on tape.  The farms
                 cannot process events if the calibrations aren't
                 prepared.

                 Calibrations (07)  163462  25-May-03


                 Current data-taking   163527

              Reprocessing:  Restarting.



Steve
</pre>
<!-- Author: wolbers -->


<!-- Date: Wed May 28 06:51:35 2003 -->
<pre>
Roman, Yen-Chu, all,

                Updated status of the farms as of :

                       7:00 A.M.  5/28/03


               Behind by approx. 11 Million on Raw Data

                 7.5 Million total events processed in last 24 hours

                 1.6 Million gjet08 processed through 4.10.4e


               Issues:

                 1. 4.10.4e testing/running.  Output is being
                    checked.

               Plan for the day:

                 1. Process raw data with 4.8.4g1.
                 2. Run streams to catch up to calibrations
                      with priority on B and G.
                 3. Run 4.10.4e reprocessing.

-----------------------------------------------------------------------
                   The following table has been modified to
                   reflect the fact that runs 162519 and
                   higher that were processed with 4.8.4g
                   need to be rerun with 4.8.4g1.
                   The histograms, etc. from 4.8.4g are available.

             Stream        Run        Date run was taken   Events
behind*

            Stream A      162498        5-May-03         1,410,665
            Stream B      163431       24-May-03           524,380
            Stream C      163431       24-May-03           331,471
            Stream D      162498        5-May-03         2,119,913
            Stream E      163431       24-May-03           466,091
            Stream G      163431       24-May-03           550,608
            Stream H      162986       15-May-03         5,218,524
            Stream J      163431       24-May-03           339,273

             * This is more or less behind raw data on tape.  The farms
                 cannot process events if the calibrations aren't
                 prepared.

                 Calibrations (07)  163474  25-May-03


                 Current data-taking   163527

              Reprocessing:  Restarting.



Steve
</pre>
<!-- Author: wolbers -->



<!-- Date: Wed May 28 09:23:18 2003 -->
<pre>Database Usage : OK      
Oracle Usage   : OK      
Checking that kahuna process is running...      
    kahuna process is running OK .      
Data Logger disks: OK </pre>
<!-- Author: Stefano -->


<!-- Date: Wed May 28 09:24:18 2003 -->
<pre>Check calibrations for the most recent runs: OK  
Updated up to run 163464.</pre>
<!-- Author: Stefano -->


<!-- Date: Wed May 28 14:53:10 2003 -->
<pre>Looked at physics plots for:  
run 163117(0x27d2d) 
run 163128(0x27d38) 
run 163136(0x27d40) 
run 163148(0x27d4c) 
run 163348(0x27e14) 
run 163349(0x27e15) 
run 163355(0x27e1b) 
run 163356(0x27e1c) 
run 163357(0x27e1d) 
run 163358(0x27e1e) 
run 163360(0x27e20) 
run 163383(0x27e37) 
run 163384(0x27e38) 
run 163385(0x27e39)  
run 163390(0x27e3e) 
run 163394(0x27e42) 
  
All the distributions OK. 
 
NOTE: 
Stn ntuple maker failed on the following runs: 
 
run 163064(0x27cf8): XRCPControlFileError (known problem)  
 
run 163113(0x27d29): ROOT/TBuffer error: 
%ERLOG-e ROOT-EventRecord:                                                               object of class EventRecord read too few bytes: 320607 instead of 559396                                                                                         GlobalLibraryLogger TBuffer::CheckByteCount() 25-May-2003 12:06:08 
         run = 163113 event = 2898738 
%ERLOG-w ROOT/TBuffer: 
         EventRecord::Streamer() not in sync with data on file, fix Streamer()           GlobalLibraryLogger TBuffer::CheckByteCount() 25-May-2003 12:06:08 
         run = 163113 event = 2898738 
%ERLOG-s Abnormal termination: caused by signal:  8 Framework abortJob() 
         25-May-2003 12:06:08 run = 163113 event = 2898836  
 
         
run 163130(0x27d3a): All events processed correctly then  
ZMthrow at the end of processing (Timer-W-UnnecessaryOp) </pre>
<!-- Author: Stefano -->


<!-- Date: Wed May 28 14:56:17 2003 -->
<pre>Database Usage : OK       
Oracle Usage   : OK       
Checking that kahuna process is running...       
    kahuna process is running OK.</pre>
<!-- Author: Stefano -->

<!-- Date: Thu May 29 07:59:22 2003 -->
<pre>
Roman, Yen-Chu, all,

                Updated status of the farms as of :

                       8:00 A.M.  5/29/03

               Behind by approx. 6.3 Million on Raw Data

                 7.5 Million total events processed in last 24 hours

                    6.6 Million raw events through 4.8.4g1
                    0.9 Million gjet08 processed through 4.10.4e


               Issues:

                 1. 4.10.4e running.  Output going to tape. 
                 2. Caught up to calibrations on most streams.  

               Plan for the day:

                 1. Process raw data with 4.8.4g1.
                 2. Run streams to catch up to calibrations
                      with priority on B and G.
                 3. Run 4.10.4e reprocessing.

-----------------------------------------------------------------------
                   The following table has been modified to
                   reflect the fact that runs 162519 and
                   higher that were processed with 4.8.4g
                   need to be rerun with 4.8.4g1.
                   The histograms, etc. from 4.8.4g are available.

             Stream        Run        Date run was taken   Events
behind*

            Stream A      162498        5-May-03         1,410,665
            Stream B      163431       24-May-03           524,380
            Stream C      163431       24-May-03           331,471
            Stream D      162498        5-May-03         2,119,913
            Stream E      163431       24-May-03           466,091
            Stream G      163431       24-May-03           550,608
            Stream H      163431       24-May-03           683,833
            Stream J      163431       24-May-03           379,415

             * This is more or less behind raw data on tape.  The farms
                 cannot process events if the calibrations aren't
                 prepared.

                 Calibrations (07)  163474  25-May-03


                 Current data-taking   163527

            



Steve
</pre>
<!-- Author: wolbers -->



<!-- Date: Thu May 29 08:41:38 2003 -->
<pre>Calibrations are 4 days behind.  Informed experts.</pre>
<!-- Author: Stefano -->


<!-- Date: Thu May 29 18:15:07 2003 -->
<pre>Check calibrations for the most recent runs: OK   
Updated up to run 163519.</pre>
<!-- Author: Stefano -->

<!-- Date: Fri May 30 12:12:17 2003 -->
<pre>
Roman, Yen-Chu, all,

                Updated status of the farms as of :

                      10:00 A.M.  5/30/03

               Behind by approx. 2 Million on Raw Data

                 4.4 Million total events processed in last 24 hours

                    4.0 Million raw events through 4.8.4g1
                    0.4 Million gjet08 processed through 4.10.4e


               Issues:

                 1. 4.10.4e running.  Output going to tape. 
                 2. Caught up to calibrations on most streams.  

               Plan for the day:

                 1. Process raw data with 4.8.4g1.
                 2. Run streams to catch up to calibrations
                      with priority on B and G.
                 3. Run 4.10.4e reprocessing.


             Stream        Run        Date run was taken  Events behind*

            Stream A      163519       27-May-03            15,488
            Stream B      163519       27-May-03            76,252
            Stream C      163519       27-May-03            48,508
            Stream D      162663        8-May-03         1,883,257
            Stream E      163519       27-May-03            53,113
            Stream G      163519       27-May-03            87,615
            Stream H      163519       27-May-03            94,891
            Stream J      163519       27-May-03            44,651

             * This is more or less behind raw data on tape.  The farms
                 cannot process events if the calibrations aren't
                 prepared.

                 Calibrations (07)  163526  27-May-03


                 Current data-taking   163527

        
Steve Wolbers
</pre>
<!-- Author: wolbers -->


<!-- Date: Sat May 31 07:03:40 2003 -->
<pre>
Roman, Yen-Chu, all,

                 Updated status of the farms as of :

                       7:00 A.M.  5/31/03

                Behind by approx. 1.4 Million on Raw Data

                  3.8 Million total events processed in last 24 hours

                     1.7 Million raw events through 4.8.4g1
                     2.1 Million gjet08 processed through 4.10.4e


                Issues:

                  1. 4.10.4e running.  Output going to tape.
                  2. Caught up to calibrations on most streams.

                Plan for the day:

                  1. Process raw data with 4.8.4g1.
                  2. Run streams to catch up to calibrations
                       with priority on B and G.
                  3. Run 4.10.4e reprocessing.


              Stream        Run        Date run was taken  Events behind*

             Stream A      163519       27-May-03            15,488
             Stream B      163519       27-May-03            76,252
             Stream C      163519       27-May-03            48,508
             Stream D      162937       14-May-03         1,086,371
             Stream E      163519       27-May-03            53,113
             Stream G      163519       27-May-03            87,615
             Stream H      163519       27-May-03            94,891
             Stream J      163519       27-May-03            44,651

              * This is more or less behind raw data on tape.  The farms
                  cannot process events if the calibrations aren't
                  prepared.

                  Calibrations (07)  163526  27-May-03

                  Current data-taking   163527

Steve Wolbers
</pre>
<!-- Author: wolbers -->


<!-- Date: Sun Jun 1 11:12:06 2003 -->
<pre>
Roman, Yen-Chu, all,

                  Updated status of the farms as of :

                        11:00 A.M.  June 01 2003

                 Behind by approx. 2.8 Million on Raw Data

                   2.0 Million total events processed in last 24 hours

                      0.0 Million raw events through 4.8.4g1
                      2.0 Million gjet08 processed through 4.10.4e


                 Issues:

                   1. 4.10.4e running.  Output going to tape.
                   2. Working on streams A and D output to tape.
                   3. Waiting for the last 2 runs' calibrations.

                 Plan for the day:

                   1. Process raw data with 4.8.4g1.
                   2. Run streams to catch up to calibrations
                        with priority on B and G.
                   3. Run 4.10.4e reprocessing.


               Stream        Run        Date run was taken  Events behind*

              Stream A      162498        5-May-03         1,410,665
              Stream B      163519       27-May-03            76,252
              Stream C      163519       27-May-03            48,508
              Stream D      162937       14-May-03         1,086,371
              Stream E      163519       27-May-03            53,113
              Stream G      163519       27-May-03            87,615
              Stream H      163519       27-May-03            94,891
              Stream J      163519       27-May-03            44,651

               * This is more or less behind raw data on tape.  The farms
                   cannot process events if the calibrations aren't
                   prepared.

                   Calibrations (07)  163526  27-May-03

                   Current data-taking   163527

Steve Wolbers
</pre>
<!-- Author: wolbers -->


<!-- Date: Sun Jun 1 11:12:06 2003 -->
<pre>
Roman, Yen-Chu, all,

                  Updated status of the farms as of :

                        11:00 A.M.  June 01 2003

                 Behind by approx. 2.8 Million on Raw Data

                   2.0 Million total events processed in last 24 hours

                      0.0 Million raw events through 4.8.4g1
                      2.0 Million gjet08 processed through 4.10.4e


                 Issues:

                   1. 4.10.4e running.  Output going to tape.
                   2. Working on streams A and D output to tape.
                   3. Waiting for the last 2 runs' calibrations.

                 Plan for the day:

                   1. Process raw data with 4.8.4g1.
                   2. Run streams to catch up to calibrations
                        with priority on B and G.
                   3. Run 4.10.4e reprocessing.


               Stream        Run        Date run was taken  Events behind*

              Stream A      162498        5-May-03         1,410,665
              Stream B      163519       27-May-03            76,252
              Stream C      163519       27-May-03            48,508
              Stream D      162937       14-May-03         1,086,371
              Stream E      163519       27-May-03            53,113
              Stream G      163519       27-May-03            87,615
              Stream H      163519       27-May-03            94,891
              Stream J      163519       27-May-03            44,651

               * This is more or less behind raw data on tape.  The farms
                   cannot process events if the calibrations aren't
                   prepared.

                   Calibrations (07)  163526  27-May-03

                   Current data-taking   163527

Steve Wolbers
</pre>
<!-- Author: wolbers -->


<!-- Date: Mon Jun 2 08:23:16 2003 -->
<pre>
Roman, Yen-Chu, all,

                  Updated status of the farms as of :

                        8:00 A.M.  June 02 2003

                 Behind by approx. 2.8 Million on Raw Data

                  4.4 Million total events processed in last 24 hours

                      0.04 Million raw events through 4.8.4g1
                      4.4 Million gjet08 processed through 4.10.4e


                 Issues:

                   1. 4.10.4e running.  Output going to tape.
                   2. Working on streams A and D output to tape.
                   3. Waiting for the last 2 runs' calibrations.

                 Plan for the day:

                   1. Process raw data with 4.8.4g1.
                   2. Run streams to catch up to calibrations
                        with priority on B and G.
                   3. Run 4.10.4e reprocessing.


               Stream        Run        Date run was taken  Events
behind*

              Stream A      162520        6-May-03         1,371,009
              Stream B      163519       27-May-03            76,252
              Stream C      163519       27-May-03            48,508
              Stream D      162937       14-May-03         1,086,371
              Stream E      163519       27-May-03            53,113
              Stream G      163519       27-May-03            87,615
              Stream H      163519       27-May-03            94,891
              Stream J      163519       27-May-03            44,651

               * This is more or less behind raw data on tape.  The
farms
                   cannot process events if the calibrations aren't
                   prepared (or if the events are not on tape).

                   Calibrations (07)  163526  27-May-03

                   Current data-taking   163958

Steve Wolbers
</pre>
<!-- Author: wolbers -->


<!-- Date: Tue Jun 3 08:33:13 2003 -->
<pre>
Roman, Yen-Chu, all,

                  Updated status of the farms as of :

                        8:00 A.M.  June 03 2003

                 Behind by approx. 0.35 Million on Raw Data

                  5.3 Million total events processed in last 24 hours

                      2.8 Million raw events through 4.8.4g1
                      1.5 Million gjet08 processed through 4.10.4e


                 Issues:

                   1. 4.10.4e running.  Output going to tape.

                 Plan for the day:

                   1. Process raw data with 4.8.4g1.
                   2. Run 4.10.4e reprocessing.


               Stream        Run        Date run was taken  Events
behind*

              Stream A      163527       27-May-03         
              Stream B      163527       27-May-03            
              Stream C      163527       27-May-03           
              Stream D      163527       27-May-03         
              Stream E      163527       27-May-03            
              Stream G      163527       27-May-03           186,087
              Stream H      163527       27-May-03           163,092
              Stream J      163527       27-May-03           

               * This is more or less behind raw data on tape.  The
farms
                   cannot process events if the calibrations aren't
                   prepared (or if the events are not on tape).

                   Calibrations (07)  163956  02-June-03

                   Current data-taking   163958

Steve Wolbers
</pre>
<!-- Author: wolbers -->


<!-- Date: Wed Jun 4 07:00:50 2003 -->
<pre>
Roman, Yen-Chu, all,

                   Updated status of the farms as of :

                         7:00 A.M.  June 04 2003

                  Behind by approx. 1.7 Million on Raw Data

                   0 Million total events processed in last 24 hours

                       0 Million raw events through 4.8.4g1
                       0 Million gjet08 processed through 4.10.4e


                  Issues:

                    1. 4.10.4e running.  Output going to tape.
                    2. Database problems.

                  Plan for the day:

                    1. Process raw data with 4.8.4g1.
                    2. Run 4.10.4e reprocessing.


                Stream        Run        Date run was taken  Events
behind*

               Stream A      163527       27-May-03
               Stream B      163527       27-May-03           349,145
               Stream C      163527       27-May-03           104,054
               Stream D      163527       27-May-03            83,971
               Stream E      163527       27-May-03           275,024
               Stream G      163527       27-May-03           374,893
               Stream H      163527       27-May-03           408,754
               Stream J      163527       27-May-03            91,958

                * This is more or less behind raw data on tape.  The
farms
                    cannot process events if the calibrations aren't
                    prepared (or if the events are not on tape).

                    Calibrations (07)  163956  02-June-03

                    Current data-taking   164110

Steve Wolbers
</pre>
<!-- Author: wolbers -->


<!-- Date: Thu Jun 5 08:00:02 2003 -->
<pre>
Roman, Yen-Chu, all,

                   Updated status of the farms as of :

                         8:00 A.M.  June 05 2003

                  Behind by approx. 2.7 Million on Raw Data

                   0 Million total events processed in last 24 hours

                       0 Million raw events through 4.8.4g1
                       0 Million gjet08 processed through 4.10.4e


                  Issues:

                    1. Waiting for database access.  

                  Plan for the day:

                    1. Process raw data with 4.8.4g1.
                    2. Run 4.10.4e reprocessing.


                Stream        Run        Date run was taken  Events
behind*

               Stream A      163527       27-May-03            58,647
               Stream B      163527       27-May-03           349,145
               Stream C      163527       27-May-03           305,191
               Stream D      163527       27-May-03           242,841
               Stream E      163527       27-May-03           275,024
               Stream G      163527       27-May-03           570,043
               Stream H      163527       27-May-03           586,811
               Stream J      163527       27-May-03           266,218

                * This is more or less behind raw data on tape.  The
farms
                    cannot process events if the calibrations aren't
                    prepared (or if the events are not on tape).

                    Calibrations (07)  163956  02-June-03

                    Current data-taking   164128

Steve Wolbers
</pre>
<!-- Author: wolbers -->


<!-- Date: Fri Jun 6 07:49:51 2003 -->
<pre>
Roman, Yen-Chu, all,

                   Updated status of the farms as of :

                         8:00 A.M.  June 06 2003

                  Behind by approx. 4.2 Million on Raw Data

                   0 Million total events processed in last 24 hours

                       0 Million raw events through 4.8.4g1
                       0 Million gjet08 processed through 4.10.4e


                  Issues:

                    1. Database access and related problems.  

                  Plan for the day:

                    1. Sort out DB and DB-related issues.
                    2. Process raw data with 4.8.4g1.
                    3. Run 4.10.4e reprocessing.


                Stream        Run        Date run was taken  Events
behind*

               Stream A      163527       27-May-03            58,647
               Stream B      163527       27-May-03           666,299
               Stream C      163527       27-May-03           441,293
               Stream D      163527       27-May-03           348,028
               Stream E      163527       27-May-03           580,195
               Stream G      163527       27-May-03           763,270
               Stream H      163527       27-May-03           900,523
               Stream J      163527       27-May-03           400,098

                * This is more or less behind raw data on tape.  The
farms
                    cannot process events if the calibrations aren't
                    prepared (or if the events are not on tape).

                    Calibrations (07)  163956  02-June-03

                    Current data-taking   164261

Steve Wolbers
</pre>
<!-- Author: wolbers -->



<!-- Date: Fri Jun 6 15:59:38 2003 -->
<pre>1) Compared the results of tracking (histograms) as run in the production executable, for the three cases of input data : ie. a) Input has compressed banks (PadSqz::COTQ, Padsqz::SVXQ) and original raw banks present, b) Input has only compressed banks but no raw banks (SIXD, ISLD, COTD) c) Input has only raw banks. 
 
KS test shows no difference for existing files. 
 
2) waiting for 4.8.4i and David's new input files to test.</pre>
<!-- Author: Elena Gerchtein -->

<!-- Date: Sat Jun 7 07:51:45 2003 -->
<pre>
Roman, Yen-Chu, all,

                    Updated status of the farms as of :

                          8:00 A.M.  June 07 2003

                   Behind by approx. 4.8 Million on Raw Data

                    5.0 Million total events processed in last 24 hours

                        0 Million raw events through 4.8.4g1
                      5.0 Million bmcl08 processed through 4.10.4e


                   Issues:

                     1. Database issues resolved.
                     2. Still need 07 calibrations to process
                        raw data further.  Some confusion here.

                   Plan for the day:

                     1. Process raw data with 4.8.4g1.
                     2. Run 4.10.4e reprocessing.


                 Stream        Run        Date run was taken   Behind*

                Stream A      163527       27-May-03            99,414
                Stream B      163527       27-May-03           766,261
                Stream C      163527       27-May-03           441,293
                Stream D      163527       27-May-03           440,821
                Stream E      163527       27-May-03           670,352
                Stream G      163527       27-May-03           856,522
                Stream H      163527       27-May-03         1,026,130
                Stream J      163527       27-May-03           533,629

     * This the number of events on tape that have not been processed.

                  Calibrations (07)  163956  02-June-03

                     Current data-taking   164274

      Reprocessed with 4.10.4e:

                   gjet08    12.9 million
                   bmcl08     5.0 million

Steve Wolbers
</pre>
<!-- Author: wolbers -->



<!-- Date: Sat Jun 7 14:27:26 2003 -->
<pre>1) run check_run.sh on june runs 163956-164302. 
 - no BADPASSCALIB errors. 
 - severe errors  
   * check_run.163959.log  
  %ERLOG-s ROOT/TUnixSystem: send (Broken pipe)  GlobalLibraryLogger  
   * check_run.164257.log 
  %ERLOG-s ROOT/TUnixSystem: send (Broken pipe)  GlobalLibraryLogger 
         TUnixSystem::UnixSend()  7-Jun-2003 14:08:27 run = 0 event = 0  
   * check_run.164200.log  
   %ERLOG-s ROOT/TUnixSystem: send (Broken pipe)  GlobalLibraryLogger 
         TUnixSystem::UnixSend()  7-Jun-2003 14:09:44 run = 0 event = 0  
   * check_run.164109.log      
%ERLOG-s ROOT/TUnixSystem: send (Broken pipe)  GlobalLibraryLogger 
         TUnixSystem::UnixSend()  7-Jun-2003 14:12:37 run = 0 event = 0 </pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Sat Jun 7 15:51:22 2003 -->
<pre>4.8.4i 10K test results. 
 
1) no core dumps 
2) no severe errors 
3) errors: 
   * MuonStubModule MuonStubFinder () - about 40 times 
     Err:1: Too many hits in region 
   * CT_TrackingModule - 3 times ( it looks more like warning ) 
 Err:1: 
         /home/cdfsoft/dist/packages/TrackingMods/V00-02-78/src/CT_TrackingModule.cc 
        : 847  Skipping event with high multiplicity and writing ErrorSummary 
         object. The sum of the maximum number of contiguous wires hit 
         in the 3 outer superlayers, minus the midpoint is  1345 
        .  A typical hardware failure will have a value of  2655  
4) Output files are readable.</pre>
<!-- Author: Elena Gerchtein -->

<!-- Date: Sun Jun 8 08:23:14 2003 -->
<pre>
Roman, Yen-Chu, all,

                     Updated status of the farms as of :

                           8:00 A.M.  June 08 2003

                    Behind by approx. 5.9 Million on Raw Data

                     3.6 Million total events processed in last 24 hours

                         0 Million raw events through 4.8.4g1
                       3.6 Million bmcl08 processed through 4.10.4e


                    Issues:

                      1. Database issues resolved.
                      2. Still need 07 calibrations to process
                         raw data further.

                    Plan for the day:

                      1. Process raw data with 4.8.4g1.
                      2. Run 4.10.4e reprocessing.


                  Stream        Run        Date run was taken   Behind*

                 Stream A      163527       27-May-03           152,740
                 Stream B      163527       27-May-03           962,071
                 Stream C      163527       27-May-03           541,147
                 Stream D      163527       27-May-03           440,821
                 Stream E      163527       27-May-03           670,352
                 Stream G      163527       27-May-03         1,026,868
                 Stream H      163527       27-May-03         1,410,311
                 Stream J      163527       27-May-03           716,090

      * This the number of events on tape that have not been processed.

                   Calibrations (07)  163956  02-June-03

                      Current data-taking   164313

       Reprocessed with 4.10.4e:

                    gjet08    12.9 million
                    bmcl08     8.6 million

Steve Wolbers
</pre>
<!-- Author: wolbers -->



<!-- Date: Sun Jun 8 15:14:26 2003 -->
<pre>1) AC++DUMP concatenation_test.tcl does not work in 4.8.4i - 
   Liz Sexton is informed. 
2) sent request to restart kahuna. 
3) DB usage is OK.</pre>
<!-- Author: Elena Gerchtein -->

<!-- Date: Sun Jun 8 16:28:21 2003 -->
<pre>
Target Name : cdfofprd.fnal.gov 
Target Type : Database 
Node Name : fcdfora1.fnal.gov 
Status : COMPLETED 



From wolbers@fnal.gov  Sun Jun  8 16:27:35 2003
Received: from woozle.fnal.gov (woozle.fnal.gov [131.225.9.22])
	by b0dau30.fnal.gov (SGI-8.9.3/8.9.3) with SMTP id QAA95023
	for <cdf_elog@b0dau30.fnal.gov>; Sun, 8 Jun 2003 16:27:35 -0500 (CDT)
Received: from fnal.gov (12-251-115-5.client.attbi.com [12.251.115.5])
 by woozle.fnal.gov
 (iPlanet Messaging Server 5.2 HotFix 1.10 (built Jan 23 2003))
 with ESMTPSA id <0HG500BGEXIZII@woozle.fnal.gov> for
 cdf_elog@b0dau30.fnal.gov; Sun, 08 Jun 2003 07:35:00 -0500 (CDT)
Date: Sun, 08 Jun 2003 07:32:56 -0500
From: Stephen Wolbers <wolbers@fnal.gov>
Subject: Farms status June 08 2003
To: chenyc@fnal.gov, lysak@fnal.gov, rharris@fnal.gov, murat@fnal.gov,
        fkw@fnal.gov, roser@fnal.gov, mlindgre@fnal.gov,
        guima@huhepl.harvard.edu, cdf_offline_operations@fnal.gov,
        tml@uiuc.edu, cdf_elog@b0dau30.fnal.gov, lmiller@fnal.gov,
        boj@fnal.gov, krg@fnal.gov
Message-id: <3EE32CF8.4080609@fnal.gov>
MIME-version: 1.0
Content-type: text/plain; format=flowed; charset=us-ascii
Content-transfer-encoding: 7bit
X-Accept-Language: en-us
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.2)
 Gecko/20010726 Netscape6/6.1

<pre>
Roman, Yen-Chu, all,

                     Updated status of the farms as of :

                           8:00 A.M.  June 08 2003

                    Behind by approx. 5.9 Million on Raw Data

                     3.6 Million total events processed in last 24 hours

                         0 Million raw events through 4.8.4g1
                       3.6 Million bmcl08 processed through 4.10.4e


                    Issues:

                      1. Database issues resolved.
                      2. Still need 07 calibrations to process
                         raw data further.

                    Plan for the day:

                      1. Process raw data with 4.8.4g1.
                      2. Run 4.10.4e reprocessing.


                  Stream        Run        Date run was taken   Behind*

                 Stream A      163527       27-May-03           152,740
                 Stream B      163527       27-May-03           962,071
                 Stream C      163527       27-May-03           541,147
                 Stream D      163527       27-May-03           440,821
                 Stream E      163527       27-May-03           670,352
                 Stream G      163527       27-May-03         1,026,868
                 Stream H      163527       27-May-03         1,410,311
                 Stream J      163527       27-May-03           716,090

      * This the number of events on tape that have not been processed.

                   Calibrations (07)  163956  02-June-03

                      Current data-taking   164313

       Reprocessed with 4.10.4e:

                    gjet08    12.9 million
                    bmcl08     8.6 million

Steve Wolbers
</pre>
<!-- Author: wolbers -->


<!-- Date: Mon Jun 9 08:18:49 2003 -->
<pre>
Roman, Yen-Chu, all,

                     Updated status of the farms as of :

                           8:00 A.M.  June 09 2003

                    Behind by approx. 8.1 Million on Raw Data

                  0.6 Million total events processed in last 24 hours

                         0 Million raw events through 4.8.4g1
                       0.6 Million bmcl08 processed through 4.10.4e


                    Issues:

                      1. Database issues resolved.
                      2. Still need 07 calibrations to process
                         raw data further?  Author is confused. 
                         Clarification hopefully to come later today.

                    Plan for the day:

                      1. Process raw data with 4.8.4g1.
                      2. Run 4.10.4e reprocessing.


                  Stream        Run        Date run was taken   Behind*

                 Stream A      163527       27-May-03           196,525
                 Stream B      163527       27-May-03         1,322,259
                 Stream C      163527       27-May-03           691,882
                 Stream D      163527       27-May-03           578,113
                 Stream E      163527       27-May-03           976,799
                 Stream G      163527       27-May-03         1,458,630
                 Stream H      163527       27-May-03         1,979,505
                 Stream J      163527       27-May-03           870,624

      * This the number of events on tape that have not been processed.

                   Calibrations (07)  163956 (164052?) 02-June-03

                      Current data-taking   164354 

       Reprocessed with 4.10.4e:

                    gjet08    12.9 million
                    bmcl08     9.3 million

Steve Wolbers
</pre>
<!-- Author: wolbers -->


<!-- Date: Tue Jun 10 07:37:10 2003 -->
<pre>
Roman, Yen-Chu, all,

             Updated status of the farms as of :

                   7:30 A.M.  June 10 2003

             Behind by approx. 9.8 Million on Raw Data

           3.0 Million total events processed in last 24 hours

                  0 Million raw events through 4.8.4g1
                3.0 Million blpc08 processed through 4.10.4e


             Issues:

               1. Problems with reading calibrations.

             Plan for the day:

               1. Process raw data with 4.8.4g1.
               2. Run 4.10.4e reprocessing.


           Stream        Run        Date run was taken   Behind*

          Stream A      163527       27-May-03           196,525
          Stream B      163527       27-May-03         1,512,939
          Stream C      163527       27-May-03           892,675
          Stream D      163527       27-May-03           747,400
          Stream E      163527       27-May-03         1,147,306
          Stream G      163527       27-May-03         1,690,308
          Stream H      163527       27-May-03         2,556,128
          Stream J      163527       27-May-03         1,053,815

   * This the number of events on tape that have not been processed.

            Calibrations (07)  163956 (164052?) 02-June-03

               Current data-taking   164393

   Reprocessed with 4.10.4e:

             gjet08    12.9 million
             bmcl08     9.3 million
             blpc08     3.0 million

Steve Wolbers
</pre>
<!-- Author: wolbers -->


<!-- Date: Tue Jun 10 12:59:25 2003 -->
<pre>
Roman, Yen-Chu, all,

              Updated status of the farms as of :

                    7:30 A.M.  June 10 2003

              Behind by approx. 9.8 Million on Raw Data

            3.0 Million total events processed in last 24 hours

                   0 Million raw events through 4.8.4g1
                 3.0 Million blpc08 processed through 4.10.4e


              Issues:

                1. Problems with reading calibrations
           -->     (silicon calibrations missing)

              Plan for the day:

                1. Process raw data with 4.8.4g1.
                2. Run 4.10.4e reprocessing.


            Stream        Run        Date run was taken   Behind*

           Stream A      163527       27-May-03           196,525
           Stream B      163527       27-May-03         1,512,939
           Stream C      163527       27-May-03           892,675
           Stream D      163527       27-May-03           747,400
           Stream E      163527       27-May-03         1,147,306
           Stream G      163527       27-May-03         1,690,308
           Stream H      163527       27-May-03         2,556,128
           Stream J      163527       27-May-03         1,053,815

    * This the number of events on tape that have not been processed.

             Calibrations (07)  163956 (164052?) 02-June-03

                Current data-taking   164393

    Reprocessed with 4.10.4e:

              gjet08    12.9 million
              bmcl08     9.3 million
              blpc08     3.0 million

Steve Wolbers

-->  Beginning June 16, 2003 Steve Wolbers will be on vacation (for 3
-->  weeks).  Yen-Chu Chen will take over the daily reports.

</pre>
<!-- Author: wolbers -->



<!-- Date: Tue Jun 10 15:46:17 2003 -->
<pre>db - ok 
kahuna was restarted 2 times.</pre>
<!-- Author: Elena Gerchtein -->

<!-- Date: Wed Jun 11 08:08:40 2003 -->
<pre>
Roman, Yen-Chu, all,

               Updated status of the farms as of :

                     8:00 A.M.  June 11 2003

               Behind by approx. 11.7 Million on Raw Data

           6.7 Million total events processed in last 24 hours

                  0 Million raw events through 4.8.4g1
                6.7 Million blpc08 processed through 4.10.4e


          Issues:

             1. Waiting for calibrations.

               Plan for the day:

             1. Process raw data with 4.8.4g1.
             2. Run 4.10.4e reprocessing.


           Stream        Run        Date run was taken   Behind*

          Stream A      163527       27-May-03           263,352
          Stream B      163527       27-May-03         1,708,450
          Stream C      163527       27-May-03         1,065,337
          Stream D      163527       27-May-03           887,167
          Stream E      163527       27-May-03         1,322,551
          Stream G      163527       27-May-03         1,887,977
          Stream H      163527       27-May-03         3,325,237
          Stream J      163527       27-May-03         1,219,036

     * This the number of events on tape that have not been processed.

              Calibrations (07)  163956 02-June-03

                 Current data-taking   164451

     Reprocessed with 4.10.4e:

               gjet08    12.9 million
               bmcl08     9.3 million
               blpc08     9.9 million

Steve Wolbers

-->  Beginning June 16, 2003 Steve Wolbers will be on vacation (for 3
-->  weeks).  Yen-Chu Chen will take over the daily reports.

</pre>
<!-- Author: wolbers -->


<!-- Date: Thu Jun 12 07:05:41 2003 -->
<pre>
Roman, Yen-Chu, all,

                Updated status of the farms as of :

                      7:00 A.M.  June 12 2003

                Behind by approx. 13.1 Million on Raw Data

            5.1 Million total events processed in last 24 hours

                 0.3 Million raw events through 4.8.4g1
                 4.8 Million blpc08 processed through 4.10.4e


           Issues:

              1. Calibrations available for June data.
              2. Emergency CDF switch work.

                Plan for the day:

              1. Process raw data with 4.8.4g1.
              2. Run 4.10.4e reprocessing.


            Stream        Run        Date run was taken   Behind*

           Stream A      163527       27-May-03           337,082
           Stream B      163956        2-Jun-03         1,870,535
           Stream C      163527       27-May-03         1,266,613
           Stream D      163527       27-May-03         1,050,092
           Stream E      163527       27-May-03         1,500,027
           Stream G      164079        3-Jun-03         1,979,928
           Stream H      163527       27-May-03         3,714,962
           Stream J      163527       27-May-03         1,404,567

      * This the number of events on tape that have not been processed.

               Calibrations (07)  164386 09-June-03

                  Current data-taking   164517

      Reprocessed with 4.10.4e:

                gjet08    12.9 million
                bmcl08     9.3 million
                blpc08    14.4 million

Steve Wolbers

-->  Beginning June 16, 2003 Steve Wolbers will be on vacation (for 3
-->  weeks).  Yen-Chu Chen will take over the daily reports.

</pre>
<!-- Author: wolbers -->



<!-- Date: Thu Jun 12 15:26:12 2003 -->
<pre>4.8.4i (with additional patch) 10K test. 
 
1) I've cheched the Production log files for 4 files 
from each collection: 
  compress_nodrop 
  compress_rawdrop 
  nocompress 
( the rest is still running ) 
 
There is an error message 
  COTD/Q not found by CT_HitSet for compress_rawdrop files. 
 
2) all files are readable with AC++Dump 
 
3) the output of compress_rawdrop is different from compress_nodrop and 
   nocompress, something is missing in compress_nodrop files, 
   even the size of the concatenated hist file is different, some 
   histograms are empty. I used compare_prod_hists as on the 
   web page. 
 
   the concatenated hist files are in 
   /cdf/scratch/cdfopr/testRel/4.8.4i/results</pre>
<!-- Author: Elena Gerchtein -->

<!-- Date: Fri Jun 13 07:19:56 2003 -->
<pre>
Roman, Yen-Chu, all,

                 Updated status of the farms as of :

                       7:00 A.M.  June 13 2003

                 Behind by approx. 8.3 Million on Raw Data

              5.8 Million total events processed in last 24 hours

                   5.7 Million raw events through 4.8.4g1
                   0.1 Million blpc08 processed through 4.10.4e


            Issues:

               1. Finished all currently requested 4.10.4e
                  reprocessing requests.

            Plan for the day:

               1. Process raw data with 4.8.4g1.



             Stream        Run        Date run was taken   Behind*

            Stream A      164386        9-Jun-03           142,341
            Stream B      164386        9-Jun-03           524,376
            Stream C      164303        7-Jun-03           871,441
            Stream D      164200        4-Jun-03           958,138
            Stream E      164274        6-Jun-03         1,011,826
            Stream G      164354        9-Jun-03           987,539
            Stream H      164261        6-Jun-03         3,172,067
            Stream J      164354        9-Jun-03           637,572

       * This the number of events on tape that have not been processed.

                Calibrations (07)  164451 11-June-03

                   Current data-taking   164625

       Reprocessed with 4.10.4e:

                 gjet08    12.9 million
                 bmcl08     9.3 million
                 blpc08    14.6 million

Steve Wolbers

-->  Beginning June 16, 2003 Steve Wolbers will be on vacation (for 3
-->  weeks).  Yen-Chu Chen will take over the daily reports.

</pre>
<!-- Author: wolbers -->



<!-- Date: Fri Jun 13 10:46:52 2003 -->
<pre>Trained new offline shifter.</pre>
<!-- Author: Elena Gerchtein -->


<!-- Date: Fri Jun 13 16:15:56 2003 -->
<pre>- Ran the "10K production executable test run" up to step 5 (including) - not finished 
- Changed /cdf/scratch/cdfopr/Sntuple/val/compare_stau_hist.cc  (so that only folders with bad histograms are displayed)</pre>
<!-- Author: Peter Wagner -->

<!-- Date: Sat Jun 14 07:27:15 2003 -->
<pre>
Roman, Yen-Chu, all,

              Updated status of the farms as of :

                    7:30 A.M.  June 14 2003

              Behind by approx. 2.8 Million on Raw Data

           6.3 Million total events processed in last 24 hours

                6.3 Million raw events through 4.8.4g1
                0.0 Million blpc08 processed through 4.10.4e


         Issues:

            1. Finished all currently requested 4.10.4e
               reprocessing requests.

         Plan for the day:

            1. Process raw data with 4.8.4g1.


           Stream        Run        Date run was taken   Behind*

          Stream A      164451       10-Jun-03            80,318
          Stream B      164451       10-Jun-03           384,369
          Stream C      164451       10-Jun-03           256,620
          Stream D      164451       10-Jun-03           205,250
          Stream E      164451       10-Jun-03           278,771
          Stream G      164451       10-Jun-03           425,336
          Stream H      164451       10-Jun-03           822,367
          Stream J      164451       10-Jun-03           315,519

     * This the number of events on tape that have not been processed.

               Calibrations (07)  164509 11-June-03

               Current data-taking   164724

      Reprocessed with 4.10.4e:

               gjet08    12.9 million
               bmcl08     9.3 million
               blpc08    14.6 million

Steve Wolbers

-->  Beginning June 16, 2003 Steve Wolbers will be on vacation (for 3
-->  weeks).  Yen-Chu Chen will take over the daily reports.

</pre>
<!-- Author: wolbers -->



<!-- Date: Sat Jun 14 16:51:11 2003 -->
<pre>- ProductionExe tarfile created on lnx3 for patch 4.8.4 i 
- discovered error: files in LIST_OF_PROCESSED_RUNS were not processed correctly by ~/scripts/make_stntuple.csh due to a "ls" alias in .cshrc  =&gt; some files lost</pre>
<!-- Author: Peter -->

<!-- Date: Sun Jun 15 08:12:25 2003 -->
<pre>
Roman, Yen-Chu, all,

               Updated status of the farms as of :

                     8:00 A.M.  June 15 2003

               Behind by approx. 4.0 Million on Raw Data

            0.0 Million total events processed in last 24 hours

                 0.0 Million raw events through 4.8.4g1
                 0.0 Million blpc08 processed through 4.10.4e


          Issues:

             1. Finished all currently requested 4.10.4e
                reprocessing requests.
             2. Processing caught up to calibrations.

          Plan for the day:

             1. Process raw data with 4.8.4g1.


            Stream        Run        Date run was taken   Behind*

           Stream A      164451       10-Jun-03           128,670
           Stream B      164451       10-Jun-03           582,777
           Stream C      164451       10-Jun-03           256,620
           Stream D      164451       10-Jun-03           205,250
           Stream E      164451       10-Jun-03           435,789
           Stream G      164451       10-Jun-03           637,192
           Stream H      164451       10-Jun-03         1,348,503
           Stream J      164451       10-Jun-03           465,739

      * This the number of events on tape that have not been processed.

                Calibrations (07)  164509 11-June-03

                Current data-taking   164737

       Reprocessed with 4.10.4e:

                gjet08    12.9 million
                bmcl08     9.3 million
                blpc08    14.6 million

Steve Wolbers
</pre>
<!-- Author: wolbers -->



<!-- Date: Sun Jun 15 16:58:47 2003 -->
<pre>- moved /cdf/scratch/cdfopr/mon -contents to /cdf/scratch/cdfopr/val 
- inserted new histogram in "physmon.root"</pre>
<!-- Author: Peter -->


<!-- Date: Tue Jun 17 17:29:47 2003 -->
<pre>- setup of a test release of cdfsoft from v. 4.11.1 
- experienced problems during installation of Stntuple (cdfsoft v4.11.1) 
- trying to fix problems in LIST_OF_PROCESSED_RUNS; got again a cron job error message by mail</pre>
<!-- Author: Peter -->


<!-- Date: Tue Jun 17 17:35:42 2003 -->
<pre>Concerning yesterday: 
- moved all ntuple folders to fcdfdata033 
- tried to fix bug in the creation of ntuple files from raw data (see LIST_OF_PROCESSED_RUNS)</pre>
<!-- Author: Peter -->


<!-- Date: Wed Jun 18 16:05:53 2003 -->
<pre>- modified cronjob "run_stnmaker.sh" to not to use rootd to get raw-data files, but to use "rcp" instead</pre>
<!-- Author: Peter -->

<!-- Date: Thu Jun 19 12:05:30 2003 -->
<pre>

               Updated status of the farms as of :

                     9:30 A.M.  June 19 2003

               Behind by approx. 13.0 Million on Raw Data

            0.5 Million total events processed in last 24 hours

                 0.5 Million raw events through 4.8.4g1


          Issues:

             1. Processing caught up to calibrations.

          Plan for the day:

             1. Process raw data with 4.8.4g1.


            Stream        Run        Date run was taken   Behind*

           Stream A      164451       10-Jun-03           353,557
           Stream B      164729       14-Jun-03         1,215,907
           Stream C      164451       10-Jun-03         1,149,184
           Stream D      164451       10-Jun-03           857,408
           Stream E      164451       10-Jun-03         1,189,966
           Stream G      164729       14-Jun-03         1,726,535
           Stream H      164451       10-Jun-03         4,219,663
           Stream J      164451       10-Jun-03         1,293,734

      * This the number of events on tape that have not been processed.

                Calibrations (07)  164819 16-June-03

                Current data-taking   164916

       Reprocessed with 4.10.4e:

                gjet08    12.9 million
                bmcl08     9.3 million
                blpc08    14.6 million

Yen-Chu Chen
</pre>
<!-- Author: Yen-Chu -->



<!-- Date: Thu Jun 19 16:14:58 2003 -->
<pre>- fixed run_stnmaker.sh and make_stntuple.csh</pre>
<!-- Author: Peter -->


<!-- Date: Fri Jun 20 11:57:02 2003 -->
<pre>- trained new shifter</pre>
<!-- Author: Peter -->

<!-- Date: Sat Jun 21 09:03:37 2003 -->
<pre>

               Updated status of the farms as of :

                     8:30 A.M.  June 21 2003

               Behind by approx. 10.0 Million on Raw Data

            0.005 Million total events processed in last 24 hours

                 0.005 Million raw events through 4.8.4g1

            (Note that in the previous day, about 5 Million events
             were processed. For which I didn't get chance to give
             the status report.)


          Issues:

             1. Processing caught up to calibrations.

          Plan for the day:

             1. Process raw data with 4.8.4g1.


            Stream        Run        Date run was taken   Behind*

           Stream A      164799       16-Jun-03           199,984
           Stream B      164729       14-Jun-03         1,624,424
           Stream C      164799       16-Jun-03           606,497
           Stream D      164799       16-Jun-03           453,141
           Stream E      164799       16-Jun-03           853,148
           Stream G      164799       16-Jun-03         1,491,891
           Stream H      164799       16-Jun-03         2,712,251
           Stream J      164799       16-Jun-03           936,215

      * This the number of events on tape that have not been processed.

                Calibrations (07)  164959 16-June-03

                Current data-taking   165063

       Reprocessed with 4.10.4e:

                gjet08    12.9 million
                bmcl08     9.3 million
                blpc08    14.6 million

Yen-Chu Chen
</pre>
<!-- Author: Yen-Chu -->



<!-- Date: Tue Jun 24 13:27:21 2003 -->
<pre>-------------------------- building 4.8.4i tarball for the Farm 
- checked out 4.8.4i.patches -r 1.2 
- Production/setup_input.tcl -r 1.26 is the same as -r v4_8_4, 
  use the one from v4_8_4 branch 
- tarball built, farmers can find it in FCDFLNX3: 
  /cdf/scratch/cdfopr/tarballs/ProductionExe_4.8.4i.tar.gz 
</pre>
<!-- Author: murat -->


<!-- Date: Wed Jul 2 09:34:35 2003 -->
<pre>running 10K test on 5.0.0int2</pre>
<!-- Author: erin abouzaid -->


<!-- Date: Thu Jul 3 15:09:27 2003 -->
<pre>running 10k test on 4.8.5</pre>
<!-- Author: erin abouzaid -->


<!-- Date: Fri Jul 4 14:58:48 2003 -->
<pre>running 10k test on 4.8.4,  
using run 165466 (0x02865a)</pre>
<!-- Author: Zongru -->


<!-- Date: Sat Jul 5 12:14:04 2003 -->
<pre>using run 165466 (0x02865a), running 10k test on 4.8.4g1 and 
4.8.5, the comparison is done: the histograms are identical.</pre>
<!-- Author: Zongru -->


<!-- Date: Mon Jul 7 13:27:55 2003 -->
<pre>10K test with version 5.0.0int3, 
the comparison between kai and gcc is done, 
the results are identical.</pre>
<!-- Author: Zongru -->


<!-- Date: Tue Jul 22 15:48:11 2003 -->
<pre>Complained to CSL pager person about empty disk logger plots. He's looking into it...</pre>
<!-- Author: collin wolfe -->


<!-- Date: Wed Jul 23 09:16:11 2003 -->
<pre>Found that Kahuna was down. Emailed cdfdh_oper. Kahuna was  
restarted.  
 
Disk logging plots now being filled again. </pre>
<!-- Author: collin -->


<!-- Date: Sun Aug 17 10:43:50 2003 -->
<pre>- work in 5.1.0pre5 to check reproducibility of the  
  crashes. Will make another entry when 5.1.0pre5 is  
  released. -Pasha</pre>
<!-- Author: murat -->


<!-- Date: Sun Aug 17 11:38:48 2003 -->
<pre>- released 5.1.0pre5 area; Tomonobu continues looking at  
  the rest crashes</pre>
<!-- Author: murat -->


<!-- Date: Sat Sep 27 09:03:28 2003 -->
<pre>Checked ORACLE DB usage: 17 ora1, 11 ora1 
kahuna is OK 
 
Two text files generated giving the status of the input and ouptut of the stntuple merging project.  They are found in /cdf/scratch/cdfopr/testRel/5.1.0pre10/sewk_input_092703.txt 
/cdf/scratch/cdfopr/testRel/5.1.0pre10/sewk_output_092703.txt 
on fcdflnx3. </pre>
<!-- Author: Stephen Wolbers -->


<!-- Date: Sat Sep 27 15:45:28 2003 -->
<pre>Worked with David Dagenhart today on the stntuple merge project. Will write a complete report on Sunday with overall plans and status.   
Frank Wuerthwein showed me how to get quota and other useful information for fcdfdata033.  I will write that up on Sunday as well. 
 
Did the usual ORACLE and DIM checks.  All looks OK. 
 
</pre>
<!-- Author: Stephen Wolbers -->


<!-- Date: Sat Sep 27 15:47:31 2003 -->
<pre>Two additional files withwith lists of input and output files from the next set of stntuple merging project can be found at: 
 
/cdf/scratch/cdfopr/testRel/5.1.0pre10/wolbers_countevents_set2.C 
/cdf/scratch/cdfopr/testRel/5.1.0pre10/wolbers_countevents_sewk_set2.C </pre>
<!-- Author: Stephen Wolbers -->


<!-- Date: Sun Sep 28 08:14:27 2003 -->
<pre>ORACLE check: 
 
    23 connections to fcdfora1, look OK 
    11 connections for replica 
 
    No serious CPU offenders on cdfrep1 
 
 DIM is working</pre>
<!-- Author: Stephen Wolbers -->


<!-- Date: Sun Sep 28 16:02:06 2003 -->
<pre>End of shift. 
 
ORACLE looks OK.  18 and 45 connections. 
No big CPU users. 
 
DIM is OK.</pre>
<!-- Author: Stephen Wolbers -->


<!-- Date: Tue Sep 30 07:59:38 2003 -->
<pre>Checked ORACLE DB 
    16  ora1 
    14  rep1  
 
No big CPU users of rep1</pre>
<!-- Author: Stephen Wolbers -->


<!-- Date: Tue Sep 30 14:27:47 2003 -->
<pre>Check DB 
      28 ora1 
      35 rep1  
 
Nobody above 4000 seconds 
   
DIM OK </pre>
<!-- Author: Stephen Wolbers -->


<!-- Date: Tue Sep 30 15:50:40 2003 -->
<pre>ORACLE check : 13 and 17  
No big CPU users 
  
DIM OK</pre>
<!-- Author: Stephen Wolbers -->


<!-- Date: Wed Oct 1 08:35:12 2003 -->
<pre>ORACLE check OK: 18 and 31 connections on ora1 and rep1. 
No big CPU users. 
 
Farm is running farmlet "Stream F", which is beamline executable. 
 
cdfopr on fcdflnx3 is running root_exe.sh jobs to concatenate STNTUPLEs and feed them to fcdfdata034.</pre>
<!-- Author: Stephen Wolbers -->


<!-- Date: Wed Oct 1 15:34:49 2003 -->
<pre>Checked DB use: 19 and 21 connections on ora1 and rep1. 
    One user was over 4000 seconds on rep1.  Sent mail. 
 
DIM is working fine. 
 
</pre>
<!-- Author: Stephen Wolbers -->


<!-- Date: Thu Oct 2 08:24:56 2003 -->
<pre>Art Kreymer brought an issue of load on fcdfora1 to our attention. 
 
The DB connections right now are: 
 
fcdfora1: 64 and dropping (it was 114+ earlier) 
fcdfrep1: 31 
 
Otherwise fcdfrep1 looks OK. 
 
The large load on fcdfora1 is from the farms: beamlines processing. Will send mail to farms and beamlines folks to see if things are working properly.</pre>
<!-- Author: Stephen Wolbers -->


<!-- Date: Thu Oct 2 09:42:33 2003 -->
<pre>After much excitement the DB load on ora1 looks better now.  Still not clear whether the farm is using more than the "allowed" number of connections, but that will be discussed later.</pre>
<!-- Author: Stephen Wolbers -->


<!-- Date: Thu Oct 2 12:08:51 2003 -->
<pre>Checked DB:  22 and 34 are the number of connections for the ora1 and rep1 DB.   
 
The DIM looks OK after fcdfsgi2 came back up. 
 
The load on rep1 looks OK -- no user is over 4000 seconds over the past 4 hours. 
 
The farms may or may not have had problems due to the high load on ora1.  People are hopefully investigating.</pre>
<!-- Author: Stephen Wolbers -->


<!-- Date: Thu Oct 2 15:43:48 2003 -->
<pre>DB check is fine (20 and 17) 
 
DB CPU is fine. 
 
DIM is fine.</pre>
<!-- Author: Stephen Wolbers -->


<!-- Date: Fri Oct 3 08:47:40 2003 -->
<pre>------------------------------------ 2003-10-02 15:33 - from Dustin 
Hi all, 
Pass 11 has now been filled upto the avaliable beamlines (run 160157). 
Pass10 (which had some broken used_sets for runs 164xxx+) was fixed by  
Rob yesterday and is now avaliable for all runs upto the shutdown. 
 
Reminder of what's in these passtables: 
 
Both pass10 and pass11 contain.... 
the latest alignments 
the new TOF tables 
the new SiChipOn table 
SiStipMask table for runs that were missing it. 
updated Calorimeter tables (updated offline lers for later runs) 
 
and then 
Pass10 contains history code 20 beamlines 
Pass11 contains history code 21 beamlines 
 
Thanks, 
Dustin 
</pre>
<!-- Author: murat -->


<!-- Date: Fri Oct 3 08:51:14 2003 -->
<pre>Date: Thu, 02 Oct 2003 12:42:58 -0500 
From: Dmitri Litvintsev &lt;litvinse@fnal.gov&gt; 
Subject: Re: DB connection problems for BeamExe on the farm 
To: Alexey Ershov &lt;ershov@fnal.gov&gt; 
Cc: Stephen Wolbers &lt;wolbers@fnal.gov&gt;, 
 krg@andrew.cmu.edu, 
 Pasha Murat &lt;murat@fnal.gov&gt;, 
 Art Kreymer &lt;kreymer@fnal.gov&gt; 
----------------------- 
Hi Alexey, 
 
Yes we have to re-run on these runs. But first I need to understand why 
this error took place (look at the code and make suggestions) 
 
Dmitri Litvintsev 
 
/----------------------------------------------------------\ 
| Tel:       (630) 840 5005                                | 
| FAX:       (630) 840 2968                                | 
| office:    149-L CDF Trailers                            | 
| E-mail:    litvinse@fnal.gov                             | 
\----------------------------------------------------------/ 
 
On Thu, 2 Oct 2003, Alexey Ershov wrote: 
 
&gt; Dmitri, 
&gt; It turned out that we did have DB connection problems when running on 
&gt; the farm overnight. We have never had these problems before. I assume it 
&gt; has to do with the increased throughput of the farm (we now run on 250 
&gt; nodes simultaneously). 
&gt; 
&gt; 
&gt; Please take a look at logfiles 
&gt; /cdf/opr/cdfprod0/5.1.0_beam_a_4/Stream_F/logs 
&gt; grep -i erlog-e * 
&gt; A couple of messages are shown below. 
&gt; -------------------------------------------------------------------------------------- 
&gt; OCI Error 
&gt; sql: INSERT INTO FILECATALOG.CDF2_RUNSECTION_LIVETIMES ( RUN_NUMBER, 
&gt; SECTION_NUMBER, TRIGGER_INDEX, COUNT_BEFORE_PS_0, COUNT_BEFORE_PS, 
&gt; COUNT_AFTER_PS_0, COUNT_AFTER_PS, FRACTION) VALUES 
&gt; (165941,242,1764,14750139,0,59002,0,0.0040001) 
&gt; DFCDynamicPrescalesOTL::doPut: Error = 
&gt; OCI Error 
&gt;  Failed to add dynamic Prescales 
&gt; SQL function call error 
&gt; OCI Error 
&gt; sql: cant connect: user='cdf_reader' passwd='reader' tnsname='cdfofprd' 
&gt; %ERLOG-e connection_failed:  failed to connect to database 
&gt;          cdf_reader/reader@cdfofprd  retrying in  10  seconds.... 
&gt;          DBObjects::dbOTL 02-Oct-2003 05:12:35 CST run = 165941 event = 
&gt; 1860256 
&gt; %ERLOG-e DBCON_F: ConFail:  23,  fncdf110_28982_1065089238 , 
&gt;          cdf_reader/xxxxx@cdfofprd ,  1,  0 DBEventLogger 
&gt;          02-Oct-2003 05:12:35 CST run = 165941 event = 1860256 
&gt; 
&gt; ---------------------------------------- 
&gt; OCI Error 
&gt; sql: cant connect: user='' passwd='' tnsname='cdfofwrite' 
&gt; %ERLOG-e connection_failed:  failed to connect to database  /@cdfofwrite 
&gt;           retrying in  10  seconds....  DBObjects::dbOTL 
&gt;          02-Oct-2003 06:12:30 CST run = 165949 event = 1611413 
&gt; %ERLOG-e DBCON_F: ConFail:  51,  fncdf73_23178_1065090439 , 
&gt; /@cdfofwrite ,  1 
&gt;         ,  0 DBEventLogger 02-Oct-2003 06:12:30 CST run = 165949 event = 
&gt; 1611413 
&gt; OCI Error 
&gt; sql: cant connect: user='' passwd='' tnsname='cdfofwrite' 
&gt; %ERLOG-e connection_failed:  failed to connect to database  /@cdfofwrite 
&gt;           retrying in  46  seconds....  DBObjects::dbOTL 
&gt;          02-Oct-2003 06:12:40 CST run = 165949 event = 1611413 
&gt; %ERLOG-e DBCON_F: ConFail:  53,  fncdf73_23178_1065090439 , 
&gt; /@cdfofwrite ,  2 
&gt;         ,  0 DBEventLogger 02-Oct-2003 06:12:40 CST run = 165949 event = 
&gt; 1611413 
&gt; ----------------------------------------- 
&gt; 
&gt; 
&gt; How serious are these problems? Should we re-run on the affected runs so 
&gt; that PreScaleModule and SvtBeamModule write to the DB correctly? 
&gt; Should we limit the number of jobs we run simultaneously on the farm? 
&gt; 
&gt; Thank you. 
&gt; -Alexey 
&gt; 
</pre>
<!-- Author: murat -->


<!-- Date: Fri Oct 3 09:02:40 2003 -->
<pre>Checked ORACLE DB: 
 
fcdfora1 : 54 connections, quickly reduced to 30 
fcdfrep1 : 18 connections, looks OK 
 
CPU use is not a problem on fcdfrep1. 
 
DIM is fine.</pre>
<!-- Author: Stephen Wolbers -->


<!-- Date: Fri Oct 3 11:51:02 2003 -->
<pre> 
DB usage check:  
 
fcdfora1:  16 processes 
replica 1: 23 processes</pre>
<!-- Author: Steve Takach -->


<!-- Date: Fri Oct 3 12:04:23 2003 -->
<pre>CPU usage check: no violations of the 4000sec/4hr limit (except for cafuser yesterday, 2 October 2003).</pre>
<!-- Author: Steve Takach -->


<!-- Date: Fri Oct 3 15:02:55 2003 -->
<pre> 
Check of DB usage: 
 
fcdfora1:   21 connections 
fcdfrep1:   23 connections 
 
CPU usage: no violations b/t 10:00 and 14:00 today of the 4000sec/4hr cpu time limit.</pre>
<!-- Author: Steve Takach -->


<!-- Date: Sat Oct 4 08:41:08 2003 -->
<pre>DB usage check:  
 
fcdfora1:     18 connections, mostly idle oracle and sam processes. 
fcdfrep1:      8 connections 
 
Disk usage check:   nothing full, not much variation in the past 24 hours. 
 
fcdfsgi1 stage log:  I note the following entries from the last 15 hrs or so:  
 
Whom should be notified about the error messages below? 
 
2003-Oct-03 17:25:54 filesets CB1278.0 spooled to tape CB1278 (encp SUCCESS) 
2003-Oct-03 18:04:17 filesets CB1275.0 CB1275.1 spooled to tape CB1275 (encp SUCCESS) 
2003-Oct-03 18:04:24 failed to catalog fileset CB1275.1 
2003-Oct-03 18:13:00 filesets CB1276.0 spooled to tape CB1276 (encp SUCCESS) 
2003-Oct-03 18:13:02 failed to catalog fileset CB1276.0 
 
fcdfsgi2 kahuna process check:   Here's the output....everything appears ok. 
 
Checking that kahuna process is running... 
------------------------------------------ 
 08:30:24 [N] Attemping to contact Kahuna on hostname 'localhost' 
 08:30:25 [N] Server comms established OK. Server version 1.3.3 
 08:30:25 [N] Kahuna is answering queries on 'localhost' 
 
 
kahuna is running OK, checking tape queue status (should be active)... 
---------------------------------------------------------------------- 
This rsh session is using DES encryption for all data transmissions. 
QUEUE_NAME      PRIO STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN  SUSP 
tape             80   Open:Active      2    1    -    -     4     3     1     0 
 
 
Checking status of jobs in tape queue... 
---------------------------------------- 
This rsh session is using DES encryption for all data transmissions. 
JOBID USER     STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME 
635387 mjkim    RUN   tape       fcdfsgi2    fcdfsgi2    * PS2338.0 Oct  3 16:43 
635388 mjkim    PEND  tape       fcdfsgi2                * PS2332.0 Oct  3 16:43 
635389 mjkim    PEND  tape       fcdfsgi2                * PS1867.0 Oct  3 16:43 
635390 mjkim    PEND  tape       fcdfsgi2                * PS1686.0 Oct  3 16:43 
 
cdfrep1 CPU usage check: 
 
cafuser = 10791sec during 03-OCT-2003 22:10:17 to 04-OCT-2003 02:00:17 
 
anikeev = 7928sec during 03-OCT-2003 18:10:04 to 03-OCT-2003 22:00:04 
 
no violations of 4000sec/4hr limit in other time periods over the last 28 hours. 
 
Sent a form email to anikeev@fnal.gov as per the offline instructions posted on the web.</pre>
<!-- Author: Steve Takach -->


<!-- Date: Sun Oct 5 09:09:24 2003 -->
<pre>DB Usage Check: 
 
fcdfora1: 14 connections, all but 2 are inactive. 
fcdfrep1: 24 connections, most inactive. 
 
fcdfrep1 CPU Check: 
 
For time period 04-OCT-2003 14:10:05 
             to 04-OCT-2003 18:00:05 
 
 
Top 20 CPU users on database instance cdfrep01 by user's login name 
 
User's login name   CPU Used (sec)   Cpu per session (sec) 
-----------------   --------------   --------------------- 
anikeev                       7382                   1.192 
cafuser                       1849                   1.042 
varganov                      1472                   0.772 
cdfopr                        1363                 681.375 
hkg                           1129                   0.544 
jpursley                       886                   0.561 
anyes                          783                   1.083 
tuttle                         399                   0.494 
eikoyu                         279                   0.722 
takimoto                       256                   0.996 
jkim                           246                   1.758 
ggiurgiu                       195                   0.273 
masato                         179                   0.108 
yik                            140                   0.201 
litvinse                       102                   0.040 
vacavant                        93                   0.234 
dwjang                          90                   0.203 
yuji                            85                   4.047 
jeans                           69                   0.601 
terashi                         57                   0.168 
 
 
Top 5 CPU users on database instance cdfrep01 by Oracle account 
 
Oracle account      CPU Used (sec)   Cpu per session (sec) 
--------------      --------------   --------------------- 
CDF_READER                   16860                   1.190 
FILECATALOG_READUSER           492                   0.049 
DBREADER                        34                   0.224 
REPADMIN_PRD                     5                   0.320 
TOOLMAN                          3                   0.201 
 
For time period 04-OCT-2003 10:10:05 
             to 04-OCT-2003 14:00:05 
 
 
Top 20 CPU users on database instance cdfrep01 by user's login name 
 
User's login name   CPU Used (sec)   Cpu per session (sec) 
-----------------   --------------   --------------------- 
anikeev                       5647                   1.228 
cafuser                       2389                   1.436 
varganov                      1678                   0.778 
kennedy                        660                   0.028 
hkg                            513                   0.468 
jkim                           279                   1.714 
eikoyu                         254                   0.693 
masato                         177                   0.096 
mey                            158                   0.719 
terashi                        149                   0.301 
yik                            136                   0.197 
ggiurgiu                       133                   0.290 
yuji                           111                   5.067 
vacavant                       100                   0.273 
blocker                         71                   0.521 
dwjang                          57                   0.196 
takimoto                        56                   0.923 
wicklund                        53                   0.356 
twright                         43                   0.381 
mjkim                           40                   1.735 
 
 
Top 5 CPU users on database instance cdfrep01 by Oracle account 
 
Oracle account      CPU Used (sec)   Cpu per session (sec) 
--------------      --------------   --------------------- 
CDF_READER                   11850                   1.126 
FILECATALOG_READUSER          1003                   0.035 
DBREADER                        36                   0.232 
REPADMIN_PRD                     4                   0.241 
TOOLMAN                          3                   0.191 
 
 
Sent anikeev form email as per instructions on web. 
 
Data Logger Disk Check:   no problems. 
 
 
Raw data tapes: a few errors noted (see copied records below) 
 
2003-Oct-03 17:25:54 filesets CB1278.0 spooled to tape CB1278 (encp SUCCESS) 
2003-Oct-03 18:04:17 filesets CB1275.0 CB1275.1 spooled to tape CB1275 (encp SUCCESS) 
2003-Oct-03 18:04:24 failed to catalog fileset CB1275.1 
2003-Oct-03 18:13:00 filesets CB1276.0 spooled to tape CB1276 (encp SUCCESS) 
2003-Oct-03 18:13:02 failed to catalog fileset CB1276.0 
 
---&gt; sent email to cdfdh_oper@fnal.gov and to litvense@fnal.gov reporting this problem. 
 
Kahuna Check: 
Checking that kahuna process is running... 
------------------------------------------ 
 09:07:24 [N] Attemping to contact Kahuna on hostname 'localhost' 
 09:07:24 [N] Server comms established OK. Server version 1.3.3 
 09:07:24 [N] Kahuna is answering queries on 'localhost' 
 
 
kahuna is running OK, checking tape queue status (should be active)... 
---------------------------------------------------------------------- 
This rsh session is using DES encryption for all data transmissions. 
QUEUE_NAME      PRIO STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN  SUSP 
tape             80   Open:Active      2    1    -    -     0     0     0     0 
 
 
Checking status of jobs in tape queue... 
---------------------------------------- 
This rsh session is using DES encryption for all data transmissions. 
No unfinished job found in queue &lt;tape&gt; 
Checking that kahuna process is running... 
------------------------------------------ 
 09:07:24 [N] Attemping to contact Kahuna on hostname 'localhost' 
 09:07:24 [N] Server comms established OK. Server version 1.3.3 
 09:07:24 [N] Kahuna is answering queries on 'localhost' 
 
 
kahuna is running OK, checking tape queue status (should be active)... 
---------------------------------------------------------------------- 
This rsh session is using DES encryption for all data transmissions. 
QUEUE_NAME      PRIO STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN  SUSP 
tape             80   Open:Active      2    1    -    -     0     0     0     0 
 
 
Checking status of jobs in tape queue... 
---------------------------------------- 
This rsh session is using DES encryption for all data transmissions. 
No unfinished job found in queue &lt;tape&gt; 
 
 
 
 
Checking that kahuna process is running... 
------------------------------------------ 
 09:07:24 [N] Attemping to contact Kahuna on hostname 'localhost' 
 09:07:24 [N] Server comms established OK. Server version 1.3.3 
 09:07:24 [N] Kahuna is answering queries on 'localhost' 
 
 
kahuna is running OK, checking tape queue status (should be active)... 
---------------------------------------------------------------------- 
This rsh session is using DES encryption for all data transmissions. 
QUEUE_NAME      PRIO STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN  SUSP 
tape             80   Open:Active      2    1    -    -     0     0     0     0 
 
 
Checking status of jobs in tape queue... 
---------------------------------------- 
This rsh session is using DES encryption for all data transmissions. 
No unfinished job found in queue &lt;tape&gt; 
 
</pre>
<!-- Author: Steve Takach -->


<!-- Date: Sun Oct 5 14:08:54 2003 -->
<pre>Oracle DB Usage Check: 
 
fcdfora1: 16 connections 
fcdfrep1: 13 connections 
 
For time period 05-OCT-2003 10:10:06 
             to 05-OCT-2003 14:00:06 
 
 
 
Top 20 CPU users on database instance cdfrep01 by user's login name 
 
User's login name   CPU Used (sec)   Cpu per session (sec) 
-----------------   --------------   --------------------- 
cafuser                       3558                   0.793 
varganov                       937                   0.643 
tuttle                         373                   0.493 
hkg                            306                   0.572 
eikoyu                         276                   0.799 
jkim                           205                   1.680 
vacavant                       181                   0.057 
terashi                        142                   0.302 
cdfopr                         137                   0.038 
masato                         114                   0.151 
dwjang                          72                   0.187 
anyes                           63                   0.300 
yuji                            54                   2.973 
boveia                          49                   0.575 
twright                         36                   0.371 
litvinse                        34                   0.029 
toolman                         33                   0.198 
wicklund                        16                   0.078 
robine                          15                   0.179 
slai                            14                   0.089 
 
 
 
 
Top 5 CPU users on database instance cdfrep01 by Oracle account 
 
Oracle account      CPU Used (sec)   Cpu per session (sec) 
--------------      --------------   --------------------- 
CDF_READER                    5929                   0.601 
FILECATALOG_READUSER           690                   0.080 
DBREADER                        31                   0.200 
REPADMIN_PRD                     4                   0.241 
TOOLMAN                          3                   0.180 
 
 
---&gt; no violations of the 4000sec limit in the last period. 
 
Data Logger Disk Usage Check: no problems as would be expected. 
 
Raw data tapes written: 
  No problems other than what have been noted in previous, recent log entries. 
 
Kahuna check: 
 
Checking that kahuna process is running... 
------------------------------------------ 
 14:07:38 [N] Attemping to contact Kahuna on hostname 'localhost' 
 14:07:38 [N] Server comms established OK. Server version 1.3.3 
 14:07:38 [N] Kahuna is answering queries on 'localhost' 
 
 
kahuna is running OK, checking tape queue status (should be active)... 
---------------------------------------------------------------------- 
This rsh session is using DES encryption for all data transmissions. 
QUEUE_NAME      PRIO STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN  SUSP 
tape             80   Open:Active      2    1    -    -     0     0     0     0 
 
 
Checking status of jobs in tape queue... 
---------------------------------------- 
This rsh session is using DES encryption for all data transmissions. 
No unfinished job found in queue &lt;tape&gt; 
</pre>
<!-- Author: Steve Takach -->


<!-- Date: Mon Oct 6 08:22:57 2003 -->
<pre>Oracle DB check: 
 
fcdfora1:  23 connections 
fcdfrep1:  19 connections 
 
CPU usage check: 
 
no violations in the last 12 hours of the 4000 sec limit. 
 
Data Logger Disks:   no problems  
 
Raw data tapes:   nothing new on the web pages since 2003.10.03 
 
Check Kahuna: 
 
fcdflnx3.fnal.gov&gt; dhmon 
 
Checking that kahuna process is running... 
------------------------------------------ 
 08:21:29 [N] Attemping to contact Kahuna on hostname 'localhost' 
 08:21:29 [N] Server comms established OK. Server version 1.3.3 
 08:21:29 [N] Kahuna is answering queries on 'localhost' 
 
 
kahuna is running OK, checking tape queue status (should be active)... 
---------------------------------------------------------------------- 
This rsh session is using DES encryption for all data transmissions. 
QUEUE_NAME      PRIO STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN  SUSP 
tape             80   Open:Active      2    1    -    -    30    29     1     0 
 
 
Checking status of jobs in tape queue... 
---------------------------------------- 
This rsh session is using DES encryption for all data transmissions. 
JOBID USER     STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME 
635610 manca    RUN   tape       fcdfsgi2    fcdfsgi2    * GI0749.3 Oct  5 23:09 
635611 manca    PEND  tape       fcdfsgi2                * GI0749.4 Oct  5 23:09 
635612 manca    PEND  tape       fcdfsgi2                * GI0749.2 Oct  5 23:09 
635613 manca    PEND  tape       fcdfsgi2                * GI0749.1 Oct  5 23:09 
635614 manca    PEND  tape       fcdfsgi2                * GI0749.0 Oct  5 23:09 
635615 manca    PEND  tape       fcdfsgi2                * GI0747.5 Oct  5 23:09 
635616 manca    PEND  tape       fcdfsgi2                * GI0747.4 Oct  5 23:09 
635617 manca    PEND  tape       fcdfsgi2                * GI0747.3 Oct  5 23:09 
635618 manca    PEND  tape       fcdfsgi2                * GI0747.2 Oct  5 23:09 
635619 manca    PEND  tape       fcdfsgi2                * GI0747.1 Oct  5 23:09 
635620 manca    PEND  tape       fcdfsgi2                * GI0744.4 Oct  5 23:09 
635621 manca    PEND  tape       fcdfsgi2                * GI0744.5 Oct  5 23:09 
635622 manca    PEND  tape       fcdfsgi2                * GI0747.0 Oct  5 23:09 
635623 manca    PEND  tape       fcdfsgi2                * GI0744.2 Oct  5 23:09 
635624 manca    PEND  tape       fcdfsgi2                * GI0744.3 Oct  5 23:09 
635625 manca    PEND  tape       fcdfsgi2                * GI0744.0 Oct  5 23:09 
635626 manca    PEND  tape       fcdfsgi2                * GI0741.3 Oct  5 23:09 
635627 manca    PEND  tape       fcdfsgi2                * GI0741.2 Oct  5 23:09 
635628 manca    PEND  tape       fcdfsgi2                * GI0741.1 Oct  5 23:09 
635629 manca    PEND  tape       fcdfsgi2                * GI0741.0 Oct  5 23:09 
635630 manca    PEND  tape       fcdfsgi2                * GI0945.2 Oct  5 23:09 
635631 manca    PEND  tape       fcdfsgi2                * GI1239.1 Oct  5 23:09 
635632 manca    PEND  tape       fcdfsgi2                * GI0776.2 Oct  5 23:09 
635633 manca    PEND  tape       fcdfsgi2                * GI0776.0 Oct  5 23:09 
635634 manca    PEND  tape       fcdfsgi2                * GI0744.1 Oct  5 23:09 
635635 manca    PEND  tape       fcdfsgi2                * GI0776.3 Oct  5 23:09 
635636 manca    PEND  tape       fcdfsgi2                * GI0776.1 Oct  5 23:09 
635637 manca    PEND  tape       fcdfsgi2                * GI0750.0 Oct  5 23:09 
635638 manca    PEND  tape       fcdfsgi2                * GI0741.4 Oct  5 23:09 
635639 manca    PEND  tape       fcdfsgi2                * GI1241.0 Oct  5 23:09 
 
</pre>
<!-- Author: Steve Takach -->


<!-- Date: Mon Oct 6 15:14:54 2003 -->
<pre> 
DB check: 
 
fcdfora1:    20 connections. 
fcdfrep1:    19 connections. 
 
fcdfrep1 CPU Usage:   other than cafuser, no violations of the 4000sec/4 hr limit for the last 8 hrs. 
 
Data logger disks: no problems, data volume on disk for the last 24 hours or so for disks4,6,7 and home. 
 
raw data tape check: no write errors since last check this morning (as expected since no data taking!) 
 
kahuna running ok..... 
 
fcdflnx3.fnal.gov&gt; dhmon 
 
Checking that kahuna process is running... 
------------------------------------------ 
 15:13:25 [N] Attemping to contact Kahuna on hostname 'localhost' 
 15:13:25 [N] Server comms established OK. Server version 1.3.3 
 15:13:25 [N] Kahuna is answering queries on 'localhost' 
 
 
kahuna is running OK, checking tape queue status (should be active)... 
---------------------------------------------------------------------- 
This rsh session is using DES encryption for all data transmissions. 
QUEUE_NAME      PRIO STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN  SUSP 
tape             80   Open:Active      2    1    -    -    46    45     1     0 
 
 
Checking status of jobs in tape queue... 
---------------------------------------- 
This rsh session is using DES encryption for all data transmissions. 
JOBID USER     STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME 
635623 manca    RUN   tape       fcdfsgi2    fcdfsgi2    * GI0744.2 Oct  5 23:09 
635624 manca    PEND  tape       fcdfsgi2                * GI0744.3 Oct  5 23:09 
635625 manca    PEND  tape       fcdfsgi2                * GI0744.0 Oct  5 23:09 
635626 manca    PEND  tape       fcdfsgi2                * GI0741.3 Oct  5 23:09 
635627 manca    PEND  tape       fcdfsgi2                * GI0741.2 Oct  5 23:09 
635628 manca    PEND  tape       fcdfsgi2                * GI0741.1 Oct  5 23:09 
635629 manca    PEND  tape       fcdfsgi2                * GI0741.0 Oct  5 23:09 
635630 manca    PEND  tape       fcdfsgi2                * GI0945.2 Oct  5 23:09 
635631 manca    PEND  tape       fcdfsgi2                * GI1239.1 Oct  5 23:09 
635632 manca    PEND  tape       fcdfsgi2                * GI0776.2 Oct  5 23:09 
635633 manca    PEND  tape       fcdfsgi2                * GI0776.0 Oct  5 23:09 
635634 manca    PEND  tape       fcdfsgi2                * GI0744.1 Oct  5 23:09 
635635 manca    PEND  tape       fcdfsgi2                * GI0776.3 Oct  5 23:09 
635636 manca    PEND  tape       fcdfsgi2                * GI0776.1 Oct  5 23:09 
635637 manca    PEND  tape       fcdfsgi2                * GI0750.0 Oct  5 23:09 
635638 manca    PEND  tape       fcdfsgi2                * GI0741.4 Oct  5 23:09 
635639 manca    PEND  tape       fcdfsgi2                * GI1241.0 Oct  5 23:09 
635705 manca    PEND  tape       fcdfsgi2                * GI1241.0 Oct  6 09:46 
635706 manca    PEND  tape       fcdfsgi2                * GI1239.1 Oct  6 09:46 
635707 manca    PEND  tape       fcdfsgi2                * GI0776.2 Oct  6 09:46 
635708 manca    PEND  tape       fcdfsgi2                * GI0776.3 Oct  6 09:46 
635709 manca    PEND  tape       fcdfsgi2                * GI0776.1 Oct  6 09:46 
635710 manca    PEND  tape       fcdfsgi2                * GI0945.2 Oct  6 09:46 
635711 manca    PEND  tape       fcdfsgi2                * GI0943.0 Oct  6 09:46 
635712 manca    PEND  tape       fcdfsgi2                * GI0943.2 Oct  6 09:46 
635713 manca    PEND  tape       fcdfsgi2                * GI0744.5 Oct  6 09:46 
635714 manca    PEND  tape       fcdfsgi2                * GI0744.4 Oct  6 09:46 
635715 manca    PEND  tape       fcdfsgi2                * GI0744.3 Oct  6 09:46 
635716 manca    PEND  tape       fcdfsgi2                * GI0744.2 Oct  6 09:46 
635717 manca    PEND  tape       fcdfsgi2                * GI0744.1 Oct  6 09:46 
635718 manca    PEND  tape       fcdfsgi2                * GI0744.0 Oct  6 09:46 
635719 manca    PEND  tape       fcdfsgi2                * GI0741.4 Oct  6 09:46 
635720 manca    PEND  tape       fcdfsgi2                * GI0741.3 Oct  6 09:46 
635721 manca    PEND  tape       fcdfsgi2                * GI0741.1 Oct  6 09:46 
635722 manca    PEND  tape       fcdfsgi2                * GI0741.2 Oct  6 09:46 
635723 manca    PEND  tape       fcdfsgi2                * GI0741.0 Oct  6 09:46 
635724 manca    PEND  tape       fcdfsgi2                * GI0750.0 Oct  6 09:46 
635725 manca    PEND  tape       fcdfsgi2                * GI0747.0 Oct  6 09:46 
635726 manca    PEND  tape       fcdfsgi2                * GI0747.1 Oct  6 09:46 
635727 manca    PEND  tape       fcdfsgi2                * GI0747.3 Oct  6 09:46 
635728 manca    PEND  tape       fcdfsgi2                * GI0747.4 Oct  6 09:46 
635729 manca    PEND  tape       fcdfsgi2                * GI0747.2 Oct  6 09:46 
635730 manca    PEND  tape       fcdfsgi2                * GI0747.5 Oct  6 09:46 
635731 manca    PEND  tape       fcdfsgi2                * GI0749.0 Oct  6 09:46 
635732 manca    PEND  tape       fcdfsgi2                * GI0749.1 Oct  6 09:46 
635733 manca    PEND  tape       fcdfsgi2                * GI0776.0 Oct  6 09:46 
 
 
 
 
</pre>
<!-- Author: Steve Takach -->


<!-- Date: Tue Oct 7 12:33:28 2003 -->
<pre>DB Usage check: 
 
fcdfora1:   21 connections 
fcdfrep1:   23 connections 
 
fcdfrep1 CPU usage:   no violations in the last 28 hours of the 4000sec/4hr limit, except for cafuser. 
 
Data logger disk check:  no problems. 
 
raw data tapes: no failures since 3 October. 
 
check KAHUNA: seems ok. 
 
fcdflnx3.fnal.gov&gt; dhmon 
  
Checking that kahuna process is running... 
------------------------------------------ 
rsh: kcmd to host fcdfsgi2 failed - No credentials cache file found 
trying normal rsh (/usr/bin/rsh)  WARNING: NO ENCRYPTION! 
rsh: invalid option -- N 
usage: rsh [-nd] [-l login] host [command] 
  
  
kahuna is running OK, checking tape queue status (should be active)... 
---------------------------------------------------------------------- 
rsh: kcmd to host fcdfsgi2 failed - No credentials cache file found 
trying normal rsh (/usr/bin/rsh)  WARNING: NO ENCRYPTION! 
rsh: invalid option -- N 
usage: rsh [-nd] [-l login] host [command] 
  
  
Checking status of jobs in tape queue... 
---------------------------------------- 
rsh: kcmd to host fcdfsgi2 failed - No credentials cache file found 
trying normal rsh (/usr/bin/rsh)  WARNING: NO ENCRYPTION! 
rsh: invalid option -- N 
usage: rsh [-nd] [-l login] host [command] 
</pre>
<!-- Author: Steve Takach -->


<!-- Date: Wed Oct 8 08:28:58 2003 -->
<pre>DB Usage check: 
 
fcdfora1:  27 connections 
fcdfrep1:  23 connections 
 
fcdfrep1 CPU usage:  1 violation of 4000 sec limit (see below) 
 
For time period 07-OCT-2003 18:10:05 
             to 07-OCT-2003 22:00:05 
 
 
 
Top 20 CPU users on database instance cdfrep01 by user's login name 
 
User's login name   CPU Used (sec)   Cpu per session (sec) 
-----------------   --------------   --------------------- 
orejudos                      4787                   1.409 
cafuser                       3911                   0.377 
bhatti                         740                   0.621 
ben                            690                   0.937 
lmiller                        628                   0.633 
eikoyu                         616                   0.522 
fernand                        592                   0.482 
dauria                         543                   0.801 
tafirout                       509                   0.404 
sidoti                         470                   0.879 
anikeev                        400                   1.191 
zrwan                          374                   6.333 
chlebana                       350                   0.503 
cdfprod0                       313                   0.086 
sabik                          295                   0.206 
jkim                           228                   1.724 
carott                         223                   1.713 
rescigno                       206                  51.462 
thomsone                       152                   2.372 
dwjang                         139                   0.399 
 
 
 
 
Top 5 CPU users on database instance cdfrep01 by Oracle account 
 
Oracle account      CPU Used (sec)   Cpu per session (sec) 
--------------      --------------   --------------------- 
CDF_READER                   16214                   0.975 
FILECATALOG_READUSER           848                   0.053 
DBREADER                        45                   0.294 
REPADMIN_PRD                     4                   0.251 
TOOLMAN                          3                   0.184 
 
----&gt; sent form email to orejudos@fnal.gov as per web page instructions. 
 
Data logger disk check: activity on disk 6; no apparent problems. 
 
Data tapes written: Tapes being written on 7 October. No apparent problems. 
 
KAHUNA check:  running ok.</pre>
<!-- Author: Steve Takach -->


<!-- Date: Thu Oct 9 09:17:25 2003 -->
<pre>DB usage check: 
 
fcdfora1:    26 connections 
fcdfrep1:    13 connections 
 
fcdfrep1 CPU usage check: 
 
For time period 09-OCT-2003 02:10:06 
             to 09-OCT-2003 06:00:06 
 
 
Top 20 CPU users on database instance cdfrep01 by user's login name 
 
User's login name   CPU Used (sec)   Cpu per session (sec) 
-----------------   --------------   --------------------- 
frisch                        9569                   0.859 
cafuser                       6208                   0.937 
behari                        1522                   0.610 
hkg                           1399                   0.771 
sidoti                         613                   0.629 
bello                          612                   0.335 
vallecor                       233                   0.661 
dwjang                         223                   0.596 
slai                           219                   0.159 
usynin                         203                   1.369 
litvinse                       184                   0.205 
eikoyu                         136                   0.766 
jkim                           111                   1.791 
anikeev                         98                   0.204 
gervasio                        78                   0.422 
varganov                        64                   0.569 
jpursley                        63                   0.168 
jahred                          53                   0.334 
toolman                         50                   0.297 
herndon                         45                   1.094 
 
 
Top 5 CPU users on database instance cdfrep01 by Oracle account 
 
Oracle account      CPU Used (sec)   Cpu per session (sec) 
--------------      --------------   --------------------- 
CDF_READER                   21736                   0.921 
FILECATALOG_READUSER           262                   0.031 
DBREADER                        47                   0.307 
REPADMIN_PRD                     3                   0.231 
TOOLMAN                          3                   0.198 
 
 
1 violdation of the 4000sec/4hr limit ---&gt; sent form email notification to frisch@fnal.gov, frisch@hep.uchicago.edu 
 
Data logger disk check:   activity on disk6, but no problems there or elsewhere. 
 
written data tape check:  no errors in last 24 hours. 
 
KAHUNA check:  running ok. 
 
</pre>
<!-- Author: Steve Takach -->


<!-- Date: Thu Oct 9 15:02:23 2003 -->
<pre>DB Usage check 
 
fcdfora1:   27 connections 
fcdfrep1:   21 connections 
 
fcdfrep1 CPU check:   no violations of the 4000sec limit in the last 8 hours. 
 
Data logger disk check:  seem ok disk6 data volume growing to about 450 Gb of 676 Gb max. 
 
 
written tape check: no errors in recent tape writes for the last 24 hours. 
 
kahuna check:  seems ok 
fcdflnx3.fnal.gov&gt; dhmon 
 
Checking that kahuna process is running... 
------------------------------------------ 
 15:01:53 [N] Attemping to contact Kahuna on hostname 'localhost' 
 15:01:53 [N] Server comms established OK. Server version 1.3.3 
 15:01:53 [N] Kahuna is answering queries on 'localhost' 
 
 
kahuna is running OK, checking tape queue status (should be active)... 
---------------------------------------------------------------------- 
This rsh session is using DES encryption for all data transmissions. 
QUEUE_NAME      PRIO STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN  SUSP 
tape             80   Open:Active      2    1    -    -     1     0     1     0 
 
 
Checking status of jobs in tape queue... 
---------------------------------------- 
This rsh session is using DES encryption for all data transmissions. 
JOBID USER     STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME 
636694 orejudos RUN   tape       fcdfsgi2    fcdfsgi2    * GI2430.1 Oct  9 14:46 
 
 
 
</pre>
<!-- Author: Steve Takach -->


<!-- Date: Fri Oct 10 12:59:36 2003 -->
<pre> 
  10 Oct 2003 12:55 
 
DB usage check: 
    fcdfora1  20 connections 
    fcdfrep1  19 connections 
 
    fcdfrep1 CPU check  none &gt; 4000 sec. 
 
Kahuna check: OK 
</pre>
<!-- Author: Jeremy Lys -->


<!-- Date: Fri Oct 10 17:30:53 2003 -->
<pre> 
DB usage check: 
    fcdfora1  22 connections 
    fcdfrep1  11 connections 
   
    fcdfrep1 CPU check: none &gt; 4000 sec. 
 
Kahuna check:  OK 
</pre>
<!-- Author: Jeremy Lys -->


<!-- Date: Sat Oct 11 08:42:39 2003 -->
<pre>DB usage check:  fcdfora1 25 connections 
                 fcdfrep1 20 connections 
fcdfrep1 CPU check: none &gt; 4000 sec (except cafuser) 
Kahuna check: OK 
</pre>
<!-- Author: Jeremy Lys -->


<!-- Date: Sat Oct 11 15:55:43 2003 -->
<pre>DB usage check:  fcdfora1  19 connections 
                 fcdfrep1  22 connections 
  fcdfrep1 CPU check:  OK 
Kahuna check:  OK 
</pre>
<!-- Author: Jeremy Lys -->


<!-- Date: Sun Oct 12 09:01:55 2003 -->
<pre>DB usage check:  fcdfora1  16 connections 
                 fcdfrep1  11 connections 
    fcdfrep1 CPU check:  none &gt; 4000 (except cafuser) 
Kahuna check:  OK 
</pre>
<!-- Author: Jeremy Lys -->


<!-- Date: Sun Oct 12 15:51:59 2003 -->
<pre>DB usage check:  fcdfora1:  15 connections 
                 fcdfrep1:  15 connections 
                 fcdfrep1 CPU check: none &gt; 4000 (except cafuser) 
Kahuna:  OK 
</pre>
<!-- Author: Jeremy Lys -->


<!-- Date: Mon Oct 13 16:03:19 2003 -->
<pre>DB usage check:  fcdfora  18 connections 
                 fcdfrep1 25   " 
      fcdfrep1 CPU check:  OK 
Kahuna check OK 
</pre>
<!-- Author: Jeremy Lys -->


<!-- Date: Wed Oct 15 19:05:55 2003 -->
<pre> Have done checks during past two days. No major problems. 
 
Current status: 
 
DB usage check: cdfora1 18 connections 
                cdfrep1 14     " 
  cdfrep1 CPU check: none &gt; 4000  
Kahuna check:  OK 
 
</pre>
<!-- Author: Jeremy Lys -->


<!-- Date: Fri Oct 17 08:47:42 2003 -->
<pre>DB usage check:  cdfora1  24 connections 
                 cdfrep1  26      " 
cdfrep1 CPU check: OK 
Kahuna check:  OK 
</pre>
<!-- Author: Jeremy Lys -->


<!-- Date: Sat Oct 18 15:46:03 2003 -->
<pre> checks today. no  problem.  
 
Current status:  
 
DB usage check: cdfora1 21 connections  
                cdfrep1 18     "  
  cdfrep1 CPU check: none &gt; 4000   
Kahuna check:  OK  
 
 
</pre>
<!-- Author: cuevas -->


<!-- Date: Sun Oct 19 15:49:01 2003 -->
<pre>checks today. no  problem.   
 
Current status:   
 
DB usage check: cdfora1 15 connections   
                cdfrep1 17     "   
  cdfrep1 CPU check: none &gt; 4000    
Kahuna check:  OK   
</pre>
<!-- Author: cuevas -->


<!-- Date: Thu Oct 23 16:17:45 2003 -->
<pre>checks today. no  problem.   
after rebooting, 
everything looks right!  
 
 
Current status:    
 
DB usage check: cdfora1 15 connections    
                cdfrep1 17     "    
  cdfrep1 CPU check: none &gt; 4000     
Kahuna check:  OK    
 
</pre>
<!-- Author: cuevas -->


<!-- Date: Fri Oct 31 15:45:43 2003 -->
<pre>today checks: no  problem.    
 
Current status:    
Kahuna check:  OK 
DB usage check: cdfora1 13 connections    
                cdfrep1 19 connections  
 cdfrep1 CPU check: 
anikeev = 21336 sec during 30-OCT-2003 22:10:04 to 31-OCT-2003 02:00:04 
Sent email to anikeev@fnal.gov 
 
core files from farm: 
stream_C run = 168562 
core.14245.26556.2     KalSiliconStandaloneTracker::forwardSearch 
core.10060.26556.2     ELdestination::ELdestination 
core.1264.26556.2      ELdestination::ELdestination 
core.14107.26556.2     ELdestination::ELdestination 
core.23892.26556.2     AppFrame::stackStop from AbsParmGeneral&lt;bool&gt;::restoreState from ELdestination::ELdestination 
all jobs finished with: 
bad_alloc: Could not allocate memory .. 
 
</pre>
<!-- Author: Elena Vataga -->


<!-- Date: Sun Nov 2 12:05:47 2003 -->
<pre>Kahuna check:  OK  
DB usage:      OK 
cdfrep1 CPU check: 
  email to  
dube1@fnal.gov 
tafirout@physics.utoronto.ca 
 
User's login name   CPU Used (sec)   Cpu per session (sec) 
-----------------   --------------   --------------------- 
 
dube1                         4457                   1.724 
tafirout                      4131                   3.850 
</pre>
<!-- Author: Elena Vataga -->


<!-- Date: Mon Nov 3 15:43:24 2003 -->
<pre>The DIM/Kahuna system was "permanently retired" this morning, 
made corresponding undate in shift web pages. 
DB usage check: cdfora1 11 connections     
                cdfrep1 43     "     
  cdfrep1 CPU check: none &gt; 4000     
 
============================================================= 
building 5.2.0: 
============================================================= 
error during compilation 
"/cdf/opr/cdfopr/testRel/5.2.0/Stntuple/mod/InitPhotonBlock.cc", line 183:  
error: 
          identifier "phoPesDeltaR" is undefined 
        photon-&gt;fPemPesDr = phoPesDeltaR(pho); 
                            ^ 
 
1 error detected in the compilation of  
"/cdf/opr/cdfopr/testRel/5.2.0/Stntuple/mod/InitPhotonBlock.cc". 
KCC: Compilation failed. 
================================================================ 
building 5.2.0 with maxopt 
================================================================ 
&lt;**lib**&gt; FrameMods 
&lt;**compiling**&gt; AppRootOutputModule.cc 
"ISOcxx iostream", line 32: catastrophic error: could not open source file 
          "ISOcxx/ISOcxx.h" 
    #include "ISOcxx/ISOcxx.h" 
                              ^ 
 
1 catastrophic error detected in the compilation of "/cdf/opr/cdfopr/testRel/5.2.0/FrameMods/DH/AppRootOutputModule.cc". 
Compilation terminated. 
KCC: Compilation failed. 
 
&lt;**compiling**&gt; TStnElectronID.cc 
"ISOcxx cmath", line 46: catastrophic error: could not open source file 
          "ISOcxx/ISOcxx.h" 
    #include "ISOcxx/ISOcxx.h" 
                              ^ 
 
1 catastrophic error detected in the compilation of "/cdf/opr/cdfopr/testRel/5.2.0/Stntuple/alg/TStnElectronID.cc". 
Compilation terminated. 
KCC: Compilation failed. 
 
</pre>
<!-- Author: Elena Vataga -->


<!-- Date: Fri Nov 7 09:55:31 2003 -->
<pre>Crashes returning -1 event number appear fixed as of 
last night.  CdfMetModule.cc endRun was the problem.</pre>
<!-- Author: Dagenhart -->


<!-- Date: Sun Nov 9 11:27:19 2003 -->
<pre>On Friday I sent mail to ikrav regading database CPU usage 
 
For time period 06-NOV-2003 22:10:06 
            to 07-NOV-2003 02:00:06 
 
Top 20 CPU users on database instance cdfrep01 by user's login name 
 
User's login name   CPU Used (sec)   Cpu per session (sec) 
-----------------   --------------   --------------------- 
ikrav                         4910                   0.872 
bernd                         3212                   1.149 
 
 
Sent another to ikrav on Saturday: 
For time period 08-NOV-2003 10:10:03 
             to 08-NOV-2003 14:00:03 
 
 
Top 20 CPU users on database instance cdfrep01 by user's login name 
 
User's login name   CPU Used (sec)   Cpu per session (sec) 
-----------------   --------------   --------------------- 
ikrav                         6169                   0.536 
cafuser                       1438                   0.215 
 
On Sunday sent message to igv regarding database CPU usage 
 
For time period 08-NOV-2003 18:10:03 
             to 08-NOV-2003 22:00:03 
 
 
Top 20 CPU users on database instance cdfrep01 by user's login name 
 
User's login name   CPU Used (sec)   Cpu per session (sec) 
-----------------   --------------   --------------------- 
igv                          14543                   1.295 
cafuser                       1757                   0.251 
 
</pre>
<!-- Author: Heather Gerberich -->


<!-- Date: Sun Nov 9 14:17:50 2003 -->
<pre>Sent mail to ceballos regarding database CPU usage. 
 
For time period 09-NOV-2003 10:10:04 
             to 09-NOV-2003 14:00:04 
 
 
Top 20 CPU users on database instance cdfrep01 by user's login name 
 
User's login name   CPU Used (sec)   Cpu per session (sec) 
-----------------   --------------   --------------------- 
ceballos                      7467                   1.105 
varganov                      1845                   0.629 
</pre>
<!-- Author: Heather Gerberich -->


<!-- Date: Mon Nov 10 12:12:05 2003 -->
<pre>Sent email to ceballos regarding database cpu usage. 
 
For time period 10-NOV-2003 06:10:06 
             to 10-NOV-2003 10:00:06 
 
 
Top 20 CPU users on database instance cdfrep01 by user's login name 
 
User's login name   CPU Used (sec)   Cpu per session (sec) 
-----------------   --------------   --------------------- 
ceballos                      5653                   1.195 
cafuser                       4516                   0.471 
 
</pre>
<!-- Author: Heather Gerberich -->


<!-- Date: Mon Nov 10 14:17:38 2003 -->
<pre>Send email to ceballos regarding database cpu usage 
 
For time period 10-NOV-2003 10:10:06 
             to 10-NOV-2003 14:00:06 
 
 
Top 20 CPU users on database instance cdfrep01 by user's login name 
 
User's login name   CPU Used (sec)   Cpu per session (sec) 
-----------------   --------------   --------------------- 
ceballos                      5401                   1.181 
cafuser                       2107                   0.481</pre>
<!-- Author: Heather Gerberich -->


<!-- Date: Wed Nov 12 08:25:30 2003 -->
<pre>sent email to jdlee regarding database cpu usage: 
 
 
 
For time period 11-NOV-2003 14:10:04 
             to 11-NOV-2003 18:00:04 
 
 
Top 20 CPU users on database instance cdfrep01 by user's login name 
 
User's login name   CPU Used (sec)   Cpu per session (sec) 
-----------------   --------------   --------------------- 
jdlee                        10361                   1.781 
 
For time period 11-NOV-2003 18:10:04 
             to 11-NOV-2003 22:00:04 
 
 
Top 20 CPU users on database instance cdfrep01 by user's login name 
 
User's login name   CPU Used (sec)   Cpu per session (sec) 
-----------------   --------------   --------------------- 
jdlee                         8100                   1.930 
 
For time period 11-NOV-2003 22:10:02 
             to 12-NOV-2003 02:00:02 
 
 
Top 20 CPU users on database instance cdfrep01 by user's login name 
 
User's login name   CPU Used (sec)   Cpu per session (sec) 
-----------------   --------------   --------------------- 
jdlee                         5859                   1.457 
</pre>
<!-- Author: Heather Gerberich -->


<!-- Date: Fri Dec 12 13:47:26 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-63&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-63  width='300' align='top'></a></td><td>A common character of events in Stream_E that crashed ProductionExe : busy COT - i.e  
 
number of TOF hits: 119, in both PMTs: 105 
 number of Phi and Sas Silicon hits 0 
 number of COT tracks 0 
 number of COT Hits 17138 
 size of CLC list 1 
 size of BSC list 1 
 missing Et 9.05487 
 
However, running ProductionExe on those events doesn't cause crash.  
 
</td></tr></table>
<!-- Author: Yanwen  -->
<!-- Comment: Fri Dec 12 15;02;01 comment by...Yanwen -->
<pre>I guess I was wrong. All the events can be reconstructed by ProductionExe one by one -- tracks look beautiful in the output file, even though the COT hits looked such a mess.  </pre>


<!-- Date: Fri Dec 12 17:55:02 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-67&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-67  width='300' align='top'></a></td><td>I could not reproduce on single crash in a whole day.  
This is an crashed event. I show the raw COT hits and the constructed tracks after I ran ProductionExe on it. </td></tr></table>
<!-- Author: Yanwen  -->


<!-- Date: Fri Dec 12 17:56:11 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-68&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-68  width='300' align='top'></a></td><td>Raw COT hits : where I thought too messy. </td></tr></table>
<!-- Author: Yanwen  -->


<!-- Date: Mon Dec 15 08:36:12 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-70&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-70  width='300' align='top'></a></td><td>couldn't reproduce the crash. But something must have gone very wrong with plug calorimeter in this event.  
enjoy. </td></tr></table>
<!-- Author: cdfopr -->


<!-- Date: Sat Dec 20 16:07:38 2003 -->
<pre>/cdf/opr at fcdflnx3 is full. 
 
/dev/sdd1  977988884 878630884         0 100% /cdf/opr 
 
e-mail sent to sexton@fnal.gov and wdd@fnal.gov</pre>
<!-- Author: Youngdo Oh -->


<!-- Date: Sun Dec 21 09:42:51 2003 -->
<pre>run 176576 has time delay greater than 72 hours tagged 
as PASS 11 
 
mail sent to calibration experts, Ben Cooper. 
 
</pre>
<!-- Author: Youngdo Oh -->


<!-- Date: Sat Jan 3 08:44:09 2004 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>I have two things which in my mind would make work of the 
offline shifter more efficient: 
 
1. Start a new file for the offline e-log (the current one grew 
to big, and requires a lot of scrolling to get to the most  
recent entries. New Year is another good reason to start a new 
one. 
 
2. Konqueror is a 'recommended' browser for the offline shifter, 
but it crashes/freezes saying smth. about JavaScript being off, 
when one tries to open the e-log link in it ... not very helpful</pre></td></tr></td></table>
<!-- Author: Konstantin Anikeev -->


<!-- Date: Sat Jan 3 08:48:50 2004 -->
<pre>there are calibrations (Pass 11) missing for the following runs: 
 
176651, 176655, 176695, 176696, 
 
and will officially start to be "missing" in less than 2 hours 
for runs 
 
177140, 177150 
 
E-mail sent to Ben Cooper &lt;bdc@hep.ucl.ac.uk&gt; and cc'ed to 
others...</pre>
<!-- Author: Konstantin Anikeev -->


<!-- Date: Sat Jan 3 09:02:55 2004 -->
<pre>No single user exceeded 6000 sec DB CPU usage in the last 
24hr period. No action taken. </pre>
<!-- Author: Konstantin Anikeev -->
<!-- Comment: Sat Jan  3 09;12;09 comment by...Konstantin Anikeev -->
<pre>Also the number of connections is 14-15 &lt; 100.  OK.</pre>
<!-- Date: Sat Jan 3 09:08:40 2004 -->
<pre>Deginning of the shift DH system check: 
 
1. disk3-disk9 are Ok.  (well, disk3 is ~60% full but not 75%) 
2. NO errors in writing raw data to tapes 
 
will check again in the middle of the shift</pre>
<!-- Author: Konstantin Anikeev -->