<!-- Title: 11:56:23  Mon Jan 28 2002 -->
<!-- Crewchief: mirsi -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>CDF Offline Log</pre>


<!-- Date: Mon Jan 28 11:59:03 2002 -->
<pre>I've created new page, since the old one was getting too long. 
 
At the Farm we are processign the data till the run 137023 with 4.2.0c and we'd like to switch to 4.3.0something/4.3.1 after this run.  Also, we'll switch to concatenation on the worker nodes.</pre>
<!-- Author: mirsi -->

<!-- Date: Tue Jan 29 15:59:38 2002 -->
<pre>

1) 10K test at fcdflnx1 using 4.3.1pre2 is cancelled pending
   delivery of 4.3.1pre3

2) As a preparation to 10K test at fcdflnx1 I copied the following
   files of newly taken data in the directory  /cdf/scratch/cdfprod0/10Kevents

   these are the files:

        -rw-r--r--   1 cdfprod0 cdf      1072324357 Jan 29 15:12 ar021b3e.017cphys
        -rw-r--r--   1 cdfprod0 cdf      1072010598 Jan 29 15:16 ar021b3e.0001phys
        -rw-r--r--   1 cdfprod0 cdf      1073143808 Jan 29 15:18 ar021b3e.0321phys
        -rw-r--r--   1 cdfprod0 cdf      18340442 Jan 29 15:18 ar021b3e.0506phys

        -rw-r--r--   1 cdfprod0 cdf      1108803610 Jan 29 14:53 br021b3e.0001phys
        -rw-r--r--   1 cdfprod0 cdf      1027803368 Jan 29 15:13 br021b3d.0065phys
        -rw-r--r--   1 cdfprod0 cdf      1008530665 Jan 29 15:24 br021b3d.006bphys
        -rw-r--r--   1 cdfprod0 cdf      1008400057 Jan 29 15:36 br021b3d.0071phys

        -rw-r--r--   1 cdfprod0 cdf      1070462193 Jan 29 14:44 jr021b3e.0001phys

        -rw-r--r--   1 cdfprod0 cdf      1087051413 Jan 29 14:48 gr021b3e.0001phys
        -rw-r--r--   1 cdfprod0 cdf      1022582130 Jan 29 15:19 gr021b3e.002ephys
        -rw-r--r--   1 cdfprod0 cdf      1061959301 Jan 29 15:28 gr021b3e.003fphys
        -rw-r--r--   1 cdfprod0 cdf      1021005766 Jan 29 15:36 gr021b3e.0094phys

</pre>
<!-- Author: Dmitry Litvintsev -->


<!-- Date: Tue Jan 29 16:46:10 2002 -->
<pre>

 10K test of 4.3.1pre3 started @ fcdflnx1


</pre>
<!-- Author: Dmitry Litvintsev -->



<!-- Date: Thu Jan 31 15:34:20 2002 -->
<pre>From: Rob Snihur &lt;snihur@fnal.gov&gt; 
To: "cdf_offline_operations@fnal.gov" &lt;cdf_offline_operations@fnal.gov&gt; 
Cc: Avi Yagil &lt;yagil@fnal.gov&gt; 
Subject: more v15 calibrations 
Date: Thu, 31 Jan 2002 15:01:19 -0600 
Sender: owner-cdf_offline_operations@listserv.fnal.gov 
X-Mailer: Mozilla 4.73 [en] (X11; U; Linux 2.2.14-1.3.0f2 i686) 
 
 Hi, 
 Version 15 of the used_set with process_name=PROD_PHYSICS_CDF 
 now exists for all good runs between 122621 and 138104 inclusive 
 (as listed at http://fnpcc.fnal.gov/run_info.php). 
 This includes Avi's favorite run 138045. 
 
 thanks, 
 -rob 
</pre>
<!-- Author: murat -->
<!-- Comment: Thu Jan 31 15;35;11 comment by...murat -->
<pre>should be 138046.</pre>


<!-- Date: Fri Feb 1 13:27:26 2002 -->
<pre>CDF-BUGS: 
 
PR#1896: miconfigured rcpdb on NCDF09. - fixed 
PR#1894: not reproducible (probably caused by the stale object 
         files 
PR#1900: rcp question 
PR#1902: request for good run filter (on Bill Badgett's to do list)</pre>
<!-- Author: murat -->


<!-- Date: Fri Feb 1 15:15:37 2002 -->
<pre>FCDFLNX1 was added to the list of hosts allowed to open 
X-connections with B0 machines over the firewall by Sergej 
Burov - now we can do debugging on FCDFLNX1 from B0.</pre>
<!-- Author: murat -->


<!-- Date: Fri Feb 1 16:35:33 2002 -->
<pre>PR#1905: unresolved reference during the link, user advised</pre>
<!-- Author: murat -->


<!-- Date: Fri Feb 1 16:38:41 2002 -->
<pre>ProductionExe 4.3.1pre3 crashed twice when it ran on run=138046 
(Miro said he processed about 600K events from this run) 
 
crash#1 : run 138046, ev 241484 
crash#2 : run 138046, ev 1854260 
 
In both cases executable crashed in the same place: 
 
KalTracking/src/KalNewExperimentalTracker.cc, line 278 
 
I ran ProductionExe on these events on FCDFLNX1 and was able  
to process both successfully, though I noticed that processing 
took about 10 minutes in one case and 5 minutes in another, all 
time spent in SVX tracking. 
 
I looked at the offending events with the event display - they  
are overpopulated, the crashes happen because processing time 
on the Farm exceeded 15min (according to Miro).  
 
</pre>
<!-- Author: murat -->


<!-- Date: Fri Feb 1 16:39:56 2002 -->
<pre>Art started build of 4.3.1</pre>
<!-- Author: murat -->


<!-- Date: Fri Feb 1 17:11:32 2002 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-2&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-2  width='300' align='top'></td><td>SVX display of run 138046 ev 1854260   (took about 10 min to process)</a></td></tr></table>
<!-- Author: murat -->


<!-- Date: Fri Feb 1 17:12:48 2002 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-3&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-3  width='300' align='top'></td><td>COT display of run=138046 event=1854260</a></td></tr></table>
<!-- Author: murat -->


<!-- Date: Sat Feb 2 09:20:36 2002 -->
<pre>one more crash from run 138046 processing: event 2038100 
same place as previous 2, event display shows very high occupancy 
in SVX. Processed successfully on fcdflnx1, timing for SVX  
modules: 
SiClusteringMod 1       28.340000+/-0.000000   33.402140+/-0.000000   28.340 
SiPatternRecMod 1       353.85000+/-0.000000   353.62980+/-0.000000  353.850 
seems to be another "Farm's done timeout" 
</pre>
<!-- Author: murat -->


<!-- Date: Sat Feb 2 10:19:16 2002 -->
<pre>CDF-BUGS: first report for today - PR#1906 (question on how to  
request a new CVS package) - answered</pre>
<!-- Author: murat -->


<!-- Date: Sun Feb 3 09:09:38 2002 -->
<pre>-------------------- Beginning of Suday shift 
- CDF-BUGS: PR#1907 (problem with RCP, user asked for reproducible example) 
- FARM: processed 4M events with 4.3.1, 57 crashes, proceed with analysing  
   them 
 
 
</pre>
<!-- Author: murat -->


<!-- Date: Sun Feb 3 09:15:07 2002 -->
<pre>update on the crash rate - on FCDFLNX1 we currently have : 
- stream A: 7 crashes 
- stream B: 41 
- stream G: 25 
- stream J:  8 
-------------- 
total:          81 
</pre>
<!-- Author: murat -->


<!-- Date: Sun Feb 3 14:44:27 2002 -->
<pre>The Farm has ended the first phase of the reconstruction with 4.3.1 on the following streams/runs: 
 
A - runs from 133896 to run 137559 (aphysd) 
B - runs from 137024 to run 137559 (btst1d,2d) 
G - runs from 137024 to run 137559 (gtst1d,2d,3d) 
J - runs from 137024 to run 137559 (jphysd) 
 
This needs to be concatenated. Some portion of the stream A is done, but all of the other need to be done. 
 
We've switched from 4.2.0b on the run 137023, which showed to be the last run without too many problems.</pre>
<!-- Author: mirsi -->


<!-- Date: Sun Feb 3 15:59:30 2002 -->
<pre>Summary of crash analysis (3.96 Mln events processed with 4.3.1) 
 
---------------------------------------------------------------------------- 
stream  N(crashes) N(FTO)       Comment 
------------------------------------------------------------------------------ 
A        8           7     many events with 100MBytes/ev ISL printout 
B       46          46     many events with 100MBytes/ev ISL printout 
G       26          26     many events with 100MBytes/ev ISL printout 
J        8           8     many events with 100MBytes/ev ISL printout 
------------------------------------------------------------------------------- 
FTO: "Farms Timeout" - event processing took more than 15 min 
 
The bottom line: 
 
- all but one crashes are due to the Farms "done timeout"caused by the  
  long processing time in SVX reconstruction code. 
- uncommented printout somewhere in TrackingObjects/src/TRYRun2SiStripSet.cc 
  generated gigabytes of output, which may indicate problem with the data. 
- corresponding experts are contacted 
</pre>
<!-- Author: murat -->


<!-- Date: Sun Feb 3 16:07:10 2002 -->
<pre>processing of the data with 4.3.1 ran into a few operational problems: 
 
- there are [over-populated] events, processing of which takes 
  more than 15 minutes on the Production Farm. All the time is spent 
  inside the SVX pattern recognition. Examples can be found,  
  in  
       fcdflnx1:/cdf/scratch/cdfprod0/4.3.1_d/Stream_J/crashed 
 
  directory. The code needs to be protected against such events ASAP. 
 
- because of unprotected 'couts' in TrackingObjects/src/TRYRun2SiStripSet.cc 
  the log files generated by the Production Farm reach 500MBytes in size 
  which causes the Farms nodes to collapse on these events. 
  In particular, there are zillions of  
 
   ISL chip Id -712 is invalid on output 
 
  lines in the log files. This also needs to be fixed ASAP. 
  Examples can be found in  
 
     fcdflnx1:/cdf/scratch/cdfprod0/4.3.1_d/Stream_J/logs 
 
  directory - take any log file with the size &gt; 100 MBytes. 
 
- As already noticed by Miro, PROD_PHYSICS_CDF v15 calibrations are missing  
  for at least following runs included into the good run list: 
 
136885 
136953 
136919 
136920 
136922 
136923 
136949 
136950 
136952 
</pre>
<!-- Author: murat -->
<!-- Comment: Sun Feb  3 16;29;35 comment by...Rob Snihur -->
<pre>Calibrations: It looks like I missed a block of runs
when I made the recent version.  I'll do that now.

I admit that I did not check all 711 goodruns 
(between 122621-138104) last time.
I just did right now, and those 9 are the only runs not at v15.
Of course, this will not catch runs which enter the goodrun
list after I copy-and-paste from the farms webpage
http://fnpcc.fnal.gov/run_info.php
Is this the definitive list which drives the farms,
or should I monitor something else?
</pre>


<!-- Comment: Mon Feb  4 12;26;42 comment by...murat  -->
<pre>PROD_PHYSICS_CDF also missing for run 138335

</pre>
<!-- Date: Sun Feb 3 17:05:45 2002 -->
<pre>The good run list is the definite list that the Farm is using for the processing. </pre>
<!-- Author: mirsi -->


<!-- Date: Mon Feb 4 09:07:34 2002 -->
<pre>================================ start of Monday shift 
- CDF-BUGS: 
     PR#1908: forwarded to Liz Sexton 
     PR#1909: forwarded to missing Et experts 
     PR#1910: crash of CDFSIM - attempt to reproduce 
- Farm:  
     waiting for the tracking experts to provide patches for 4.3.1</pre>
<!-- Author: murat -->
<!-- Comment: Mon Feb  4 12;10;35 comment by...murat -->
<pre>PR#1910: failed to reproduce


</pre>
<!-- Date: Mon Feb 4 11:49:49 2002 -->
<pre>Reconstruction continues with 4.3.1 on the following runs: 
 
Stream A - runs 137713 to 138335 
Stream B - runs 137713 to 138104 
Stream G - runs 137713 to 138104 
Stream J - runs 137713 to 138104 
 
The range is affected by the data availability on tape. 
</pre>
<!-- Author: mirsi -->




<!-- Date: Tue Feb 5 09:07:12 2002 -->
<pre>============================== start of the shift   
- CDF-BUGS:  first report: PR#1922  
- Farms: new core files from last night filled up fcdflnx1:/cdf/scratch by 100%</pre>
<!-- Author: murat -->


<!-- Date: Tue Feb 5 11:35:19 2002 -->
<pre>latest filesets on tape:  
----------------  
APHYSR: CA5255.4 2002 Feb 03 13:53CST  
BPHYSR: CA5275.4 2002 Feb 04 20:06CST  
CPHYSR: CA5040.0 2002 Jan 24 14:55CST  
DPHYSR: CA5017.0 2002 Jan 22 12:03CST  
EPHYSR: CA5290.4 2002 Feb 05 10:05CST  
FPHYSR: CA3202.5 2001 Mar 31 17:23CST  
GPHYSR: CA5288.4 2002 Feb 05 07:04CST  
HPHYSR: CA5289.4 2002 Feb 05 09:45CST  
IPHYSR: CA5044.4 2002 Jan 24 17:16CST  
JPHYSR: CA5287.4 2002 Feb 05 06:10CST  
</pre>
<!-- Author: murat -->


<!-- Date: Tue Feb 5 14:03:04 2002 -->
<pre>- building new version of the ProductionExe with Karlsruhe patches in  
   TrackingKal/TrackingKal/KalSiHitSet.hh    -r 1.11  
   TrackingKal/src/KalSiHitSet.cc            -r 1.13  
</pre>
<!-- Author: murat -->


<!-- Date: Tue Feb 5 17:34:14 2002 -->
<pre>everything prefixed with FCDFLNX1:/cdf/scratch/cdfprod0/4.3.1_d  
  
  
----------------------------------------------------------------  
Stream_J/crashed/137404.122319.crashed.jr0218bc.0042phys.15608.3  
----------------------------------------------------------------  
SiClusteringMod 1       27.800000+/-0.000000   52.655484+/-0.000000   27.800  
SiPatternRecMod 1        5.790000+/-0.000000    5.800094+/-0.000000    5.790  
  
-------------------------------------------------------------------------------  
Stream_B/crashed/137549.64546.crashed.br02194d.001aphys.15765.10  
-------------------------------------------------------------------------------  
SiClusteringMod 1       29.500000+/-0.000000   50.399400+/-0.000000   29.500  
SiPatternRecMod 1      106.700000+/-0.000000  107.847661+/-0.000000  106.700  
  
-------------------------------------------------------------------------------  
Stream_G/crashed/137484.62245.crashed.gr02190c.0021phys.15668.8  
-------------------------------------------------------------------------------  
SiClusteringMod 1       28.820000+/-0.000000   54.001663+/-0.000000   28.820  
SiPatternRecMod 1       47.290000+/-0.000000   62.488730+/-0.000000   47.290  
</pre>
<!-- Author: murat -->
<!-- Comment: Tue Feb  5 17;35;33 comment by...murat -->
<pre>this is timing for Si reco modules after applying Karlsruhe patch

</pre>

<!-- Date: Wed Feb 6 14:15:26 2002 -->
<pre>
The super-express stream has been restarted with 4.3.1 with calibration
version v15.

The first file being reconstructed is

          ar021c02.0002phys

</pre>
<!-- Author: Yen-Chu -->
<!-- Comment: Fri Feb  8 10;24;20 comment by...Rob Snihur -->
<pre>(Super) express production should use version 11 of the
calibrations!  Version 15 has the "correct" COTStage0 &
SVX calibrations, which are not yet available on the time-scale
(~30 min) required for express production.
See:
http://nucl02.fnal.gov/~snihur/calib/#Production
</pre>
<!-- Date: Thu Feb 7 11:22:44 2002 -->
<pre>---------------------- beginning of the shift  
-ProductionFarm: processed runs up to 138335.  
                 down for the memory upgrade  
- CDF-BUGS: today's reports start from PR#1939  
- Matt's patch to ISLD I/O code didn't help - code goes  
  into an infinite loop, waiting for the new patch  
- continue with the analysis of the new core files   
  from yesterday</pre>
<!-- Author: murat -->


<!-- Date: Thu Feb 7 14:22:43 2002 -->
<pre>- several crashes from Feb 06 analyzed, all - Farm "done shutdowns"  
- processing times with the recent patch - 100-250 sec/event  
- proceed with making a new patched executable - 4.3.1a - for the Farm</pre>
<!-- Author: murat -->


<!-- Date: Fri Feb 8 08:03:02 2002 -->
<pre>- 10k test with 4.3.1a (run 138046) succeded  
- MuonStubModule takes 1sec/event - experts contacted  
- will see if we could include latest patches to the   
  CES/CPR code  
</pre>
<!-- Author: murat -->


<!-- Date: Fri Feb 8 09:11:36 2002 -->
<pre>Farm is down till noon. We expect that we have all of the Farm operation then and we will proceed with data processing up to run 138424 (switch to 8x20). 
 
If we can get 4.3.1a by that time, we'd like to switch to that together with 8x20. 
</pre>
<!-- Author: mirsi -->


<!-- Date: Fri Feb 8 12:57:18 2002 -->
<pre>
The super-express stream has been restarted with 4.3.1 with calibration
  ProcessName set PROD_PHYSICS_CDF
  Version set 11
  parmSetName set PROD_PHYSICS_CDF11


The first file to be reconstructed is

          ar021c60.0001phys

Note that although ar021c61.0001phys was reconstructed properly, to avoid
confusion I renamed the produced file aa021c60.0001phys to be
aa021c61.0001phys.save and resubmitted it.

</pre>
<!-- Author: Yen-Chu -->



<!-- Date: Fri Feb 8 18:13:48 2002 -->
<pre>ProductionExe 4.3.1b tarball available on FCDFLNX1  
  
cdfsoft@fcdflnx1 4.3.1b]$ dir -l $PWD/*.gz -rw-r--r--   1 cdfsoft  cdfsoft  40010280 Feb  8 18:08 /cdf/code/cdfsoft/dist/cdfpatch/4.3.1b/ProductionExe_4.3.1b.tar.gz  
  
latest patches were tested on about 1000 events (3 jobs x 300 events each) of run 138046  
  
</pre>
<!-- Author: murat -->


<!-- Date: Sat Feb 9 07:58:16 2002 -->
<pre>=&gt; start of the day shift    
- The Farm finishes processing of 4x8 data    
looking at the log files copied over to FCDFLNX1 (which may not account for    
all the data):  after yesterday's memory upgrade and until now the Farm has    
processed :    
    
stream A :   0 files   - see processed runs up to 138244, asked Miro   
                                  to double check   
stream B : 249 files     
stream G :  89 files      
stream J :  34 files    - done (run 138424 processed)   
-------------------------    
- if we take 2500 events/file, then 270 files x 2,500 = 675,000 events    
- no crashes so far    
    
</pre>
<!-- Author: murat -->


<!-- Date: Sun Feb 10 09:44:12 2002 -->
<pre>---- start of the shift      
-Farm: - finished with 4x8 for the streams A,G,J (no more crashes)      
            - stream B: 36 new crashes, all - "15min timeouts" (event       
               processing time exceeded 15 min, job was killed by the Farm       
               control software)      
            - processed so far with 4.3.1:       
                  stream A :   975,250      
                  stream B : 8,694,214      
                  stream G : 3,323,887      
                  stream J:   1,343,785      
                  --------------      
                   total  :   14,337,136    
   
            - reported crashes: 245  (appx 1 per 58,500 processed events)   
      
                                                                 </pre>
<!-- Author: murat -->


<!-- Date: Sun Feb 10 10:12:21 2002 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-4&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-4  width='300' align='top'></td><td>15min Farm done timeouts: distribution for the event size</a></td></tr></table>
<!-- Author: murat -->


<!-- Date: Sun Feb 10 14:13:16 2002 -->
<pre>- got 36 new crashes, all in stream B (the rest streams are done?)  
- all are 15min timeouts, RAW events are large ( &gt;500 KBytes)  
- ran on 2 events on FCDFLNX1:  
  
** Feb  9 12:09 138236.193360.crashed.br021bfc.0036phys.16321.7  
- event size: 521395  
- processing time:  
 SiClusteringMod 1    26.080000+/-0.000000   30.750074+/-0.000000   26.080  
 SiPatternRecMod 1   243.740000+/-0.000000  243.769868+/-0.000000  243.740  
- memory consumption (by top): 380M  
  
** Feb  9 09:35 138240.313370.crashed.br021c00.005ephys.16329.4  
- event size : 726376  
- processing times:   
 SiClusteringMod 1    26.390000+/-0.000000   30.881654+/-0.000000   26.390  
 SiPatternRecMod 1   998.740000+/-0.000000  998.858888+/-0.000000  998.740  
  
- the 1st event has been processed in &lt; 5min,   
  why did it take the Farm node 15 min?  - swap?   
  
- lots of "GOOD ELECTRON" printouts - somebody left uncommented "cout"  
  in the validation module  
</pre>
<!-- Author: murat -->


<!-- Date: Mon Feb 11 14:13:15 2002 -->
<pre>- some info on the Production Farm timing (from Miro)  
  the bottom line is that for the recent runs the processing  
  times vary, depending on the node, from 10 to 20 sec/event  
  with roughly 80% of this time spent in SVX reconstruction  
  code ( 20 sec/event is 4 times above the original specs)  
 +-------------------------------------------------+-------------------+---------+  
 | time/job| fname             |  ndname   | time/event  | lostevnts | num_event |  
 +--------------------------------------------+-------------------+-----------+---  
 |   41219 | br021bfc.0001phys |  fncdf54  |      13.05  |         0 |      3158 |  
 |   68667 | br021bfc.0006phys |  fncdf55  |      22.03  |         1 |      3117 |  
 |   30996 | br021bfc.000aphys |  fncdf149 |      10.23  |         0 |      3031 |  
 |   29952 | br021bfc.000ephys |  fncdf148 |      10.86  |         0 |      2758 |  
 |   71406 | br021bfc.0012phys |  fncdf51  |      24.48  |         2 |      2917 |  
 |   27562 | br021bfc.0016phys |  fncdf145 |       9.68  |         0 |      2846 |  
 |   30125 | br021bfc.001aphys |  fncdf144 |      10.47  |         0 |      2877 |  
 |   37434 | br021bfc.001ephys |  fncdf63  |      13.20  |         0 |      2836 |  
 |  121096 | br021bfc.0022phys |  fncdf21  |      40.79  |         2 |      2969 |  
 |  181194 | br021bfc.0026phys |  fncdf6   |       NULL  |         2 |         0 |  
 |   88597 | br021bfc.002aphys |  fncdf4   |      29.52  |         1 |      3001 |  
 |  181310 | br021bfc.002ephys |  fncdf5   |      60.44  |         3 |      3000 |  
 |   59978 | br021bfc.0032phys |  fncdf2   |      19.89  |         0 |      3015 |  
 |   79444 | br021bfc.0036phys |  fncdf3   |      29.59  |         2 |      2685 |  
 |  136153 | br021bfc.003aphys |  fncdf1   |      45.57  |         2 |      2988 |  
 |  116090 | br021bfc.003ephys |  fncdf9   |      38.11  |         1 |      3046 |  
 |   32370 | br021bfc.0043phys |  fncdf103 |      10.99  |         0 |      2945 |  
 |   30748 | br021bfc.0047phys |  fncdf102 |      11.30  |         0 |      2721 |  
 |   30202 | br021bfc.004bphys |  fncdf105 |      11.01  |         0 |      2742 |  
 |   48996 | br021bfc.004fphys |  fncdf104 |      15.87  |         1 |      3087 |  
 |   35190 | br021bfc.0054phys |  fncdf107 |      12.11  |         0 |      2906 |  
 |   25455 | br021bfc.0058phys |  fncdf106 |      12.91  |         0 |      1972 |  
 +-------+-------------------+-----------+-------------------------------------+  
</pre>
<!-- Author: murat -->


<!-- Date: Mon Feb 11 14:23:25 2002 -->
<pre>To explain the numbers a little bit - nodes from fncdf1 up to fncdf50 are PIII/500MHz, 512MB memory. Fncdf51-90 are PIII/800MHz, 512MB memory. 
 
Fncdf91-154 are PIII 1GHz, 1GB memory. Times in the time/event column should be divided by (lostevnts+1) in order to obtain real processing time without multiple reprocessings. 
 
The table then should read: 
+-------------------------------------------------+-------------------+----------+---------------------------------------------------------------------------+-----------+ 
| unix_timestamp(ts_done)-unix_timestamp(ts_exec) | fname             | ndname   | (unix_timestamp(ts_done)-unix_timestamp(ts_exec))/num_event/(lostevnts+1) | num_event | 
+-------------------------------------------------+-------------------+----------+---------------------------------------------------------------------------+-----------+ 
|                                           41219 | br021bfc.0001phys | fncdf54  |                                                                   13.0522 |      3158 | 
|                                           68667 | br021bfc.0006phys | fncdf55  |                                                                   11.0149 |      3117 | 
|                                           30996 | br021bfc.000aphys | fncdf149 |                                                                   10.2263 |      3031 | 
|                                           29952 | br021bfc.000ephys | fncdf148 |                                                                   10.8600 |      2758 | 
|                                           71406 | br021bfc.0012phys | fncdf51  |                                                                    8.1598 |      2917 | 
|                                           27562 | br021bfc.0016phys | fncdf145 |                                                                    9.6845 |      2846 | 
|                                           30125 | br021bfc.001aphys | fncdf144 |                                                                   10.4710 |      2877 | 
|                                           37434 | br021bfc.001ephys | fncdf63  |                                                                   13.1996 |      2836 | 
|                                          121096 | br021bfc.0022phys | fncdf21  |                                                                   13.5956 |      2969 | 
|                                          181194 | br021bfc.0026phys | fncdf6   |                                                                      NULL |         0 | 
|                                           88597 | br021bfc.002aphys | fncdf4   |                                                                   14.7612 |      3001 | 
|                                          181310 | br021bfc.002ephys | fncdf5   |                                                                   15.1092 |      3000 | 
|                                           59978 | br021bfc.0032phys | fncdf2   |                                                                   19.8932 |      3015 | 
|                                           79444 | br021bfc.0036phys | fncdf3   |                                                                    9.8627 |      2685 | 
|                                          136153 | br021bfc.003aphys | fncdf1   |                                                                   15.1889 |      2988 | 
|                                          116090 | br021bfc.003ephys | fncdf9   |                                                                   19.0561 |      3046 | 
|                                           32370 | br021bfc.0043phys | fncdf103 |                                                                   10.9915 |      2945 | 
|                                           30748 | br021bfc.0047phys | fncdf102 |                                                                   11.3003 |      2721 | 
|                                           30202 | br021bfc.004bphys | fncdf105 |                                                                   11.0146 |      2742 | 
|                                           48996 | br021bfc.004fphys | fncdf104 |                                                                    7.9359 |      3087 | 
|                                           35190 | br021bfc.0054phys | fncdf107 |                                                                   12.1094 |      2906 | 
|                                           25455 | br021bfc.0058phys | fncdf106 |                                                                   12.9082 |      1972 | 
+-------------------------------------------------+-------------------+----------+---------------------------------------------------------------------------+-----------+ 
Since the times are more than 10s on a 1GHZ machine, that translates to more than 20s on a PIII 1GHz, which is a correct statement that it is 4 times more than the design. 
 
 
</pre>
<!-- Author: mirsi -->


<!-- Date: Mon Feb 11 14:29:48 2002 -->
<pre>Sorry, once again: 
+----------------+-------------------+----------+----------------+-----------+ 
| execution time | fname             | ndname   | time per event | num_event | 
+----------------+-------------------+----------+----------------+-----------+ 
|          41219 | br021bfc.0001phys | fncdf54  |        13.0522 |      3158 | 
|          68667 | br021bfc.0006phys | fncdf55  |        11.0149 |      3117 | 
|          30996 | br021bfc.000aphys | fncdf149 |        10.2263 |      3031 | 
|          29952 | br021bfc.000ephys | fncdf148 |        10.8600 |      2758 | 
|          71406 | br021bfc.0012phys | fncdf51  |         8.1598 |      2917 | 
|          27562 | br021bfc.0016phys | fncdf145 |         9.6845 |      2846 | 
|          30125 | br021bfc.001aphys | fncdf144 |        10.4710 |      2877 | 
|          37434 | br021bfc.001ephys | fncdf63  |        13.1996 |      2836 | 
|         121096 | br021bfc.0022phys | fncdf21  |        13.5956 |      2969 | 
|          88597 | br021bfc.002aphys | fncdf4   |        14.7612 |      3001 | 
|         181310 | br021bfc.002ephys | fncdf5   |        15.1092 |      3000 | 
|          59978 | br021bfc.0032phys | fncdf2   |        19.8932 |      3015 | 
|          79444 | br021bfc.0036phys | fncdf3   |         9.8627 |      2685 | 
|         136153 | br021bfc.003aphys | fncdf1   |        15.1889 |      2988 | 
|         116090 | br021bfc.003ephys | fncdf9   |        19.0561 |      3046 | 
|          32370 | br021bfc.0043phys | fncdf103 |        10.9915 |      2945 | 
|          30748 | br021bfc.0047phys | fncdf102 |        11.3003 |      2721 | 
|          30202 | br021bfc.004bphys | fncdf105 |        11.0146 |      2742 | 
|          48996 | br021bfc.004fphys | fncdf104 |         7.9359 |      3087 | 
|          35190 | br021bfc.0054phys | fncdf107 |        12.1094 |      2906 | 
|          25455 | br021bfc.0058phys | fncdf106 |        12.9082 |      1972 | 
+----------------+-------------------+----------+----------------+-----------+ 
</pre>
<!-- Author: mirsi -->


<!-- Date: Tue Feb 12 18:09:27 2002 -->
<pre>Here is the list of runs and the corresponding module where reconstruction has takes nore than 3 times the average time per event: 
The first value is the time value for the given run and the second value is the average for all of the runs. 
 
 
This statistics has been created for stream B, 4.3.1b. 
 
Run 137024 module CESQModule is above 3x average: 0.0165194092933 0.00509427925171 
Run 137025 module CESQModule is above 3x average: 0.0165712043939 0.00509427925171 
Run 137026 module CESQModule is above 3x average: 0.0166599435256 0.00509427925171 
Run 137027 module CESQModule is above 3x average: 0.0167131344045 0.00509427925171 
Run 137024 module CentralStripClu is above 3x average: 0.106414752572 0.0225424393652 
Run 137025 module CentralStripClu is above 3x average: 0.106794821499 0.0225424393652 
Run 137026 module CentralStripClu is above 3x average: 0.107603872529 0.0225424393652 
Run 137027 module CentralStripClu is above 3x average: 0.107381076256 0.0225424393652 
Run 137990 module MuonStubModule is above 3x average: 1.48883069977 0.483868473073 
Run 137991 module MuonStubModule is above 3x average: 1.50278632736 0.483868473073 
Run 137992 module MuonStubModule is above 3x average: 1.49163758906 0.483868473073 
Run 137994 module MuonStubModule is above 3x average: 1.48892216048 0.483868473073 
Run 138021 module MuonStubModule is above 3x average: 1.48660876135 0.483868473073 
Run 138041 module MuonStubModule is above 3x average: 1.49405636554 0.483868473073 
Run 138044 module MuonStubModule is above 3x average: 1.48930756114 0.483868473073 
Run 138045 module MuonStubModule is above 3x average: 1.49490008065 0.483868473073 
Run 138046 module MuonStubModule is above 3x average: 1.49620039447 0.483868473073 
Run 138050 module MuonStubModule is above 3x average: 1.49068877593 0.483868473073 
Run 138052 module MuonStubModule is above 3x average: 1.49821979111 0.483868473073 
Run 138076 module MuonStubModule is above 3x average: 1.51822879224 0.483868473073 
Run 138077 module MuonStubModule is above 3x average: 1.51543654644 0.483868473073 
Run 138081 module MuonStubModule is above 3x average: 1.51172443176 0.483868473073 
Run 138082 module MuonStubModule is above 3x average: 1.52240192412 0.483868473073 
Run 138100 module MuonStubModule is above 3x average: 1.52030802892 0.483868473073 
Run 138104 module MuonStubModule is above 3x average: 1.51486525073 0.483868473073 
Run 137024 module PlugStripCluste is above 3x average: 0.0181458653048 0.00329876861614 
Run 137025 module PlugStripCluste is above 3x average: 0.0180886622205 0.00329876861614 
Run 137026 module PlugStripCluste is above 3x average: 0.0183027430415 0.00329876861614 
Run 137027 module PlugStripCluste is above 3x average: 0.0183248150982 0.00329876861614 
Run 138236 module SiPatternRecMod is above 3x average: 14.212884766 2.25726744684 
Run 138239 module SiPatternRecMod is above 3x average: 13.6344796536 2.25726744684 
Run 138240 module SiPatternRecMod is above 3x average: 13.1830294013 2.25726744684 
Run 138241 module SiPatternRecMod is above 3x average: 13.8678470596 2.25726744684 
Run 138242 module SiPatternRecMod is above 3x average: 14.0338048584 2.25726744684 
Run 138243 module SiPatternRecMod is above 3x average: 12.8385694703 2.25726744684 
Run 138244 module SiPatternRecMod is above 3x average: 10.8076177421 2.25726744684 
</pre>
<!-- Author: mirsi -->


<!-- Date: Tue Feb 12 23:35:16 2002 -->
<pre>- started 10K test with 4.4.0pre2 on FCDFLNX1. Use data  
  from run=138046 (files gr021b3e.0001phys, gr021b3e.002ephys, 
  gr021b3e.0094phys)</pre>
<!-- Author: murat -->


<!-- Date: Thu Feb 14 08:21:36 2002 -->
<pre>Yesterday evening Farm processed all of the 4x8. I've flushed all of the concatenated data to tapes. </pre>
<!-- Author: mirsi -->


<!-- Date: Fri Feb 15 15:15:43 2002 -->
<pre>Beginning of the shift. 
 
Debugging runMC for 4.4.0.</pre>
<!-- Author: lena -->


<!-- Date: Sat Feb 16 08:54:22 2002 -->
<pre>Debugging runMC - continuing. 
For 4.4.0 runMC will be used with herwig generator 
instead of pythia. Tcl file is provided by Tony.</pre>
<!-- Author: lena -->


<!-- Date: Sat Feb 16 15:08:29 2002 -->
<pre>working on 4.4.0 build errors.</pre>
<!-- Author: Lena -->


<!-- Date: Sun Feb 17 08:57:47 2002 -->
<pre>starting 10K test with 4.4.0</pre>
<!-- Author: lena -->
<!-- Comment: Sun Feb 17 09;15;58 comment by...murat -->
<pre>created entry for Lena in fcdflnx1:~cdfprod0/.k5login</pre>

<!-- Comment: Sun Feb 17 09;51;00 comment by...lena -->
<pre>copied the following data file on fcdflnx1 for 10K test:
ar02200e.0147phys br02200e.0001phys gr02200e.0001phys 
from  run # 139278</pre>

<!-- Comment: Sun Feb 17 11;29;29 comment by...lena -->
<pre>10K test ended with three core dumps. Debugging. </pre>

<!-- Comment: Sun Feb 17 11;40;01 comment by...lena -->
<pre>All core dumps are in PhotonValidation.
Restarting 10K test with disabled PhotonValidation.</pre>

<!-- Comment: Sun Feb 17 12;15;26 comment by...lena -->
<pre>Restarting 10K test with disabled ElectronValidation
which also core dump.</pre>


<!-- Date: Sun Feb 17 16:23:01 2002 -->
<pre>The Farm has finished reconstructin all of the events up to run 138868 and is ready to continue. 
 
File er021d72.0001phys was not possible to reconstruct since it crashed mroe that 4 times when trying to reconstruct. 
</pre>
<!-- Author: mirsi -->


<!-- Date: Mon Feb 18 11:58:51 2002 -->
<pre>Patched 4.4.0pre3: 
Production/setup_monitors.tcl 
disabled ElectronValidation and PhotonValidation</pre>
<!-- Author: lena -->


<!-- Date: Tue Feb 19 11:51:51 2002 -->
<pre>working on FarmTools - log analizing script</pre>
<!-- Author: lena -->


<!-- Date: Thu Feb 21 12:03:49 2002 -->
<pre>preparing another patch for 4.4.0 ( bug in simulation: 
passing null pointer )</pre>
<!-- Author: Lena -->


<!-- Date: Sat Feb 23 17:45:22 2002 -->
<pre>Preparing 4.3.1c patch that will become 4.3.2 release. 
- Add SvxDaqObjects/SvxDaqObjects/SiEventPedCalculator.hh file 
- Add TauObjects that are required by patched Tau 
- ExoticMods and TopMods libraries must be added to link 
  ProductionExe 
 
Running 10K test on files: 
gr01f5f5.00b7phys 
br021b3e.0001phys  
ar02200e.0147phys 
different streams, different runs... 
</pre>
<!-- Author: Fedor -->


<!-- Date: Sat Feb 23 21:56:37 2002 -->
<pre>4.3.2 10K test is still running... So far so good. 
Looking into output I note the following: 
 
1- HUGE amount of ERLOG-w messages. Mostly from silicon. 
  Probably it is not dangerous, but so annoying! 
2- "GOOD ELECTRON" printout from PhotonValidation - looks 
  like debugging printout still uncommented 
3- ERLOG-e messages like 
%ERLOG-e [OUT_NOT_IN]: 
         /home/cdfsoft/dist/packages/TofAlgs/V00-00-44/src/TofGeometricHitFinder.cc 
        : 245  Track came out of side  -1  of bar  37 but did not go in. 
         GlobalLibraryLogger Helix::getPhiAtR(rho)() 23-Feb-2002 14:43:38 
4- ERLOG-s messages like 
%ERLOG-s CalDataMaker: /cdf/scratch/cdfprod0/4.3.1c/Calor/src/CalDataMaker.cc : 
         502   unpack HATD bank : more than 8 hits =&gt; Data probably corrupted 
         GlobalLibraryLogger Helix::getPhiAtR(rho)() 23-Feb-2002 15:50:14 
5- ERLOG-S messages like 
%ERLOG-S CT_TrackingModule: 
         /cdf/scratch/cdfprod0/4.3.1c/TrackingMods/src/CT_TrackingModule.cc : 
         818  Skipping event with high hit multiplicity.  13937 
          hits in super layer  7 CT_TrackingModule 23-Feb-2002 14:21:01 
 
</pre>
<!-- Author: Fedor -->


<!-- Date: Sun Feb 24 01:39:48 2002 -->
<pre>10338 events of different streams and runs (see above) were 
processed without crash with 4.3.1c patch. 
Output files are readable by AC++Dump. 
This patch as ready for declaring as a new release: 
code management has been requested. 
I failed to add patch file to Release/patches because 
"ratnikov" is not authorized to write into Release.  
 
10K test log statistics (warning and higher) is the following: 
 
   29 Chip Invalid error   -w GlobalLibraryLog TRYRun2SiStripSe   1242*    1242 
   30 Chip Side Error      -w GlobalLibraryLog TRYRun2SiStripSe  18228*   18228 
   31 Chip Side Error      -w GlobalLibraryLog TRYRun2SiStripSe  44080*   44080 
   32 DHInputDataList-&gt;    -w GlobalLibraryLog Helix::getPhiAtR      1        1 
   33 DaqInfoSet:chip stat -w GlobalLibraryLog SiVrbInfoSet::se 369669*  369669 
   34 First word in HDI bl -w GlobalLibraryLog TRYRun2SiStripSe  12550*   12550 
   35 First word in HDI bl -w GlobalLibraryLog TRYRun2SiStripSe  84374*   84374 
   36 Fit did not converge -w GlobalLibraryLog CentralMuonStub2      1        1 
   37 Hit order error      -w GlobalLibraryLog CMU_PairSorter::     10       10 
   38 JetCluAlgorithm      -w GlobalLibraryLog CMU_PairSorter::      7        7 
   39 JetCluAlgorithm      -w GlobalLibraryLog Helix::getPhiAtR   7184*    7184 
   40 JetCluModule         -w JetCluModule                         828*     828 
   41 JetCluModule         -w JetCluModule-con                     828*     828 
   42 JetCluModule         -w JetCluModule-con                     828*     828 
   43 JetMetValidation     -w JetMetValidation                     931*     931 
   44 No Track-Plane Inter -w ClusterMonitor   ClusterMonitor::     42*      42 
   45 No Track-Plane Inter -w TrackMonitor     TrackMonitor::ca     42*      42 
   46 No Track-Plane Inter -w TrackMonitor     TrackMonitor::fi     42*      42 
   47 Pes2dCluster collect -w GlobalLibraryLog CalDrivenCdfEmOb   1243*    1243 
   48 RCP put failure      -w CentralStripClus                       1        1 
   49 RCP put failure      -w CprClusterModule                       1        1 
   50 RCP put failure      -w CprClusterMon                          1        1 
   51 RCP put failure      -w GeometryManager                        2        2 
   52 RCP put failure      -w HepRootManager                         2        2 
   53 RCP put failure      -w PhotonValidation                       1        1 
   54 RCP put failure      -w SiClusteringModu                       1        1 
   55 Readout errors detec -w SiClusteringModu                   10338*   10338 
   56 TRKQ_ESTIM_NEG       -w GlobalLibraryLog TRKQCotCorrectio    194*     194 
   57 TRKQ_MATRIX_INV      -w GlobalLibraryLog TRKQCotCorrectio    194*     194 
   58 There are no more fi -w DHInput                                1        1 
   59 TofAbstAssoc::HUGE_D -w GlobalLibraryLog Helix::getPhiAtR      1        1 
   60 TofIterativeT0Calcul -w GlobalLibraryLog CMU_PairSorter::      2        2 
   61 TofIterativeT0Calcul -w GlobalLibraryLog Helix::getPhiAtR    652*     652 
   62 Too many hits in reg -w MuonStubModule                        19*      19 
   63 Unknown record type  -w CdfEmObjectModul CdfEmObjectModul      1        1 
   64 Unrecognized ISLD wo -w GlobalLibraryLog TRYRun2SiStripSe    188*     188 
   65 Unrecognized SIXD wo -w GlobalLibraryLog TRYRun2SiStripSe  46916*   46916 
   66 [SCALED_NOISE_IS_0]  -w GlobalLibraryLog                     102*     102 
   67 [SCALED_NOISE_IS_0]  -w GlobalLibraryLog CMU_PairSorter::    173*     173 
   68 [SCALED_NOISE_IS_0]  -w GlobalLibraryLog Helix::getPhiAtR 497095*  497095 
   69 [rho out of bounds]  -w GlobalLibraryLog Helix::getL2DAtR  19020*   19020 
   70 [rho out of bounds]  -w GlobalLibraryLog Helix::getPhiAtR  59610*   59610 
   71 [workaround illegal  -w GlobalLibraryLog Helix::getL2DAtR 120389*  120389 
   72 divide by 0          -w CMU_DtoEModule                        28*      28 
   73 grid seed error      -w GlobalLibraryLog PEMChiSquare::Dp     54*      54 
   74 jetReclustering fail -w TopEventModule                       827*     827 
   75 [OUT_NOT_IN]         -e GlobalLibraryLog Helix::getPhiAtR      4        4 
   76 CalDataMaker         -s GlobalLibraryLog Helix::getPhiAtR      1        1 
   77 [PlugStripClusterMak -s GlobalLibraryLog Helix::getPhiAtR      2        2 
   78 CT_TrackingModule    -S CT_TrackingModul                      20*      20 
 
IMHO such a HUUUUUGE number ammount of warnings indicates that they are actually 
just information messages. And must be qualifyed accordingly. Otherwise 
several true warnings if any will be definitely lost in this heap... 
 
</pre>
<!-- Author: Fedor -->


<!-- Date: Mon Feb 25 10:48:56 2002 -->
<pre>Answer about "%ERLOG-s CalDataMaker" error from Beate: 
this is a problem which is not yet understood. In principle more than 8  
hits should NEVER happen. There is someone looking into it We would like  
to keep the severe error messsage though. The data that are stored are  
still useful though. So, I will weaken the text message.</pre>
<!-- Author: Fedor -->


<!-- Date: Mon Feb 25 19:07:19 2002 -->
<pre>4.3.2 status 
Beate verified that message is harmless and commented it out 
Bob cleaned printout in PhotonValidation 
I've just started another 10K test, but it must be OK as far as 
nothing except printout has been changed scince previous test 
Request code management to generate release.</pre>
<!-- Author: Fedor -->


<!-- Date: Wed Feb 27 15:15:55 2002 -->
<pre>Start 100K test of 4.3.1b with new tcl: 
 
the only difference with the standard 4.3.1b tarball is that it is using 
Production/setup_trkReco.tcl revision 1.31, which is claimed to be the  
following: 
 
---------------------------- 
revision 1.31 
date: 2002/02/18 20:40:51;  author: herndon;  state: Exp;  lines: +5 -4 
Production tcl for kal tracking 
--------------------------- 
 
</pre>
<!-- Author: Fedor -->


<!-- Date: Wed Feb 27 16:50:29 2002 -->
<pre>Running 10K test for 4.3.2pre1 on fcdflnx1. See data files above.</pre>
<!-- Author: Fedor -->


<!-- Date: Thu Feb 28 00:53:15 2002 -->
<pre>4.3.1pre1 10K test was reproduced.Ran without crashes. 
Preparing tarball to run 100K test on farm... 
 
Warnings/error summary is the following: 
 
   28 Chip Invalid error   -w GlobalLibraryLog TRYRun2SiStripSe   1234*    1234 
   29 Chip Side Error      -w GlobalLibraryLog TRYRun2SiStripSe  18228*   18228 
   30 Chip Side Error      -w GlobalLibraryLog TRYRun2SiStripSe  42664*   42664 
   31 DHInputDataList-&gt;    -w GlobalLibraryLog Helix::getPhiAtR      1        1 
   32 DaqInfoSet:chip stat -w GlobalLibraryLog SiVrbInfoSet::se 330570*  330570 
   33 First word in HDI bl -w GlobalLibraryLog TRYRun2SiStripSe  12502*   12502 
   34 First word in HDI bl -w GlobalLibraryLog TRYRun2SiStripSe  69124*   69124 
   35 Hit order error      -w GlobalLibraryLog CMU_PairSorter::     10       10 
   36 JetCluAlgorithm      -w GlobalLibraryLog CMU_PairSorter::      7        7 
   37 JetCluAlgorithm      -w GlobalLibraryLog Helix::getPhiAtR   4313*    4313 
   38 JetCluModule         -w JetCluModule                          55*      55 
   39 JetCluModule         -w JetCluModule-con                      55*      55 
   40 JetCluModule         -w JetCluModule-con                      55*      55 
   41 No Track-Plane Inter -w ClusterMonitor   ClusterMonitor::     27*      27 
   42 No Track-Plane Inter -w TrackMonitor     TrackMonitor::ca     27*      27 
   43 No Track-Plane Inter -w TrackMonitor     TrackMonitor::fi     27*      27 
   44 Pes2dCluster collect -w GlobalLibraryLog CalDrivenCdfEmOb   1202*    1202 
   45 Readout errors detec -w SiClusteringModu                    6117*    6117 
   46 TRKQ_ESTIM_NEG       -w GlobalLibraryLog TRKQCotCorrectio    181*     181 
   47 TRKQ_MATRIX_INV      -w GlobalLibraryLog TRKQCotCorrectio    181*     181 
   48 There are no more fi -w DHInput                                1        1 
   49 TofAbstAssoc::HUGE_D -w GlobalLibraryLog Helix::getPhiAtR      1        1 
   50 TofIterativeT0Calcul -w GlobalLibraryLog CMU_PairSorter::      2        2 
   51 TofIterativeT0Calcul -w GlobalLibraryLog Helix::getPhiAtR    324*     324 
   52 Too many hits in reg -w MuonStubModule                        16*      16 
   53 Unknown record type  -w CdfEmObjectModul CdfEmObjectModul      1        1 
   54 Unrecognized ISLD wo -w GlobalLibraryLog TRYRun2SiStripSe      4        4 
   55 Unrecognized SIXD wo -w GlobalLibraryLog TRYRun2SiStripSe  45588*   45588 
   56 [SCALED_NOISE_IS_0]  -w GlobalLibraryLog                     102*     102 
   57 [SCALED_NOISE_IS_0]  -w GlobalLibraryLog CMU_PairSorter::    173*     173 
   58 [SCALED_NOISE_IS_0]  -w GlobalLibraryLog Helix::getPhiAtR 482892*  482892 
   59 [rho out of bounds]  -w GlobalLibraryLog Helix::getL2DAtR  12095*   12095 
   60 [rho out of bounds]  -w GlobalLibraryLog Helix::getPhiAtR  37748*   37748 
   61 [workaround illegal  -w GlobalLibraryLog Helix::getL2DAtR  76274*   76274 
   62 divide by 0          -w CMU_DtoEModule                        13*      13 
   63 grid seed error      -w GlobalLibraryLog PEMChiSquare::Dp     30*      30 
   64 jetReclustering fail -w TopEventModule                        58*      58 
   65 [OUT_NOT_IN]         -e GlobalLibraryLog Helix::getPhiAtR      4        4 
   66 CT_TrackingModule    -S CT_TrackingModul                      12*      12 
</pre>
<!-- Author: Fedor -->


<!-- Date: Thu Feb 28 00:56:26 2002 -->
<pre>4.3.1b_kal 100K test passed on farm. Single crash in stream E - 
will study in the morning...</pre>
<!-- Author: Fedor -->

<!-- Date: Thu Feb 28 12:04:23 2002 -->
<pre>
The super-express stream was restarted with 4.4.0.

The first file reconstructed by 4.4.0 was

          aa0221fe.0001phys

</pre>
<!-- Author: Yen-Chu -->



<!-- Date: Sun Mar 3 12:11:16 2002 -->
<pre>Farm test of patched 4.3.2pre1 is completed on the farm 
successfully. 
It is original 4.3.2pre1 tag + patches: 
#------------------------------------------------------------------------------ 
#   Avoid filtering by TopEventModule  
#------------------------------------------------------------------------------ 
Production/setup_path.tcl -r 1.31 
Production/setup_output.tcl -r 1.32 
Production/setup_input.tcl -r 1.17 
 
#------------------------------------------------------------------------------ 
# Clean up of warning messages 
#------------------------------------------------------------------------------ 
Electron/src/CalDrivenCdfEmObject.cc -r 1.50 
 
Trajectory/src/Helix.cc -r 1.12 
 
The following amount of error messages is generated  
 
%ERLOG-S CT_TrackingModule: 3922 
%ERLOG-s [PlugStripMaker]: 18 
%ERLOG-e [OUT_NOT_IN]: 368 
%ERLOG-e [[IN_NOT_OUT]: 70 
%ERLOG-e [NONPHYS_TRACK_COVARIANCE]: 22 
 
They all are well known. 
 
Remaining problem: 
MuonValidation filters events out from the monitoring 
path. Downstream TauValidation and TauTriggerValidation 
lack these events. 
</pre>
<!-- Author: Fedor -->

<!-- Date: Thu Mar 7 11:12:43 2002 -->
<pre>
The super-express stream has been restarted after today's fcdfsgi2
maintanence.

There was a file, ar022419.0001phys, got interupted in reconstruction.
Resubmitted.

According to the job report this file took a long time to reconstruct and
was not finished before the shutdown.

Job </cdf/data01/stream_a/.express_yc/lsf_job/ar022419.0001phys.tcl.csh>
was submitted from host <fcdfsgi2> by user <chenyc>.
Job was executed on host(s) <fcdfsgi2>, in queue <esap>, as user <chenyc>.
</cdf/home/chenyc> was used as the home directory.
</cdf/data01/stream_a/.express_yc> was used as the working directory.
Started at Tue Mar  5 22:20:44 2002
Results reported at Thu Mar  7 10:28:18 2002

The run number is 140313 which is not in the good run list.

</pre>
<!-- Author: Yen-Chu -->


<!-- Date: Thu Mar 7 13:12:41 2002 -->
<pre>

It is quiet time, no data coming in.

The super-express stream has been restarted again but with 4.3.2.

'4.3.2' is the current version used on the production farm.

The last file reconstructed by 4.4.0 was ar0224c3.0001phys.

</pre>
<!-- Author: Yen-Chu -->