<!-- Title: 08:39:03  Thu Mar 21 2002 -->
<!-- Crewchief: Andy Beretvas -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>The current status of the 10 K test: 
The problem with invalid command name "SAS3DPVFinder" 
has been fixed by Liz Sexton. However a new problem has appeared 
TRYGenPrimaryVertexSet was unable to read from OBSP/OBSV! 
 
In addition the problem with stnmaker.exe which causes a core dump on event 506( for the br set) still remains. 
 
Some progress on gcc conversion now only 95 Modules have problems. Some code like hepToyz and EdmUtilities now works. 
</pre>

<!-- Date: Thu Mar 21 12:05:36 2002 -->
<pre>

The super-express stream has now six allowed processes in the queue, which
means the CPU power is doubled.

At the moment the files in the look area are either reconstructed or
reconstructing, except for the files smaller than 1 MB.

For bookkeeping purpose the name of the reconstructed files are stored in
the following file,

   /cdf/data01/stream_a/reconstructed_files.

</pre>
<!-- Author: Yen-Chu -->



<!-- Date: Sat Mar 23 11:17:36 2002 -->
<pre>Copied 4.3.2a exe: 
 
rcp /cdf/code/cdfsoft/dist/cdfpatch/4.3.2a/ProductionExe_4.3.2a.tar.gz  
 
cdfprod0@cdffarm1:/home/cdfprod0/thread/executables/ 
 
 
 
Started  Test1 according to miro's instruction: 
 
"The run that I know is unbiased Si run - 138676. I would  
suggest that you run this test on good runs: 
 
138671,138672,138673,138674,138676,138677,140127,140819 
 
as we did not run on these so far, since there were at the  
time missing calibrations as well. This should be quite 
sufficient, as it is around 700k events in total." 
 
I have verified that version 15 of calibrations in 4.3.2a and  
the calibrations seem to exist for the runs above.</pre>
<!-- Author: rick st. denis -->
<!-- Comment: Sat Mar 23 11;55;22 comment by...rob snihur -->
<pre>Version 15 of calibrations exists up to and including goodrun 141218.  Waiting on SVX calibrations for runs thereafter.</pre>

<!-- Date: Mon Mar 25 02:57:27 2002 -->
<pre>

The super-express stream is restarted with 4.3.2a. The first file
reconstructed with it is

    ar0228d4.0001phys

the corresponding run number is 141524.

The change is to fix a rare bug in the TopEventModule. Which when happened
causes reading problem in that production file.

</pre>
<!-- Author: Yen-Chu -->



<!-- Date: Wed Apr 10 17:27:59 2002 -->
<pre>Hi Yen Chu,  
  
 you can pick up the tarball with 4.5.0pre3a (to be used for 100K test)  
 on FCDFLNX1. -best, Pasha  
  
 fcdflnx1:/cdf/code/cdfsoft/dist/cdfpatch/4.5.0pre3a:  
   total 46102  
   drwxr-xr-x  15 cdfsoft  cdfsoft       545 Apr 10 16:54 .  
   drwxr-xr-x  20 cdfsoft  cdfsoft      1887 Apr 10 16:31 ..  
   -rw-r--r--   1 cdfsoft  cdfsoft        10 Apr 10 15:08 .base_release  
   drwxr-xr-x  10 cdfsoft  cdfsoft       242 Apr 10 15:09 ExoticMods  
   drwxr-xr-x  10 cdfsoft  cdfsoft       233 Apr 10 15:23 FarmTools  
   drwxr-xr-x   9 cdfsoft  cdfsoft       169 Apr 10 15:52 FarmsControl  
   -rw-r--r--   1 cdfsoft  cdfsoft       154 Apr 10 15:08 GNUmakefile  
   drwxr-xr-x   7 cdfsoft  cdfsoft       884 Apr 10 15:08 Production  
   -rw-r--r--   1 cdfsoft  cdfsoft  47168671 Apr 10 16:53 ProductionExe_4.5.0pre3a.tar.gz  
   drwxr-xr-x   7 cdfsoft  cdfsoft       157 Apr 10 15:08 QCD  
   drwxr-xr-x   3 cdfsoft  cdfsoft        65 Apr 10 15:09 bin  
</pre>
<!-- Author: murat -->


<!-- Date: Fri Apr 12 10:30:04 2002 -->
<pre>Dear Yen Chu, 
 
Please pick up the tarball for 4.5.0pre3b for the 100k test. 
 
regards 
Mary (offline shift) 
 
fcdflnx1:/cdf/code/cdfsoft/dist/cdfpatch/4.5.0pre3b: 
total 46104 
drwxr-xr-x   7 cdfsoft  cdfsoft       171 Apr 12 09:37 BottomValidation 
drwxr-xr-x  10 cdfsoft  cdfsoft       242 Apr 12 09:37 ExoticMods 
drwxr-xr-x  10 cdfsoft  cdfsoft       233 Apr 12 10:05 FarmTools 
drwxr-xr-x   9 cdfsoft  cdfsoft       169 Apr 12 10:06 FarmsControl 
-rw-r--r--   1 cdfsoft  cdfsoft       154 Apr 12 09:36 GNUmakefile 
drwxr-xr-x   8 cdfsoft  cdfsoft       186 Apr 12 09:38 MuonGeometry 
drwxr-xr-x   7 cdfsoft  cdfsoft       884 Apr 12 09:37 Production 
-rw-r--r--   1 cdfsoft  cdfsoft  47164213 Apr 12 10:08 ProductionExe_4.5.0pre3b.tar.gz 
drwxr-xr-x   7 cdfsoft  cdfsoft       157 Apr 12 09:37 QCD 
drwxr-xr-x   9 cdfsoft  cdfsoft       248 Apr 12 09:37 TrackingMods 
drwxr-xr-x   3 cdfsoft  cdfsoft        65 Apr 12 09:38 bin 
-rwxr-xr-x   1 cdfsoft  cdfsoft      1279 Apr 12 10:02 bins 
-rw-r--r--   1 cdfsoft  cdfsoft       927 Apr 12 10:04 bins.log 
-rw-r--r--   1 cdfsoft  cdfsoft        55 Apr 12 10:04 current.csh 
-rw-r--r--   1 cdfsoft  cdfsoft       105 Apr 12 10:04 declare.csh 
drwxr-xr-x   2 cdfsoft  cdfsoft        35 Apr 12 09:36 doc 
-rw-r--r--   1 cdfsoft  cdfsoft     12740 Apr 12 10:02 gmake.log 
drwxr-xr-x   2 cdfsoft  cdfsoft       247 Apr 12 10:06 include 
drwxr-xr-x   3 cdfsoft  cdfsoft        65 Apr 12 09:38 lib 
drwxr-xr-x   2 cdfsoft  cdfsoft        35 Apr 12 09:36 man 
drwxr-xr-x   2 cdfsoft  cdfsoft        35 Apr 12 09:36 results 
drwxr-xr-x   3 cdfsoft  cdfsoft        65 Apr 12 09:37 tmp 
drwxr-xr-x   2 cdfsoft  cdfsoft        65 Apr 12 10:04 ups 
</pre>
<!-- Author: convery -->

<!-- Date: Tue Apr 16 13:24:26 2002 -->
<pre>

Hi Yen-Chu,

having failed to reproduce a crash by running on 10 stripped events,
Mary ran ProductionExe 4.5.0pre3b on a file hr021daf.0001phys, 
which contains one of the allegedly offending events.
he job she submitted on FCDFLNX1 finished successfully. Clearly, we need
to investigate the problem further. In order to do this, could you try 
rerunning the job on the Production Farm on the same file ot see if the problem
is reproducible on the Farms side? If you need, you can find this file 
on FCDFLNX1 in
                  /cdf/scratch/cdfprod0/4.5.0pre3a_02/test1/crashed 
directory.
					thanks, Pasha
</pre>
<!-- Author: murat -->


<!-- Date: Fri Apr 19 22:38:26 2002 -->
<pre>

> 
> > We have never seen any problems with minor kernel upgrades affecting
> > a user application up until now.
> 
> And probably not even now.
> The recent problems occur running 2.4 kernel code under 2.2,
> a major backward mismatch that could be expected to fail.

A brief update on the situation:

- yesterday Mary Convery successfully ran ProductionExe built on FCDFLNX1 
 (kernel 4.2.16) on CDFPCA (kernel 2.2.16) on an input file which was causing
  the same [unpacked from the same tarfile] executable to crash on the Production
  Farm (kernel 2.2.19, I believe)

- this morning Mary successfully ran ProductionExe 4.5.0pre3b built on 
  CDFPCA (kernel 2.2.16) on (the same!) CDFPCA on an input file which caused 
  the executable to crash on the Farm

- by accident yesterday we were able to reproduce the crash of ProductionExe on 
  FCDFSGI2, today the fix went in. The plan is that we are continuing tests (including 
  running purify-ed executable) on FCDFSGI2, the next step after we succeed on FCDFSGI2 
  will be 10K test on CDFPCA, then - 100K test on the Farm with CDFPCA used as a 
  build platform.
						-best, Pasha
</pre>
<!-- Author: murat -->



<!-- Date: Sat Apr 27 09:34:16 2002 -->
<pre>Following the guidelines for patching the productionExe as 
posted, I failed at step 8.  I do not have the kerberos 
authorization to rcp the tarball to the cdffarm1. 
 
I ran into a related problem trying to add a new patch 
into Releases:  I'm not authorized (as me) to write into 
the Releases area of the repository, though I presumably am 
authorized as part of my temporary office of offline shifter. 
Luckily Pasha was nearby and committed the addition, but this 
won't always be the case.</pre>
<!-- Author: James Bellinger -->