<!-- Title: 09:45:14  Sun Apr 28 2002 -->
<!-- Crewchief: mirsi -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Test of the 4.5.0pre4a run on FRH6.1.1 on 712k events went ok, with no crashes. Test of 7.1.1 still waits for the Executable to be used. 
 
mirsi</pre>


<!-- Date: Tue Apr 30 08:25:36 2002 -->
<pre>The 7.11 test ran fairly well.  There was a crash attributed to 
a known system bug in the kernel used on the farm.  Some histograms 
differed, usually in the direction of more clusters found under 
the 7.11 code. 
 
We used Pasha's tool to help compare the "standard" 4.3.2a and 
4.5.0pre4 validation packages.  The authors of those packages 
with differences have been notified. 
TowerValidationModule, CprClusterMon, PesValidation, 
ElectronValidation, PhotonValidation, TopValidationModule, 
TrackMonitor, TrackingMon 
In the case of TrackingMon Shin-Shan Yu says the change is 
understood as an improvement, due to tighter Z0 cuts.</pre>
<!-- Author: James Bellinger -->


<!-- Date: Sat May 11 22:12:52 2002 -->
<pre>For the past day there has been a problem accessing data from tapes, since fcdfsgi1 is inaccessible and DIM is apparently down. DH has been notified.</pre>
<!-- Author: mirsi -->


<!-- Date: Mon May 13 14:37:33 2002 -->
<pre>Upgraded Farm to FRH711. Up and running once again with fcdfsgi1 fixed. The first priority is processing of stream J and B data with 4.5.0 - good runs 138425 - 142718 In parallel we will try to catch up with raw data taking as well, but it will have lower priority.</pre>
<!-- Author: mirsi -->


<!-- Date: Mon May 13 14:41:36 2002 -->
<pre>Just to make sure that everyone is up to speed where to look for histograms and log files from the 4.5.0 stream J and B reprocessing: 
 
fcdflnx3: ~cdfprod0/4.5.0_02/general1/ for stream J 
fcdflnx3: ~cdfprod0/4.5.0_02/general2/ for stream B 
 
this is different from the typical processing, since this is pass 2 - i.e. reprocessing, in order to avoid mixing with the pass 1 data.</pre>
<!-- Author: mirsi -->


<!-- Date: Mon May 13 16:29:41 2002 -->
<pre>Removed DFC entries for files corresponding to run 138616 
(this includes data from run 138617) because raw data 
from the two runs is mixed into the same files due to 
an online problem related to the number of EndOfRun records. 
 
138616 was a short test run and 138717 was a physics run. 
 
ace@b0dau32 29% DFCFileTool -locate -db production_file_catalog -noflt -runsections "[(138616,0),(138616,65)]" 
 
FileName|FilesetName|CreationTime|FileSize|EventCount|LowRun/Event|HighRun/Event|LumSumOnline|LumSumOffline|RunSections|LifeTi 
mes 
ir021d78.0001phys|orphan_csl|1013108834|-1714|6088|[138616,1]|[138616,6088]|1.9e-05|1.98776e-05|1^[(138616,1),(138616,3)]|0 
ir021d78.0004phys|CA5388.3|1013108859|889|2270|[138616,6089]|[138616,8358]|1e-19|1.25761e-06|1^[(138616,4),(138616,4)]|0 
ir021d78.0005phys|CA5388.3|1013108884|603|1536|[138616,8359]|[138616,9895]|8e-06|7.84e-06|1^[(138616,5),(138616,5)]|0 
ir021d78.0006phys|CA5388.3|1013108918|697|1776|[138616,9896]|[138616,11671]|8e-06|7.84e-06|1^[(138616,6),(138616,6)]|0 
ir021d78.0007phys|CA5388.3|1013108958|719|1827|[138616,11672]|[138616,13498]|1e-19|1.25761e-06|1^[(138616,7),(138616,7)]|0 
ir021d78.0008phys|CA5388.3|1013108981|695|1765|[138616,13499]|[138616,15264]|1.1e-05|1.078e-05|1^[(138616,8),(138616,8)]|0 
gr021d78.0001phys|CA5319.1|1013110132|991|3275|[138616,1]|[138616,31789]|6.1e-05|6.35528e-05|1^[(138616,1),(138616,10)]|0 
hr021d78.0001phys|CA5320.0|1013110157|1001|2896|[138616,3]|[138616,31786]|6.1e-05|6.35528e-05|1^[(138616,1),(138616,10)]|0 
ir021d78.0009phys|CA5388.3|1013112188|574|1458|[138616,15265]|[138616,16753]|8e-06|7.84e-06|1^[(138616,9),(138616,9)]|0 
ir021d78.000aphys|CA5388.3|1013112191|135|345|[138616,16755]|[138616,17579]|7e-06|6.86e-06|1^[(138616,10),(138616,10)]|0 
 
DFC entries were removed with commands like the following: 
ace@b0dau32 30%  DFCFileTool -remove  -db production_file_catalog_write -file ir021d78.0001phys 
</pre>
<!-- Author: Tony Vaiciulis -->


<!-- Date: Mon May 27 09:01:23 2002 -->
<pre>The test of 4.5.1a went surpisingly well - no single crash in 643k events on the run 145077. 
 
The resulting hist,logs,... can be found at fcdflnx3:~cdfprod0/4.5.1a_02/test1/ 
</pre>
<!-- Author: mirsi -->


<!-- Date: Wed Jun 5 09:18:23 2002 -->
<pre>All of the data in between runs 138425 and 145669 (i.e. all of the  8x20 with exception of 139249 [missing SVX calib]) for streams B and J is done.  
 
Logs, core, hist can be found at: 
~cdfprod0/4.5.2_03/@fcdflnx3 
</pre>
<!-- Author: mirsi -->


<!-- Date: Fri Jun 14 09:46:24 2002 -->
<pre>Hello all, 
 
test of 4.6.0pre5 was done yesterday on the Farm - the result is 10 crashes in about 643k events in run 145077. 
 
I think we need to refine that a little bit.  
</pre>
<!-- Author: mirsi -->


<!-- Date: Tue Aug 6 09:07:40 2002 -->
<pre>- results of previous 10K test with 4.7.1pre1: 
  - 2 jobs out of 3 ran out of memory 
  - one job finished successfully 
- rebuild ProductionExe with ROOT 3.03/07c and GSL 1.2 
- disable CosmicFinderModule 
- restart 10K test on FCDFLNX3 again (3 jobs),  
  Zhibin monitors memory consumption with top</pre>
<!-- Author: murat -->


<!-- Date: Tue Aug 6 14:20:03 2002 -->
<pre>10K test ended up without crashes. TrackingKal problem has gone. 
 
The following quick checks were done after that: 
- we looked at the histograms - they all are readable, so the  
  "directory read" problem seems to be fixed 
- we ran Edm_ObjectLister - getting the following diagnostics  
  once per event: 
 
AC++&gt; cont -nev 1 
%ERLOG-w ROOT/TExMap: key 212436 not found at 170  GlobalLibraryLogger  
         TExMap::Remove()  6-Aug-2002 14:04:23 run = 147869 event = 3585144 
%ERLOG-w ROOT-TObject,:  
         reference to unavailable class TObject, pointers of this type will be 0  
          GlobalLibraryLogger TBuffer::CheckObject()  6-Aug-2002 14:04:23  
         run = 147869 event = 3585144 
 
- ran AC++Dump (linked against ROOT v3_03_07c) in "list" 
  mode - everything is fine except we were again getting the 
  same diagnostics as above 
- not to forget that when we opened the histogram file , we got  
  the same diagnostics. 
- output files to look at :  
  machine: FCDFLNX3 
 directory: 
/cdf/scratch/cdfopr/testRel/4.7.1pre1/pass1/ar02419d.03c4phys-ana 
  
 events: ar02419d.03c4phys.out 
 historgams: prodHist.root 
 
May be Rob and Philippe could comment on the meaning of this ROOT diagnostics</pre>
<!-- Author: zhibin -->
<!-- Comment: Tue Aug  6 15;06;12 comment by...murat -->
<pre>- failed to reproduce the problem with reading the hist file 

  back - could be due to the use of the ROOT 3.03.07b...



</pre>


<!-- Date: Wed Aug 7 15:04:51 2002 -->
<table><tr><td><a href=/elog/notebooks/offline/Wed_Aug__7_2002_150434.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-6  align='top'></td><td>4.7.1pre1 (with ROOT v3_03_07c, gsl v1_2 and CosmicFinder  disabled)</a></td></tr></table>
<!-- Author: zhibin -->


<!-- Date: Wed Aug 7 16:06:40 2002 -->
<pre>- started 10K test on Linux3 with 4.7.1pre3 ProductionExe</pre>
<!-- Author: murat -->


<!-- Date: Wed Aug 7 20:06:32 2002 -->
<pre>- around 19:50 all 3 jobs died for unknown reason 
- 10K test restarted</pre>
<!-- Author: murat -->


<!-- Date: Thu Aug 8 09:58:20 2002 -->
<pre>-around 11:50PM three jobs died again 
-10K test restarted</pre>
<!-- Author: zhibin -->


<!-- Date: Thu Aug 8 13:00:42 2002 -->
<pre>-the system was down. 
-the size of memory of Stream D is at certain value after 188 munite monitoring, so is that of Stream H. 
</pre>
<!-- Author: zhibin -->


<!-- Date: Thu Aug 8 14:27:32 2002 -->
<pre>10K test with 4.7.1pre3 ProductionExe restarted  
after FCDFLNX3 came back up</pre>
<!-- Author: murat -->


<!-- Date: Fri Aug 9 12:25:32 2002 -->
<pre>-10K test was done successfully around 19:30PM yesterday 
</pre>
<!-- Author: Bruce -->


<!-- Date: Fri Aug 9 15:52:23 2002 -->
<center><a href='/elog/notebooks/offline/Fri_Aug__9_2002_155025.ps'><img src='/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-8' ></a></center>
<!-- Author: Bruce -->


<!-- Date: Fri Aug 9 15:53:41 2002 -->
<table><tr><td><a href=/elog/notebooks/offline/Fri_Aug__9_2002_155335.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-12  align='top'></td><td>4.7.1pre3(with ROOT v3_03_07c) </a></td></tr></table>
<!-- Author: Bruce -->


<!-- Date: Sat Aug 10 12:06:10 2002 -->
<pre>100K test of 4.7.1pre3 finished with 2 crashes per 1.2 Mln events 
We can proceed with the final build of 4.7.1.</pre>
<!-- Author: murat -->


<!-- Date: Sun Aug 11 10:46:08 2002 -->
<pre>- running separate executable - validation modules only - on the output 
  of Production 
- all the input files are in /cdf/scratch/cdfopr/testRel/4.7.1pre3/pass1 
- disable TauValidationModule: it crashes 
#----------------------------------------------------------------------- 
#  input file: ar02419d.03c4phys.out 
#------------------------------------------------------------------------ 
24115 cdfopr     9   0 66156  64M 28252 R    99.9  0.8   0:12 prod2.exe 
24115 cdfopr     9   0  248M 248M 32832 R    95.9  3.0   1:36 prod2.exe 
24115 cdfopr     9   0  248M 248M 33016 R    94.3  3.0   4:06 prod2.exe 
24115 cdfopr    19  19  254M 254M 33348 R N  94.3  3.1  19:13 prod2.exe 
24115 cdfopr    19  19  256M 256M 33368 R N  97.4  3.1  22:32 prod2.exe 
24115 cdfopr    19  19  257M 257M 33384 R N  96.2  3.1  31:42 prod2.exe 
24115 cdfopr    19  19  257M 257M 33392 R N  94.1  3.1  36:29 prod2.exe 
24115 cdfopr    19  19  260M 260M 33392 R N  93.6  3.2  45:57 prod2.exe 
... and then it ended...  
.. memory leaks are &lt; 20MBytes/validation job for validation modules 
</pre>
<!-- Author: murat -->


<!-- Date: Sun Aug 11 10:47:30 2002 -->
<pre># 
#  running 4.7.1pre3 (reconstruction modules only) on stream A file - ar02419d.03c4phys 
# 
17774 cdfopr     9   0  412M 412M 38312 R    97.8  5.1   3:05 prod1.exe 
17774 cdfopr    10   0  414M 414M 38356 R    96.8  5.1   9:22 prod1.exe 
17774 cdfopr    19  19  426M 426M 38356 R N  99.1  5.2  12:27 prod1.exe 
17774 cdfopr    19  19  429M 429M 38544 R N  98.5  5.3  21:25 prod1.exe 
17774 cdfopr    19  19  436M 436M 38560 R N  99.7  5.4  24:56 prod1.exe 
17774 cdfopr    19  19  463M 463M 38560 R N  98.7  5.7  36:30 prod1.exe 
17774 cdfopr    19  19  464M 464M 38560 R N  98.4  5.7  42:10 prod1.exe 
17774 cdfopr    19  19  481M 481M 38560 R N  99.9  5.9  46:10 prod1.exe 
17774 cdfopr    19  19  489M 489M 38560 R N  96.6  6.0  54:18 prod1.exe 
17774 cdfopr    19  19  488M 488M 38560 R N  98.7  6.0  56:53 prod1.exe 
17774 cdfopr    19  19  493M 493M 38560 R N  96.8  6.1  61:58 prod1.exe 
17774 cdfopr    19  19  501M 501M 38560 R N  97.1  6.2  64:18 prod1.exe 
17774 cdfopr    19  19  517M 517M 38560 R N  96.4  6.4  77:22 prod1.exe 
17774 cdfopr    19  19  530M 530M 38560 R N  97.5  6.5  84:37 prod1.exe 
17774 cdfopr    19  19  540M 540M 38560 R N  97.1  6.7  90:03 prod1.exe 
17774 cdfopr    19  19  585M 585M 38560 R N  95.1  7.2 119:50 prod1.exe 
17774 cdfopr    19  19  592M 592M 38560 R N  96.5  7.3 124:37 prod1.exe 
17774 cdfopr    19  19  603M 603M 38560 R N  96.7  7.4 129:59 prod1.exe 
 
- so far ProductionExe leaks about 1.5 MBytes/minute, which is about  
  100 KBytes/event</pre>
<!-- Author: murat -->


<!-- Date: Sun Aug 11 12:22:48 2002 -->
<pre>tried running Purify v4_5 on FCDFSGI2 on the "light-weight"  
ProductionExe with the validation modules taken out form the  
build: 
 
 --------------- HepPDT Version 1.00.01 ---------------  
Purify slave: Warning: About to run out of memory. 
Using last remaining 64000 bytes. 
 
Purify slave: Error: Memory allocation failed during a request for 4636 bytes. 
This is usually caused by exhausting the available swap space. 
malloc failed 
------------------------- 
Earlier yesterday same happened to insure++ running on FCDFLNX3 
(after about an hour of waiting)</pre>
<!-- Author: murat -->


<!-- Date: Fri Aug 16 12:06:17 2002 -->
<pre> 
From Bruce Barnett (outgoing) and Sunil Somalwar (incoming) Offline CO's: results of Pasha's study to examine memory leakage 
jumps.  Plot to follow. 
</pre>
<!-- Author: Sunil Somalwar -->


<!-- Date: Fri Aug 16 12:54:36 2002 -->
<table><tr><td><a href=/elog/notebooks/offline/Fri_Aug_16_2002_125421.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-14  align='top'></td><td>Pasha's memory leak study results</a></td></tr></table>
<!-- Author: Sunil Somalwar -->


<!-- Date: Sat Aug 17 15:25:33 2002 -->
<pre>Offline database usage webpage (to be updated frequently) 
created: 
 
http://www-cdf.fnal.gov/internal/upgrades/computing/operations/offline_shifts/oracle_usage.dat</pre>
<!-- Author: Sunil Somalwar -->


<!-- Date: Sat Aug 17 15:28:51 2002 -->
<pre>In the process of investigating memory usage patterns of ProductionExe 
we found that there are events, for which SiPatternRecModule allocates 
chunks of memory of the order of 50-100 MBytes. This memory never 
gets released until the end of the job. Here is a table showing 
memory size of the ProductionExe process for one of the jobs we ran. 
(enabled were SiClustering, PVFinder, CT_Tracking, SiPatternRec and 
PadTrack modules): 
 
event number        before     before  before  before  before 
                    SiClus    PVFind   CT_Tra  SiPatR  PadTrk 
------------------------------------------------------------ 
  1                 420052    420044   420176  420180  461132 
  2                 461112    460992   461000  460996  516148 
  3                 516168    516148   516176  516156  617416 
 
The question is whether it is a memory leak or an allocation  
of extra memory, needed by the program.  Note that the events are 
special somehow, but appear normal on Event Display.</pre>
<!-- Author: Pasha and Sunil  -->


<!-- Date: Mon Aug 19 12:32:11 2002 -->
<pre>Production tarball v4.7.1a is available on fcdflnx3:  
 since Aug 16:  
  
murat@fcdflnx3:~&gt;dir -l ~cdfsoft/dist/cdfpatch/4.7.1a/*.gz  
-rw-r--r--    1 cdfsoft  cdfsoft  78941561 Aug 16 12:16 /home/cdfsoft/dist/cdfpatch/4.7.1a/ProductionExe_4.7.1a.tar.gz  
  
The Farmers can grab it right away.  
                                        -best, Pasha  
</pre>
<!-- Author: murat -->


<!-- Date: Tue Aug 27 11:43:29 2002 -->
<p> 
10K test on 4.7.0int5 
<ol> 
<li> run 147869, from July 9.  Streams a,b,d,g. 
<li> I specified stream a file incorrectly, so it didn't process any 
     events.  I corrected the file name, edited the 
     files_to_process.txt file, and ran it again. 
<li> The x windows failed to pop up.  Presuming that they were just for 
     getting the running tail of the log files, I did it myself by 
     popping up my own x windows, logging in, and using tail -f. 
     This should go into the instructions. 
<li> Got word from Pasha that we should really build the BareBones 
     executable.  Modified Production/src/GNUmakefile to build just 
     this (otherwise it would have built everything except the 
     official ProductionExe), and I'll move the executable over the 
     original ProductionExe so the other scripts will work okay. 
     In the meantime, killed the running tests from above. 
<li> Error messages in building BareBones (same as ProductionExe, 
     actually): 
  <pre> 
<**building**> BareBones 
Warning: Duplicate shared library /usr/lib/libdl.so: using first occurrence only. 
Warning: Duplicate shared library /usr/lib/libdl.so: using first occurrence only. 
/home/cdfsoft/products/tcl/v8_3_1a/Linux+2.2/lib/libtcl.a(tclUnixPipe.o): In function `TclpCreateTempFile': 
tclUnixPipe.o(.text+0xf6): the use of `tmpnam' is dangerous, better use `mkstemp' 
  </pre> 
<li> Screwed up my directories (confused by TestRel vs testRel).  So I'll 
     start from scratch, rebuilding the BareBones executable from a clean 
     4.7.0int5 release. 
<li> The log files record the following message: 
  <pre> 
cannot find user module by the name MidPointModule 
  </pre> 
<li> Beate notes that the current tcl's won't work in the new order, and 
     that she'll make a new setup_jetReco.tcl 
</ol> 
</p>  
<!-- Author: jtseng -->


<!-- Date: Tue Aug 27 12:13:58 2002 -->
<p> 
Avi and Pierre inform me that there is a prepared BareBones.tcl.  It is 
to be found in Production/user/BareBones.tcl.  I modify process10K to 
point to this tcl file, remake FarmTools.all, and then run10K -p 1, 
as directed. 
</p>  
<!-- Author: jtseng -->

<!-- Date: Tue Aug 27 22:18:54 2002 -->
<pre> 4.7.0int5 barebones tarball

Dear Farmers, please pick up the 4.7.0int5 tarball for 100K test:

[cdfopr@fcdflnx3 pasha]$ dir -l $PWD/*.gz
-rw-r--r--    1 cdfopr   cdf      70839694 Aug 27 21:54 /cdf/scratch/cdfopr/testRel/pasha/ProductionExe_4.7.0int5.tar.gz

Rob Snihur suggested that we process the same runs as during the previous 100K test:

148824 >570 nb-1
149387 >560 nb-1

Note, that the validation modules are turned off - there were problems with adding them.

							-best, Pasha
</pre>
<!-- Author: murat -->


<!-- Date: Thu Aug 29 17:17:51 2002 -->
<pre>
Adding validation modules to the 4.7.0int5 BareBones executable was
observed to cause crashes in database accesses made by the ChipStatus
constructor.  This crash happened randomly when running the 10K tests
on streams A, B, D, and G; in 4 successive passes at running all four
streams simultaneously, using the same input files, different jobs
would actually fail each time, first D and G, then just B, then all
four, and then B and D.  The crash was always the same.

When the TowerValidationModule path was removed, the crashes didn't.
When that path was put back in but the other path, with the other
validation modules, activated, then the crashes came back.

Pasha and I moved the job to fcdfsgi2 from fcdflnx3 in order to run
purify.  There were lots of UMR's (uninitialized memory reads) as
usual, followed by the crash.  It was eventually narrowed down to
occurring when the Prereq clone for the Tower path was configured;
simply cloning it and enabling the clone didn't cause the crash.

So the problem is apparently an interaction between Prereq and
ChipStatus.  We felt that at this point it was better to ask experts
to take a look.

The executable can be found on fcdfsgi2 at
/cdf/data05/s0/commission/4.7.0int5/bin/IRIX6-KCC_4_0/BareBones.
The tcl files are in
/cdf/data05/s0/commission/4.7.0int5/Production/.
setup_barebones_monitors.tcl contains the specification of the
monitoring paths.  Also in the directories are the purify executable
(BareBones.pure) and the purify output (BareBones.pure.pv).
</pre>
<!-- Author:  jtseng -->


<!-- Date: Thu Aug 29 17:48:42 2002 -->
<pre>
4.7.0int5 tarball for 100K test is available on FCDFLNX3 from

[cdfopr@fcdflnx3 ~]$ dir -l /cdf/scratch/cdfopr/testRel/pasha/*.tar.gz
-rw-r--r--    1 cdfopr   cdf      70154733 Aug 29 17:31 /cdf/scratch/cdfopr/testRel/pasha/ProductionExe_4.7.0int5.tar.gz

							-best, Pasha
</pre>
<!-- Author: murat -->



<!-- Date: Thu Aug 29 22:01:02 2002 -->
<pre>The 4.7.0int5 is ongoing and we have already produced some files. We will copy those to the designated area at fcdflnx3 soon for validation.</pre>
<!-- Author: mirsi -->


<!-- Date: Fri Aug 30 16:10:03 2002 -->
<pre>Last night during the 100K test, the following jobs had  
increased memory usage (to over 700 MB).  The files they 
were running on are also listed: 
 
job		file 
17405.7		dr024558.023ephys 
17424.7		hr024558.01a4phys 
17442.4		hr024558.02e6phys</pre>
<!-- Author: Amy Connolly -->


<!-- Date: Sat Aug 31 14:13:38 2002 -->
<pre>I was unknowingly setup in 4.7.1 during the 10K test earlier 
today, so we resubmitted it, now in 4.8.0pre1.</pre>
<!-- Author: Amy Connolly -->

<!-- Date: Sat Aug 31 22:42:28 2002 -->
<pre>
Hi All,

10K test of ProductionExe 4.8.1 finished w/o crashes. As the next check, I ran Stnmaker 
and AC++Dump on the output files - both crashed trying to access max Pt track associated 
to an electron.

Bob - could you take a look? - Looks like your help is needed. The output files to look at 
are located on FCDFLNX3:

fcdflnx3:/cdf/scratch/cdfopr/testRel/4.8.0pre1/pass1/*/*.out*

I also started 10K/pass2 with the goal of reproducing anomalous memory consumption 
during 100K test of 4.7.0int5.
						-best, Pasha
</pre>
<!-- Author: murat -->



<!-- Date: Mon Sep 2 15:11:59 2002 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-15&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-16  width='300' align='top'></td><td>This plot shows data retrieved from log files from the 100K test on 4.8.0pre1 that happened on August 30th. Each entry is another data type accessed for each file read (one for each of TOFAttLen, TOFChanOffs, ... for each file read).  The times reported in the log files are the average time  it took to execute transactions that went out to the database.  Plotted here is the average time for each "get."</a></td></tr></table>
<!-- Author: Amy Connolly -->


<!-- Date: Mon Sep 2 15:16:43 2002 -->
<pre>Oops, the previous plot is actually from the 100K test that 
occured on August 30th for 4.7.0int5, not 4.8.0pre1.</pre>
<!-- Author: Amy Connolly -->


<!-- Date: Wed Sep 4 09:16:11 2002 -->
<table><tr><td><a href=/elog/notebooks/offline/Wed_Sep__4_2002_091559.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-18  align='top'></td><td>DB access and memory usage statistics for 100K test of ProductionExe 4.7.0int5</a></td></tr></table>
<!-- Author: murat -->


<!-- Date: Wed Sep 4 12:10:34 2002 -->
<table><tr><td><a href=/elog/notebooks/offline/Wed_Sep__4_2002_121018.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-20  align='top'></td><td>memory usage and DB access times for 100K test of ProductionExe v4.8.0pre2</a></td></tr></table>
<!-- Author: murat -->
<!-- Comment: Wed Sep  4 16;21;37 comment by...murat -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>plot for memory consumption vs event number indicates 
that there may be a memory leak in one of the 4.80pre2 
ProductionExe modules.</pre></td></tr></td></table>
<!-- Date: Wed Sep 4 13:41:12 2002 -->
<pre>The histograms from the 10k test (for the validation modules listed below) of 4.8.0pre2 looked OK, 
but the histograms for the 100k test  
(fcdflnx3:~cdfprod0/4.8.0pre2_05/test1/hist) are empty! 
 
What was the difference between these two tests? 
 
val modules OK'd: 
FitBeam 
Tau 
TrackingMon 
</pre>
<!-- Author: Rob Snihur -->
<!-- Comment: Wed Sep  4 16;14;33 comment by...murat -->
<pre>the TCL file used on the Farms for 100K test of 4.8.pre2
did't have monitor path enabled.</pre>
<!-- Date: Wed Sep 4 15:58:03 2002 -->
<pre> 
I have copied over the files which core dumped during the 
100K test on 4.8.0pre2 into /cdf/scratch/cdfopr/10Kevents/ 
 
The files are: 
 
er024558.0127phys 
gr02478b.001aphys 
er024558.0260phys 
br024558.03f1phys 
er02478b.002cphys 
er024558.02e3phys 
gr024558.026bphys 
 
-Amy</pre>
<!-- Author: Amy Connolly -->




<!-- Date: Wed Sep 4 21:25:10 2002 -->
<pre>Started 4.8.0pre3 test. Looks ok.</pre>
<!-- Author: mirsi -->


<!-- Date: Thu Sep 5 16:33:59 2002 -->
<pre> 
Purify has been run on 4.8.0pre2, and the results are 
in /cdf/data05/s0/commission/4.8.0pre2/ProductionExe.pure.pv.   
It shows memory leaks related to PhysicsTowerDataMaker.   
Beate is looking at the problem.</pre>
<!-- Author: Amy Connolly -->


<!-- Date: Fri Sep 6 11:53:42 2002 -->
<pre>Ran purify on 4.8.0pre2 stnmaker.exe.  Results are in 
/cdf/data05/s0/commission/4.8.0pre2/stnmaker.exe.pure.pv. 
 
Saw a few memory overwrites that were reported to Pasha. 
Also saw memory leaks in PhysicsTowerDataMaker, same as seen in Production. 
</pre>
<!-- Author: twright -->


<!-- Date: Fri Sep 6 12:27:49 2002 -->
<pre>The following librarians were notified of errors reported by 
their modules in the 10k test with 4.8.0pre2: 
 
muons (Ken Bloom), COT (Chris Hays), SVX (Matt Herndon), TOF (Ivan Furic), SMX (Michael Riveline) 
 
The full list of errors and their frequency can be found at: 
/cdf/scratch/cdfopr/OfflineMonitoring/prod-errors-480pre2.txt 
on fcdflnx3 
</pre>
<!-- Author: twright -->
<!-- Comment: Fri Sep  6 13;17;22 comment by...twright -->
<pre>Oops, I meant the 100k event test
</pre>


<!-- Comment: Sat Sep  7 12;52;57 comment by...rob snihur -->
<pre>Michael Riveline is not with CDF any more.
I suggest bugging Steve Kuhlmann,
and maybe other folks from Argonne.
</pre>
<!-- Date: Sat Sep 7 09:21:01 2002 -->
<pre>Miro reported that FileInput module should be replaced with 
DHInput_2 in the Production tcl files - FileInput doesn't 
support so-called "rawRS" mode.  Called Liz - couldn't reach 
her, so we are going ahead with building the new tarball. 
farm_master.tcl and setup_input.tcl are modified and tagged 
as v4_8_0pre3a (branch for these files). Tarball is built. 
ash-2.04$ dir -l $PWD/*.gz 
-rw-r--r--    1 cdfsoft  cdfsoft  70431449 Sep  7 09:13 
/home/cdfsoft/dist/cdfpatch/4.8.0pre3a/ProductionExe_4.8.0pre3</pre>
<!-- Author: murat -->


<!-- Date: Sat Sep 7 09:44:23 2002 -->
<pre>Date: Fri, 06 Sep 2002 17:22:59 -0500  
 From: Dmitri Litvintsev &lt;litvinse@fnal.gov&gt;  
 To: Pasha Murat &lt;murat@fnal.gov&gt;  
 Subject: creates tst06 datasets  
  
 Hi Pasha,  
  
 as agreed:  
  
 &gt; DFCQuery "select ds_name_id, dataset_name from filecatalog.cdf2_datasets where ds_name_id like '%tst06'"  
  atst06 TEST_DATASET-A-06  
  btst06 TEST_DATASET-B-06  
  ctst06 TEST_DATASET-C-06  
  dtst06 TEST_DATASET-D-06  
  etst06 TEST_DATASET-E-06  
  ftst06 TEST_DATASET-F-06  
  gtst06 TEST_DATASET-G-06  
  htst06 TEST_DATASET-H-06  
  itst06 TEST_DATASET-I-06  
  jtst06 TEST_DATASET-J-06  
  
</pre>
<!-- Author: murat -->



<!-- Date: Mon Sep 9 10:57:45 2002 -->
<table><tr><td><a href=/elog/notebooks/offline/Mon_Sep__9_2002_105648.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-24  align='top'></td><td> </a></td></tr></table>
<!-- Author: wright -->
<!-- Comment: Mon Sep  9 11;13;28 comment by...wright -->
<pre>Plots of memory usage vs. event number, database access times, 
and DB access times vs. table number for 4.8.0pre3a_06.  The 
table number key follows:

 0: CALDigiToGeV3OTL       19: COTChanT0OTL           38: SiAlignWaferOTL
 1: CALL1Peds3OTL          20: COTStage0OTL           39: SiAlignWarpOTL
 2: CHASlewingOTL          21: CSXChASTOTL            40: SiChipPedOTL
 3: CMPAllOTL              22: CSXGlobalOTL           41: SiStripDHOTL
 4: CMPChannelOTL          23: CotBeamPositionOTL     42: SiStripMaskOTL
 5: CMPPositionOTL         24: DFCFilesOTL            43: SvxBeamPositionOTL
 6: CMPRegionOTL           25: PASSCALIBSOTL          44: TOFAttLenOTL
 7: CMPStackOTL            26: PASSESOTL              45: TOFChanOffsOTL
 8: CMUAll2OTL             27: PESCalibOTL            46: TOFLightSpdOTL
 9: CMUAllOTL              28: PESLaserOTL            47: TOFPedLookupOTL
10: CMUChannel2OTL         29: PHASlewingOTL          48: TOFResidualOTL
11: CMUChannelOTL          30: PPRLinEResponseOTL     49: TOFTWalkOTL
12: CMUPositionOTL         31: RunListOTL             50: TOFTacParmOTL
13: CMUWedgeOTL            32: RunSummaryOTL          51: TOFTimeResolOTL
14: CMXAllOTL              33: SVX_ConfigurationsOTL  52: UsedSetOTL
15: CMXChannelOTL          34: SVX_Rundate_ViewOTL    53: Used_SetsOTL
16: CMXPositionOTL         35: SiAlignBarrelOTL       54: ValidSetOTL
17: CMXQuadrantOTL         36: SiAlignFrameOTL        55: WHASlewingOTL
18: CMXWedgeOTL            37: SiAlignLadderOTL
</pre>


<!-- Date: Mon Sep 9 11:14:58 2002 -->
<center><a href='/elog/notebooks/offline/Mon_Sep__9_2002_105626.ps'><img src='/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-22' ></a></center>
<!-- Author: wright -->
<!-- Comment: Mon Sep  9 11;15;33 comment by...wright -->
<pre>Same plot as above, for 4.8.0pre3_05
</pre>


<!-- Date: Tue Sep 10 10:11:13 2002 -->
<table><tr><td><a href=/elog/notebooks/offline/Tue_Sep_10_2002_101046.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-26  align='top'></td><td>And now for the 4.8.0 StreamJ processing </a></td></tr></table>
<!-- Author: wright -->


<!-- Date: Thu Sep 12 09:28:22 2002 -->
<table><tr><td><a href=/elog/notebooks/offline/Thu_Sep_12_2002_092812.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=offline&action=view&page=-28  align='top'></td><td>Memory and database usage plots for the 4.8.1 runs </a></td></tr></table>
<!-- Author: wright -->


<!-- Date: Thu Sep 12 15:58:30 2002 -->
<pre>500k test of 4.8.2 begun.</pre>
<!-- Author: Chris -->