<!-- Title: 21:41:15  Sat Jan 20 2007 -->
<!-- Crewchief: cdf_csl -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<b><font color="#003366" face="Arial">setup csl</font></b> works!


<!-- Date: Sat Jan 20 21:48:29 2007 -->
<font face="Arial"> 
Version 2.2 is set to default csl version.<br> 
 It will be reached by following command: 
<pre> 
> setup csl 
> cd $CSL_DIR 
</pre> 
For each code updates, it needs to be build under '~cdfprod/csl' directory by user 'cdfprod'. After code built, it should be necessary to declear the version to be picked up by 'setup csl 'version'. Declearing scripts are in the ups directory. Please read <a 
href="http://www-cdfonline.fnal.gov/ops/ace2help/csl2/ExpertPages/ExpertNotes.html#ups-csl"> 
Experts Note </a> for more in detail.  
</font><br>
<!-- Author: Geumbong -->


<!-- Date: Tue Jan 23 16:03:03 2007 -->
<p>cslmoncron.pl script updated 
<ul> 
<li>cslmoncron.pl is the script that updates node activity 
    statistics for the CSL monitoring web page - load, network, 
    and disk.</li> 
<li>Logging of the activity statistics to a cumulatively  
    appended history files were removed, as the line counts 
    were getting very large. All that information is also 
    entered into the monitoring web page RRD database anyway.</li> 
</ul> 
</p>
<!-- Author: Willis -->


<!-- Date: Thu Jan 25 18:28:00 2007 -->
<b> Remained Files in .open directory </b><br><br> 
I've removed files as following: 
<li>b0csl21 : Removed then since they came from test.<br> 
<pre> 
Jan 24 20:18 /mnt/data1/logger/ace/15/.open/e_116969.1456time 
Jan 24 20:18 /mnt/data1/logger/ace/15/.open/e_116969.1533time 
</pre> 
</li> 
<li>b0csl22 : Removed & recovered 
<pre> 
Jan 25 14:31 /mnt/data1/logger/ace/00/.open/ar038dde.0002test 
/mnt/data1/logger/ace/00 
-rwxrwxr--  1 cslace cdf 983M Jan 19 21:41 ar038c2d.0126phys* 
-rwxrwxr--  1 cslace cdf 809M Jan 20 00:31 ar038c2d.0281phys* 
-rwxrwxr--  1 cslace cdf 796M Jan 23 17:19 ar038d07.0410phys* 
</pre> 
</li> 
<li>b0csl23 : Test file is removed, phys file is recovered.  
<pre> 
Jan 23 17:12 /mnt/data1/logger/ace/00/.open/br038d07.0471phys 
Jan 25 14:31 /mnt/data1/logger/ace/00/.open/br038dde.0001test 
cslace cdf 1057297300 Jan 20 00:23 br038c2d.02a3phys 
cslace cdf 1002112059 Jan 23 17:12 br038d07.045ephys 
</pre> 
</li> 
I'll move rest of them later. <br>
<!-- Author: Geumbong -->
<!-- Comment: Sun Jan 28 17;55;10 comment by...Geumbong -->
<li> Old file check text files remained in primary directory has moved to /cdf/onln/data/cslace/log/fileCheck/.</li>
<li> b0csl24: Test file is removed
<pre>
Jan 25 14:31 /mnt/data1/logger/ace/00/.open/cr038dde.0001test
</pre>
<li> b0csl25: Test file is removed and moved phys file
<pre>
Jan 23 17:18 /mnt/data1/logger/ace/00/.open/dr038d07.0472phys
Jan 25 14:31 /mnt/data1/logger/ace/00/.open/dr038dde.0001test
Jan 25 14:31 /mnt/data1/logger/ace/00/.open/dr038dde.0004test
</pre>
</li>
<li> b0csl26: Test file is removed and moved phys file
<pre>
Jan 23 17:15 /mnt/data1/logger/ace/00/.open/er038d07.0479phys
Jan 25 14:31 /mnt/data1/logger/ace/00/.open/er038dde.0001test
</pre>
</li> 
<li> b0csl27: Test file is removed and phys file is moved
<pre>
Jan 23 17:18 /mnt/data1/logger/ace/00/.open/gr038d07.0486phys
Jan 25 14:31 /mnt/data1/logger/ace/00/.open/gr038dde.0001test
</pre>
</li>
<li> b0csl28: Test file is removed and phys/cosm file is moved. It has file in primary directory w/o db file.
<pre>
Jan 23 01:57 /mnt/data1/logger/ace/00/.open/ir038cf8.0148cosm
Jan 25 14:31 /mnt/data1/logger/ace/00/.open/hr038dde.0001test
-rwxrwxr--  1 cslace cdf 992M Jan 23 01:57 ir038cf8.0132cosm*
</pre>
</li>
<li> b0csl29: Test file is removed.
<pre>
Jan 25 14:31 /mnt/data1/logger/ace/00/.open/jr038dde.0001test
</pre>
</li>
<!-- Date: Fri Jan 26 11:36:16 2007 -->
<p>CSL control script updates for CSL v2_3: 
<ul> 
<li>Stop child client loggers gracefully with SIGQUIT, now that 
    they listen for that signal.</li> 
<li>The CSL driver and child logger start, stop, and kill 
    operations now have wait loops that checks for successful 
    completion before proceeding. The wait loops have a  
    maximum loop count. An error condition is set if that 
    maximum loop count is exceeded.</li> 
</ul> 
</p>
<!-- Author: Willis -->
<!-- Comment: Fri Jan 26 15;33;16 comment by...Willis -->
<p>Ray Culbertson is in the process of modifying ProcMon to use
   ``setup csl" and the UPS versioned CSL control scripts.
   At the moment, the cslcom and childlogger scripts used in
   ProcMon are from the ~cslace/bin area, which is effectively
   v2_0, and not the current v2_2 default.
</p>



<!-- Date: Sun Jan 28 18:03:51 2007 -->
Events bursted in streamD while taking XFT test around midnight Jan 28. At that time, L3 turned to green about 10 min. after test started and there were many files left in .open area in b0csl25. Here is summary about files: 
<pre> 
XFT test - started at 11:20. This is consistent with the time that streamD bursted.  
 
Jan 27 23:25 /mnt/data1/logger/ace/00/.open/dr038e6f.0001phys --> Readible, Section 1 
Jan 27 23:25 /mnt/data1/logger/ace/00/.open/dr038e6f_1.0001phys --> Readible, Has section 1 events only 
 
Jan 27 23:25 /mnt/data1/logger/ace/00/.open/dr038e6f.0002phys --> Readible, Section 2. 
Jan 27 23:26 /mnt/data1/logger/ace/00/.open/dr038e6f_1.0002phys --> Readible, Section 2 and 3. 
Jan 27 23:26 /mnt/data1/logger/ace/00/.open/dr038e6f_2.0002phys --> Readible, Section 3 and 4. 
Jan 27 23:26 /mnt/data1/logger/ace/00/.open/dr038e6f_3.0002phys --> Error in read, size 313M 
 
Jan 27 23:39 /mnt/data1/logger/ace/00/.open/dr038e6f.001ephys --> Readible, Section 30, 31 
Jan 27 23:40 /mnt/data1/logger/ace/00/.open/dr038e6f_1.001ephys --> Readible, Section 31, 32 
Jan 27 23:40 /mnt/data1/logger/ace/00/.open/dr038e6f_2.001ephys --> Readible, Section 32, 33 
Jan 27 23:41 /mnt/data1/logger/ace/00/.open/dr038e6f_3.001ephys --> Readible, Section 33. 
Jan 27 23:41 /mnt/data1/logger/ace/00/.open/dr038e6f.0022phys --> Readible, Section 34. 
 
 L3 went green, CSL_MAX_CLIENT cause CSL being restarted. 
 Again, no end run message found from CSL receiver. 
 
Jan 28 00:08 /mnt/data1/logger/ace/00/.open/dr038e72.0001phys --> Readible, Section 1. 
Jan 28 00:08 /mnt/data1/logger/ace/00/.open/dr038e72_1.0001phys --> Readible, Section 1,2 
Jan 28 00:09 /mnt/data1/logger/ace/00/.open/dr038e72_2.0001phys --> Readible, Section 2,3 
Jan 28 00:09 /mnt/data1/logger/ace/00/.open/dr038e72_3.0001phys --> Readible, Section 3,4 
Jan 28 00:10 /mnt/data1/logger/ace/00/.open/dr038e72_4.0001phys --> Readible, Section 4,5,6  
Jan 28 00:10 /mnt/data1/logger/ace/00/.open/dr038e72_5.0001phys --> Readible, Section 6 
Jan 28 00:11 /mnt/data1/logger/ace/00/.open/dr038e72.0007phys --> Readible, Section 7, Size 990M 
 
 Same problem happen again. CSL did not receive end run message. 
 
Jan 28 00:45 /mnt/data1/logger/ace/00/.open/dr038e75.0010phys --> Readible, Section 16 
Jan 28 00:46 /mnt/data1/logger/ace/00/.open/dr038e75_1.0010phys --> Readible, Section 16, 17 
Jan 28 00:46 /mnt/data1/logger/ace/00/.open/dr038e75_2.0010phys --> Readible, Section 17, 18 
Jan 28 00:46 /mnt/data1/logger/ace/00/.open/dr038e75_3.0010phys --> Readible, Section 18  
Jan 28 00:47 /mnt/data1/logger/ace/00/.open/dr038e75.0013phys --> Readible, Section 19, Size 1.1G 
Jan 28 00:53 /mnt/data1/logger/ace/00/.open/dr038e75_1.0020phys --> Readible, Section 33.  
 
  ps. dr038e75_1.0020phys exceeds 1.8G but successfully closed and moved to FCC. funny. 
 
Jan 28 00:54 /mnt/data1/logger/ace/00/.open/dr038e75.0022phys --> Readible, Section 34, Size 1.5G 
Jan 28 00:54 /mnt/data1/logger/ace/00/.open/dr038e75.0023phys -->  
</pre> 
I moved those files to /mnt/data1/logger/ace/specialFiles/. 
<!-- Author: Geumbong -->


<!-- Date: Wed Jan 31 18:53:59 2007 -->
<b> <font color=purple> Moving Open file/Closed file with huge hidden db file </font></b><br><br> 
<b>CreateDBFile.sh</b> is modified to use only one directory  
<pre> 
/mnt/data1/logger/error 
</pre> 
for creating/correcting db files. <font color=red>Please move open files to /mnt/data1/logger/error </font> instead of /mnt/data1/logger/ace/error.  <br><br> 
<i> System Recovery/Clean up files in .open area  </i> is corrected accordingly: 
<pre> 
4. Move the data files files to 
 
   /mnt/data1/logger/error/  
</pre> 
<br> 
<b> Moved files - history </b> 
<ul> 
As talked in the meeting, file closing with huge/higgen db file happened at the same time that open files were left over. Here's the summary of moving files. <br> 
</ul> 
<li> b0csl22 : 
<pre> 
 994M Jan 29 06:34 ar038e89.04dcphys --> Open file 
 226M Jan 29 06:58 ar038e89.0561phys --> huge db file 
</pre> 
</li> 
<li> b0csl23: 
<pre> 
 957M Jan 29 06:58 br038e89.0568phys --> huge db file 
 588M Jan 31 11:26 br038ed7.0110phys --> huge db file 
 495M Jan 29 06:56 br038e89.057bphys --> open file 
 994M Jan 31 11:14 br038ed7.00eephys --> open file 
</pre> 
</li> 
<li> b0csl24: 
<pre> 
 926M Jan 29 06:49 cr038e89.056aphys --> huge db file 
 946M Jan 29 06:49 cr038e89.0575phys --> open file 
</pre> 
<li> b0csl25:  
<pre> 
 986M Jan 29 06:58 dr038e89.055fphys --> huge db file 
 
 Those files are moved to specialFile directory. What is good is that they don't have additional run section in secondary files:  
 
Jan 28 16:00 /mnt/data1/logger/ace/00/.open/dr038e79.0001phys --> Readible, Section 1 only. 
Jan 28 16:00 /mnt/data1/logger/ace/00/.open/dr038e79_1.0001phys --> Readible, Section 1 only. 
Jan 28 16:01 /mnt/data1/logger/ace/00/.open/dr038e79.0002phys --> Readible, Section 2 only. 
Jan 28 16:01 /mnt/data1/logger/ace/00/.open/dr038e79_1.0002phys --> Readible, Section 2 only. 
</pre> 
<li>b0csl26: 
<pre> 
 979M Jan 29 06:52 er038e89.0563phys --> huge db file 
 989M Jan 29 06:52 er038e89.0573phys --> open file 
</pre> 
</li> 
<li>b0csl28: 
<pre> 
 1.1G Jan 29 06:48 hr038e89.051cphys --> huge db file 
</pre> 
</li> 
<br>
<!-- Author: Geumbong -->


<!-- Date: Fri Feb 2 12:20:00 2007 -->
<b><font color="#CC0066"> CSL event flow check</font></b><br><br> 
That CSL EVENT FLOW checked if there were any csl queues stuck. It basically used <i> cslcom check </i> same as CSL MAIN. Ray  Culbertson explained that's why two checks were merged to one CSL MAIN. We don't need to worry about CSL EVENT FLOW any longer. <br><br> 
<!-- Author: Geumbong -->


<!-- Date: Tue Feb 6 09:40:08 2007 -->
<b> <font color="FF3300">v2.3</font> is running. </b> It could be picked by 'setup csl' from now on.
<!-- Author: Geumbong -->
<!-- Comment: Tue Feb  6 10;00;16 comment by...Geumbong -->
<pre>There were several falure of restarting csl.
 
 1. Restarting csl from devel version did not stop csl.
    - As we did in previous test, I restarted csl after
      setting up development version. Unfortunately
      I did not save the message. However, it worked
      stopping csl manually from setting up current version.

 2. For a few times, CSL driver received signal 28 and
     did not succeed starting processes. At that time,
     from terminal, "Some problem in config" message 
     was printed, but it's not obvious why. CSL was
     started up in the end.

 3. We may have events burst from streamD from xft test.
    According to Andrew's saying, new fix in trigger table
    has not been tested yet.     
</pre>


<!-- Date: Tue Feb 6 12:58:18 2007 -->
<p>Updated the run_cslcom.sh (cslcom start) script: 
<ul> 
<li>Attempt 5 tries, spaced 1 second apart, sending START to the 
    csl_driver process before exiting with an error.<li> 
<li>Wait up to 14 seconds for the csl_driver to spawn off the 
    necessary 8 processes for running before exiting with 
    an error. The script retries the process count once every 
    second.</li> 
</ul> 
</p>
<!-- Author: Willis -->


<!-- Date: Wed Feb 7 16:00:32 2007 -->
<pre>I changed setup_csl_calib.csh script as follows.  
 
 
envioromental variable,SYS_MAJOR, is set to 2. It is hard-coded 
Restart is sucessful after change. 
  
&lt;pre&gt; 
#set    major_version = $SYS_VERSION 
#set    major_version = $major_version:r 
#set    major_version = $major_version:r 
#set    major_version = $major_version:r 
#set    major_version = $major_version:r 
#setenv SYS_MAJOR       $major_version 
#unset  major_versio 
    setenv SYS_MAJOR '2' 
&lt;/pre&gt;</pre>
<!-- Author: tatsuya -->


<!-- Date: Mon Feb 19 13:27:57 2007 -->
CSL pager is called due to logger queue stuck. L3 turned to green and ace restarted csl. It happened when ace started L2 torture run , 234650. However, I found very weired message from csl receiver. <br><br> 
<li><b>From CSL receiver: </b> 
<pre> 
[17428] csl_receiver_run_event: event 1, byte count is -1969780175 
 Global_event_count: 1 
 ERROR : received run 234650, current run is 1000000005. 
</pre> 
</li> 
<li><b>From CSL logger(b0csl20): </b> 
<pre> 
========================================================= 
hostname:  b0csl20.fnal.gov 
process:   $Id: csl_logger.c,v 1.231 2007/02/02 02:23:12 richard Exp $ 
label:     0x20017100 
threshold: 2 
ended: (Local time) Mon Feb 19 11:54:48 2007 
********************************************************* 
[00 ] Child 21073 terminated (status = 0) at Mon Feb 19 11:54:48 2007 
 
[00 ] Data representation: 0x000e0e00 (SeqRoot little endian) 
 
[17415] ERROR--> hostname: b0csl20.fnal.gov 
         prg name: $Id: csl_logger.c,v 1.231 2007/02/02 02:23:12 richard Exp $ 
         function: csl_logger_process_BOR() 
         datetime: Mon Feb 19 12:22:29 2007 
         src file: /cdf/onln/code/cdfprod/csl/v2_3/src/programs/csl_logger.c 
         src line: 2544 
         src info: [00 ] Run number 234648, 1103521656-type event not accepted; last run = 1000000005 
[00 ] Data representation: 0x000e0e00 (SeqRoot little endian) 
[00 ] Data representation: 0x000e0e00 (SeqRoot little endian) 
.... burst of Data representation..  
</pre> 
This meant that the weird message started being written right after silicon test. 
</li> 
<br>
<!-- Author: Geumbong -->


<!-- Date: Wed Feb 21 12:55:43 2007 -->
<b><font color="666600">CSL v2_4 test in between stores</font></b><br><br> 
<li><b>Stream fail over</b>:<br> 
Once the childlogger killed, open file is closed and queues are filled up. It clears out when the logger is restarted.  
<pre> 
-rwxrwxr--  1 cslace cdf  191M Feb 21 10:16 ir039526.0001test* --> has section 1, 2 
-rwxrwxr--  1 cslace cdf  749M Feb 21 10:20 ir039526.0003test* --> starts with section 3. 
</li> 
<li><b>Events burst</b>:<br> 
<pre> 
-rw-r--r--  1 cslace cdf 1.8G Feb 21 11:16 hr039529.000ftest 
-rw-r--r--  1 cslace cdf  74M Feb 21 11:16 hr039529_1.000ftest --> has run section 15 only. 
 
After that the logger did not write out any for a while and it started with run section 22. 
 
-rw-r--r--  1 cslace cdf 1.8G Feb 21 11:19 hr039529.0016test --> starts with section 22. 
-rw-r--r--  1 cslace cdf  68M Feb 21 11:19 hr039529_1.0016test --> has only section 22. 
</pre> 
</li> 
<li><b>Bad events writing out:</b><br> 
It writes out bad events only in csl receiver scratch area as expected.  
</li><br> 
  
<!-- Author: Geumbong -->
<!-- Comment: Wed Feb 21 23;17;59 comment by...Geumbong -->
<b><font color="#993300">Additional info from test </font></b><br><br>
<b> Killed a child logger process with "-9" to reproduce failure mode</b></li>
<ul>
<ol>
<li> That killed child logger did not leave any message in log files, which is different from SIGQUIT mode. </li>
<li> All the open files were not closed either with following messages.
<pre> Main logger 

 *** Break *** write on a pipe with no one to read it
[00e] Stream has shutdown abruptly!
[00e] Run section truncated: 21
[00e] Performing connection recovery, discarding remainder of this run section.
</pre>
<pre> Other loggers were also affected by break 

[00a] New file opened: /mnt/data1/logger/ace/00/.open/ar039528.0003test
[00a] Opened logger file list /cdf/onln/data/cslace/log/fileCheck/FileList.logger.0a.1171787441 (fd = 10)
[00a] No files are open: file index = 0, section = [3, 3]
[00a] csl_logger_child_recv_in: connection to receiver dropped
[00a] csl_logger_child_recv_in: enqueueing forced CSL_STREAM_SHUTDOWN for local logger

 *** Break *** write on a pipe with no one to read it

</pre>
</li>
<li> When the child logger process was killed, rest events of that run section were discarded and it opened another file starting from next run section.
<pre>
  379K Feb 21 17:55 er039529.0001test.evt <-- .open, b0csl26
  173K Feb 21 17:35 er039529.0016test.evt <-- .open, b0csl21 (failed over)
</pre>
</li>
</ol>
</ul>
The failover thing worked as expected for the run section, however all open files were not closed. 
<br><br>
<b>Lost all events from streamH(run 234792) </b>
<ul>
<ol>
<li>The child logger process was killed soon after test run was activated.</li>
<li>Stream H was not redirected to any other streams 
<pre> Main logger(b0csl20)

 *** Break *** write on a pipe with no one to read it
[00h] Stream has shutdown abruptly!
[00h] Run section truncated: 12
[00h] Performing connection recovery, discarding remainder of this run section.
[00h]      2 events,    0.3 MB total @   0.0 MB/sec
[00j]    256 events,    0.6 MB total @   0.0 MB/sec
[00j]    256 events,    0.3 MB total @   0.0 MB/sec
[00a]   2048 events,    4.6 MB total @   0.0 MB/sec
malloc failed!: Cannot allocate memory
terminate called recursively
[00 ] Child 14300 aborted at Wed Feb 21 10:58:50 2007

[00g]   1024 events,    1.1 MB total @   0.0 MB/sec
[00b]   1024 events,    1.1 MB total @   0.0 MB/sec
[00b]   1024 events,    0.5 MB total @   0.0 MB/sec
</pre>
</li>
</ol>
</ul>
<b> Lost some run sections in the case of event burst(run 234793)</b>
<ul>
<ol>
<li> Stream H- run section 16,19,21,23 missing.</li>
<li> Child logger got segmentation violation error when closing files - new file boundary issue.<br>
<pre>
[00h] New file opened: /mnt/data1/logger/ace/00/.open/hr039529.000ftest

[26369] ERROR--> hostname: b0csl28.fnal.gov
         prg name: $Id: csl_logger_client.c,v 1.5 2007/02/17 09:48:13 richard Exp $
         function: csl_logger()
         datetime: Wed Feb 21 11:16:18 2007
         src file: /cdf/onln/code/cdfprod/csl/v2_4/src/programs/csl_logger_client.c
         src line: 1630
         src info: File size 1.8 GB.  FIXME: Sent warning to RC.
Actual size of file is 1850079423, percentage used is 0, Global_logger_size: 1048576000
Fill: Switching to new file: /mnt/data1/logger/ace/00/.open/hr039529_1.000ftest

[26369] ERROR--> hostname: b0csl28.fnal.gov
         prg name: $Id: csl_logger_client.c,v 1.5 2007/02/17 09:48:13 richard Exp $
         function: csl_logger()
         datetime: Wed Feb 21 11:16:19 2007
         src file: /cdf/onln/code/cdfprod/csl/v2_4/src/programs/csl_logger_client.c
         src line: 1630
         src info: File size 1.8 GB.  FIXME: Sent warning to RC.
Actual size of file is 1900562074, percentage used is 0, Global_logger_size: 1048576000

[26369] ERROR--> hostname: b0csl28.fnal.gov
         prg name: $Id: csl_logger_client.c,v 1.5 2007/02/17 09:48:13 richard Exp $
         function: csl_logger()
         datetime: Wed Feb 21 11:16:20 2007
         src file: /cdf/onln/code/cdfprod/csl/v2_4/src/programs/csl_logger_client.c
         src line: 1630
         src info: File size 1.8 GB.  FIXME: Sent warning to RC.
Actual size of file is 1955963383, percentage used is 0, Global_logger_size: 1048576000
[00h] File size is too large, closing file.
[00h] Writing remaining of run section to a split file.

 *** Break *** segmentation violation
 Generating stack trace...
 0x08060c2d in <unknown> from /cdf/onln/code/cdfprod/csl/v2_4/bin/i686-linux/csl_logger_client
 0x08057066 in <unknown> from /cdf/onln/code/cdfprod/csl/v2_4/bin/i686-linux/csl_logger_client
 0x08056072 in <unknown> from /cdf/onln/code/cdfprod/csl/v2_4/bin/i686-linux/csl_logger_client
 0x08056bcc in <unknown> from /cdf/onln/code/cdfprod/csl/v2_4/bin/i686-linux/csl_logger_client
 0x0805915a in <unknown> from /cdf/onln/code/cdfprod/csl/v2_4/bin/i686-linux/csl_logger_client
 0x0805366e in <unknown> from /cdf/onln/code/cdfprod/csl/v2_4/bin/i686-linux/csl_logger_client
 0x008b1341 in <unknown> from /lib/tls/libpthread.so.0
 0x00a846fe in __clone + 0x5e from /lib/tls/libc.so.6
</pre>
<li> While having the problem in child logger side, main logger dicided to discard the next run section and redirect the stream(H) to other logger. That's why we missed those 4 run sections.<br>
<pre> Main logger(b0csl20) logs

[00h] Stream has shutdown abruptly!
[00h] Run section truncated: 16
[00h] Performing connection recovery, discarding remainder of this run section.
[00h] Choosing new logger node b0csl21-private:9000 for stream h
[00h] Sending handshake for re-connection of stream h
[00h] Connection died on run section section: 16
[00h] Connection recovery resuming on run section: 17
....
[00h] Stream has shutdown abruptly!
[00h] Run section truncated: 19
[00h] Performing connection recovery, discarding remainder of this run section.
[00h] Choosing new logger node b0csl23-private:9000 for stream h
[00h] Sending handshake for re-connection of stream h
[00h] Connection died on run section section: 19
[00h] Connection recovery resuming on run section: 20
...
[00e] Stream has shutdown abruptly!
[00e] Run section truncated: 21
[00e] Performing connection recovery, discarding remainder of this run section.
[00h] Choosing new logger node b0csl28-private:9000 for stream h
[00h] Sending handshake for re-connection of stream h
[00h] Connection died on run section section: 21
[00h] Connection recovery resuming on run section: 22
...
[00h] Stream has shutdown abruptly!
[00h] Run section truncated: 23
[00h] Performing connection recovery, discarding remainder of this run section.
[00h]  65536 events, 1511.2 MB total @  54.0 MB/sec
[00h] Choosing new logger node b0csl25-private:9000 for stream h
[00h] Sending handshake for re-connection of stream h
[00h] Connection died on run section section: 23
[00h] Connection recovery resuming on run section: 24
</pre></li></ol></ul>


<!-- Date: Fri Feb 23 17:48:44 2007 -->
<pre>&lt;p&gt;Run Section database update script problem: The runSect.tcsh 
script got stuck on 19Feb07, and no run sections have been 
entered into the database since then. It appears that the 
sqlplus process it spawns off got stuck. After killing that one, 
and the runSect.tcsh script, the cron job restarted it, and 
run sections are now being entered into the database.&lt;/p&gt; 
 
&lt;p&gt;This is a mystery.&lt;/p&gt;</pre>
<!-- Author: Willis -->


<!-- Date: Wed Feb 28 18:16:05 2007 -->
<p>Additional updates to v2_4 from Rich that were added today: 
<ul> 
<li>Fixed event overflow threshold for file splitting. It will now 
overflow at 1.7GB to prevent Root from doing it first. 
<li>Tried testing for the problem where "kill -9" on one logger caused 
problems on all loggers, observed in Geumbong's test of version 2_4. 
I could not replicate it. 
<li>Made some minor bug fixes to the fail-over code. 
<li>Made some minor bug fixes to the file splitting code. 
<li>Turned on core dumps in setup_csl.sh for debugging: ulimit -c unlimited 
</ul> 
 
Geum Bong and I tested the updated v2_4 code. These are  
results summarized by Geum Bong. 
<pre> Today we had an updated v2_4 code test using quiet time. 
 It was successfully done as below, so I'll go ahead request 
end-of-store test. 
                                                                                       
 1. Stream fail over - 
                                                                                       
 previous :  When killing a logger with -9 option, all files were left 
                 open and saw "break - write on pipe". 
                                                                                       
 today     : Files from the killed logger were left open, but other streams 
                did not get affected by that. All works fine. Lost stream was 
                redirected to the spare logger. 
                                                                                       
 2. Event burst - 
                                                                                       
 Unfortunately could not make it happen during test. Let's hope 
that shift crew does it for us. Once the file size hit 1.8Gb but 
there were no "splitted" files seen.. You may check the output 
files and db files from 
                                                                                       
 b0csl28:/cdf/scratch/cslace 
-rwxr-xr--  1 cslace cdf  134 Feb 28 16:54 hr039748.0017test.db* 
-rwxr-xr--  1 cslace cdf 1.8G Feb 28 16:54 hr039748.0017test* 
-rwxr-xr--  1 cslace cdf  134 Feb 28 16:55 hr039748.0018test.db* 
-rwxr-xr--  1 cslace cdf 860M Feb 28 16:55 hr039748.0018test* 
-rwxr-xr--  1 cslace cdf  134 Feb 28 16:56 hr039748.0019test.db* 
-rwxr-xr--  1 cslace cdf 1.4G Feb 28 16:56 hr039748.0019test* 
-rwxr-xr--  1 cslace cdf  134 Feb 28 16:57 hr039748.001atest.db* 
-rwxr-xr--  1 cslace cdf 1.7G Feb 28 16:57 hr039748.001atest* 
-rwxr-xr--  1 cslace cdf  173 Feb 28 17:02 hr039748.0015test.db* 
-rwxr-xr--  1 cslace cdf 920M Feb 28 17:02 hr039748.0015test* 
</pre> 
We also noticed that in the run control CSL summary page, 
one event was lost. In previous runs, Geum Bong  
found that the data files contained all events, indicating 
accounting glitches in the shared memory memory structure. 
During the test, I noticed times when two events were lost, 
but after a while, one lost event came back.   
</p>
<!-- Author: Willis -->
<!-- Comment: Fri Mar  2 15;58;57 comment by...Geumbong -->
<font color="#006600"><b>v2_4 is linked as current csl and taking data now. v2_5 is made for next round update.</b></font> 


<!-- Date: Fri Mar 2 16:28:35 2007 -->
<font color="#996699"><b> Misteries about currupted db file correction.</b></font><br> 
<ol> 
<li>Today Willis found that there remined db files only in "/mnt/data1/logger/ace/20" where the corrected/created db files are moved to. Moreover, those db files were made silly as below: 
<pre> 
73:b0csl29 % more jr03977d.032aphys.db 
jr03977d.032aphys  966.40073967     7313   235389 13490030   235389 13788983      810      828 
15426453504   0  0  0 
</pre></li> 
<li> Those files were already moved to FCC leaving correct db files for database update last evening except two files(cr03977d.0335phys and jr03977d.0317phys (were in stager/fix_me)).</li> 
<li> Looked like that the correcting db file scirpt ran once already without moving the output data/db files to designated directory. It ran again as set by cron job today in 10am and made that silly db file. </li> 
<li> There were no run section jump found even in OLDdbfiles.</li> 
......... a lot of unsolved.. 
</ol> 
<br> 
Anyway, database info turned out correct even with missing data files. All the data files are checked in SAM. However, I'd like to suggest modifying db file correcting cron job script to check data file instead of db file. 
<!-- Author: Geumbong -->


<!-- Date: Fri Mar 9 09:37:33 2007 -->
<b><font color=red>v2_5 code test</font></b> 
<li><b><font color=blue> I could make events burst and see the files are closed as expected!!! </font></b> No file is left open.  
Great!  
<pre> 
168:b0csl24 % lsr 
total 4.2G 
-rwxrwxr--  1 cslace cdf 1.8G Mar  9 09:17 cr039a71.0001test* 
-rwxrwxr--  1 cslace cdf  602 Mar  9 09:17 cr039a71.0001test.db* 
-rwxrwxr--  1 cslace cdf 719M Mar  9 09:18 cr039a71.000dtest_split* 
-rwxrwxr--  1 cslace cdf  134 Mar  9 09:18 cr039a71.000dtest_split.db* 
-rwxrwxr--  1 cslace cdf 1.8G Mar  9 09:18 cr039a71.000etest* 
-rwxrwxr--  1 cslace cdf  134 Mar  9 09:18 cr039a71.000etest.db* 
-rwxrwxr--  1 cslace cdf  76M Mar  9 09:18 cr039a71.000etest_split* 
-rwxrwxr--  1 cslace cdf  134 Mar  9 09:18 cr039a71.000etest_split.db* 
</pre> 
</li> 
<li>There are core files made from v2_5.  
<pre> 
/cdf/onln/data/cslace/log/csl_logger 
 
-rw-------    1 cslace   cdf      65548288 Mar  9 09:21 core.18515 
-rw-------    1 cslace   cdf      78884864 Mar  9 09:21 core.18516 
-rw-------    1 cslace   cdf      82317312 Mar  9 09:21 core.18533 
-rw-------    1 cslace   cdf      80949248 Mar  9 09:21 core.18528 
-rw-------    1 cslace   cdf      93564928 Mar  9 09:21 core.18519 
-rw-------    1 cslace   cdf      65449984 Mar  9 09:29 core.22216 
-rw-------    1 cslace   cdf      65916928 Mar  9 09:29 core.22203 
-rw-------    1 cslace   cdf      65781760 Mar  9 09:29 core.22197 
-rw-------    1 cslace   cdf      68530176 Mar  9 09:29 core.22201 
-rw-------    1 cslace   cdf      72290304 Mar  9 09:29 core.22218 
-rw-------    1 cslace   cdf      75943936 Mar  9 09:29 core.22206 
</pre>
<!-- Author: Geumbong -->
<!-- Comment: Fri Mar  9 13;42;33 comment by...Willis -->
<pre>Rich points out that some of the core files listed above
are from v2_4. It appears the first ones made at 09:29
are the  v2_5 ones.</pre>


<!-- Date: Mon Mar 12 14:49:47 2007 -->
<b><font color="#6655cc"> code v2_5 test today was successful!</font></b> 
<p> 
<li><b>No core files are generated after end of run: </b> 
<ul> 
The test was taken between 1230 to 1310 during day shift. <br> 
There were no core files are made after 4 times end of run.  
</ul> 
</li> 
<li><b>Double check file boundary issue</b> 
<ul> 
It worked again as expected.  
<pre> 
53:b0csl23 % cd /mnt/data1/logger/ace/specialFiles/ 
54:b0csl23 % lsr 
total 2.9G 
-rwxr-xr--  1 cslace cdf 1.8G Mar 12 13:09 br039b28.0001test* 
-rwxr-xr--  1 cslace cdf  563 Mar 12 13:09 br039b28.0001test.db* 
-rwxr-xr--  1 cslace cdf 229M Mar 12 13:09 br039b28.000ctest_split* 
-rwxr-xr--  1 cslace cdf  134 Mar 12 13:09 br039b28.000ctest_split.db* 
-rwxr-xr--  1 cslace cdf 905M Mar 12 13:09 br039b28.000dtest* 
-rwxr-xr--  1 cslace cdf  134 Mar 12 13:09 br039b28.000dtest.db* 
</pre> 
</ul> 
</li>  
<li><b><font color="#ff3399">miscellaneous</font></b> 
<ul> 
Following files infer two things; <br> 
one is that the split files are not caught by stager and the other is that the file splitted in v2_4 code.  
<pre> 
b0csl28:/mnt/data1/logger/ace/00/ 
-rwxrwxr--  2 cslace cdf  433K Mar  5 11:11 ir0398b9.0001cosm_split* 
-rwxrwxr--  1 cslace cdf   134 Mar  5 11:11 ir0398b9.0001cosm_split.db* 
 
b0csl28:/mnt/data1/logger/ace/specialFiles 
-rw-r--r--  1 cslace cdf 1.8G Mar  5 11:12 ir0398b9.0002cosm 
-rw-r--r--  1 cslace cdf 1.8G Mar  5 11:12 ir0398b9_1.0002cosm 
-rw-r--r--  1 cslace cdf 1.8G Mar  5 11:13 ir0398b9_2.0002cosm 
-rw-r--r--  1 cslace cdf 449M Mar  5 11:13 ir0398b9_3.0002cosm 
 
</pre> 
As shown above, file boundary worked once in v2_4 for the first section event burst, but did not work from second section. It worked anytime in v2_5 though. 
<p> 
Also we should discuss how to care split file.  
</ul> 
</li>  
</ul>
<!-- Author: Geumbong -->
<!-- Comment: Wed Mar 14 14;39;10 comment by...Geumbong -->
<b> File manifest counted the split file too! </b>
<p>
Those event burst test file is found from fileCheck directory as below:
<pre>
None	1825361943	 Mon Mar 12 13:06:47 2007	1173722807	br039b28.0001test	br039b28.0001test
None	 239689592	 Mon Mar 12 13:06:51 2007	1173722811	br039b28.000ctest_br039b28.000ctest_split
None	 946927425	 Mon Mar 12 13:07:29 2007	1173722849	br039b28.000dtest	br039b28.000dtest
</b>
File name is written somewhat weirdly, however we did not lose a file!