<!-- Title: 10:44:32  Mon Nov 20 2006 -->
<!-- Crewchief: cdf_csl -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
Happy Thanksgiving!


<!-- Date: Mon Nov 20 10:51:45 2006 -->
Let's start in new page. <br><br> 
<li>To prevent last week's CSL problem(one .open directory missing), I modified child logger script to check if the directory exists.  
<pre> 
 
  if ( -e /mnt/data1/logger/ace/00/.open) then 
    echo "" 
    echo "directory exist /data1"  
    echo "" 
  else  
    mkdir /mnt/data1/logger/ace/00/.open 
  endif 
 
  if ( -e /mnt/data2/logger/ace/00/.open) then 
    echo "" 
    echo "directory exist /data2"  
    echo "" 
  else 
    mkdir  
 
</pre></li><br> 
<li> I moved all the scripts from "~cslace" to "~cslace/bin" and changed .aliases accordingly. </li>  
<!-- Author: Geumbong -->


<!-- Date: Mon Nov 20 16:31:43 2006 -->
<pre>Oh new page... 
 
I found incorrect db file on b0csl29. 
Failed to create the py file for:  /mnt/data1/logger/error/jr037c4d.048dphys (Nov 20 15:13) 
The result running Edm_Eventlister is missing 3 section. 
.... 
228429 1171 18676001 
228429 1175 18676032 
.... 
 
I modified script to make correct db file. 
How to use: 
Log on b0csl** as "cslace". 
&gt;cd Script 
&gt;Corrdbfile.sh &lt;filename&gt; 
That's all.  
We can get correct db file. 
For example, output as follows 
   
b0csl29 % ./Corrdbfile.sh jr037c4d.048dphys 
Make modify db file 
14970324109 : 1165 
Now checking event list of jr037c4d.048dphys, Wait for a few minuites 
Make db file list 
Change to correct db file, wrong one is moved to ... 
Made correct db file 
97.456u 4.046s 1:43.54 98.0%    0+0k 0+0io 24pf+0w 
 
So far, we need to move file to /mnt/data1/logger/ace/20/ after check if db file is really correct.</pre>
<!-- Author: tatsuya -->
<!-- Comment: Mon Nov 20 17;23;32 comment by...tatsuya -->
<pre>Additionaly,,
The result of event lister is kept in EventLog directry and wrong db file also moved to EventLog/OLDdbfile, so far.</pre>


<!-- Date: Tue Nov 21 08:09:58 2006 -->
<pre> While taking csl torture run for failure mode test, we got csl stuck at logger queues. No child loggers was killed at the moment. We had to restart CSL. Following message are from log of main logger. It seemed streamB logger was broken.  
 
====================================================== 
[00d]  32768 events,  137.5 MB total @  19.4 MB/sec 
[00h]    256 events,    1.3 MB total @   0.0 MB/sec 
[00j]   1024 events,    2.7 MB total @   0.0 MB/sec 
[00b]   2048 events,    2.0 MB total @   0.0 MB/sec 
[00h]    512 events,    0.6 MB total @   0.0 MB/sec 
[00d]  65536 events,  157.1 MB total @   8.7 MB/sec 
 
 *** Break *** write on a pipe with no one to read it 
[00d] Stream has shutdown abruptly! 
[00d] Performing connection recovery, discarding remainder of this run section. 
[00d] Cannot connect to logger node. 
[00d] Will retry 3 times before switching to another logger node. 
[00d] Sending handshake for re-connection of stream d 
[00c]   2048 events,    0.8 MB total @   0.0 MB/sec 
[00a]   2048 events,   11.5 MB total @   0.1 MB/sec 
[00j]   2048 events,    0.7 MB total @   0.0 MB/sec 
[00e]   4096 events,    1.2 MB total @   0.0 MB/sec 
[00h]   1024 events,    0.7 MB total @   0.0 MB/sec 
[00g]   4096 events,    2.3 MB total @   0.0 MB/sec 
[00d] Stream has shutdown abruptly! 
[00d] Performing connection recovery, discarding remainder of this run section. 
[00d] Cannot connect to logger node. 
[00d] Will retry 2 times before switching to another logger node. 
[00d] Sending handshake for re-connection of stream d 
===========================================</pre>
<!-- Author: Geumbong -->


<!-- Date: Tue Nov 21 17:34:58 2006 -->
<pre>CSL hardware/software failure test. 
Today what we did are 
1) One SATAbeast(SATA1) is down. 
In this case, CSL did not work to fail disk connection. We had to reboot all loggers, SATAbeast, fibre channel switch. 
Today it took over 1hr to recover all system. I think that we need documentation about procedure to recover more quickly. 
 
2) Kill child logger process on one nodes. 
When child logger process is killed, this stream is redirected immediately. opend data of this logger nodes are lost but CSL is able to resume taking data at once. 
We don't need to restart CSL/L3 to go back in situ. just end run and restart run are required.   </pre>
<!-- Author: tatsuya -->


<!-- Date: Wed Nov 22 11:07:36 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=cs-logger&action=view&page=-6&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=cs-logger&action=view&page=-7  width='300' align='top'></a></td><td>This is interesting plots.  
Today around 2am, number of logger nodes go up to 20. 
According to checking process, this process does not use cpu and not be zombi. 
Add this issue in the list we want to discuss with Rich. 
</td></tr></table>
<!-- Author: tatsuya -->
<!-- Comment: Wed Nov 22 11;16;47 comment by...tatsuya -->
<pre>I may understand it. ACE tried to run with another partition (no silicon run) around 2am. 
 </pre>


<!-- Date: Sat Nov 25 18:04:24 2006 -->
<pre>We got error message. 
First this message is spewed. 
Run: [228683], Partition [0].  Receiver [7]. Warning !!!! Event did NOT have a stream !!!!  Counter = [0] 
And then this message is following. 
[6851] ERROR--&gt; hostname: b0csl20.fnal.gov 
         prg name: $Id: csl_receiver.c,v 1.100.4.2 2006/11/15 16:50:23 richard Exp $ 
         function: csl_receiver_child_body() 
         datetime: Sat Nov 25 16:55:10 2006 
         src file: /cdf/onln/home/csl/csl_code_b0csl20/build_csl_2006_1117/cs-logger/src/programs/csl_receiver.c 
         src line: 1276 
         src info: no more than 16 receiver client connections available 
</pre>
<!-- Author: tatsuya -->


<!-- Date: Sun Nov 26 18:45:54 2006 -->
<big><font color=red> CSL Failure test plan </font></big> 
<br> 
<ol>  
<b><li> One SATAbeast(RAID) down </b><br> 
Disk IO error for '/mnt/data1' in b0csl22/24/26/28 and for '/mnt/data2' in b0csl23/25/27/29. <br> 
In fact, two luns in each stream are cross-linked to SATAbeast 1 and 2 for balancing raid usage.<br> 
We expect to see stream redirection if the output is not readable, but it didn't happen.<br> 
CSL kept getting stuck due to logger queues and did not work run until being restarted.<br><br> 
<font color=green><b> To recover:</b> After SATAbeast up, we had to reboot both SATAbeast, fibre channel switch and all logger to recover all system.It took 20-30 min. </font><br><br> 
<font color=blue><b> What we learned:</b> CSL should redirect stream to other lun(it will make use of the other RAID) if it could not find output directory.  </font>  
</li><br><br> 
 
<b><li> One fibre channel switch down.</b> <br> 
All logger nodes should use alternate path automatically. We will check the software behaviors. Disk IO error is expected from this failure. That means CSL may not work in disk io error until code is fixed. <br><br> 
 
<b><font color=green> To recover: </b> Need to reboot all logger, switch and SATAbeast after switch back. It takes 20-30. </font></li><br><br>  
 
<b><li> SATAbeast controller failure </b><br> 
All logger nodes should use alternate path automatically. However, disk IO error is expected. That means CSL may not work in disk io error until code is fixed.<br><br> 
 
<b><font color=green> To recover: </b> ... </font></li> 
<br><br> 
<b><li> One disk of RAID failure </b><br> 
Can be tested by pulling off one disk from raid. The spare disk should work immediately (Does it work immediately without any touch? Shouldn't we replace the disk by spare physically?).  <br>Need to check the effect for taking data using CSL software. 
<br><br> 
<font color=green><b> To recover: </b> ... </font></li><br><br> 
 
<b><li> Receiver down </b><br> 
CSL can be started immediately in spare receiver without any changes on configuration. However, the <i> L3 output node </i> and <i> CSLHost for consumer script </i> should be changed. It should be done in 10 min. <br> It was tested once in between stores. At that time, CSL was shutdown in normal way. <br> Will see how software get back to work in sudden death. <br><br> 
<font color=green><b> To recover:</b> The spare receiver should work until b0csl20 is back on work. When it's back, Restarting csl and changing CSL hostname for L3 output and consumer are only necessary. </font></li><br><br> 
 
<b><li> Logger down </b><br> 
The stream written in this logger should be redirected to another logger. 
<br><br> 
<font color=green> <b>To recover: </b> Need end run and start a new run for use. Restarting CSL is not necessary.</font></li> 
<br><br> 
<b><li> Use spare switch </b><br> 
Spare switch is already configured. To test this, it will be necessary to move cables and reboot receiver/logger machines.<br><br> 
<font color = green><b> To receiver:</b> Do it the other way around.  </font></li> 
</ol> 
 
<li> For hardware failure cases, csl can run in degraded state if software can handle failure mode. It needs to reboot all the machines to be in good state.</li><br>  
<li> Most hardware failures are related to the disk IO error. I do think "Immediate redirection" of stream is the only solution for smooth running. </li><br>
<!-- Author: Geumbong, Tatsuya -->


<!-- Date: Mon Nov 27 16:36:29 2006 -->
<pre> 
 I cleaned up the log files. 
 
 - deleted old log files which were made before Sep. 
   (CSL started valuable run from mid Oct). 
 
 - deleted unused directories. 
 
 - tar and gziped logs from prototype machine(b0csl02-06) 
   and moved them to b0gateway:/cdf/scratch/cslace/. 
 
 Scripts will be available soon to zip files if it got 
more than 2 mo. old. </pre>
<!-- Author: Geumbong -->


<!-- Date: Tue Nov 28 23:43:08 2006 -->
<b> CSL software test </b><br><br> 
New csl code(build_csl_2006_1128_test1) was made from Rich's latest fix and tested in low lum. I put the test results after Rich's comment about fix.<br> 
<pre> 
1. DB Files should be fixed in most circumstances. 
     I have not tested if they still have problems with interrupted  
run sections because I did not explicitly fix this problem yet.  
Please check though just in case. 
</pre> 
 --> We tried to make run section gap by halting run for 2~3 minutes, but it did not make the section gap. Run section was continuous in both db file and Eventlister. Test run number was x37e0a and x37e0b.  
<br><br> 
<pre> 
2. Auto-creation of output directories should work properly. The  
check should run each time a file is opened.</pre> 
 --> It worked! Child logger made ".open" directory if it's not there. "Starting script" checked and made ".open" directory when the child logger got started.<br><br> 
<pre> 
3. The csl_receiver connection limit was increased. </pre> 
 --> We will try to confuse csl in between stores. This was not tested. <br><br> 
<pre> 
4. Preliminary support for failover if the all available LUNs are  
filled, and if output directories are not accessible for some reason.  
This support has not been tested yet. 
</pre> 
 --> We made disk I/O error by moving output directory(/mnt/data1/logger/ace to /mnt/data1/logger/ace2) in the middle of data taking. When it moved suddenly, logger started using other lun(/mnt/data2/logger/ace). That worked very well! In  current run(#228878) streamA is using "/mnt/data2/logger/ace". CSL will use /mnt/data1/logger/ace when new run starts.   
<!-- Author: Geumbong, Tatsuya -->


<!-- Date: Wed Nov 29 18:14:23 2006 -->
Today's CSL problem.. 
<br><br> 
 
We've got pattern for l3-csl problem which lead ace to restart csl. Some problem made in L2 level caused L3 send BAD EVENTS to CSL. At this time(See elog at 2006.11.29 1000hr), CSL logger queues were full with streamD(bad event monitoring) events, which was checked from run summary. We concluded that csl did not handle bad events appropriately. <br><br> 
In this morning, Tatsuya happend to pass by control room and he restarted csl immediately. <font color = red> If ace did not recognize fully stuck logger queues and tried to start new run, I presume that there would happen the "MAX CLIENT" error from duplicated L3 output. </font> 
 
<br><br> 
<b>Comments from Mako and Kevin</b> 
<pre> 
 
> I believe they were archived to a special area 
> in the old system on a one event per one file basis. 
 
This is true; it will be quite slow. 
 
> I don't remember where that code lived, although the error that they 
> appear to be causing suggests that it used to live in the old cs_logger 
> and has somehow been effectively disabled by the software upgrades. 
 
Error files are handled by functions in src/library/BadEvent.c. 
If calls to BadEvent_* functions are gone, we need to 
put them back in. This would be the receiver node's 
job --- do we have disk space there? 
 
I believe the logger will also honour the stream 
bits for these events, so they will also be written 
in the raw data files, if the Level-3 chooses to 
set one or more stream bits. (Error events are 
"converted" to normal events once error file is 
written.) 
                         mako 
</pre>
<!-- Author: Geumbong -->


<!-- Date: Thu Nov 30 15:06:29 2006 -->
<pre>Today's CSL test summary, 
Try to do harware test, logger node is down, using new code. After unplugging private network on b0csl27, logger queue filled up immedietely and CSL stopped.  
Interesthing thign is that CSL worked without restarting after end of run (It works from config state).  
To check behaivour, Rich modify debbugin code to fix it.  </pre>
<!-- Author: tatsuya -->


<!-- Date: Fri Dec 1 21:11:33 2006 -->
<b> CSL TEST TODAY</b> 
<br><br> 
<ol> 
<b><li> File manifest checking: </b> <br><br> 
In final version, it started working. It records all opened/closed files as below: <br><br> 
<pre> 
 Filelist location : /cdf/onln/data/cslace/log/fileCheck 
 File Names **************************************8 
-rwxrwxr--    1 cslace   cdf          1.1K Dec  1 16:23 FileList.logger.15a.1163993417* 
-rwxrwxr--    1 cslace   cdf           623 Dec  1 18:59 FileList.logger.15a.1165011904* 
-rwxrwxr--    1 cslace   cdf           356 Dec  1 19:01 FileList.logger.0j.1165017022* 
-rwxrwxr--    1 cslace   cdf           534 Dec  1 19:01 FileList.logger.0h.1165017021* 
-rwxrwxr--    1 cslace   cdf           445 Dec  1 19:10 FileList.logger.0a.1165017022* 
-rwxrwxr--    1 cslace   cdf            89 Dec  1 19:26 FileList.logger.0i.1165022525* 
-rwxrwxr--    1 cslace   cdf           623 Dec  1 20:15 FileList.logger.0c.1165017021* 
-rwxrwxr--    1 cslace   cdf           534 Dec  1 20:15 FileList.logger.0g.1165017021* 
-rwxrwxr--    1 cslace   cdf           534 Dec  1 20:15 FileList.logger.0d.1165017021* 
-rwxrwxr--    1 cslace   cdf           534 Dec  1 20:15 FileList.logger.0e.1165017021* 
-rwxrwxr--    1 cslace   cdf           534 Dec  1 20:16 FileList.logger.0b.1165017021* 
******************************************************* 
 File contents **************************************** 
Tape     261750266       Fri Dec  1 18:08:19 2006       1165018099      jr037f32.0001phys       jr037f32.0001phys 
Tape      27511914       Fri Dec  1 18:14:56 2006       1165018496      jr037f34.0001phys       jr037f34.0001phys 
Tape      25337081       Fri Dec  1 18:21:26 2006       1165018886      jr037f35.0001phys       jr037f35.0001phys 
Tape      44868485       Fri Dec  1 19:01:38 2006       1165021298      jr037f3a.0001phys       jr037f3a.0001phys 
******************************************************* 
</pre> 
In file contents, first file name is opened file name and later one is closed file name. If later file name is blank, this means the file is not closed yet.. <br><font color=red>We hope that stager edit this for "Tape|Look|None" check. </font></li><br><br> 
<b><li> db file curruption </b><br><br> 
DB file curruption has not seen since the test on 11/28/06. Rich said that he modified the db file early in this week. We will keep an eye on this.. </li><br><br> 
<b><li> Bad event </b><br><br> 
Unfortunately we could not make bad event today. We tried to follow exactly what ace did when bad events were made, but it did not happen today. I'll summerize the symptom and get advised from L2/L3 experts. <br> 
Rich is trying to figure it out logically from code.</li><br><br>  
<b><li> logger disconnection during data taking: </b><br><br> 
This is not solved yet. CSL got stuck due to full logger queues.  
Sometimes message queues got cleaned up by itself, but sometimes it needs csl restart. I believe Rich got enough information for this problem. </li></ol>
<!-- Author: Geumbong -->


<!-- Date: Mon Dec 4 17:17:05 2006 -->
<b> CSL OPEN FILE CLEANUP POLICY </b> 
<br><br> 
Many Open files from previous test(2006.12.01) were deleted because they were null files (nothing written inside). Here I'd like to write open file cleanup policy. 
<br> 
<pre> 
 - Delete files if it is null file (size = 306B). They can be made from CSL test.  
 - Move sizable files to  "/mnt/data1/logger/ace/error/." and make db files and resend to "/mnt/data1/logger/ace/20/." to be picked up. If we can fake the complex number in head of each run section summary line, it will be moved to FCC.  
</pre> 
 I rarely expect to have open files from normal running since new code closes files in any problems. However this can be happened in the case of <font color=blue> Stream Redirection</font>. Please leave your comments about this. <br><br>
<!-- Author: Geumbong -->


<!-- Date: Mon Dec 4 22:42:03 2006 -->
<pre>Spare CSL switch is working!! 
We tried to swap swith to spare one. What we did were switch all cable to spare one. 
We don't need to reboot any machines although L3 and CSL restart is required. 
And then we did network test quickly to check configurations. 
We obtained thoughput over 200MB/s and it works well. 
So We keep using spare one for next store.</pre>
<!-- Author: tatsuya,guembong -->


<!-- Date: Tue Dec 5 22:32:56 2006 -->
<pre>CSL got bad events again in this morning(1006.12.05.0900). 
Restarting CSL was not successful due to defunct mornitoring process. 'restartCSL' script was modified to issue possible command in case of problems. Nevertheless I got paged for mis-restarting CSL. I'll work on it. </pre>
<!-- Author: Geumbong -->


<!-- Date: Wed Dec 6 23:41:09 2006 -->
Message from Pasha about run section info.<br>  
<pre> 
 
Dear All, 
 
CDF2_RUNSECTIONS table is important and needs to be filled. 
 
It is being used in more than one place in the offline, 
luminosity group updates the offline luminosity. 
 
                                 -best, Pasha 
 
root [0] cdfofprd.DescribeTable("CDF2_RUNSECTIONS") 
RUN_NUMBER                     NUMBER 
SECTION_NUMBER                 NUMBER 
BIRTH_TIME                     NUMBER 
LOW_EVENT                      NUMBER 
HIGH_EVENT                     NUMBER 
LUM_AVERAGE_ONLINE             NUMBER 
LUM_INTEGRAL_ONLINE            NUMBER 
LUM_AVERAGE_OFFLINE            NUMBER 
LUM_INTEGRAL_OFFLINE           NUMBER 
DATA_QUALITY                   RAW 
LUMINOSITY_VERSION             NUMBER 
LIVETIME                       NUMBER 
RUNTIME                        NUMBER 
ID                             NUMBER 
</pre>
<!-- Author: Geumbong -->


<!-- Date: Wed Dec 6 23:42:47 2006 -->
Report about Bad Event test from Vadim<br> 
<pre> 
 
Just to let people know what was the result of my attempt to break L3: 
 
I was trying to set the pointer to the L2 trigger bits in the TL2D to 
some huge/absurd number. That was the way that we broke L3 in the past, 
because L3 looks for prereq in the L2 bits. However, what I forgot was 
that Bill B. and I we put some code in the fer code that relies on  
this  when we padded the bank  in order to accomodate more than 150 bits 
some months ago. So all I was doing was to break the readout code in the 
crate before even getting to the L3. I will try to come up with some 
other way tomorrow, but at this point, if we could load a previous fer 
version into that crate that would do it. 
 
Vadim 
 
</pre>
<!-- Author: Geumbong -->


<!-- Date: Fri Dec 8 02:05:35 2006 -->
<pre>Go back to main switch. 
I swap switch from spare to main. 
I made sure performance using CSL torture. It works well. </pre>
<!-- Author: tatsuya -->


<!-- Date: Sun Dec 10 16:14:03 2006 -->
One file was left open for 21 hours,ir0381f0.0001cosm. I paste the error log about the run. <font color=red> Amazingly, this file is closed aroun 1500 today with termination of logger process.</font>  
 
<pre> 
(RUN Comment) rundb.RunComment@5e2517 
  RunNumber   229872 
  CreateTime  2006-12-09 18:28:28.0 
  TextComment RunSet: ACE_COSMIC_NOTRACKS  Trigger: COSMICS_NOTRACKS [9,731,639]  
  Name        Nagesh 
  State       ACTIVATE 
  ELog        true 
CT: 2006.12.09 18:28:54 
28'54" 2 crate/s: b0svt09(115), b0svt08(117),  in error. [RXPT] 
(RC)   18:29:18 End -> IDLE 
</pre> 
<br> 
At that time, ace used partition 1 and terminated that run soon after activating. He went back to partition 0. <br>I could not find any clue from log files why the file is not closed at that time.   
  
<!-- Author: Geumbong -->


<!-- Date: Tue Dec 12 14:35:20 2006 -->
<b><font color="#006666">Script is ready to use for modifying/creating db files!</font></b><br><br> 
Check "System Recovery" process first. <br><br> 
<li>Modify db file - truncated db file case</li> 
<pre> 
63:b0csl22 % ./CreateDBFile.sh 
usage ./CreateDBFile.sh <Filename> 
 
64:b0csl22 % ./CreateDBFile.sh ar03827a.03e5phys 
Modify db file 
15073936357 : 997 
Now checking event list of ar03827a.03e5phys, Wait for a few minutes 
Make db file list 
Change to correct db file, wrong one is moved to /cdf/onln/home/cslace/Script/EventLog/OLDdbfile/ 
Made correct db file 
Moved to /mnt/data1/logger/ace/20/ --------> DONE 
118.912u 4.394s 2:05.81 98.0%   0+0k 0+0io 169pf+0w 
</pre></li><br> 
<li>Create db file - after move open file to /mnt/data1/logger/ace/error/ <br> 
Make sure the file is readible before creating db file. Once db file is made, this file will be moved to FCC. 
<pre>  
57:b0csl23 % lsr /mnt/data1/logger/ace/error 
total 925M 
-rw-r--r--  1 cslace cdf 925M Dec 10 14:50 br038228.001fphys 
58:b0csl23 % ./CreateDBFile.sh br038228.001fphys 
Create db file 
Now checking event list of br038228.001fphys, Wait for a few minutes 
 
70:b0csl23 % ./CreateDBFile.sh br038228.001fphys 
Create db file 
Now checking event list of br038228.001fphys, Wait for a few minutes 
Make db file list 
root [0] 
Processing Createdbfile.C("br038228.001fphys",229928,969285999,6166)... 
Made new db file 
Created db file is saved in /cdf/onln/home/cslace/Script/EventLog/NEWdbfile/ 
Moved to /mnt/data1/logger/ace/20/ --------> DONE 
82.693u 4.001s 1:28.55 97.8%    0+0k 0+0io 2pf+0w 
 
71:b0csl23 % lsr /mnt/data1/logger/ace/20 
total 1.2G 
-rw-r--r--  1 cslace cdf 244M Nov 15 10:54 br037a34.0001phys 
-rw-r--r--  1 cslace cdf 925M Dec 10 14:50 br038228.001fphys 
-rw-r--r--  1 cslace cdf  892 Dec 12 13:47 br037a34.0001phys.db 
-rw-r--r--  1 cslace cdf  494 Dec 12 14:11 br038228.001fphys.db 
</pre></li>  
<!-- Author: Geumbong -->


<!-- Date: Tue Dec 12 15:25:32 2006 -->
<b> File size exceeds 1GB. </b><br><br> 
No sign is found from run control. Child logger was crashed when the file size exceeds 1GB. Fortunately, child logger was revived right after craching. Will check others too.     
 
From stream c child logger log files: 
<pre> 
[00c] New file opened: /mnt/data1/logger/ace/00/.open/cr03827a.0268phys 
 
[27987] ERROR--> hostname: b0csl24.fnal.gov 
         prg name: $Id: csl_logger_client.c,v 1.1.2.27 2006/12/11 18:05:23 richard Exp $ 
         function: csl_logger() 
         datetime: Tue Dec 12 03:46:26 2006 
         src file: /cdf/onln/home/csl/csl_code_b0csl20/build_csl_2006_1211/cs-logger/src/programs/csl_logger_client.c 
         src line: 1477 
         src info: File size 1.8 GB.  FIXME: Sent warning to RC. 
Actual size of file is 1886049296, percentage used is 0, Global_logger_size: 1048576000 
Fill: Switching to new file: /mnt/data1/logger/ace/00/.open/cr03827a_1.0268phys 
 
[27987] ERROR--> hostname: b0csl24.fnal.gov 
         prg name: $Id: csl_logger_client.c,v 1.1.2.27 2006/12/11 18:05:23 richard Exp $ 
         function: csl_logger() 
         datetime: Tue Dec 12 03:46:28 2006 
         src file: /cdf/onln/home/csl/csl_code_b0csl20/build_csl_2006_1211/cs-logger/src/programs/csl_logger_client.c 
         src line: 1477 
         src info: File size 1.8 GB.  FIXME: Sent warning to RC. 
Actual size of file is 1971315444, percentage used is 0, Global_logger_size: 1048576000 
 
[27987] ERROR--> hostname: b0csl24.fnal.gov 
         prg name: $Id: csl_logger_client.c,v 1.1.2.27 2006/12/11 18:05:23 richard Exp $ 
         function: csl_logger() 
         datetime: Tue Dec 12 03:46:29 2006 
         src file: /cdf/onln/home/csl/csl_code_b0csl20/build_csl_2006_1211/cs-logger/src/programs/csl_logger_client.c 
         src line: 1477 
         src info: File size 1.8 GB.  FIXME: Sent warning to RC. 
Actual size of file is 2057139832, percentage used is 0, Global_logger_size: 1048576000 
[00c] File closed: /mnt/data1/logger/ace/00/cr03827a.025aphys 
[00c] [8841 events; 1003347516 bytes:written; 1004688459 bytes:filesize; 1 MB/s] 
[00c] DB file written: /mnt/data1/logger/ace/00/cr03827a.025aphys.db 
 
[00c] New file opened: /mnt/data1/logger/ace/00/.open/cr03827a.0276phys 
 
 *** Break *** segmentation violation 
 Generating stack trace... 
 0x08060459 in <unknown> from /cdf/onln/home/csl/csl_code_b0csl20/newcsl/cs-logger/bin/i686-linux/csl_logger_client 
 0x08056ba2 in <unknown> from /cdf/onln/home/csl/csl_code_b0csl20/newcsl/cs-logger/bin/i686-linux/csl_logger_client 
 0x08055fc6 in <unknown> from /cdf/onln/home/csl/csl_code_b0csl20/newcsl/cs-logger/bin/i686-linux/csl_logger_client 
 0x08056722 in <unknown> from /cdf/onln/home/csl/csl_code_b0csl20/newcsl/cs-logger/bin/i686-linux/csl_logger_client 
 0x08058be8 in <unknown> from /cdf/onln/home/csl/csl_code_b0csl20/newcsl/cs-logger/bin/i686-linux/csl_logger_client 
 0x08053690 in <unknown> from /cdf/onln/home/csl/csl_code_b0csl20/newcsl/cs-logger/bin/i686-linux/csl_logger_client 
 0x00b82341 in <unknown> from /lib/tls/libpthread.so.0 
 0x010326fe in __clone + 0x5e from /lib/tls/libc.so.6 
</pre> 
 
That child logger process was aborted : 
<pre> 
[00c] Run number is 230010 
[00c] Lookarea policy 3 in effect. 
[00c] Run type index 1 in effect. 
[00c] Run type phys in effect. 
[N/A] Child 27987 aborted at Tue Dec 12 03:51:24 2006 
</pre> 
 
Could not find any other message from log files.. 
<!-- Author: Geumbong -->