<!-- Title: 10:26:36  Sat Dec  1 2001 -->
<!-- Crewchief: CO -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
CO LOG


<!-- Date: Sat Dec 1 10:27:14 2001 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-147&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-147' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-148&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-148' width='300'></a></center>
<!-- Author: hray/level 2 -->
<!-- Comment: Sat Dec  1 10;31;00 comment by...hray/level2 -->
CO's Please Note that LEVEL 2 IS NOW WORKING!!  These two plots from TrigMon are what you should always see.   The left plot is the clustering boards for L2.  If you see anything off the diagonal, please page L2.  The right plot is the number of words each board sends to the decision board in the l2 decision crate.  We have a description of what you should see at the top of the plot.  Please pay attention to this!  Last night for the 1M MINIMUM_BIAS run there was no information from level2 - none of the boards in the decision crate were sending.  Again, if you see some boards not sending that are supposed to be (ie boards in slots 0-1,1-2,2-3,3-4) please page L2


<!-- Date: Sat Dec 1 20:30:51 2001 -->
<pre>Kaori just recompiled Stage0 consumer 
Instructions to run it: 
0) use the b0dap56 machine 
1) open new terminal 
2) check if Stage0 is running (check with ps -ef | grep Stage0). If so kill it  
3) source ~cdfsoft/cdf2.cshrc 
4) setup cdfsoft2 4.1.0 
5) cd /data1/consumer/newstage0 
6) srt_setup -a 
7) setenv CONSUMER_WIRTE_SUMMARY_DB sammy 
8) $&gt; ./Consumer/scripts/runStage0 &lt;partition_Id&gt;</pre>
<!-- Author: Antonio S -->
<!-- Comment: Sun Dec  2 00;02;41 comment by...Antonio -->
<pre>No longer in use. Start in the ways suggested by kaori some entries below</pre>
<!-- Date: Sat Dec 1 21:32:04 2001 -->
<table><tr><td><a href=/elog/notebooks/consumer-ops/Sat_Dec__1_2001_213052.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-150  align='top'></td><td>Next run with data and with COT on; we should check if the low timing phenomenon occurs again... (plot from Stefano)</a></td></tr></table>
<!-- Author: Antonio -->


<!-- Date: Sat Dec 1 23:50:05 2001 -->
<pre>When the collisions start, and we start taking data with 
the new physics trigger table, please use: 
 
./startMon_new.sh consumername &lt;partitionID&gt; 
 
to start online monitors,  and 
 
./startEvd.sh &lt;partitionID&gt; 
 
for eventdisplay,  and 
 
./startDisplay.sh consumername 
 
to start HistoDisplayMain (GUI for the consumer Histo-Display)</pre>
<!-- Author: kaori -->


<!-- Date: Sun Dec 2 00:01:39 2001 -->
<pre>Monitor XMon memory usage 
do on a xterm on b0dap55 
top 
then press M (the processes will be sorted in memory) 
if size of XMon (on top) should oscillate between 90 and ~150M 
but restart XMOn (ctrl-c on b0dap51 and then start again XMon) if in the Swap free shrinks and used inflates! 
 
 
</pre>
<!-- Author: Antonio -->



<!-- Date: Sun Dec 2 02:12:13 2001 -->
<pre>XMon died. This is the screen dump: 
 
ConsumerInput Begin processing 8800th record. Run 13396 
3, Trigger 43768 
%ERLOG-e XMon: Prefred trigger 0  EM8 has increased to  
44.1202  
         Hz, over 50% of it's initial rate of 0 Hz  XMo 
n SL1DMonitor()  
          2-Dec-2001 01:17:17 run = 133963 event = 4428 
4 
%ERLOG-e XMon: Prefred trigger 0  EM8 has increased to  
1.01593e+11  
         nb, over 90% of it's initial x-section of 0 nb 
  XMon SL1DMonitor()  
          2-Dec-2001 01:17:17 run = 133963 event = 4428 
4 
%ERLOG-e XMon: Level 1 trigger 0  L1_EM12_&_CLC has inc 
reased to 0.00234607  
         Hz, over 50% of it's initial rate of 0 Hz  XMo 
n SL1DMonitor()  
          2-Dec-2001 01:17:17 run = 133963 event = 4428 
4 
 
%ERLOG-e XMon: Level 1 trigger 0  L1_EM12_&_CLC has inc 
reased to 5.40216e+06  
         nb, over 90% of it's initial x-section of 0 nb 
  XMon SL1DMonitor()  
          2-Dec-2001 01:17:17 run = 133963 event = 4428 
4 
Socket not valid. 
%ERLOG-e XMon: Prefred trigger 1  MB_CLC has increased  
to 0.101663  
         Hz, over 50% of it's initial rate of 0 Hz  XMo 
n SL1DMonitor()  
          2-Dec-2001 01:17:17 run = 133963 event = 4428 
4 
%ERLOG-e XMon: Prefred trigger 1  MB_CLC has increased  
to 2.34094e+08  
         nb, over 90% of it's initial x-section of 0 nb 
  XMon SL1DMonitor()  
          2-Dec-2001 01:17:17 run = 133963 event = 4428 
4 
%ERLOG-e XMon: Level 1 trigger 1  L1_EM8_&_CLC_PS1 has  
increased to 0.00234607  
         Hz, over 50% of it's initial rate of 0 Hz  XMo 
n SL1DMonitor()  
          2-Dec-2001 01:17:17 run = 133963 event = 4428 
4 
%ERLOG-e XMon: Level 1 trigger 1  L1_EM8_&_CLC_PS1 has  
increased to 5.40216e+06  
         nb, over 90% of it's initial x-section of 0 nb 
  XMon SL1DMonitor()  
          2-Dec-2001 01:17:17 run = 133963 event = 4428 
4 
%ERLOG-e XMon: Prefred trigger 2  JET10 has increased t 
o 38.134  
         Hz, over 50% of it's initial rate of 0 Hz  XMo 
n SL1DMonitor()  
          2-Dec-2001 01:17:17 run = 133963 event = 4428 
4 
Socket not valid. 
%ERLOG-e XMon: Prefred trigger 2  JET10 has increased t 
o 8.78088e+10  
         nb, over 90% of it's initial x-section of 0 nb 
  XMon SL1DMonitor()  
          2-Dec-2001 01:17:17 run = 133963 event = 4428 
4 
Socket not valid. 
Socket not valid. 
%ERLOG-e XMon: Level 1 trigger 4  L1_JET10_&_CLC has in 
creased to 1.44058e+07  
         nb, over 90% of it's initial x-section of 0 nb 
  XMon SL1DMonitor()  
 
          2-Dec-2001 01:17:17 run = 133963 event = 4428 
4 
Socket not valid. 
Socket not valid. 
%ERLOG-e XMon: Level 1 trigger 7  L1_JET5_&_CLC_PS200 h 
as increased to 300120  
         nb, over 90% of it's initial x-section of 0 nb 
  XMon SL1DMonitor()  
          2-Dec-2001 01:17:17 run = 133963 event = 4428 
4 
%ERLOG-s Abnormal termination: caused by signal:  11 Fr 
amework abortJob()  
          2-Dec-2001 01:17:17 run = 133963 event = 4428 
4 
Calling abortJob for DummyInput... 
Calling abortJob for FileInput... 
Calling abortJob for DHInput... 
********************* DHInputModule ******************* 
***** 
Calling abortJob for *      !!! Job is aborted !!!      
                         * 
* The reason is most likely not relevant to DHInput its 
elf * 
******************************************************* 
***** 
YbosDiskFileInput... 
Calling abortJob for FileInputModule... 
Calling abortJob for XXXDiskFileInput... 
Calling abortJob for ConsumerInput... 
Calling abortJob for ErrorLoggerManager... 
Calling abortJob for CalibrationManager... 
Calling abortJob for GeometryManager... 
Calling abortJob for SignalManager... 
Calling abortJob for ConsumerErrorModule... 
Calling abortJob for XMon... 
%ERLOG?? XMon Summary: ::Evaluation:  0 ::evtProcessed: 
  8000::Comment: None   
         XMon XMon()  2-Dec-2001 01:17:18 run = 133963  
event = 44284 
Socket not valid. 
Dec  2 01:17:18 2001 Opening connection to database &lt;cd 
fonprd.fnal.gov&gt; 
Dec  2 01:17:18 2001Accessing data base string: 
INSERT INTO CONSUMER_SUMMARY (RC_RUNNUMBER,NAME,CID,PRO 
CEVT,EVALUATION,COMMENTS) VALUES (133963,'XMon',CONSUME 
R_SEQUENCE.NEXTVAL,8000,'UNDEFINED','None') 
XMon: Root file /data1/consumer/results/XMon/XMon133963 
aa.root saved. 
 
 
Saved the log file in the run area as XMon.cdfdaq_133963.log 
 
 
</pre>
<!-- Author: Igor -->


<!-- Date: Sun Dec 2 05:18:02 2001 -->
<pre>ObjectMon died. Here is the terminal dump: 
 
ConsumerInput Begin processing 35800th record. Run 1339 
63, Trigger 105861 
================================================== 
JetMonitor::summary(): 
For detailed definition of the checks and the histogram 
s we refer 
you to the JetMonitor web page: 
/consumer/objectmon/objectmo 
n.html 
--------------------------------------- 
A) Q-checks: 
      Histogram name      Check #   Result     
      nJets                 0       NOT enough entries 
      nJets                 1       NOT enough entries 
      nJets                 2       NOT enough entries 
 
      logSumEt              0       NOT enough entries 
 
      logEt                 0       NOT enough entries 
 
      eta                   0       NOT enough entries 
      eta                   1       NOT enough entries 
      eta                   2       NOT enough entries 
 
--------------------------------------- 
B) PHI-checks: 
      Histogram name                Result              
 Chi^2      
  0.) phi Nj All                    NOT enough entries  
  
  1.) phi Nj Barrel                 NOT enough entries  
  
  2.) phi Nj Forwd                  NOT enough entries  
  
  3.) phi Nj Bacwd                  NOT enough entries  
  
  4.) phi Et All                    NOT enough entries  
  
  5.) phi Et Barrel                 NOT enough entries  
  
  6.) phi Et Forwd                  NOT enough entries  
  
  7.) phi Et Bacwd                  NOT enough entries  
  
================================================== 
 FCN=148.537 FROM MIGRAD    STATUS=CONVERGED      12 CA 
LLS          13 TOTAL 
                     EDM=1.2699e-15    STRATEGY= 1      
 ERROR MATRIX ACCURATE  
  EXT PARAMETER                                   STEP  
        FIRST    
  NO.   NAME      VALUE            ERROR          SIZE  
     DERIVATIVE  
   1  p0           1.52919e+00   2.15265e-01   1.28534e 
-03   2.34113e-07 
 FCN=505.484 FROM MIGRAD    STATUS=CONVERGED      12 CA 
LLS          13 TOTAL 
                     EDM=8.3619e-21    STRATEGY= 1      
 ERROR MATRIX ACCURATE  
  EXT PARAMETER                                   STEP  
        FIRST    
  NO.   NAME      VALUE            ERROR          SIZE  
     DERIVATIVE  
   1  p0           1.51590e+02   1.74121e+00   1.91339e 
-02   7.42706e-11 
 FCN=0 FROM MIGRAD    STATUS=CONVERGED      39 CALLS    
       40 TOTAL 
                     EDM=0    STRATEGY= 1  ERROR MATRIX 
 UNCERTAINTY 100.0 per cent 
  EXT PARAMETER                APPROXIMATE        STEP  
        FIRST    
  NO.   NAME      VALUE            ERROR          SIZE  
     DERIVATIVE  
   1  p0           0.00000e+00   1.41421e+00   0.00000e 
+00   0.00000e+00 
 FCN=505.484 FROM MIGRAD    STATUS=CONVERGED      12 CA 
LLS          13 TOTAL 
                     EDM=8.3619e-21    STRATEGY= 1      
 ERROR MATRIX ACCURATE  
  EXT PARAMETER                                   STEP  
        FIRST    
  NO.   NAME      VALUE            ERROR          SIZE  
     DERIVATIVE  
   1  p0           1.51590e+02   1.74121e+00   1.91339e 
-02   7.42706e-11 
 FCN=0 FROM MIGRAD    STATUS=CONVERGED      39 CALLS    
       40 TOTAL 
                     EDM=0    STRATEGY= 1  ERROR MATRIX 
 UNCERTAINTY 100.0 per cent 
  EXT PARAMETER                APPROXIMATE        STEP  
        FIRST    
  NO.   NAME      VALUE            ERROR          SIZE  
     DERIVATIVE  
   1  p0           0.00000e+00   1.41421e+00   0.00000e 
+00   0.00000e+00 
======================================================= 
= 
TrackMonitor::summary(): Run133963 Events 0 
======================================================= 
= 
Generic Checker: 
      Histogram name      Result         Chi^2          
  
------------------------------------------------------- 
- 
        Track Z_0         No Events        
        Impact Para       No Events        
        Track Rapidity    No Events        
        Track Phi         No Events        
        Sing(Q)/Track pT  No Events (before Quality Cut 
s) 
        Sing(Q)/Track pT  No Events (after Quality Cuts 
) 
======================================================= 
= 
 JpsiMonitor::summary() Run 133963 events 33124 
 Check 1 
dimuon mass                                   : Fit WAS 
 NOT PERFORMED  due to insufficient statistics 
 Check 2 
cmu dimuon mass                               : Fit WAS 
 NOT PERFORMED  due to insufficient statistics 
 Check 3 
cmp dimuon mass                               : Fit WAS 
 NOT PERFORMED  due to insufficient statistics 
 Check 4 
cmx dimuon mass                               : Fit WAS 
 NOT PERFORMED  due to insufficient statistics 
 
 Check 5 
at least one cmu  dimuon mass                 : Fit WAS 
 NOT PERFORMED  due to insufficient statistics 
 Check 6 
at least one cmp  dimuon mass                 : Fit WAS 
 NOT PERFORMED  due to insufficient statistics 
 Check 7 
at least one cmx  dimuon mass                 : Fit WAS 
 NOT PERFORMED  due to insufficient statistics 
 Check 8 
everything muon + only cmu dimuon mass        : Fit WAS 
 NOT PERFORMED  due to insufficient statistics 
 Check 9 
everything muon + only cmp dimuon mass        : Fit WAS 
 NOT PERFORMED  due to insufficient statistics 
 Check 10 
everything muon + only cmx dimuon mass        : Fit WAS 
 NOT PERFORMED  due to insufficient statistics 
%ERLOG?? ObjectMon Summary: ::Evaluation:  1 ::evtProce 
ssed:  33124::Comment:  
         Still at developing phase.JET:CANNOT TELL;Stil 
l at developing phase.Still at developing phase.MUON:CA 
NNOTTELL|CMU phi (Stub):Low Stat|CMP phi (Stub):Low Sta 
t|CMU Z (Stub):Low Stat|Charge/p_{T}:Low Stat|CMU dx:Lo 
w Stat|CMP dx:Low Stat|CMX dx:Low Stat;Track::UNDEFINED 
:[Z0:No Events][D0:No Events][Eta:No Events][Phi:No Eve 
nts][Charge/pT:No Events];  
          ObjectMon ObjectMon()  2-Dec-2001 04:11:32 ru 
n = 133963 event = 0 
ConsumerSummaryDB::insertRunSummary(): Comment string o 
f Run Summary had to be 
truncated to 200 characters. Larger comments cannot be  
stored in the data base. 
Dec  2 04:11:32 2001 Opening connection to database &lt;cd 
fonprd.fnal.gov&gt; 
Dec  2 04:11:33 2001Accessing data base string: 
INSERT INTO CONSUMER_SUMMARY (RC_RUNNUMBER,NAME,CID,PRO 
CEVT,EVALUATION,COMMENTS) VALUES (133963,'ObjectMon',CO 
NSUMER_SEQUENCE.NEXTVAL,33124,'CANNOTTELL','Still_at_de 
veloping_phase.JET:CANNOT_TELL;Still_at_developing_phas 
e.Still_at_developing_phase.MUON:CANNOTTELL|CMU_phi_(St 
ub):Low_Stat|CMP_phi_(Stub):Low_Stat|CMU_Z_(Stub):Low_S 
tat|Charge/p_{T}:Low_Sta') 
ObjectMon: Root file /data1/consumer/results/ObjectMon/ 
ObjectMon133963aa.root saved. 
ObjectMon: summary file /data1/consumer/results/ObjectM 
on/ObjectMon133963.sum written. 
SiAlignmentManager at beginRun: no DB requested 
ConsumerErrorAdministrator::attachStat(): ERROR 
Could not create _destStatCTRL instance. Pointer not NU 
LL. 
ConsumerInput Begin processing 35900th record. Run 1339 
64, Trigger 634 
ConsumerInput Begin processing 36000th record. Run 1339 
64, Trigger 2339 
%ERLOG-s Abnormal termination: caused by signal:  8 Fra 
mework abortJob()  
          2-Dec-2001 05:09:15 run = 133964 event = 3756 
Calling abortJob for DummyInput... 
 
Calling abortJob for FileInput... 
Calling abortJob for DHInput... 
 
 
Hope this helps to debug the problem. The autors should 
probably enable the core dumps in order to deal with these 
troubles. For that, they need to undo the signal catching 
performed by the SignalManager -- the best place to do it 
is in their AppUserBuild class. See, for example, AppUserBuild 
for SVXMon. 
</pre>
<!-- Author: Igor -->
<!-- Comment: Sun Dec  2 06;11;12 comment by...Igor -->
<pre>
ObjectMon died again, with similar printout. Copied the log file to /data1/consumer/runarea/ObjectMon/ObjectMon.cdfdaq_133964.log
</pre>
<!-- Comment: Sun Dec  2 06;22;42 comment by...Igor -->
<pre>
ObjectMon died yet again, after about 300 events

</pre>
<!-- Comment: Sun Dec  2 06;36;36 comment by...Igor -->
<pre>ObjectMon died again. I am giving up on it.
</pre>
<!-- Date: Sun Dec 2 05:24:38 2001 -->
<pre>XMon seems to have a memory leak somewhere. Here is the output 
of the "top" program: 
 
9650 cdfdaq     2   0  246M 193M  2732 S       0  0.7 38.4   5:53 HistoDisplayMai 
8199 cdfdaq    14   0  308M 160M  5092 R       0 35.9 31.7  46:41 XMon 
8201 cdfdaq     2   0 95764  83M   724 R       0  6.5 16.5   4:44 Server 
25230 root      19   0 68320  60M  1004 R       0 10.7 12.0  47:17 X 
 
XMon keeps growing... 
</pre>
<!-- Author: Igor -->



<!-- Date: Sun Dec 2 06:15:01 2001 -->
<pre>ObjectMon died immediately after restart. Here is the display output: 
 
======================================================= 
==================== 
 
DisplayServer::openClientServerSocket(): Server socket  
for HISTO DISPLAY 
               clients at PORT 9091 available. 
 
               Use   b0dap56.fnal.gov:9091 
               as input stream to connect from the Hist 
oDisplay 
               to the Display Server. 
 
======================================================= 
==================== 
 
SiAlignmentManager: turning off alignments ... source = 
 none 
SiAlignmentManager at beginRun: no DB requested 
%ERLOG-w track view not found: can not find default tra 
ck view  ObjectMon  
         TrackMonitor()  2-Dec-2001 06:12:19 run = 1339 
64 event = 437721 
Socket not valid. 
%ERLOG-w track view not found: can not find default tra 
ck view  ObjectMon  
         TrackMonitor()  2-Dec-2001 06:12:19 run = 1339 
64 event = 438018 
%ERLOG-w track view not found: can not find default tra 
ck view  ObjectMon  
         TrackMonitor()  2-Dec-2001 06:12:19 run = 1339 
64 event = 438100 
%ERLOG-w No CalData: calStatus =  0  != OK.  ObjectMon  
JetMonitor()  
          2-Dec-2001 06:12:21 run = 133964 event = 4382 
58 
%ERLOG-w track view not found: can not find default tra 
ck view  ObjectMon  
         TrackMonitor()  2-Dec-2001 06:12:21 run = 1339 
64 event = 438258 
%ERLOG-w track view not found: can not find default tra 
ck view  ObjectMon  
         TrackMonitor()  2-Dec-2001 06:12:22 run = 1339 
64 event = 438280 
Socket not valid. 
%ERLOG-w track view not found: can not find default tra 
ck view  ObjectMon  
         TrackMonitor()  2-Dec-2001 06:12:22 run = 1339 
64 event = 438329 
%ERLOG-w No CalData: calStatus =  0  != OK.  ObjectMon  
JetMonitor()  
          2-Dec-2001 06:12:22 run = 133964 event = 4383 
42 
%ERLOG-w track view not found: can not find default tra 
ck view  ObjectMon  
         TrackMonitor()  2-Dec-2001 06:12:23 run = 1339 
64 event = 438342 
%ERLOG-w track view not found: can not find default tra 
ck view  ObjectMon  
         TrackMonitor()  2-Dec-2001 06:12:23 run = 1339 
64 event = 438409 
%ERLOG-w track view not found: can not find default tra 
ck view  ObjectMon  
         TrackMonitor()  2-Dec-2001 06:12:23 run = 1339 
64 event = 438587 
Socket not valid. 
%ERLOG-w track view not found: can not find default tra 
ck view  ObjectMon  
         TrackMonitor()  2-Dec-2001 06:12:23 run = 1339 
64 event = 438618 
%ERLOG-w No CalData: calStatus =  0  != OK.  ObjectMon  
JetMonitor()  
          2-Dec-2001 06:12:23 run = 133964 event = 4386 
30 
%ERLOG-w No CalData: calStatus =  0  != OK.  ObjectMon  
JetMonitor()  
          2-Dec-2001 06:12:26 run = 133964 event = 4388 
54 
%ERLOG-w track view not found: can not find default tra 
ck view  ObjectMon  
         TrackMonitor()  2-Dec-2001 06:12:27 run = 1339 
64 event = 438908 
%ERLOG-w No CalData: calStatus =  0  != OK.  ObjectMon  
JetMonitor()  
 
          2-Dec-2001 06:12:28 run = 133964 event = 4391 
11 
Socket not valid. 
%ERLOG-w No CalData: calStatus =  0  != OK.  ObjectMon  
JetMonitor()  
          2-Dec-2001 06:12:28 run = 133964 event = 4393 
03 
ConsumerInput Begin processing 100th record. Run 133964 
, Trigger 440112 
%ERLOG-s Abnormal termination: caused by signal:  8 Fra 
mework abortJob()  
          2-Dec-2001 06:12:39 run = 133964 event = 4401 
53 
Calling abortJob for DummyInput... 
</pre>
<!-- Author: Igor -->



<!-- Date: Sun Dec 2 06:29:51 2001 -->
<pre>BeamMon died. Copied the log file to 
/data1/consumer/runarea/BeamMon/BeamMon.cdfdaq_133964.log 
There appears to be no useful info about the reason for 
the crash in the terminal output. 
</pre>
<!-- Author: Igor -->



<!-- Date: Sun Dec 2 10:03:36 2001 -->
<pre>Dear Cos and Kaori, 
our Stage0 expert (Joao) told me that we 'CANNOT RUN 
STAGE0 from startMon_new.sh. The instructions attached 
close to the SVXMON display terminal should be followed. 
These are the ones reported from Antonio last evening. 
Joao restarted the stage0 server on b0dap56 and everything 
should be ok now. Keep in touch for further developments. 
 
			Stefano</pre>
<!-- Author: Stefano -->


<!-- Date: Sun Dec 2 14:36:33 2001 -->
<pre>I have restarted all Displays and All consumers BUT STAGE0 
since the shot setup is underway and we were finding  
continuos problem with Lummon.  
As Kaori told me I used ./startMon_new.sh 
after having commented out the relevant part 
for stage0. Now in  run 133970  (cosmic_notracks)  
everything looks ok. The only problem (if any) I 
could see is in TrigMon. In the consumer window there 
are errors for each event as: 
"in CdfJet: particle with p = pz ..." 
 
This fortunately does not look to go in the log file 
otherwise it could fill up some disks. I have checked 
with: 
df -k /data1 
that we still have 25 GB available. 
 
If the error persists during shot setup we should 
send an e-mail to trgmon experts. For the moment, 
I advised Heather Ray which is in the control room. 
 
</pre>
<!-- Author: Stefano -->


<!-- Date: Sun Dec 2 18:48:30 2001 -->
<pre>ObjectMon: checkFreq set 3000 for TrackMonitor, should reduce the crash rate. File /data1/consumer/runarea/ObjectMon/Consumer/ObjectMon/runObjectMon.tcl</pre>
<!-- Author: slava77 -->


<!-- Date: Sun Dec 2 23:52:28 2001 -->
<pre>Put a folder on CO tables. Migh be useful to put hardcopies of notes, plots etc.... 
Cheers, 
Antonio</pre>
<!-- Author: Antonio -->


<!-- Date: Mon Dec 3 07:53:56 2001 -->
<pre>ObjectMon died. Here are a few lines of output just before 
abort was called: 
 
SiAlignmentManager at beginRun: no DB requested 
ConsumerErrorAdministrator::attachStat(): ERROR 
Could not create _destStatCTRL instance. Pointer not NU 
LL. 
ConsumerInput Begin processing 190900th record. Run 133 
989, Trigger 1695 
%ERLOG-s Abnormal termination: caused by signal:  8 Fra 
mework abortJob()  
          3-Dec-2001 07:48:08 run = 133989 event = 5524 
Calling abortJob for DummyInput... 
 
</pre>
<!-- Author: Igor -->


<!-- Date: Mon Dec 3 11:29:33 2001 -->
<pre>Running on store 800. Situation for consumers is reasonable. 
The only problem I have is ObjectMon. I called the expert 
since I do not observe any tracks (this looks correlated 
with the same problem in EvtDisplay where only XFT tracks 
are shown). The expert told me that they are working on  
this and that it depends on the reconstruction version 
used 4.1.0intxx vs the one used in the L3. 
 
In the last 10minutes objmon died 2 times. 
This is the error list: 
------------------------------------------------------------ 
%ERLOG-w track view not found: can not find default tra 
ck view  ObjectMon  
         TrackMonitor()  3-Dec-2001 11:16:04 run = 1339 
96 event = 533903 
%ERLOG-w No CalData: calStatus =  0  != OK.  ObjectMon  
JetMonitor()  
          3-Dec-2001 11:16:05 run = 133996 event = 5339 
55 
%ERLOG-w No CalData: calStatus =  0  != OK.  ObjectMon  
JetMonitor()  
          3-Dec-2001 11:16:06 run = 133996 event = 5340 
00 
%ERLOG-w No CalData: calStatus =  0  != OK.  ObjectMon  
JetMonitor()  
          3-Dec-2001 11:16:06 run = 133996 event = 5342 
31 
ConsumerInput Begin processing 100th record. Run 133996 
, Trigger 534582 
DisplayServer: connectClient(): Client connected from b 
0dap56.fnal.gov on port 2102. 
%ERLOG-s Abnormal termination: caused by signal:  8 Fra 
mework abortJob()  
          3-Dec-2001 11:17:37 run = 133996 event = 5346 
29 
Socket not valid. 
Calling abortJob for DummyInput... 
Calling abortJob for FileInput... 
Calling abortJob for DHInput... 
********************* DHInputModule ******************* 
***** 
Calling abortJob for YbosDiskFileInput... 
Calling abortJob for ConsumerInput... 
Calling abortJob for ErrorLoggerManager... 
Calling abortJob for PuffModule... 
Calling abortJob for *      !!! Job is aborted !!!   
---------------------------------------------------------- 
I will restart it again. 
	Please give a look. 
			Stefano</pre>
<!-- Author: Stefano -->


<!-- Date: Mon Dec 3 16:12:36 2001 -->
<pre>ObjectMon: disabling TrackMonitor in the /data1/consumer/runarea/ObjectMon/Consumer/ObjectMon/runObjectMon.tcl. Crashes happen while checking the histograms, but the check frequency is uncontrollable via talk-to (can not be formally turned off), hence, turning off the whole monitor.</pre>
<!-- Author: slava77 -->


<!-- Date: Mon Dec 3 17:24:14 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-151&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-151  width='300' align='top'></td><td>Seems that tracks on COT   are back (ons stage0 as well as on Event Display)</a></td></tr></table>
<!-- Author: Antonio -->


<!-- Date: Mon Dec 3 18:04:37 2001 -->
<pre>ObjectMon: disabled the PuffModule again. The reason: TCL_TAG has changed to 422 .. leave it disabled till we can set the TCL_TAG automatically/interactively when starting the monitor.</pre>
<!-- Author: slava77 -->


<!-- Date: Mon Dec 3 19:37:01 2001 -->
<pre>XMon crashed... 
Last seconds before crash... 
log file in 
/data1/consumer/runarea/XMon/XMon.cdfdaq_134004.log 
 
(...) 
ConsumerInput Begin processing 12300th record. Run 1340 
04, Trigger 603818 
ConsumerInput Begin processing 12400th record. Run 1340 
04, Trigger 615508 
Display Server: Got signal SIGPIPE (13). 
SysError in &lt;TUnixSystem::UnixSend&gt;: send (Broken pipe) 
Error in &lt;TUnixSystem::SendRaw&gt;: cannot send buffer 
DisplayServer::pollClients(): ERROR: pollClients(): Soc 
ket-&gt;Send returned with -1. 
               ==&gt; The socket will be closed. 
Fatal in &lt;TStorage::ReAlloc&gt;: storage exhausted 
aborting 
%ERLOG-s Abnormal termination: caused by signal:  6 Fra 
mework abortJob()  
          3-Dec-2001 19:32:55 run = 134004 event = 6221 
83 
Calling abortJob for DummyInput... 
Calling abortJob for FileInput... 
Calling abortJob for DHInput... 
********************* DHInputModule ******************* 
***** 
*      !!! Job is aborted !!!                           
    * 
* The reason is most likely not relevant to DHInput its 
elf * 
******************************************************* 
***** 
Calling abortJob for YbosDiskFileInput... 
Calling abortJob for FileInputModule... 
Calling abortJob for XXXDiskFileInput... 
Calling abortJob for ConsumerInput... 
Calling abortJob for ErrorLoggerManager... 
Calling abortJob for CalibrationManager... 
Calling abortJob for GeometryManager... 
Calling abortJob for SignalManager... 
Calling abortJob for ConsumerErrorModule... 
Calling abortJob for XMon... 
%ERLOG?? XMon Summary: ::Evaluation:  0 ::evtProcessed: 
  5090::Comment: None   
         XMon XMon()  3-Dec-2001 19:32:56 run = 134004  
event = 622183 
Socket not valid. 
Dec  3 19:32:56 2001 Opening connection to database &lt;cd 
fonprd.fnal.gov&gt; 
Dec  3 19:32:58 2001Accessing data base string: 
INSERT INTO CONSUMER_SUMMARY (RC_RUNNUMBER,NAME,CID,PRO 
CEVT,EVALUATION,COMMENTS) VALUES (134004,'XMon',CONSUME 
R_SEQUENCE.NEXTVAL,5090,'UNDEFINED','None') 
Error in &lt;TFile::TFile&gt;: file /data1/consumer/results/X 
Mon/XMon134004aa.root already exists 
Fatal in &lt;TStorage::ReAlloc&gt;: storage exhausted 
aborting 
DisplayServer: ERROR reading object messages! 
DisplayServer: Closing socket! 
Display Server: End with signal 11 ( = SIGSEGV ) 
Report end to the StateManager 
sockets closed 
runXMon: Consumer ended with value  0. 
 
</pre>
<!-- Author: Antonio -->


<!-- Date: Mon Dec 3 21:20:53 2001 -->
<pre>MOnitoring Xmon memory usage... 
Swap: 514040K  236156 used   277884 free             
               PRI  NI SIZE RSS  SHARE         CPU   M     TIME   
5382 cdfdaq    19   0  177M 177M 23040 R     0 72.7 35.1   5:56 
               15      195  195  23040         43.5 38.8  17:38 
	        0   0  201  199   8316         50.9 40.9  33:45  
               13   0  221M 198M 11408 S     0 63.9 39.2  45:39 
               17   0  231M 199M 11408 R     0 74.9 39.5  52:14  
               19   0  255M 204M  4188 R     0 47.3 40.4  66:28 
for the moment no crash!</pre>
<!-- Author: Antonio -->


<!-- Date: Mon Dec 3 21:26:52 2001 -->
<pre> 
BeamMon crashed 
log file in 
/data1/consumer/runarea/BeamMon/BeamMon.cdfdaq_134004.log 
 
Last minutes of life... 
rate of change [cm/s,1/s]: 
            0 
            0 
            0 
            0 
 
z (gaussian fit): 
mean z: 2.74514cm   width:28.0607cm 
rate of change [cm/s]   z0rate:0 widthrate:0 
z (see cdf5094): 
Zzero:4.23251cm  width: 45.4393cm   Bstar:38.3068cm  Zm 
in:0.682372cm 
rate of change [cm/s] Zzero: 0  widthrate: 0   Bstarrat 
e:0    Zminrate:0 
%ERLOG?? BeamMon Summary: ::Evaluation:  2 ::evtProcess 
ed:  55821::Comment:  
         d-phi:db filled.vx-cot:db filled./  BeamMon Be 
amMon()  
          3-Dec-2001 21:15:31 run = 134004 event = 1235 
115 
Dec  3 21:15:31 2001 Opening connection to database &lt;cd 
fonprd.fnal.gov&gt; 
Dec  3 21:15:31 2001Accessing data base string: 
INSERT INTO CONSUMER_SUMMARY (RC_RUNNUMBER,NAME,CID,PRO 
CEVT,EVALUATION,COMMENTS) VALUES (134004,'BeamMon',CONS 
UMER_SEQUENCE.NEXTVAL,55821,'OK','d-phi:db_filled.vx-co 
t:db_filled./') 
Error in &lt;TFile::TFile&gt;: file /data1/consumer/results/B 
eamMon/BeamMon134004aa.root already exists 
Error in &lt;TFile::TFile&gt;: file /data1/consumer/results/B 
eamMon/BeamMon134004ab.root already exists 
*** Error OCI_ERROR  [-1] [1] 
*** Error: ORA-00001: unique constraint (RUNDB_PRD.CNS_ 
UNIQUE) violated 
BeamMon: Root file /data1/consumer/results/BeamMon/Beam 
Mon134004ac.root saved. 
BeamMon: summary file /data1/consumer/results/BeamMon/B 
eamMon134004.sum written. 
DisplayServer::pollConsumer(): INFO: BeamMon sent messa 
ge: END CONNECTION 
                             ==&gt; Going to terminate con 
nection to BeamMon running on 
                             b0dap66.fnal.gov 
Calling abortJob for cotvxprim-svxvxprim... 
Calling abortJob for DummyOutput... 
Calling abortJob for FileOutput... 
Calling abortJob for DHOutput... 
cotvxprim-svxvxprim: 
calls:0 
runBeamMon: Consumer ended with value  0.</pre>
<!-- Author: Antonio -->


<!-- Date: Tue Dec 4 00:39:07 2001 -->
<pre>All JetMonitor plots are empty in the ObjectMon. Are they intentionally disabled? 
</pre>
<!-- Author: Igor -->
<!-- Comment: Tue Dec  4 12;10;11 comment by...Igor -->
<pre>Here is the reply from Wolfgang:

Hi Igor,

the problem is known. It is related to a buggy Puff module for the jets.
Slava has been working on that. Switching Puff module on and off.
Currently, I do not oversee the situation (releases etc.) clearly.
I guess we have to wait until the framework code gets fixed.

Thanks for the notice,

Wolfgang 
</pre>
<!-- Date: Tue Dec 4 01:42:47 2001 -->
<pre>BeamMon died, without any useful diagnostics printed to the terminal: 
 
ConsumerInput Begin processing 99900th record. Run 1340 
06, Trigger 1171034 
%ERLOG-s Abnormal termination: caused by signal:  8 Fra 
mework abortJob()  
          4-Dec-2001 01:27:27 run = 134006 event = 1171 
816 
Calling abortJob for DummyInput... 
 
 
It may be useful to enable core file generation instead of callind the AC++ AbortJob sequence... 
</pre>
<!-- Author: Igor -->
<!-- Comment: Tue Dec  4 03;06;16 comment by...Igor -->
<pre>BeamMon died again. Restarted.
</pre>


<!-- Date: Tue Dec 4 06:05:35 2001 -->
<pre>ObjectMon died in the middle of the run (134006). Here is 
some terminal output: 
 
In CdfJet: particle with p = 0 - could not return theta 
In CdfJet: particle with p = pz  - could not return eta 
In CdfJet: particle with p = pz  - could not return eta 
In CdfJet: particle with p = 0 - could not return theta 
ConsumerInput Begin processing 248000th record. Run 134 
006, Trigger 2234544 
%ERLOG-s Abnormal termination: caused by signal:  8 Fra 
mework abortJob()  
          4-Dec-2001 05:55:53 run = 134006 event = 2234 
551 
Calling abortJob for DummyInput... 
 
 
</pre>
<!-- Author: Igor -->



<!-- Date: Tue Dec 4 17:44:48 2001 -->
<pre>All the consumers just keep crashing</pre>
<!-- Author: Antonio -->
<!-- Comment: Tue Dec  4 18;51;47 comment by...Antonio -->
<pre>They crash as soon as we are taking events (cosmic)</pre>

<!-- Comment: Tue Dec  4 19;01;47 comment by...Antonio -->
<pre>OK, seems we have understood. Thanks kaori! I'll insert an entry explaining the measures to take..</pre>


<!-- Date: Tue Dec 4 19:34:12 2001 -->
<pre>If all consumers keep crashing that probably means that the L3 manager doesn't run. This could happen if the DAQ ACE is running with the Software Event Builder. In this case ask the DAQ ace to turn L3 manager on. 
THis souldn't happen if DAQ is run with the Hardware Event Builder as, per default the L3 manager is started.,</pre>
<!-- Author: Antonio -->


<!-- Date: Tue Dec 4 22:38:33 2001 -->
<pre>Two brand new folders are on the CO tables. Insert plots from consumers that might have an interest (after talking with experts) for cosmic runs as well as collision runs. 
</pre>
<!-- Author: Antonio -->


<!-- Date: Wed Dec 5 07:09:28 2001 -->
<pre>CDF is running with three "guinea pig" silicon ladder. Both SVXMon 
and SiliMon are running, no problems so far... 
 
</pre>
<!-- Author: Igor -->


<!-- Date: Thu Dec 6 01:06:09 2001 -->
<pre>I just made a new cdfsoftb0 (4.2.0c) and tested with  
physics table run, and all seems to run ok.  So, now this version 
is decleared to be the default cdfsoftb0 version. 
We no longer need to start Stage0 separately. 
We should be able to see L3 reconstructed tracks in EventDisplay. 
 
To start monitor, please use: 
 
./startMon.sh consumername &lt;partitionID&gt; 
 
To start Event Displlay, please use: 
 
./startEvd.sh &lt;partitionID&gt; 
</pre>
<!-- Author: kaori -->


<!-- Date: Thu Dec 6 01:12:26 2001 -->
<pre>We have two new machines for monitoring. Kaori has moved 
Stage0 and DAQMon to b0dap69, XMon to b0dap70. 
</pre>
<!-- Author: Igor -->


<!-- Date: Thu Dec 6 03:36:02 2001 -->
<pre>ObjectMon crashed. Here is some printout just before it happened: 
 
In CdfJet: particle with p = pz  - could not return eta 
In CdfJet: particle with p = pz  - could not return eta 
In CdfJet: particle with p = 0 - could not return theta 
%ERLOG-s Abnormal termination: caused by signal:  8 Fra 
mework abortJob()  
          6-Dec-2001 03:31:14 run = 134220 event = 4050 
35 
Calling abortJob for DummyInput... 
Calling abortJob for FileInput... 
Calling abortJob for DHInput... 
********************* DHInputModule ******************* 
***** 
Calling abortJob for YbosDiskFileInput... 
Calling abortJob for ConsumerInput... 
</pre>
<!-- Author: Igor -->
<!-- Comment: Thu Dec  6 05;45;37 comment by...Igor -->
<pre>Crashed again with similar printout. Restarted.
</pre>


<!-- Date: Thu Dec 6 07:06:53 2001 -->
<pre>BeamMon died (no useful info on the standard output) 
</pre>
<!-- Author: Igor -->


<!-- Date: Thu Dec 6 08:50:15 2001 -->
<pre>Hi, 
thanks to Kaori and the other monitor-experts, now the 
situation looks much better from the point of view of 
the consumers. 
I have few observation on current run (134220) and 
I would like the experts to give a look offline: 
- ObjectMon: 
   i) there is not anymore the TrackMonitor slides! 
   ii) the jet-monitor is "empty". 
- Stage0: 
	Monitor All Hits/All GoodHits/ UsedHitd vs cell  
		empty! 
- Ymon 
	the "title" on the summary slides is really 
	too "small". Can we make it bigger in the pad. 
 
			Stefano 
   </pre>
<!-- Author: stefano -->


<!-- Date: Thu Dec 6 12:57:13 2001 -->
<pre>By mistake, I had the monitors running on 4.2.0a, instead of 
the new default 4.2.0c, since last night.  This was corrected 
around 11:30 am today.  The strange Stage0 behaviour was due 
to this.  Stage0 is ok now.  YMon also now see the new histograms 
Greg put in yesterday afternoon.  LumMon should be not affected 
by this.</pre>
<!-- Author: kaori -->


<!-- Date: Thu Dec 6 13:03:35 2001 -->
<pre>Dear CO people, 
 
XMon, XMon Server (both running on b0dap70), and 
HistoDisplayMain displaying XMon (running on b0dap55, upper) 
have memory usage problem. 
Please stop and restart XMon, after about 5000 events 
processed on b0dap70.  Please stop/restart the GUI 
for the XMon every once a while.   thanks.</pre>
<!-- Author: Kaori -->


<!-- Date: Thu Dec 6 13:07:09 2001 -->
<pre>I have modified ObjectMon tcl in the Consumer/ObjectMon area 
enabling puffmodule, enabling TrackingMonitor.  Please be 
aware, if the ObjectMon starts crashing again, that probably 
means that rcp ID number has changed.  (this will happen if 
new L3/TT runs, with different TCL_Tag).  Anyway, if ObjectMon 
start crashing again, please contact me or ObjectMon people.</pre>
<!-- Author: kaori -->


<!-- Date: Fri Dec 7 04:38:01 2001 -->
<pre>ObjectMon just crashed. Restarted.</pre>
<!-- Author: Igor -->


<!-- Date: Fri Dec 7 05:09:16 2001 -->
<pre>BeamMon crashed. Restarted. 
</pre>
<!-- Author: Igor -->


<!-- Date: Fri Dec 7 10:12:01 2001 -->
<pre>As Kaori's request, we stopped XMon once at the middle of the 
run 134303, and restart it, and compare L1 and L2 rates. 
Almost triggers look fine, but only L2_AUTO_L1_CMU6_PT8 looks 
different 904 (old) - 506 (new) 
</pre>
<!-- Author: Satoru & Antonio -->


<!-- Date: Fri Dec 7 11:11:34 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-152&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-152  width='300' align='top'></td><td>It should be interesting to look at the same plot  (the CSX start for XW) during a cosmic run to  see if the spikes are caused by bunch crossings.</a></td></tr></table>
<!-- Author: Antonio & Satoru -->


<!-- Date: Fri Dec 7 18:06:36 2001 -->
<pre>Kaori updated trig table to phys [4,224] and trig mon table 
of run 134323 was placed in the online monitor reference  output 
file.</pre>
<!-- Author: akiya -->


<!-- Date: Fri Dec 7 20:39:19 2001 -->
<pre>Ymon monitor does not get any event during cosmic runs, due to  
it accepts only minmum bias event trigger, and YMon display  
dies finally.  It is left as it is until experts fix the problem.</pre>
<!-- Author: Akiya -->


<!-- Date: Fri Dec 7 21:20:40 2001 -->
<pre>Run 134355 starts with Physics trigger table.  All displays are 
restarted and YMon is working now.</pre>
<!-- Author: Akiya -->


<!-- Date: Fri Dec 7 23:06:37 2001 -->
<pre>BeamMon died when run 134356 started</pre>
<!-- Author: Akiya -->


<!-- Date: Fri Dec 7 23:29:33 2001 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-153&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-153' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-154&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-154' width='300'></a></center>
<!-- Author: Akiya -->


<!-- Date: Fri Dec 7 23:31:55 2001 -->
<pre>As seen above figure, East side TOF PMTs of about ch 52 to 108 
are off.  Number of hits of TOF counters is also different from  
reference figure.  It was said a power of a TOF crate is off. 
Needs to be checked by experts.</pre>
<!-- Author: Akiya -->


<!-- Date: Sat Dec 8 00:20:42 2001 -->
<pre>BeamMon crashed. This is the output we got: 
 
ConsumerInput Begin processing 18000th record. Run 1343 
56, Trigger 716133 
ConsumerInput Begin processing 18100th record. Run 1343 
56, Trigger 719537 
%ERLOG-s Abnormal termination: caused by signal:  8 Fra 
mework abortJob()  
          8-Dec-2001 00:14:44 run = 134356 event = 7225 
68 
Calling abortJob for DummyInput... 
Calling abortJob for FileInput... 
Calling abortJob for DHInput... 
Calling abortJob for ConsumerInput... 
Calling abortJob for ErrorLoggerManager... 
Calling abortJob for PuffModule... 
Calling abortJob for CalibrationManager... 
Calling abortJob for GeometryManager... 
Calling abortJob for SignalManager... 
Calling abortJob for BeamMonFilter... 
Calling abortJob for cotvxprim... 
Calling abortJob for ConsumerErrorModule... 
Calling abortJob for BeamMon... 
 
</pre>
<!-- Author: Ramon Miquel -->
<!-- Comment: Sat Dec  8 00;27;24 comment by...Ramon -->
<pre>BeamMon crashed again after about 700 events.</pre>


<!-- Date: Sat Dec 8 00:42:13 2001 -->
<table><tr><td><a href=/elog/notebooks/consumer-ops/Sat_Dec__8_2001_004150.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-156  align='top'></td><td>ObjectMon seems to complain about a couple of histograms</a></td></tr></table>
<!-- Author: Ramon -->


<!-- Date: Sat Dec 8 09:01:38 2001 -->
<pre>Dear CO people, 
 
This morning around 7:45 am, I replaced online monitor release 
(cdfsoftb0) from 4.2.0c to 4.2.0d.  Main difference is only in 
YMon.  We can now see in YMon, HTDC plots, and aslo the Trigger 
folder to make sure YMon is receiving L1_MB_CLC (bit 21 in the 
[4,224] table). 
Please start YMon, the same way as you would start all the 
other consumers.  (./startMon.sh YMon 0). 
DO NOT use (.startMon_test....)  anymore.  (I actually removed 
this script from the consumer_scripts area). 
 
Now, here is a little tricky part: 
We like to send MB events to YMon for the physics run, but during 
the cosmic run (or Rick's run), there are no MB events, so 
we like to send All events to YMon.  Currently, there are no 
easy automatic way of doing that (yes, we are working on it). 
So, we need to ask you following: 
 
in /data1/consumer/runarea/YMon/Consumer/YMon   area, 
 
there are 3 files: 
runYMon.tcl   
runYMon_420d_MB.tcl   
runYMon_420d_all.tcl 
 
When we start up YMon, YMon will automatically pick up 
runYMon.tcl in this area.  For the physics run, please make 
sure runYMon.tcl is the same as runYMon_420d_MB.tcl, while 
during the other run (like cosmic run with no MB events), 
please copy runYMon_420d_all.tcl to runYMon.tcl. 
Please remember to copy runYMon_420d_MB.tcl to runYMon.tcl 
before the start of any physics run.  Again, please check the 
YMon Trigger folder trigger bits plot time to time  
and make sure bit 21 is on (=1). 
is = 1 during the physics run 
</pre>
<!-- Author: kaori -->


<!-- Date: Sat Dec 8 12:42:49 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-158&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-158  width='300' align='top'></td><td>I found XFT SL1 occupancy plot from TrigMon have a green stripe. Discussed with shift crues, and we have found that it is due to COT HV change in this morning. </a></td></tr></table>
<!-- Author: Satoru -->


<!-- Date: Sat Dec 8 13:13:53 2001 -->
<pre>BeamMon crashed with following error messages. 
Just restarted it. 
 
------------------------------------------------------ 
 
Dec  8 13:08:52 2001Accessing data base string: 
INSERT INTO CONSUMER_SUMMARY (RC_RUNNUMBER,NAME,CID,PROCEVT,EVALUATION,COMMENTS) 
 VALUES (134363,'BeamMon',CONSUMER_SEQUENCE.NEXTVAL,58117,'OK','d-phi:db_filled. 
vx-cot:db_filled./') 
BeamMon: Root file /data1/consumer/results/BeamMon/BeamMon134363aa.root saved. 
BeamMon: summary file /data1/consumer/results/BeamMon/BeamMon134363.sum written. 
DisplayServer::pollConsumer(): INFO: BeamMon sent message: END CONNECTION 
                             ==&gt; Going to terminate connection to BeamMon runnin 
g on 
                             b0dap66.fnal.gov 
Calling abortJob for cotvxprim-svxvxprim... 
cotvxprim-svxvxprim: 
calls:0 
Calling abortJob for DummyOutput... 
Calling abortJob for FileOutput... 
Calling abortJob for DHOutput... 
runBeamMon: Consumer ended with value  0. 
</pre>
<!-- Author: Satoru -->


<!-- Date: Sat Dec 8 13:44:09 2001 -->
<pre>BeamMon crashed again. restarted. 
Error message is partly different with previous crash. 
 
----------------------------------------------------- 
Error in &lt;TFile::TFile&gt;: file /data1/consumer/results/BeamMon/BeamMon134363aa.ro 
ot already exists 
*** Error OCI_ERROR  [-1] [1] 
*** Error: ORA-00001: unique constraint (RUNDB_PRD.CNS_UNIQUE) violated 
</pre>
<!-- Author: Satoru -->


<!-- Date: Sat Dec 8 14:07:57 2001 -->
<pre>BeamMon died again.</pre>
<!-- Author: Satoru -->


<!-- Date: Sat Dec 8 14:51:21 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-160&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-160  width='300' align='top'></td><td>There is a hot channel at leftside of  CMP occupancy plot part S.</a></td></tr></table>
<!-- Author: Satoru -->


<!-- Date: Sat Dec 8 16:29:39 2001 -->
<pre>YmonDisplay hang up while looking figures in beam off time. 
message says.  The last message of ymon display console is, 
Warning in &lt;TObjectTabke::Remove&gt;: 0x08b2d310 not found at 23749 
Is there way to kill only hunged up ymon display, instead of doing killALlDisplay.sh ? 
 
TrigMon also died when calibration run started. 
 
</pre>
<!-- Author: Akiya -->


<!-- Date: Sat Dec 8 18:28:40 2001 -->
<pre>DBANA hungs up and later X window of DBANA disapperared  
with a message as follows 
 
 By run date19691131 20010909 20011110 
Error in &lt;RootX11IOErrorHandler&gt;: fatal X11 error (connection to server lost?!) 
 
**** Save data and exit application **** 
 
 
</pre>
<!-- Author: Akiya -->


<!-- Date: Sat Dec 8 21:03:35 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-161&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-161  width='300' align='top'></td><td>One crate in WHA does not have power and I hope the dead region in the plot corresponds to it. </a></td></tr></table>
<!-- Author: Akiya -->


<!-- Date: Sat Dec 8 21:26:32 2001 -->
<pre>YMon hubgs up after printing a lot of message like, 
 
Warning in &lt;TObjectTable::Remove&gt;: 0x08a1e980 not found at 19715 
 
</pre>
<!-- Author: Akiya -->


<!-- Date: Sat Dec 8 21:50:19 2001 -->
<pre>YMon hungs up again.  It seems that YMon hungs up when I open  
some of the histograms in Calorimeter/HATD/Expert directory</pre>
<!-- Author: Akiya -->


<!-- Date: Sat Dec 8 23:07:37 2001 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-162&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-162' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-163&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-163' width='300'></a></center>
<!-- Author: AKiya -->
<!-- Comment: Sat Dec  8 23;09;02 comment by...Akiya -->
<pre>Several hot counters are seen in HAT plot of CHA and PHA.
I can't decide they are OK or not as there are not reference
figures for them</pre>


<!-- Date: Sun Dec 9 02:46:00 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-164&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-164  width='300' align='top'></td><td>There seems to be a hot channel in PHA.</a></td></tr></table>
<!-- Author: Ramon -->
<!-- Comment: Sun Dec  9 02;48;06 comment by...Ramon -->
<pre>There are no reference histograms for this alide.</pre>

<!-- Comment: Sun Dec  9 07;10;32 comment by...Ramon -->
<pre>Problem mostly disappeared in run 134414, the first one with physics data. The run in the previous plot was a cosmic run</pre>


<!-- Date: Sun Dec 9 07:12:31 2001 -->
<pre>Noticed L3RegionalMon not running. Tried to start it unsuccesfully: no error message but the task is not there.</pre>
<!-- Author: Ramon -->


<!-- Date: Sun Dec 9 07:30:53 2001 -->
<pre>XMON operates very very slowly. I killed both the consumer and the display and started them again. It didn't make things better.</pre>
<!-- Author: Ramon -->


<!-- Date: Sun Dec 9 08:49:22 2001 -->
<pre>SiliMon died with following messages. 
 
SiliMonModule: Root file /data1/consumer/results/SiliMon/SiliMon134414aa.root saved. 
SiliMonModule: summary file /data1/consumer/results/SiliMon/SiliMon134414.sum written. 
DisplayServer::pollConsumer(): INFO: SiliMon sent message: END CONNECTION 
                             ==&gt; Going to terminate connection to SiliMon running on 
                             b0dap67.fnal.gov 
Calling abortJob for DummyOutput... 
Calling abortJob for FileOutput... 
Calling abortJob for DHOutput... 
runSiliMon: Consumer ended with value  0. 
</pre>
<!-- Author: Satoru -->


<!-- Date: Sun Dec 9 10:41:45 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-165&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-165  width='300' align='top'></td><td>It is the CSX start time plot for Cosmic runs  which is mentioned above. No spikes appeared since cosmic data has no bunch crossing.</a></td></tr></table>
<!-- Author: Satoru -->


<!-- Date: Sun Dec 9 11:26:25 2001 -->
<pre>BeamMon crashed during run134414. Restarted. 
</pre>
<!-- Author: Satoru -->


<!-- Date: Sun Dec 9 11:54:38 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-166&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-166  width='300' align='top'></td><td>XMon complains that the trigger rate of L2_AUTO_L1_CMU6_PT8 is too low. (run 134404) The cross section is about 240 nb**-1. I checked the cross section of L1_CMU6_PT8, and it is about 945nb**-1. L2_AUTO should be auto accept, so it is strange that these two values are inconsistent. </a></td></tr></table>
<!-- Author: Satoru -->


<!-- Date: Sun Dec 9 13:15:46 2001 -->
<pre>All consumer died (except SVXMon) during waiting time 
for the next run. 
SVXMon was running, however, dumping large amount of error 
messages. 
So I killed SVXMon, and restarted all consumers. 
</pre>
<!-- Author: Satoru -->


<!-- Date: Sun Dec 9 13:24:03 2001 -->
<pre>The consumer srashes was caused by the crash of CSL.</pre>
<!-- Author: Satoru -->


<!-- Date: Sun Dec 9 14:27:46 2001 -->
<pre>ObjectMon crashed. Error messages are as follows. 
 
%ERLOG?? ObjectMon Summary: ::Evaluation:  3 ::evtProcessed:  2150::Comment:  
         Still at developing phase.JET:PROBLEMPhi failed for 2 fits.;Still at de 
veloping phase.Still at developing phase.MUON:OK;Track::PROBLEM:[Z0:Good][D0:Goo 
d][Eta:Good][Phi:Good][Charge/pT:Bad];  
          ObjectMon ObjectMon()  9-Dec-2001 14:23:56 run = 134418 event = 54743 
Dec  9 14:23:56 2001 Opening connection to database &lt;cdfonprd.fnal.gov&gt; 
%ERLOG?? ObjectMon Summary: ::Evaluation:  3 ::evtProcessed:  2150::Comment:  
         Still at developing phase.JET:PROBLEMPhi failed for 2 fits.;Still at de 
veloping phase.Still at developing phase.MUON:OK;Track::PROBLEM:[Z0:Good][D0:Goo 
d][Eta:Good][Phi:Good][Charge/pT:Bad];  
          ObjectMon ObjectMon()  9-Dec-2001 14:23:56 run = 134418 event = 54743 
Dec  9 14:23:56 2001 Opening connection to database &lt;cdfonprd.fnal.gov&gt; 
Dec  9 14:23:56 2001Accessing data base string: 
INSERT INTO CONSUMER_SUMMARY (RC_RUNNUMBER,NAME,CID,PROCEVT,EVALUATION,COMMENTS) 
 VALUES (134418,'ObjectMon',CONSUMER_SEQUENCE.NEXTVAL,2150,'PROBLEM','Still_at_d 
eveloping_phase.JET:PROBLEMPhi_failed_for_2_fits.;Still_at_developing_phase.Stil 
l_at_developing_phase.MUON:OK;Track::PROBLEM:[Z0:Good][D0:Good][Eta:Good][Phi:Go 
od][Charge/pT:Bad];') 
ObjectMon: Root file /data1/consumer/results/ObjectMon/ObjectMon134418aa.root sa 
ved. 
ObjectMon: summary file /data1/consumer/results/ObjectMon/ObjectMon134418.sum wr 
itten. 
DisplayServer: pollConsumer(): Expected an object. 
DisplayServer: pollConsumer(): Expected an object. 
DisplayServer: ERROR reading object messages! 
DisplayServer: Closing socket! 
Display Server: End with signal 11 ( = SIGSEGV ) 
Report end to the StateManager 
 
b0dap51:185:~&gt; cat objmon_err.log 
%ERLOG?? ObjectMon Summary: ::Evaluation:  3 ::evtProcessed:  2150::Comment:  
         Still at developing phase.JET:PROBLEMPhi failed for 2 fits.;Still at developing phase.Still at developing phase.MUON:OK;Track::PROBLEM:[Z0:Good][D0:Good][Eta:Good][Phi:Good][Charge/pT:Bad];  
          ObjectMon ObjectMon()  9-Dec-2001 14:23:56 run = 134418 event = 54743 
Dec  9 14:23:56 2001 Opening connection to database &lt;cdfonprd.fnal.gov&gt; 
Dec  9 14:23:56 2001Accessing data base string: 
INSERT INTO CONSUMER_SUMMARY (RC_RUNNUMBER,NAME,CID,PROCEVT,EVALUATION,COMMENTS) VALUES (134418,'ObjectMon',CONSUMER_SEQUENCE.NEXTVAL,2150,'PROBLEM','Still_at_developing_phase.JET:PROBLEMPhi_failed_for_2_fits.;Still_at_developing_phase.Still_at_developing_phase.MUON:OK;Track::PROBLEM:[Z0:Good][D0:Good][Eta:Good][Phi:Good][Charge/pT:Bad];') 
ObjectMon: Root file /data1/consumer/results/ObjectMon/ObjectMon134418aa.root saved. 
ObjectMon: summary file /data1/consumer/results/ObjectMon/ObjectMon134418.sum written. 
DisplayServer: pollConsumer(): Expected an object. 
DisplayServer: pollConsumer(): Expected an object. 
DisplayServer: ERROR reading object messages! 
DisplayServer: Closing socket! 
Display Server: End with signal 11 ( = SIGSEGV ) 
Report end to the StateManager 
 
Running time: 4027 s 
Calling abortJob for DummyOutput... 
Calling abortJob for FileOutput... 
Calling abortJob for DHOutput... 
sockets closed 
runObjectMon: Consumer ended with value  0. 
 
</pre>
<!-- Author: Satoru -->


<!-- Date: Sun Dec 9 14:29:47 2001 -->
<pre>SiliMon crashed, too. 
</pre>
<!-- Author: Satoru -->


<!-- Date: Sun Dec 9 22:38:14 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-167&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-167  width='300' align='top'></td><td>The problem of L2_AUTO_L1_CMU6_PT8 reported this moring  is gone in this run.</a></td></tr></table>
<!-- Author: Akiya -->


<!-- Date: Sun Dec 9 23:03:24 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-168&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-168  width='300' align='top'></td><td>There are only 32 bunches in this fill.</a></td></tr></table>
<!-- Author: Akiya -->


<!-- Date: Sun Dec 9 23:12:44 2001 -->
<pre>BeamMon died and restarted</pre>
<!-- Author: Akiya -->


<!-- Date: Sun Dec 9 23:42:19 2001 -->
<pre>We tried to stop VTXMon which are running with partion 4 for debugging.  But, SVXMon does not respond to "stop" command. 
The message was, 
 
SVXMon processed 80 events 
SVXMon processed 85 events 
SVXMon processed 90 events 
SVXMon processed 95 events 
SVX_Tcl&gt; stop 
 
DisplayServer: Got an unexpected string from the consum 
er. 
 
 
 
We could also anable to find process ID to kill SVXMon process on 
b0daq66.  To kill them we close the xterm windows on b0daq51. 
 
 
 
</pre>
<!-- Author: Akiya -->


<!-- Date: Mon Dec 10 00:49:59 2001 -->
<pre>SiliMon died. Restarted</pre>
<!-- Author: Ramon -->


<!-- Date: Mon Dec 10 03:35:15 2001 -->
<pre>BeamMon died. Restarted.</pre>
<!-- Author: Ramon -->


<!-- Date: Mon Dec 10 03:37:25 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-169&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-169  width='300' align='top'></td><td>ObjectMon complains about phi distribution in MET slide.</a></td></tr></table>
<!-- Author: Ramon -->


<!-- Date: Mon Dec 10 03:56:19 2001 -->
<pre>BeamMon died again. Restarted.</pre>
<!-- Author: Ramon -->


<!-- Date: Mon Dec 10 06:33:28 2001 -->
<pre>BeamMon died again. Restarted.</pre>
<!-- Author: Ramon -->


<!-- Date: Mon Dec 10 07:00:22 2001 -->
<pre>L2 rate for L2_AUTO_L1_CMU6_PT8 low. Reported to Scico and Ops. Mgr. </pre>
<!-- Author: Ramon -->


<!-- Date: Mon Dec 10 09:36:11 2001 -->
<pre>BeamMon died. Restarted.</pre>
<!-- Author: Satoru -->


<!-- Date: Mon Dec 10 12:08:57 2001 -->
<pre>BeamMon crashed again.Restarted.</pre>
<!-- Author: Satoru -->


<!-- Date: Mon Dec 10 12:23:17 2001 -->
<pre>BeamMon died again!</pre>
<!-- Author: Satoru -->


<!-- Date: Mon Dec 10 15:26:50 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-170&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-170  width='300' align='top'></td><td>BSU ocuppancy rate plot has no entry. Thin problem is seems to be happened between run 134462 and 134468, and also seen in 134471. </a></td></tr></table>
<!-- Author: Satoru -->


<!-- Date: Mon Dec 10 15:47:24 2001 -->
<pre>I mailed to Guram Chlachidze about the BSU problem. 
</pre>
<!-- Author: Satoru -->


<!-- Date: Mon Dec 10 18:36:56 2001 -->
<pre>BSU occupancy rate plot seen in the previous run is not 
exist in this run(run 134504)</pre>
<!-- Author: Akiya -->


<!-- Date: Mon Dec 10 18:42:57 2001 -->
<pre>BeamMon died and restarted.</pre>
<!-- Author: Akiya -->


<!-- Date: Mon Dec 10 20:29:04 2001 -->
<pre>Sent three mails asking questions of funny distribution of histograms. 
(1) Phi distribution of MET as Ramon wrote, sent mail to Henneck 
(2) Hot counters in PHA and CHA, sent mail to calorimeter people 
(3) Strange trigger pattern to Carla. 
 
Carla just call back. She said it may be the version of TrigMon 
is old.  She is going to check by herself. 
</pre>
<!-- Author: Akiya -->


<!-- Date: Mon Dec 10 23:26:09 2001 -->
<pre>BeamMon died twice in last thirty minuites.</pre>
<!-- Author: Akiya -->


<!-- Date: Mon Dec 10 23:44:24 2001 -->
<pre>Jean requested CO to read  
/~svxii/sionmon/sionmon.html 
where CO instruction to checki silicon is described.</pre>
<!-- Author: Akiya -->


<!-- Date: Tue Dec 11 00:38:08 2001 -->
<pre>BeamMon died. Restarted.</pre>
<!-- Author: Ramon -->


<!-- Date: Tue Dec 11 01:12:17 2001 -->
<pre>BeamMon died again. Restarted.</pre>
<!-- Author: Ramon -->


<!-- Date: Tue Dec 11 02:20:50 2001 -->
<pre>BeamMon died again. Restarted.</pre>
<!-- Author: Ramon -->


<!-- Date: Tue Dec 11 02:39:41 2001 -->
<pre>BeamMon died again. Restarted.</pre>
<!-- Author: Ramon -->


<!-- Date: Tue Dec 11 02:49:15 2001 -->
<pre>BeamMon died again. Restarted. This is becoming ridiculous...</pre>
<!-- Author: Ramon -->


<!-- Date: Tue Dec 11 04:05:16 2001 -->
<pre>XMon complains about L2_AUTO_L1_CMU6_PT8. Noted in the shift e-log. Should keep an eye on it for next runs.</pre>
<!-- Author: Ramon -->


<!-- Date: Tue Dec 11 05:54:35 2001 -->
<pre>All monitoring programs died as a consequence of DAQ problems that stopped the run. Restarted them all. Will wait to start displays until run starts once DAQ problem are solved.</pre>
<!-- Author: Ramon -->
<!-- Comment: Tue Dec 11 06;44;21 comment by...Ramon -->
<pre>YMon isn't getting any events. Suspected reason: we are running with a test trigger (not for physics), with no trigger bit 21, which is the one YMON is expecting in the current configuration. I do nothing.</pre>
<!-- Comment: Tue Dec 11 07;11;56 comment by...Ramon -->
<pre>YMon ok once back to normal running.</pre>
<!-- Date: Tue Dec 11 06:42:06 2001 -->
<pre>The event display was left in a funny state as a consequence of the problem mentioned before. I killed it and restarted it, but no events were seen. After restarting it three times, it finally works. Problem fixed but not understood.</pre>
<!-- Author: Ramon -->




<!-- Date: Wed Dec 12 13:02:24 2001 -->
<pre>TrigMon running in partition 14 crashed. 
</pre>
<!-- Author: Satoru -->


<!-- Date: Wed Dec 12 23:08:04 2001 -->
<pre>SiliMon froze due to data format error of silicon, after showing 
following message.   This happens as long as there are error in  
silicon data and we will ask Calra to modifiy the program so as  
not to freeze SiliMon display.  When forzen, type &lt;ctrl&gt;-C in  
SiliMon terminal, then SiliMon display program is killed.  SiliMon it self does not freeze by this error. 
 
 
HistoDisplay: The requested object 'unbiasedchipIB0W5L3NC2(4062)_ERROR' is not available fro 
m the  
Display Server. 
 Failed to get unbiasedchipIB0W5L3NC2(4062)_ERROR 
 Not updated.  
HistoDisplay: The requested object 'unbiasedchipIB0W6L0PC1(4773)_ERROR' is not available fro 
m the  
Display Server. 
 Failed to get unbiasedchipIB0W6L0PC1(4773)_ERROR 
 Not updated.  
HistoDisplay: The requested object 'unbiasedchipIB5W5L4PC1(5049)_ERROR' is not available fro 
m the  
Display Server. 
 Failed to get unbiasedchipIB5W5L4PC1(5049)_ERROR 
 Not updated.  
</pre>
<!-- Author: Akiya -->


<!-- Date: Thu Dec 13 01:02:27 2001 -->
<pre>SiliMon died. Restarted.</pre>
<!-- Author: Ramon -->


<!-- Date: Thu Dec 13 02:49:32 2001 -->
<pre>Stage0 died. Restarted.</pre>
<!-- Author: Ramon -->


<!-- Date: Thu Dec 13 03:07:22 2001 -->
<pre>SiliMon died again. Restarted.</pre>
<!-- Author: Ramon -->


<!-- Date: Thu Dec 13 03:33:01 2001 -->
<pre>SiliMon died again. Restarted.</pre>
<!-- Author: Ramon -->


<!-- Date: Thu Dec 13 04:04:52 2001 -->
<pre>After SiliMon died, I was unable to get its display to work. It hungs as soon as I try to connect it to SiliMon. SVXMon, however, seems to work fine. For the moment, I give up on SiliMon</pre>
<!-- Author: Ramon -->


<!-- Date: Thu Dec 13 07:26:41 2001 -->
<pre>SiliMon died. Restarted.</pre>
<!-- Author: Ramon -->


<!-- Date: Thu Dec 13 07:30:48 2001 -->
<pre>SiliMon died again. This is the message: 
Error in &lt;TFile::TFile&gt;: file /data1/consumer/results/SiliMon/SiliMon134774aa.root already exists 
*** Error OCI_ERROR  [-1] [1] 
*** Error: ORA-00001: unique constraint (RUNDB_PRD.CNS_UNIQUE) violated 
SiliMonModule: Root file /data1/consumer/results/SiliMon/SiliMon134774ab.root saved. 
SiliMonModule: summary file /data1/consumer/results/SiliMon/SiliMon134774.sum written. 
DisplayServer::pollConsumer(): INFO: Consumer sent message: END CONNECTION 
                             ==&gt; Going to terminate connection to Consumer running on 
                             b0dap67.fnal.gov 
ConsumerExport#sigChildHandler: Abnormal termination, signal number = 110 
Calling abortJob for DummyOutput... 
Calling abortJob for FileOutput... 
Calling abortJob for DHOutput... 
runSiliMon: Consumer ended with value  0. 
 
Restarted. 
</pre>
<!-- Author: Ramon -->


<!-- Date: Thu Dec 13 07:46:24 2001 -->
<pre>SiliMon doesn't crash now, but doesn't seem to be consuming any events. I will wait until Kaori arrives and will let her know. </pre>
<!-- Author: Ramon -->
<!-- Comment: Thu Dec 13 08;01;20 comment by...Ramon -->
<pre>SiliMon seems to be working fine now, including the display. Kaori phoned, she said she will have a look into SiliMon during the day.</pre>


<!-- Date: Thu Dec 13 08:16:16 2001 -->
<pre>TrigMon died.Restarted.</pre>
<!-- Author: Satoru -->


<!-- Date: Thu Dec 13 08:56:39 2001 -->
<pre>BeamMon died. Restarted. 
</pre>
<!-- Author: Satoru -->


<!-- Date: Thu Dec 13 09:14:27 2001 -->
<pre>BeamMon died again. restarted.</pre>
<!-- Author: Satoru -->


<!-- Date: Thu Dec 13 09:33:49 2001 -->
<pre>BeamMon and SiliMon died. Restarted.</pre>
<!-- Author: Satoru -->
<!-- Comment: Thu Dec 13 15;49;21 comment by...Satoru -->
<pre>The problem that BeamMon crashes so often is
fixed by expert.</pre>
<!-- Date: Thu Dec 13 11:20:39 2001 -->
<pre>SiliMon died. Following are the error messages. 
 
%ERLOG-e CotBeam not valid:  
         /home/cdfsoft/dist/packages/TrackingMods/V00-02-22/src/CT_TrackingModule.cc  
        : 325   CT_TrackingModule CT_TrackingModule::beginRun ()  
         13-Dec-2001 11:16:14 run = 134794 event = 0 
doing unkeyed get 
 ave calib time = 305.742 
exiting CT_Cal class 
exiting CT_Cal class 
CT_Calibration::_setDriftModelConstants: Using values in database 
CT_Calibration::_setDriftModelConstants: Doing unkeyed get from DB  
CT_Calibration::_setDriftModelConstants: Found Run 134525      Version 1       Status COMPLETE 
KalDetectorNode::set(): Unhandled Material "ALUMINUM in physical volume CentralFlangePhys0"  (defaults to silicon.) 
%ERLOG-s Abnormal termination: caused by signal:  8 Framework abortJob()  
         13-Dec-2001 11:16:15 run = 134794 event = 0 
Calling abortJob for DummyInput... 
Calling abortJob for FileInput... 
Calling abortJob for DHInput... 
********************* DHInputModule ************************ 
*      !!! Job is aborted !!!                              * 
* The reason is most likely not relevant to DHInput itself * 
************************************************************ 
Calling abortJob for ConsumerInput... 
Calling abortJob for ErrorLoggerManager... 
Calling abortJob for PuffModule... 
Calling abortJob for CalibrationManager... 
Calling abortJob for GeometryManager... 
Calling abortJob for SignalManager... 
Calling abortJob for ConsumerErrorModule... 
Calling abortJob for SiClusteringModule... 
Calling abortJob for CT_TrackingModule... 
Calling abortJob for SiPatternRecModule... 
Calling abortJob for SiliMonModule... 
%ERLOG?? SiliMon Summary: ::Evaluation:  3 ::evtProcessed:  0::Comment:  
         No clusters detected in layer   0    1    2    3    4    5    6    7  No ADC counts detected in layer   0    1    2    3    4    5    6    7  No tracks detected in layer   0    1    2    3    4    5    6    7    
          SiliMonModule SiliMonModule() 13-Dec-2001 11:16:46  
         run = 134794 event = 0 
ConsumerSummaryDB::insertRunSummary(): Comment string of Run Summary had to be 
truncated to 200 characters. Larger comments cannot be stored in the data base. 
Dec 13 11:16:46 2001 Opening connection to database &lt;cdfonprd.fnal.gov&gt; 
Dec 13 11:16:46 2001Accessing data base string: 
INSERT INTO CONSUMER_SUMMARY (RC_RUNNUMBER,NAME,CID,PROCEVT,EVALUATION,COMMENTS) VALUES (134791,'SiliMon',CONSUMER_SEQUENCE.NEXTVAL,0,'PROBLEM','No_clusters_detected_in_layer___0____1____2____3____4____5____6____7__No_ADC_counts_detected_in_layer___0____1____2____3____4____5____6____7__No_tracks_detected_in_layer___0____1____2____3____4____5__') 
SiliMonModule: Root file /data1/consumer/results/SiliMon/SiliMon134791aa.root saved. 
SiliMonModule: summary file /data1/consumer/results/SiliMon/SiliMon134791.sum written. 
Calling abortJob for DummyOutput... 
Calling abortJob for FileOutput... 
Calling abortJob for DHOutput... 
DisplayServer::pollConsumer(): INFO: SiliMon sent message: END CONNECTION 
                             ==&gt; Going to terminate connection to SiliMon running on 
                             b0dap67.fnal.gov 
runSiliMon: Consumer ended with value  0. 
Press C-d to close window. 
</pre>
<!-- Author: Satoru -->


<!-- Date: Thu Dec 13 13:02:41 2001 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-171&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=consumer-ops&action=view&page=-171  width='300' align='top'></td><td>This is a plots from DAQMon. There are two peaks in upper plot COT region. And also a peak at bottom plot COT region. It is a problem of COT crate, and will cause DAQ problem. Please notify the experts if you see such a peak.</a></td></tr></table>
<!-- Author: Satoru -->
<!-- Comment: Thu Dec 13 20;33;41 comment by...Arnd -->
Actually the peaks at IP~78 are fine: there is an additional
FADC board in this crate (b0cot18), causing somewhat increased
readout time and data size. IP~75 (b0cot15) is a real chamber
problem, present since some time already, and will be taken care
of at the TDC level soon.

<!-- Date: Thu Dec 13 19:13:36 2001 -->
<pre>Jean left us the instruction to monitor silicons in a black binder.  For every runs that lasts more than 30 minutes and has more than 1000 events, CO must fill a form, "CO Run cvhecklist for silicon", which is in a black binder and in case of a new error, page silicon pager, 218-8227. 
 
Tara and Kaori also came and check that SiliMon does not crash. 
New SiliMon display pops up a red windows in case it finds a error to page silicon pager.</pre>
<!-- Author: Akiya -->



<!-- Date: Thu Dec 13 21:13:27 2001 -->
<pre>Restart SVXMon as it was not running since 134835</pre>
<!-- Author: Akiya -->


<!-- Date: Thu Dec 13 21:18:30 2001 -->
<pre>SiliMon died, restarted</pre>
<!-- Author: Akiya -->


<!-- Date: Thu Dec 13 23:28:39 2001 -->
<pre>Serious error were found in the SiliMon error log of run 134847: 
No Data and Out of Sync.  I paged silicon pager, but no answer. 
I told Amit as he is in the control room.</pre>
<!-- Author: Akiya -->
<!-- Comment: Thu Dec 13 23;33;47 comment by...Akiya -->
<pre>I filled "CO Run checklist for silicon" for run 134847, but it took about 30 min !!.  Too long time.</pre>


<!-- Date: Fri Dec 14 06:02:00 2001 -->
<pre>Filled silicon checklist and run chelist for run 134848. Found some problems in SVXMon. Reported them in shift e-log. Paged silicon expert. He will investigate.</pre>
<!-- Author: Ramon -->


<!-- Date: Fri Dec 14 06:55:16 2001 -->
<pre>TrigMon died. Restarted.</pre>
<!-- Author: Ramon -->


<!-- Date: Fri Dec 14 07:38:41 2001 -->
<pre>TrigMon died again. Restarted.</pre>
<!-- Author: Ramon -->


<!-- Date: Fri Dec 14 15:19:37 2001 -->
<pre>Copied /data1/consumer/runarea/YMon/Consumer/YMon/runYMon_420d_all.tcl 
to runYMon.tcl for cosmic run. 
 
run 134903: SiliMon Error window: 
Error in RawMonitor 
All ADC zero in: 
IB0W3L0...L4...(P and N, C0-C4) 
IB0W4L3(P and N, C0-C4) 
page silicon 
 
paged Chris, he came and observed the error. 
 
</pre>
<!-- Author: Yuri Gotra -->


<!-- Date: Fri Dec 14 16:02:51 2001 -->
<pre>created file ~/consumer_scripts&gt;startDisplay_yuri.sh with  
modifications to open Display xterm in the upper right corner, 
think it may be more convenient and leaves more space for plots.</pre>
<!-- Author: Yuri Gotra -->


<!-- Date: Fri Dec 14 17:45:18 2001 -->
<pre>Yuri wrote a nice program to scan svxmon log file. 
It is /cdf/people1/cdfdaq/consumer_scripts/checksilogs.sh. 
This will list only lines which has error. 
To use it, don't forget to change the date in the first line of the script.</pre>
<!-- Author: Akiya, Yuri, Zhaoou -->


<!-- Date: Fri Dec 14 20:22:22 2001 -->
<pre>I did some improvement on the script, putting in "TODAY" so that people don't need to type in the date. The original one was renamed "checksilogs.sh.bak". The current one is checksilogs.sh 
</pre>
<!-- Author: Zhaoou -->


<!-- Date: Sat Dec 15 02:09:57 2001 -->
<pre>Switched YMon to Minimum Bias mode. I forgot to do so when we started data taking at around midnight.</pre>
<!-- Author: Ramon -->
<!-- Comment: Sat Dec 15 02;22;57 comment by...Ramon -->
<pre>Actually, we didn't take data for more than about 15 minutes between midnight and now because of Daq problems...</pre>


<!-- Date: Sat Dec 15 02:35:41 2001 -->
<pre>SiliMom crashed and restarted</pre>
<!-- Author: Giuseppe -->


<!-- Date: Sat Dec 15 03:23:44 2001 -->
<pre>SiliMon crashed and restarted</pre>
<!-- Author: Giuseppe -->


<!-- Date: Sat Dec 15 09:35:22 2001 -->
<pre>SiliMon crashed/restarted</pre>
<!-- Author: Yuri Gotra -->


<!-- Date: Sat Dec 15 10:04:37 2001 -->
<pre>Silimon craashed/restarted</pre>
<!-- Author: Yuri -->


<!-- Date: Sat Dec 15 12:07:50 2001 -->
<pre>restarted SVXMon by request of J.P.F. 
restarted XMon display and SiliMon</pre>
<!-- Author: Yuri -->


<!-- Date: Sat Dec 15 12:58:35 2001 -->
<pre>ObjectMon, SiliMon died, restarted</pre>
<!-- Author: Yuri -->


<!-- Date: Sun Dec 16 01:38:58 2001 -->
<pre>YMon, TrigMon crashed/restarted</pre>
<!-- Author: Giuseppe -->


<!-- Date: Sun Dec 16 01:47:22 2001 -->
<pre>SiliMon crashed/restored</pre>
<!-- Author: Giuseppe -->


<!-- Date: Sun Dec 16 04:22:58 2001 -->
<pre>Stage0 crashed/restored</pre>
<!-- Author: Giuseppe -->


<!-- Date: Sun Dec 16 07:53:51 2001 -->
<pre>NOTE: 
no beam during the owl shift; 
runYMon_420d_all.tcl copied to runYMon.tcl for cosmic runs  
( in /data1/consumer/runarea/YMon/Consumer/YMon/ ). 
runYMon_420d_MB.tcl needed back to runYMon.tcl before the beam  
data acquisition will start.</pre>
<!-- Author: Giuseppe -->


<!-- Date: Sun Dec 16 09:19:14 2001 -->
<pre>Stage0 mon died/restarted</pre>
<!-- Author: Yuri -->


<!-- Date: Mon Dec 17 04:41:57 2001 -->
<pre>TrigMon crashed/restarted</pre>
<!-- Author: Giuseppe -->


<!-- Date: Mon Dec 17 04:53:42 2001 -->
<pre>TrigMon,SiliMon crashed/restarted several times. 
Probably connected with NO data acquisition......</pre>
<!-- Author: Giuseppe -->


<!-- Date: Mon Dec 17 09:17:03 2001 -->
<pre>created scripts to change YMon tcl file for physics/cosmic run: 
b0dap51:~/consumer_scripts&gt;./runCosmic.sh 
b0dap51:~/consumer_scripts&gt;./runPhysics.sh 
</pre>
<!-- Author: Yuri -->


<!-- Date: Mon Dec 17 09:18:58 2001 -->
<pre>stopped Xmon to allow debugging of the new version</pre>
<!-- Author: Yuri -->