<!-- Title: Eve shift. Wed Feb  5, 2003 -->
<!-- SciCo: R. Wagner -->
<!-- DAQAce: Alberto Annnovi -->
<!-- MonAce: &nbsp; -->
<!-- CO: &nbsp; -->
<!-- OpManager: &nbsp; -->
<!-- Notes: -->
<pre>Cosmics with partitions 5, 6, 7 (1 hour each). 
Level 2 torture with partition 3.</pre>

<!-- Date: Wed Feb 5 16:14:08 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158170">158170</A>
 Activated at 2003.02.05 16:12:48
<!-- Author: RunControl -->


<!-- Date: Wed Feb 5 16:14:32 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158170">158170</A>
ACTIVATE: Starting Cosmics run in partition 5
<!-- Author: Alberto -->



<!-- Date: Wed Feb 5 16:49:45 2003 -->
<pre>Current status of the Global Alarms page: 
 
After the iFix upgrade the hearbeat monitoring it's not working. 
But the HV monitoring & control should work. 
 
There is some problem with CEM CHA WHA HVs. 
Now CEM and WHA are off. 
 
Steve is working on both problems.</pre>
<!-- Author: Alberto -->


<!-- Date: Wed Feb 5 16:56:22 2003 -->
Took HTDC slot #3 tdc out (serial # 280) and replaced with #318. I cannot run TDCtest on this TDC however. I'll look into this more later tonite. Slot#3 has been marked online in DB again and the crate should be useable.		
<!-- Author: dave t. and dan c. -->


<!-- Date: Wed Feb 5 17:14:46 2003 -->
Replaced old transition board by new one in Plug (west) for EMTiming.
<!-- Author: Sungwon/Fabio/Dave/Slava -->

<!-- Date: Wed Feb 5 17:21:23 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158170">158170</A>
 Terminated at 2003.02.05 17:19:52
<!-- Author: RunControl -->


<!-- Date: Wed Feb 5 17:21:41 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158170">158170</A>
TERMINATE: Ending Cosmics on P5
<!-- Author: Alberto -->



<!-- Date: Wed Feb 5 17:28:25 2003 -->
<p>Many new problems, some related to VNODE1 being updated to 3.0 (maybe), and some unrelated.</p> 
 
<p>First, Jim Loskot has updated the Global Alarm and HV Summary pages so they are now one large page over the two screens on VNODE1.  I changed the puke background color to something readable from a distance.  One apparent consequence of updating VNODE1 to iFIX 3.0 is that now there are <b>no</b> heartbeats, and the HBM and VOI boxes are in alarm.  We spent most of the afternoon investigating this without much progress beyond finding that the heartbeat values are never written into VNODE1's database.  These are generated in the other nodes and sent to VNODE1, so why did they all stop simultaneously?  I have tried restarting PISABOX, but that did not restore heartbeats, either.   
Otherwise, everything is functional in the global alarms and the HV summary.  We will continue to investigate tomorrow.</p> 
 
<p>Starting with the cycle of PMT readout for the CEM CHA WHA starting at 1315, the PMT readout process on PISABOX died.  After many restarts and some panicing, I realized that an essential file (WHA_LID.HVR) was not present in the desired directory.  Nikolay put back a copy, but it's a mystery currently why this file would have disappeared.  This means monitoring was not working until just recently; however, no HVs were turned off.</p> 
 
<p>We (Mark Jim Marcin and I) also spent more time investigating the PSM status which is reported differently by VNODE1 and VNODE2.  During this period, the SVX alarm sounded on VNODE2, but an hour after the alarm actually occured as reported by the silicon power supply GUI.  Is this also a symptom of having the global alarms present on both VNODE1 and VNODE2?  Not wanting to duplicate this experience, we've gone back to running only on VNODE1.</p> 
 
<p>And one last item: the GAM box is light blue (value out of range).  However, the value of the ONOFF tag in the database is fine.  Nikolay pointed out that the "Stat Val" in the HV Summary is 0 instead of 9 like all the other systems.  Maybe this is the cause of the problem.  I continue to work on this.</p>
<!-- Author: Steve Hahn -->


<!-- Date: Wed Feb 5 17:29:13 2003 -->
We will start updating PISABOX at 0600 tomorrow, having spent all of today working on VNODE1 issues.
<!-- Author: Steve Hahn -->

<!-- Date: Wed Feb 5 17:32:04 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158174">158174</A>
ACTIVATE: plug laser with emt system
<!-- Author: slava77 -->



<!-- Date: Wed Feb 5 17:38:44 2003 -->
Steve Kuhlman called.  He replaced a fiber today and would like another Isolist run.  It is now on the board for evening shift.
<!-- Author: R. Wagner -->

<!-- Date: Wed Feb 5 17:50:30 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158175">158175</A>
 Activated at 2003.02.05 17:48:34
<!-- Author: RunControl -->



<!-- Date: Wed Feb 5 17:50:30 2003 -->
3 fib crates (b0fib00,04,06) failed to coldstart once. 
They remained yellow.
<!-- Author: Alberto -->

<!-- Date: Wed Feb 5 17:51:37 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158175">158175</A>
ACTIVATE: Starting Cosmics run on partition 7
<!-- Author: Alberto -->


<!-- Date: Wed Feb 5 17:55:37 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158174">158174</A>
TERMINATE: finished	

<!-- Author: slava77 -->



<!-- Date: Wed Feb 5 18:09:51 2003 -->
DOTO from COT_17. Bunchcounters mismatch in slot 3 and 14.
<!-- Author: Alberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158175' target=_top>158175</a>) -->
<!-- Comment: Wed Feb  5 18;37;15 comment by...Alberto -->
Got one more from COT_17. Bunchcounters mismatch in slot 3 and 15.
<!-- Date: Wed Feb 5 18:20:16 2003 -->
<pre>Several DOTO from CCAL06. 
HRR did'nt work but Reboot, Reset and recor did. 
 
 
b0ccal06:Messenger:6:15:53 PM-&gt;SMXR Not Responding event 60423, slot 3, bunch ctr 77, L2buffer 0, header 0x3dfe34d 
</pre>
<!-- Author: Alberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158175' target=_top>158175</a>) -->


<!-- Date: Wed Feb 5 18:23:52 2003 -->
<pre>The ErrorHandler has been promoted to  
state checking.  That is, RunControl will  
now check the returned state from the Error Handler  
with the expected value based on the State Manager  
in use. 
 
(Donatella mentioned that the problems this  
weekend were not related to this feature;  those  
problems have been fixed (see earlier days' e-log)) 
</pre>
<!-- Author: W.Badgett -->
<!-- Comment: Wed Feb  5 18;24;25 comment by...WFB -->
<pre>Existing RunControls will have to go thru the Partition
to pick up this change.</pre>


<!-- Date: Wed Feb 5 18:55:30 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158175">158175</A>
 Terminated at 2003.02.05 18:54:02
<!-- Author: RunControl -->


<!-- Date: Wed Feb 5 18:55:31 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158175">158175</A>
TERMINATE: Ending cosmics on P7
<!-- Author: Alberto -->


<!-- Date: Wed Feb 5 19:13:57 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158176">158176</A>
 Activated at 2003.02.05 19:12:42
<!-- Author: RunControl -->


<!-- Date: Wed Feb 5 19:17:03 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158176">158176</A>
ACTIVATE: Starting cosmics run on Partion 6.
<!-- Author: Alberto -->



<!-- Date: Wed Feb 5 19:32:20 2003 -->
Some reason, consmer status page was unable to display consumer status.  I checked /www mount from b0dap32 and other things, and 
they are all ok.  Stopped and restarted consumerstatemanager, then the status display started displaying ok.
<!-- Author: kaori -->


<!-- Date: Wed Feb 5 20:02:24 2003 -->
<pre>Looks like tomcat was not running... this would effect the  
servlets. 
 
Restarted it. 
 
</pre>
<!-- Author: Frank Chlebana -->

<!-- Date: Wed Feb 5 20:16:31 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158176">158176</A>
 Terminated at 2003.02.05 20:14:24
<!-- Author: RunControl -->


<!-- Date: Wed Feb 5 20:16:32 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158176">158176</A>
TERMINATE: Ending Cosmics run on P6
<!-- Author: Alberto -->



<!-- Date: Wed Feb 5 20:23:14 2003 -->
Steve Hahn forgot to mention that the <a href= 
'/cgi/elog/elog.pl?nb=2003&action=view&page=95&frame=2&anchor=09:12:23'>problem</a> noted by Steve Nahn on Sat. was also fixed by the upgrade today. Clicking on D in the bsu/tsu display brought up what it was supposed to.
<!-- Author: Phil Schlabach -->

<!-- Date: Wed Feb 5 20:27:03 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158177">158177</A>
 Activated at 2003.02.05 20:25:15
<!-- Author: RunControl -->


<!-- Date: Wed Feb 5 20:27:04 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158177">158177</A>
ACTIVATE: Starting Isolist test
<!-- Author: Alberto -->
<!-- Comment: Wed Feb  5 20;28;48 comment by...Alberto -->
<pre>Crate b0svt06 didn't coldstart. It was red.
Then I tried Reboot, Reset and Recover but it did not work.
The crate failed again to coldstart but it was yellow.

Then I did reset_crate and Reboot, Reset and Recover and it worked.</pre>


<!-- Date: Wed Feb 5 22:04:49 2003 -->
Done Timeout from PCAL_09. 
 
b0pcal09:Messenger:10:02:08 PM->Runtime Error 1, Event 452731: See nonsequential event number (old 10, new 10) in L2 message  
<!-- Author: Alberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158177' target=_top>158177</a>) -->

<!-- Date: Wed Feb 5 22:33:53 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158177">158177</A>
 Terminated at 2003.02.05 22:32:31
<!-- Author: RunControl -->


<!-- Date: Wed Feb 5 22:34:44 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158177">158177</A>
TERMINATE: Ending Isolist test
<!-- Author: Alberto -->


<!-- Date: Wed Feb 5 22:49:25 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158178">158178</A>
ACTIVATE: Starting L2 torture w/o FCAL01
<!-- Author: Alberto -->
<!-- Comment: Wed Feb  5 22;52;28 comment by...Alberto -->
L1 rate is 44 KHz
L2 rate is 315 Hz
L3 rate	is  56 Hz

Dead time 10%


<!-- Comment: Wed Feb  5 22;57;24 comment by...Alberto -->
<pre>Human readable:
L1 rate is 44 KHz
L2 rate is 315 Hz
L3 rate is 56 Hz
Dead time 10%</pre>


<!-- Date: Wed Feb 5 23:03:44 2003 -->
Done Timeout from CCAL_13. 
 
PM->SMXR Not Responding event 239853, slot 3, bunch ctr 12, L2buffer 1, header 0x3bfe30c  
<!-- Author: Alberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158178' target=_top>158178</a>) -->


<!-- Date: Wed Feb 5 23:21:23 2003 -->
<pre>Done Timeout from PCAL09: 
 
(MLE) b0pcal09:Messenger:11:15:56 PM-&gt;Runtime Error 1, Event 484061: See nonsequential event number (old 12, new 8) in L2 message 
(MLE) b0pcal09:Messenger:11:15:56 PM-&gt;Event 484061: Bunchcounters in slot 14 (BC=62) and slot 15 (BC=71) disagree (at least) 
(MLE) b0pcal09:Messenger:11:15:56 PM-&gt;Runtime Error 2, Event 484061: Bunch counter mismatch, mismatch count = 4 
(MLE) b0l3gate1.fnal.gov:main:11:15:56 PM-&gt;Host b0eb16.fnal.gov, task tRec_0 SCPU-P6-E-VrbHeader: Dump of header words for event 484061 from VRB in slot 12: 0x00000000 0x000012e8 0x002f00ef 0x02ec0320 0x031c02ec 0x036c0338 0x00000000 0x0 
0000000 
</pre>
<!-- Author: Alberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158178' target=_top>158178</a>) -->


<!-- Date: Wed Feb 5 23:25:12 2003 -->
<pre>Done Timeout from CCAL13: 
PM-&gt;SMXR Not Responding event 579521, slot 3, bunch ctr 27, L2buffer 1, header 0x3bfe31b</pre>
<!-- Author: Alberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158178' target=_top>158178</a>) -->


<!-- Date: Wed Feb 5 23:43:27 2003 -->
<pre>Hardware EVB erro. Auto HRR recovered it. 
 
4 of this messages: 
SCPU-P6-E-VrbHeader: Dump of header words for event 796798 from VRB in slot 10: 0x00000000 0x000016a8 0x003c00bc 0x00f80100 0x01380118 0x010000f8 0x01000720 0x07240100 
 
and 1 of this: 
34'51" b0l3gate1.fnal.gov:main:11:34:49 PM-&gt;Host b0eb13.fnal.gov, task tRec_0 SCPU-P6-E-TracerEventId: Event 796802, crate 53, channel 4 has either bad Tracer ID or bad markers around Tracer word. Tracer word event ID is 5, should be 2. 
</pre>
<!-- Author: Alberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158178' target=_top>158178</a>) -->

<!-- Date: Wed Feb 5 23:56:56 2003 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>23:55:01</TD>                                                       
<TD ALIGN=CENTER>::</TD>                                                
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD ALIGN=RIGHT></TD>                                                   
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Thu Feb 6 00:00:48 2003 -->
<font color=red>Shift Summary: </font><pre>Ran cosmics with partitions 5, 6, and 7. 
 
Steve Kuhlman istalled a fiber today, so another Isolist run was taken. 
 
Running Level 2 Torture tests 
 
As usual, see Steve Hahn's comments on IFIX status 
 
PISABOX IFIX mods will really be done tomorrow at 06:00.</pre> 
<!-- Author: R. Wagner -->


<!-- Date: Thu Feb 6 00:05:00 2003 -->
<pre>Done Timeout from CCAL13: 
 
PM-&gt;SMXR Not Responding event 1202475, slot 3, bunch ctr 9, L2buffer 0, header 0x3bfe309</pre>
<!-- Author: Alberto :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=158178' target=_top>158178</a>) -->
<!-- Comment: Thu Feb  6 00;05;25 comment by...Alberto -->
<table border=0><td><tr><td bgcolor=#e0e030>We have got 3 of this errors during this run all of them from CCAL13 and slot 3.</td></tr></td></table>