<!-- Title: Eve shift. Fri May 16, 2003 -->
<!-- SciCo: Kirsten Tollefson -->
<!-- DAQAce: Thorsten Walter -->
<!-- MonAce: Steve Vejcik -->
<!-- CO: Claudio Ferretti -->
<!-- OpManager: Masa Tanaka -->
<!-- Notes: -->
<pre>Enheriting store 2551.  Kaori writing entry about muon problem. 
 
Plan to take data with default table.</pre>


<!-- Date: Fri May 16 16:16:53.642 2003 -->
<pre>PSM is "Gray" on Global Alarms with messages on 
the PC(3rd Floor) of its being locked by someone 
else.  Monitoring ace from previous shift indicated 
that this is a know problem being worked on by 
Mark Knapp & company.</pre>
<!-- Author: steve -->
<!-- Comment: Fri May 16 17;39;36 comment by...steve -->
<pre>Steve Hahn recruited/volunteered to come check out the PSM 
incommunicado problem/issue.</pre>
<!-- Date: Fri May 16 16:17:39.826 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-5156&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-5156  width='300' align='top'></td><td>CLAD upper left plot is different from reference plot</a></td></tr></table>
<!-- Author: claudiof ferretti -->


<!-- Date: Fri May 16 16:39:49.236 2003 -->
(MLE) b0cot10:Messenger:4:34:30 PM->Event 1071842: Bunchcounters in slot 5 (BC=127) and slot 11 (BC=9) disagree (L2B=0)
<!-- Author: Thorsten :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->
<!-- Comment: Fri May 16 20;44;14 comment by...Thorsten -->
(MLE) b0cot10:Messenger:8:43:05 PM->Event 4501901: Bunchcounters in slot 5 (BC=12) and slot 11 (BC=65) disagree (L2B=2) 
<!-- Comment: Fri May 16 20;57;07 comment by...Thorsten -->
(MLE) b0cot10:Messenger:8:55:57 PM->Event 4673410: Bunchcounters in slot 5 (BC=115) and slot 10 (BC=15) disagree (L2B=1) 
(MLE) b0cot10:Messenger:8:55:57 PM->Runtime Error 3, Event 4673410: Bunch counter mismatch, mismatch count = 1
<!-- Date: Fri May 16 16:59:53.349 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-5157&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-5157  width='300' align='top'></td><td>Losses since (roughly) start of store 2551. Observed a spike > 25 KHz.</a></td></tr></table>
<!-- Author: Steve -->


<!-- Date: Fri May 16 17:01:11.002 2003 -->
<pre>Called Keith about accelerator run plan.  They will drop this  
store at 8am and then have a 1hr access for D0.  </pre>
<!-- Author: Kirsten -->


<!-- Date: Fri May 16 17:04:41.450 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5158&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5158' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5159&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5159' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5160&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5160' width='300'></a></center>
<!-- Author: Steve -->
<!-- Comment: Fri May 16 17;06;26 comment by...steve -->
<pre>Hourly plots.
Note that there is a SIGNIFICANT spike around 16:44 in both 
B0ILUM and B0PBSM (off scale).</pre>


<!-- Comment: Fri May 16 17;18;58 comment by...steve -->
<pre>Note the *huge* spike in b0bpsm does not quite line
up with instantaneous luminosity spike. The latter
trails the former by about 1-2 minutes.</pre>
<!-- Date: Fri May 16 17:06:29.362 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-5161&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-5161  width='300' align='top'></td><td>checkcalib.sh report this plot as an error on CALQIE for the previous run dedicated to CALQIE calibration</a></td></tr></table>
<!-- Author: claudiof ferretti -->
<!-- Comment: Fri May 16 17;13;38 comment by...claudiof ferretti -->
Theother plots look fine.
<!-- Date: Fri May 16 17:08:58.825 2003 -->
<pre>Global alarms indicated trip in COT but NO INHIBIT 
and then it appeared to clear itself.  Indication of 
Inhibit was the red on COT in GLOBAL ALARMS and the 
HV bars going red.</pre>
<!-- Author: steve :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->


<!-- Date: Fri May 16 17:10:17.669 2003 -->
<pre>1 crate/s: b0cot17(65),  Done TO.[RXPT]b0cot17:Messenger:5:05:27 PM-&gt;Runtime Error 1, Event 1507214: Timeout on global TDC Done 
 
Sherpherd crate cot17 successfully</pre>
<!-- Author: Thorsten  :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->
<!-- Comment: Fri May 16 20;30;59 comment by...Thorsten -->
(MLE) b0cot17:Messenger:8:28:39 PM->Event 2803179: Bunchcounters in slot 3 (BC=106) and slot 19 (BC=18) disagree (L2B=0) 
<!-- Comment: Fri May 16 21;20;40 comment by...Thorsten -->
(MLE) b0cot17:Messenger:9:15:55 PM->Event 3430284: Bunchcounters in slot 3 (BC=139) and slot 18 (BC=53) disagree (L2B=0) 
<!-- Date: Fri May 16 17:16:12.930 2003 -->
<table border=0><td><tr><td bgcolor=#e0e030>If you receive errors with L3, go to START state in RC, STOP L3 PROXY using the Ace Control Panel and then START L3 PROXY using the Ace Control Panel. It should fix the problem. 
Always remember to check on the L3 display that L3 completed the transition  before clicking on RC buttons asking for other transitions. To tell the truth, I don't expect you'll have any problem.</td></tr></td></table>
<!-- Author: Alberto &Guillelmo & Nuno (&lt;-the lucky L3 pager for this week!) -->
<!-- Comment: Fri May 16 17;43;58 comment by...Alberto -->
<table border=0><td><tr><td bgcolor=#e0e030>I'm sorry, my previous entry may not be clear. I mean the Ace Controls for EVB and L3 Farm panel (the one almost always hidden below the trigger scaler monitor or the L3 display...) </td></tr></td></table>
<!-- Date: Fri May 16 17:17:35.096 2003 -->
(MLE) b0cot19:Messenger:5:16:40 PM->Event 1638422: Bunchcounters in slot 5 (BC=77) and slot 8 (BC=12) disagree (L2B=2) 
<!-- Author: Thorsten :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->
<!-- Comment: Fri May 16 17;49;38 comment by...Thorsten -->
<pre>(MLE) b0cot19:Messenger:5:48:03 PM-&gt;Event 2050400: Bunchcounters in slot 5 (BC=130) and slot 6 (BC=12) disagree (L2B=1) 
(MLE) b0cot19:Messenger:5:48:03 PM-&gt;Event 2050400: Bunchcounters in slot 5 (BC=130) and slot 7 (BC=12) disagree (L2B=1) 
(MLE) b0cot19:Messenger:5:48:03 PM-&gt;Event 2050400: Bunchcounters in slot 5 (BC=130) and slot 8 (BC=12) disagree (L2B=1) 
(MLE) b0cot19:Messenger:5:48:03 PM-&gt;Event 2050400: Bunchcounters in slot 5 (BC=130) and slot 9 (BC=12) disagree (L2B=1) 
(MLE) b0cot19:Messenger:5:48:03 PM-&gt;Event 2050400: Bunchcounters in slot 5 (BC=130) and slot 10 (BC=12) disagree (L2B=1) 
(MLE) b0cot19:Messenger:5:48:03 PM-&gt;Event 2050400: Bunchcounters in slot 5 (BC=130) and slot 11 (BC=12) disagree (L2B=1) 
(MLE) b0cot19:Messenger:5:48:03 PM-&gt;Event 2050400: Bunchcounters in slot 5 (BC=130) and slot 12 (BC=12) disagree (L2B=1) 
(MLE) b0cot19:Messenger:5:48:03 PM-&gt;Event 2050400: Bunchcounters in slot 5 (BC=130) and slot 13 (BC=12) disagree (L2B=1) 
(MLE) b0cot19:Messenger:5:48:03 PM-&gt;Event 2050400: Bunchcounters in slot 5 (BC=130) and slot 14 (BC=12) disagree (L2B=1) 
(MLE) b0cot19:Messenger:5:48:04 PM-&gt;Event 2050400: Bunchcounters in slot 5 (BC=130) and slot 15 (BC=12) disagree (L2B=1) 
(MLE) b0cot19:Messenger:5:48:04 PM-&gt;Event 2050400: Bunchcounters in slot 5 (BC=130) and slot 16 (BC=12) disagree (L2B=1) 
(MLE) b0cot19:Messenger:5:48:04 PM-&gt;Event 2050400: Bunchcounters in slot 5 (BC=130) and slot 17 (BC=12) disagree (L2B=1) 
(MLE) b0cot19:Messenger:5:48:04 PM-&gt;Event 2050400: Bunchcounters in slot 5 (BC=130) and slot 18 (BC=12) disagree (L2B=1) 
(MLE) b0cot19:Messenger:5:48:04 PM-&gt;Event 2050400: Bunchcounters in slot 5 (BC=130) and slot 19 (BC=12) disagree (L2B=1) 
(MLE) b0cot19:Messenger:5:48:04 PM-&gt;Event 2050400: Bunchcounters in slot 5 (BC=130) and slot 20 (BC=12) disagree (L2B=1) </pre>
<!-- Comment: Fri May 16 18;56;19 comment by...Thorsten -->
(MLE) b0cot19:Messenger:6:55:11 PM->Event 2969178: Bunchcounters in slot 5 (BC=65) and slot 8 (BC=109) disagree (L2B=1)
<!-- Comment: Fri May 16 19;00;14 comment by...Thorsten -->
<pre>(MLE) b0cot19:Messenger:6:57:34 PM-&gt;Event 3002247: Bunchcounters in slot 5 (BC=18) and slot 6 (BC=124) disagree (L2B=2) 
(MLE) b0cot19:Messenger:6:57:34 PM-&gt;Event 3002247: Bunchcounters in slot 5 (BC=18) and slot 7 (BC=124) disagree (L2B=2) 
(MLE) b0cot19:Messenger:6:57:34 PM-&gt;Event 3002247: Bunchcounters in slot 5 (BC=18) and slot 8 (BC=124) disagree (L2B=2) 
(MLE) b0cot19:Messenger:6:57:34 PM-&gt;Event 3002247: Bunchcounters in slot 5 (BC=18) and slot 9 (BC=124) disagree (L2B=2) 
(MLE) b0cot19:Messenger:6:57:34 PM-&gt;Event 3002247: Bunchcounters in slot 5 (BC=18) and slot 10 (BC=124) disagree (L2B=2)
 etc., etc. </pre>
<!-- Comment: Fri May 16 19;21;30 comment by...Thorsten -->
MLE) b0cot19:Messenger:7:20:31 PM->Event 3346629: Bunchcounters in slot 5 (BC=118) and slot 7 (BC=115) disagree (L2B=1) 
<!-- Comment: Fri May 16 19;32;17 comment by...Thorsten -->
<pre>MLE) b0cot19:Messenger:7:30:02 PM-&gt;Event 3481596: Bunchcounters in slot 5 (BC=86) and slot 6 (BC=27) disagree (L2B=0) 
(MLE) b0cot19:Messenger:7:30:02 PM-&gt;Event 3481596: Bunchcounters in slot 5 (BC=86) and slot 7 (BC=27) disagree (L2B=0) 
(MLE) b0cot19:Messenger:7:30:02 PM-&gt;Event 3481596: Bunchcounters in slot 5 (BC=86) and slot 8 (BC=27) disagree (L2B=0) 
(MLE) b0cot19:Messenger:7:30:02 PM-&gt;Event 3481596: Bunchcounters in slot 5 (BC=86) and slot 9 (BC=27) disagree (L2B=0)
etc., etc.</pre>
<!-- Comment: Fri May 16 19;33;38 comment by...Thorsten -->
(MLE) b0cot19:Messenger:7:32:36 PM->Event 3515520: Bunchcounters in slot 5 (BC=24) and slot 10 (BC=127) disagree (L2B=3) 
<!-- Comment: Fri May 16 20;35;08 comment by...Thorsten -->
(MLE) b0cot19:Messenger:8:34:06 PM->Event 4379748: Bunchcounters in slot 5 (BC=121) and slot 7 (BC=3) disagree (L2B=0)
<!-- Date: Fri May 16 17:21:51.853 2003 -->
(MLE) b0cot18:Messenger:5:20:26 PM->Event 1687125: Bunchcounters in slot 4 (BC=86) and slot 7 (BC=15) disagree (L2B=1) 
<!-- Author: Thorsten :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->



<!-- Date: Fri May 16 17:40:17.893 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-5162&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-5162  width='300' align='top'></td><td>YMon COT occupancy per cell plot #2 shows spikes not only for SL6 and SL7 ("normal" in last runs) but also for SL8.</a></td></tr></table>
<!-- Author: claudio ferretti -->
<!-- Comment: Fri May 16 17;46;27 comment by...Ch. Currat (owl CO) -->
<pre>Yep, don't worry it always happen especially at the beginning of a store. Some luminosity dependence. The occupancies baseline is a bit on the edge of the ref criteria. </pre>
<!-- Date: Fri May 16 17:42:59.725 2003 -->
<pre>        L3_REFORMAT_ERROR (Parsed Info):  
         RAWREF Error - VRB-DLINCO - with code 18 -- START 
         Error count: 1 of 1 
         Reject rate: 0.028 percent 
          (see Error Handler log file for full message) 
L2 Decision Timeout.[RXPT] 
(MLE) b0dap73.fnal.gov:Thread-30799:5:36:32 PM-&gt;  Level 2 Decision Timeout 
 
HRR recovered that</pre>
<!-- Author: Thorsten  :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->


<!-- Date: Fri May 16 17:43:50.192 2003 -->
<pre>(MLE) b0wcal06:Messenger:5:42:21 PM-&gt;Event 1978201: Bunchcounters in slot 16 (BC=56) and slot 8 (BC=0) disagree (L2B=1)  
L2B 16:  27  56 139   0 
L2B  8:  27   0   9 109 
EHx: 55C80017 54190017 54220017 543F0017</pre>
<!-- Author: Thorsten :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->
<!-- Comment: Fri May 16 19;53;00 comment by...Thorsten -->
<pre>MLE) b0wcal05:Messenger:7:50:15 PM-&gt;Event 3769772: Bunchcounters in slot 16 (BC=106) and slot 9 (BC=62) disagree (L2B=1) 
L2B 16:  12 106  65 130
L2B  9:  12  62  65 130
EHx: CE0B0016 CE1C0016 CE2A0016 CE370016</pre>
<!-- Comment: Fri May 16 19;58;32 comment by...Thorsten -->
<pre>(MLE) b0wcal02:Messenger:7:57:29 PM-&gt;Event 3871860: Bunchcounters in slot 5 (BC=71) and slot 19 (BC=18) disagree (L2B=0) 
L2B  5:  71 106 115   9
L2B 19:  18 106 115   9
EHx: 7E040013 7E130013 7FE20013 7E3F0013</pre>
<!-- Comment: Fri May 16 21;51;09 comment by...Thorsten -->
<pre>(MLE) b0wcal05:Messenger:9:49:31 PM-&gt;Event 5367304: Bunchcounters in slot 16 (BC=86) and slot 9 (BC=112) disagree (L2B=0) 
L2B 16:  86 127 115 136
L2B  9: 112  71 115 136
EHx: CE080016 CE170016 CE240016 CFF50016</pre>
<!-- Date: Fri May 16 18:03:04.780 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-5163&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-5163  width='300' align='top'></td><td>Hadron TDC timing dustributions clearly shows a shift.</a></td></tr></table>
<!-- Author: claudio ferretti -->
<!-- Comment: Fri May 16 23;03;21 comment by...Kirsten -->
The clock shifted by 2ns between stores this afternoon so we 
should see a shift in the TDC timing.

<!-- Date: Fri May 16 18:03:48.396 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5164&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5164' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5165&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5165' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5166&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5166' width='300'></a></center>
<!-- Author: steve -->
<!-- Comment: Fri May 16 18;04;12 comment by...steve -->
Hourly ACNET loss plots.


<!-- Date: Fri May 16 18:04:48.878 2003 -->
b0dap63.fnal.gov:ConsumerErrorRe:6:03:24 PM->Runtime Error 1, Event 3409, RunNum 163012: SvxMon Halt Recover Run: Stuck Cellid in 5 events in Silicon/S/B2/W7/L1/C0-2 .[MLE]  
<!-- Author: Thorsten :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->


<!-- Date: Fri May 16 18:07:51.037 2003 -->
<pre>CMP HV Trip. Bring it back up-appears to work. 
Note beam *DOES* appear to be a bit ragged...</pre>
<!-- Author: steve :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->
<!-- Comment: Fri May 16 18;11;12 comment by...Thorsten -->
run was paused during the trip recovery


<!-- Date: Fri May 16 18:25:16.027 2003 -->
(MLE) b0cot01:Messenger:6:23:35 PM->Event 2515354: Bunchcounters in slot 2 (BC=30) and slot 5 (BC=136) disagree (L2B=0) 
<!-- Author: Thorsten :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->


<!-- Date: Fri May 16 18:44:53.212 2003 -->
<pre>Got warning clcreceive missing, reset the crate 
  
b0clc00-&gt; tt receive 
288aec vxTaskEntry    +60 : ea1154 () 
ea1154 clcreceive     +654: TipcSrvMsgNext () 
1e3cb8 TipcSrvMsgNext +204: TipcConnMsgNext (e8e878) 
17005c TipcConnMsgNext+110: 16fa68 (e8e878) 
16fd94 TipcConnMsgWrite+1644: TipcConnCheck (e8e878, 0) 
16e010 TipcConnCheck  +8c : 16d810 (e8e878, 0) 
16dcf4 TipcConnKeepAlive+894: TipcLinkCheck (e82288, 0) 
17ff34 TipcLinkCheck  +d4 : TipcLinkSocketCheck (e82288, 0) 
1ee398 TipcLinkSocketCheck+a8 : TutSocketCheck (13, 0) 
15cea8 TutSocketCheck +264: select () 
25c35c select         +308: semTake () 
280bcc semTake        +50 : semBTake () 
value = 0 = 0x0 
</pre>
<!-- Author: Thorsten :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->
<!-- Comment: Fri May 16 20;07;38 comment by...Frank Chlebana -->
<pre>did you do 

b0clc00-&gt; tt receive 

before or after rebooting the crate?

You should first check to see if the receive process is suspended
using

b0clc00-&gt; i

then do 

b0clc00-&gt; tt receive

before rebooting the crate.
</pre>
<!-- Comment: Fri May 16 21;19;19 comment by...Thorsten -->
yes, that is the output before shepherding the crate.
And when I hit "i" the process showed up. But since the error message "CLC receive missing" persisted, I rebooted the crate.
 
<!-- Comment: Fri May 16 21;30;08 comment by...Thorsten -->
And it was not suspended. 
<!-- Comment: Fri May 16 22;18;32 comment by...Thorsten -->
<pre>The error log showed the following sequence:
(MLE) b0clc00:Messenger:9:11:15 PM-&gt;FER_E_MISC, Warning: restarted clcreceive task!
(MLE) b0dap74.fnal.gov:Thread-168:6:37:08 PM-&gt;No receive/clcreceive task on node b0clc00
...
(MLE) b0dap74.fnal.gov:Thread-168:6:40:09 PM-&gt;No receive/clcreceive task on node b0clc00
but the b0clc00-&gt;i showed:
  NAME        ENTRY       TID    PRI   STATUS      PC       SP     ERRNO  DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----

receive    clcreceive     e9fe80 200 PEND+T       280574   e9fa38      46  1633
</pre>
<!-- Comment: Fri May 16 23;20;37 comment by...Thorsten -->
(MLE) b0clc00:Messenger:7:06:42 PM->FER_E_MISC, Warning: restarted clcreceive task!
(MLE) b0dap74.fnal.gov:Thread-168:11:10:07 PM->No receive/clcreceive task on node b0clc00

together with CLC receive missing

again vxlogin shows:
receive    clcreceive     e80900 200 PEND+T       280574   e804b8      46  1513

but this time the error did not persist.
<!-- Comment: Fri May 16 23;28;28 comment by...Frank Chlebana -->
<pre>The front end checks for the clc receive process. If it does not
find it it will restart it.

I think that if you log into the clc crate and see that the 
receive process is there then the error indicating 

No receive/clcreceive task...

should go away...

If you are too quick you might be rebooting the clc crate
after the receive process has already been restarted.

Usually an indication of problems is when the receive process
is in in the suspended state.

If it is in PEND+T you might try waiting a bit to see if the 
complaint about No receive/clcreceive task  goes away.</pre>
<!-- Date: Fri May 16 18:51:27.853 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5167&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5167' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5168&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5168' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5169&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5169' width='300'></a></center>
<!-- Author: steve -->
<!-- Comment: Fri May 16 18;52;00 comment by...steve -->
Usual hourly ACNET plots




<!-- Date: Fri May 16 19:16:47.560 2003 -->
(MLE) b0cot02:Messenger:7:15:46 PM->Event 3278914: Bunchcounters in slot 2 (BC=118) and slot 13 (BC=86) disagree (L2B=2) 
<!-- Author: Thorsten :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->



<!-- Date: Fri May 16 19:25:31.336 2003 -->
<pre>(MLE) b0pcal05:Messenger:7:24:13 PM-&gt;Event 3399282: Bunchcounters in slot 15 (BC=56) and slot 17 (BC=9) disagree (L2B=1)  
L2B 15: 109  56 112 139 
L2B 17: 109   9 112 139</pre>
<!-- Author: Thorsten :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->
<!-- Comment: Fri May 16 21;22;35 comment by...Thorsten -->
<pre>(MLE) b0pcal07:Messenger:9:20:49 PM-&gt;Event 5000091: Bunchcounters in slot 15 (BC=6) and slot 19 (BC=3) disagree (L2B=2) 
L2B 15:   6  65   6 130
L2B 19:  53  65   3 130</pre>
<!-- Date: Fri May 16 19:31:06.237 2003 -->
CDFSGI1 tape became red in ProcMon, the Daq-help-pages suggest to wait for 4h and to check if the problem persists before taking actions. This problem was also reported in  one of the previous shifts.
<!-- Author: Thorsten :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->
<!-- Comment: Fri May 16 19;38;30 comment by...Thorsten -->
The process is fine now
<!-- Comment: Fri May 16 19;51;51 comment by...Thorsten -->
The process became red again
<!-- Comment: Fri May 16 20;09;45 comment by...Thorsten -->
It is happy now.
<!-- Comment: Fri May 16 21;28;58 comment by...Thorsten -->
It is red again
<!-- Comment: Fri May 16 21;39;40 comment by...Thorsten -->
It is happy again
<!-- Comment: Fri May 16 22;06;56 comment by...Thorsten -->
It is red again.
<!-- Comment: Fri May 16 22;22;55 comment by...Thorsten -->
It is green again
<!-- Date: Fri May 16 19:37:46.052 2003 -->
<pre>(MLE) b0ccal12:Messenger:7:33:59 PM-&gt;Event 3532799: Bunchcounters in slot 17 (BC=77) and slot 19 (BC=127) disagree (L2B=0)  
L2B 17:  77 139  12  62 
L2B 19: 127 139  12  62 
EHx: 7E0F000D 7E1E000D 7E28000D 7E36000D</pre>
<!-- Author: Thorsten :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->
<!-- Comment: Fri May 16 22;38;20 comment by...Thorsten -->
<pre>(MLE) b0ccal01:Messenger:10:36:35 PM-&gt;Event 5957560: Bunchcounters in slot 16 (BC=136) and slot 19 (BC=124) disagree (L2B=2) 
L2B 16: 124  71 136 109
L2B 19: 124  71 124 109
EHx: 54070002 54160002 54280002 543C0002</pre>
<!-- Comment: Fri May 16 23;47;46 comment by...Thorsten -->
<pre>(MLE) b0ccal01:Messenger:11:46:49 PM-&gt;Event 6814408: Bunchcounters in slot 16 (BC=80) and slot 18 (BC=3) disagree (L2B=2) 
L2B 16:  86 124  80 118
L2B 18:  86 124   3 118
EHx: 54070002 54150002 54280002 54370002</pre>
<!-- Date: Fri May 16 19:57:31.549 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5170&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5170' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5171&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5171' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5172&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5172' width='300'></a></center>
<!-- Author: steve -->
<!-- Comment: Fri May 16 19;59;29 comment by...steve -->
Usual hourly ACNET loss plots

<!-- Date: Fri May 16 19:58:58.524 2003 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2003&action=view&page=-5173&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2003&action=view&page=-5173  width='300' align='top'></td><td>lostp,lostpbar for our shift. Note the increasing rate of spikes and general broadening of losses</a></td></tr></table>
<!-- Author: steve -->





<!-- Date: Fri May 16 20:14:05.253 2003 -->
<pre>MLE) b0l3gate1.fnal.gov:main:8:10:51 PM-&gt;Host b0eb11.fnal.gov, task tRec_0 
SCPU-P0-E-VrbHeader: Dump of header words for event 4060784 from VRB in slot 12: 
0x00000020 0x9600007e 0x53564444 0x00080003 0x00000000 0x00000003 0x00000003 0x00000000 
b0l3gate1.fnal.gov:main:8:10:51 PM-&gt;Host b0eb11.fnal.gov, task tRec_0 
SCPU-P0-E-BadVrbByteCount: Event 4060784 total byte count is lt 32, gt 65528, or not divisible by 8 for VRB in slot 12.</pre>
<!-- Author: Thorsten :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->
<!-- Comment: Fri May 16 20;32;20 comment by...Thorsten -->
<pre>MLE) b0l3gate1.fnal.gov:main:8:28:56 PM-&gt;Host b0eb11.fnal.gov, task tRec_0
SCPU-P0-E-VrbHeader: Dump of header words for event 4310621 from VRB in slot 10:
0x00000000 0x000012e0 0x002100e1 0x02140220 0x00e000e8 0x018401e8 0x00f40238 0x06200000
b0l3gate1.fnal.gov:main:8:28:56 PM-&gt;Host b0eb11.fnal.gov, task tRec_0
SCPU-P0-E-BadVrbByteCount: Event 4310621 total byte count is lt 32, gt 65528, or not divisible by 8 for VRB in slot 12.</pre>



<!-- Comment: Fri May 16 20;45;56 comment by...Thorsten -->
<pre>(MLE) b0l3gate1.fnal.gov:main:8:44:24 PM-&gt;Host b0eb11.fnal.gov, task tRec_0
SCPU-P0-E-VrbHeader: Dump of header words for event 4516741 from VRB in slot 10:
0x00000000 0x00001448 0x003600f6 0x023c0224 0x00f400e0 0x017801f8 0x01380238 0x07080000
b0l3gate1.fnal.gov:main:8:44:24 PM-&gt;Host b0eb11.fnal.gov, task tRec_0
SCPU-P0-E-BadVrbByteCount: Event 4516741 total byte count is lt 32, gt 65528, or not divisible by 8 for VRB in slot 12.</pre>



<!-- Comment: Fri May 16 21;07;19 comment by...Thorsten -->
<pre>SCPU-P0-E-VrbHeader: Dump of header words for event 4809226 from VRB in slot 10:
0x00000000 0x000012a0 0x00060086 0x021401bc 0x01600104 0x011c01d4 0x01ac0234 0x05600000
(MLE) b0l3gate1.fnal.gov:main:9:06:08 PM-&gt;Host b0eb11.fnal.gov, task tRec_0
SCPU-P0-E-VrbHeader: Dump of header words for event 4809226 from VRB in slot 12:
0x00000000 0xaaaaaaaa 0x55555555 0x541a007c 0x80000000 0x0000004d 0x541a007d 0x53565444
b0l3gate1.fnal.gov:main:9:06:08 PM-&gt;Host b0eb11.fnal.gov, task tRec_0
SCPU-P0-E-BadVrbByteCount: Event 4809226 total byte count is lt 32, gt 65528, or not divisible by 8 for VRB in slot 12.</pre>
<!-- Comment: Fri May 16 22;05;59 comment by...Thorsten -->
<pre>MLE) b0l3gate1.fnal.gov:main:10:04:43 PM-&gt;Host b0eb11.fnal.gov, task tRec_0
SCPU-P0-E-VrbHeader: Dump of header words for event 5557639 from VRB in slot 10:
0x00000000 0x00001328 0x003e007e 0x020802ac 0x00e000e4 0x013001e4 0x018801f4 0x05ec0000
b0l3gate1.fnal.gov:main:10:04:43 PM-&gt;Host b0eb11.fnal.gov, task tRec_0
SCPU-P0-E-BadVrbByteCount: Event 5557639 total byte count is lt 32, gt 65528, or not divisible by 8 for VRB in slot 12.</pre>
<!-- Comment: Fri May 16 23;22;50 comment by...Thorsten -->
<pre>(MLE) b0l3gate1.fnal.gov:main:11:18:08 PM-&gt;Host b0eb11.fnal.gov, task tRec_0
SCPU-P0-E-VrbHeader: Dump of header words for event 6462223 from VRB in slot 10:
0x00000000 0x00001388 0x00150055 0x022c0238 0x00f800e0 0x01540208 0x01200268 0x06400000
(MLE) b0l3gate1.fnal.gov:main:11:18:08 PM-&gt;Host b0eb11.fnal.gov, task tRec_0
SCPU-P0-E-VrbHeader: Dump of header words for event 6462223 from VRB in slot 12:
0x000000dd 0x00011940 0x0001b3b4 0x0000de30 0x00003534 0x00000074 0x00000000 0x00000000

see errorlog</pre>
<!-- Date: Fri May 16 20:58:23.229 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5174&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5174' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5175&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5175' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5176&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5176' width='300'></a></center>
<!-- Author: steve -->
<!-- Comment: Fri May 16 20;58;47 comment by...steve -->
Usual hourly ACNET loss plots.



<!-- Date: Fri May 16 21:18:02.661 2003 -->
<pre>LostP getting very spikey with some over 30kHz.  I called MCR  
and they are looking into it.</pre>
<!-- Author: Kirsten -->





<!-- Date: Fri May 16 21:27:55.926 2003 -->
<pre>TOF alarm on Global Alarm and High Voltage Summary pages 
indicating trip.  Inhibit induced but it was all short-lived 
and cleared of its own accord within 1 minute.</pre>
<!-- Author: steve :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->





<!-- Date: Fri May 16 21:46:38.815 2003 -->
<pre>Max from MCR called - they retuned and the LOSTP's look  
much better now.  The baseline has dropped below 10 kHz  
and the spikes are only to 13kHz.</pre>
<!-- Author: Kirsten -->



<!-- Date: Fri May 16 21:51:30.726 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5177&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5177' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5178&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5178' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5179&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5179' width='300'></a></center>
<!-- Author: steve -->
<!-- Comment: Fri May 16 21;52;05 comment by...steve -->
<pre>Usual hourly ACNET loss plots.
Look what a phone call from the Sci-co can do!</pre>




<!-- Date: Fri May 16 22:15:46.947 2003 -->
<pre>Node unreachable error on Global Alarms page:PISAbox systems 
(CEM,CHA,WHA,GAM) 
Cleared itself on next update. (2' later)</pre>
<!-- Author: steve :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->





<!-- Date: Fri May 16 22:57:13.146 2003 -->
<pre>2nd TOF Temporary inhibit tonight from a "trip". As previous, 
it recovered by itself. Beam spiked a bit but not as high as 
it was for most of evening.</pre>
<!-- Author: steve :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->






<!-- Date: Fri May 16 23:33:24.319 2003 -->
<pre>3RD TIME tonight, a TOF "Trip". Alarm registered as Red 
on GLOBAL ALARMS page, Red bars on High Voltag summary and 
a Trigger Inhibit (at Trigger Inhibit Status page and at 
LED above DAQ Ace consoles). Again, it cleared itself within 
30 seconds.</pre>
<!-- Author: Steve :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->


<!-- Date: Fri May 16 23:45:06.674 2003 -->
<center><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5180&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5180' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5181&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5181' width='300'></a><a href='/cgi/elog/elog.pl?nb=2003&action=view&page=-5182&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2003&action=view&page=-5182' width='300'></a></center>
<!-- Author: steve -->
<!-- Comment: Fri May 16 23;45;26 comment by...steve -->
Usual ACNET loss plots for past 2 hours


<!-- Date: Fri May 16 23:46:14.504 2003 -->
<TABLE bgcolor=#cdc9a5> 
<TR><TH>Date</th><th>Time</th><th>BLM</th><th colspan=2>Dose</th></tr> 
<tr><td>2003.05.16</td><td>23:45:16</td><td>W Inner BLM</td><td>358.84</td><td>RADS</td></tr> 
<tr><td>2003.05.16</td><td>23:45:16</td><td>W Outer BLM</td><td>0.00</td><td>RADS</td></tr> 
<tr><td>2003.05.16</td><td>23:45:16</td><td>E Inner BLM</td><td>5.00</td><td>RADS</td></tr> 
<tr><td>2003.05.16</td><td>23:45:16</td><td>E Outer BLM</td><td>358.84</td><td>RADS</td></tr> 
</TABLE>  
<!-- Author: steve -->



<!-- Date: Fri May 16 23:52:23.722 2003 -->
<pre>Had trip of L00 HV.  Mon Ace reset but tripped again.  
Have paged expert. 
 
</pre>
<!-- Author: Kirsten -->

<!-- Date: Fri May 16 23:56:38.192 2003 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012>
163012</A> 
x27CC4</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_04 [9,288,373]</TD>                                       
<TD ALIGN=CENTER>15:21:40</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>07:56:00</TD>                                          
<TD ALIGN=RIGHT>    421,032,475</TD>                                    
<TD ALIGN=RIGHT>     6,810,739</TD>                                     
<TD ALIGN=RIGHT>   1,877,457</TD>                                       
<TD ALIGN=RIGHT>  826.028</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>23:55:02</TD>                                                       
<TD ALIGN=CENTER>07:56:00</TD>                                          
<TD ALIGN=RIGHT>     421,032,475</TD>                                   
<TD ALIGN=RIGHT>      6,810,739</TD>                                    
<TD ALIGN=RIGHT>     1,877,457</TD>                                     
<TD ALIGN=RIGHT>  826.028</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Fri May 16 23:59:15.198 2003 -->
Layer 00 Silicon Trip and Inhibit. 
L00 B1W3 Layer3 HIgh DVDD Current Software Protection trip. 
Went through procedure of resetting trips and attempting 
to turn it back on as instructed (from halt state). 
Power appeared to come back up but inhibit remained. 
Then trip again.  
consulting expert(Gino) by phone/page.
<!-- Author: steve :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012' target=_top>163012</a>) -->


<!-- Date: Sat May 17 00:03:35.171 2003 -->
<font color=red>Shift Summary: </font><pre>Highlights 
---------- 
16:00 Inherited store 2551 with lum at 35e30 and stacking rate  
at 11 mA/h. 
 
Expect to drop store at 8am, one hour access for D0 and then  
shot set-up. 
 
Stores 
------ 
Store #2551 
 
Runs 
---- 
Run #163012 
 
Problems 
-------- 
 
First half of shift the LOSTP's were very spikey and 
occassionally going over 25kHz.  Called MCR, they  
retuned and losses dropped.  Keep an eye on the spikes  
and the SVX DANGER in ProcMon. 
 
Several bunchcounter mismatch errors coming from  
several crates and slots -b0cot10, b0cot17, b0cot18,  
b0cot19, b0cot02, b0wcal06, b0wcal05, b0wcal02, b0pcal05,  
b0pcal07, b0ccal12, b0ccal01 
  
CDFSGI1 tape goes red for several minutes in ProcMon but 
always clears.  DAQ help pages say don't page unless  
problem for 4 hours.  This problem was seen on previous shift  
but also always cleared itself. 
 
TOF HV has tripped and caused a trigger inhibit 3 times tonight,  
the trip clears itself and never lasts for more than a second. 
 
     delivered    809.13 nb-1    
     acquired     744.79 nb-1    
     efficiency   92.05 %     </pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   163012
Delivered Luminosity   809  
Acquired Luminosity    744  
Efficiency             92
   
</td></tr></table></pre>
<!-- Author: Kirsten -->

<!-- Date: Sat May 17 00:07:39.737 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012">163012</A>
 Terminated at 2003.05.17 00:07:32
<!-- Author: RunControl -->



<!-- Date: Sat May 17 00:07:55.473 2003 -->
<TABLE bgcolor=#cdc9a5> 
<tr><th colspan=2>Totals</th></tr> 
<tr><td>Date:</td><td>2003.05.16</td> 
<tr><td>Shift:</td><td>eve</td> 
<tr><td>Delivered luminosity:</td><td>809.13 nb<sup>-1</sup></td></tr> 
<tr><td>Acquired luminosity:</td><td>744.79 nb<sup>-1</sup></td></tr> 
<tr><td>Efficiency:</td><td>92.05</td></tr> 
</TABLE>  
<!-- Author: steve -->

<!-- Date: Sat May 17 00:09:05.997 2003 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163012">163012</A>
TERMINATE: Aborting run due to L00 problems.
<!-- Author: ambra/maria 2080 -->