<!-- Title: Day shift. Sat Aug 13, 2005 -->
<!-- SciCo: Soo-Bong Kim -->
<!-- DAQAce: Greg Williams -->
<!-- MonAce: Sasha Baroiant -->
<!-- CO: Ben Brau -->
<!-- OpManager: SungWon Lee -->
<!-- Notes: -->
<pre>Store 4330 Colliding; L=65e30</pre>


<!-- Date: Sat Aug 13 08:16:54.324 2005 -->
There is a Plug HV problem, the monitoring ace went upstairs to look at the plug HV pc.  It looks like 1 plug wedge & crate are flashing red on the iFix display:  Crate 10 & East Wedge 18.  Halted Run Control.
<!-- Author: Greg -->
<!-- Comment: Sat Aug 13 08;44;14 comment by...sasha -->
<pre>Following crate recovery procedure on plug HV problem troubleshooting page.
Check Passed LED is on for crate 10. Brought HV to standby and back to on with iFix, but HV status was still red. Then brought to standby, to off, then back on, with similar results.
Pressed Crate Reset button, Check-Passed LED came back on, so power cycled crate. After bringing HV back up, HV status was OK.</pre>
<!-- Date: Sat Aug 13 08:21:13.042 2005 -->
<pre>SRC Fatal Error!  Silicon Resonance Detected. 
Details from ErrLog: 
 
CT: 2005.08.13 08:07:36 
7'36" b0svx02:Messenger:Aug 13, 2005 8:07:37 AM->SRC Fatal Error:Sl 6 Resonance Detected  
 -->   
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error from b0svx02: Sl 6 Resonance Detected  
  
 - HALT the run                    
 - record any relevant information in e-log      
 - RECOVER - RUN      
[MLE] 
(RC)   8:07:37 Halt -> HALTED 
CT: 2005.08.13 08:07:38 
7'38" 1 crate/s: b0svx02(112),  in error. [RXPT] 
(MLW) b0dap73.fnal.gov:Thread-28:Aug 13, 2005 8:11:02 AM->Inhibit flag SET  
(MLW) b0dap73.fnal.gov:Thread-28:Aug 13, 2005 8:11:06 AM->Inhibit flag CLEAR  
 
--> There does not appear to be any byte counts</pre>
<!-- Author: Greg :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202735' target=_top>202735</a>) -->


<!-- Date: Sat Aug 13 08:41:40.682 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14210&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14210' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14211&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14211' width='300'></a></center>
<!-- Author: Steve Hahn -->
<!-- Comment: Sat Aug 13 08;56;43 comment by...Steve Hahn -->
<p>At 0626 on the owl shift, the CEM NW Gamma HV supply died.
Marjorie called me, and I walked Teresa through power cycling
the supply without success.  I came in.  Checked the fuse (yes,
Dervin, I used a DVM), it was fine.  Swapped out the Gamma--old,
FNAL# 52516; new, FNAL# 52522.  Then I used the Gamma supply
monitoring (that is, the Gamma iFIX picture) to set the HV;
bad idea, the voltage was high about 20 V.  This is expected--it's known that the voltage divider that generates the 
monitoring voltage on the back of the supply is only approximate.   At this point, the HV difference from the set 
value looked like the first of the above plots.  Strangely, the
distribution has two peaks: the one nearer 18 V high is for 
wedges 00W-05W, and the one nearer 15 V high is for wedges 18W-23W.  I lowered the voltage by 17.0 V, at which point I plotted
the second plot above.  Without any action on my part, the two
peaks had merged.  At this point, I lowered the voltage by 
another 1.5 V; right now it's at 1255.5 V.  I'll probably have
to tweak a few individual PMTs, but it looks pretty good.</p>

<p>I suggested to Soo-Bong that they mark the previous run
bad for CCAL, since most of that short run the voltage was
high by about 20 V.  I had already told Marjorie to mark the 
previous run bad, while the Gamma was dead.</p>
<!-- Date: Sat Aug 13 08:43:48.619 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202735">202735</a>
 Terminated at 2005.08.13 08:43:13
<!-- Author: RunControl -->


<!-- Date: Sat Aug 13 08:43:49.426 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202735">202735</a>
TERMINATE: End Run with bad PMT voltages
<!-- Author: Greg x2080 -->



<!-- Date: Sat Aug 13 08:46:42.886 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202736">202736</a>
 Activated at 2005.08.13 08:45:57
<!-- Author: RunControl -->


<!-- Date: Sat Aug 13 08:46:43.883 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202736">202736</a>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_06 [1,683,585] 
<!-- Author: Greg x2080 -->


<!-- Date: Sat Aug 13 08:55:19.631 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202735">202735</a>
RUNSTATUS: <PRE>Marked Bad, explanation:
CCAL - incorrect HV values
PCAL - HV tripped
</pre>
<!-- Author: cdfscico -->



<!-- Date: Sat Aug 13 09:01:55.944 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202736">202736</a>
 Terminated at 2005.08.13 09:01:30
<!-- Author: RunControl -->


<!-- Date: Sat Aug 13 09:01:57.952 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202736">202736</a>
TERMINATE: EVB needs to be cleaned up
<!-- Author: Greg x2080 -->



<!-- Date: Sat Aug 13 09:05:12.303 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14212&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14212' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14213&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14213' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14214&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14214' width='300'></a></center>
<!-- Author: sasha -->

<!-- Date: Sat Aug 13 09:10:21.260 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202737">202737</a>
 Activated at 2005.08.13 09:09:28
<!-- Author: RunControl -->


<!-- Date: Sat Aug 13 09:10:22.732 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202737">202737</a>
 Terminated at 2005.08.13 09:09:58
<!-- Author: RunControl -->


<!-- Date: Sat Aug 13 09:14:26.164 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202737">202737</a>
TERMINATE: EVB Problem still
<!-- Author: Greg x2080 -->


<!-- Date: Sat Aug 13 09:14:27.189 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202737">202737</a>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_06 [1,683,585] Cleaned up EVB
<!-- Author: Greg x2080 -->


<!-- Date: Sat Aug 13 09:19:55.780 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202734">202734</a>
cal: <tt>CEM Power Supply failure</tt>
<!-- Author: Consumer Operator(Ben Brau) -->


<!-- Date: Sat Aug 13 09:20:58.229 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202738">202738</a>
 Activated at 2005.08.13 09:20:41
<!-- Author: RunControl -->


<!-- Date: Sat Aug 13 09:20:59.312 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202738">202738</a>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_06 [1,683,585] 
<!-- Author: Greg x2080 -->



<!-- Date: Sat Aug 13 09:32:15.461 2005 -->
<pre>Event Builder Problems: 
1- Event Builder Error!, problem with b0eb13 
2- Busy Timeout, b0eb13 
3- Scanner Manager Error!, recommends HRR; then cleanup EVB if it fails 
 
We tried HRR -> it failed.   
We ended the run, reset, Cleaned up EVB, and did Partition-Config-Activate.  After a few seconds in the active state, there were at least 10 popup error messages, mostly:  
- Event Builder Error, problem with b0eb13  
- Busy Timeouts - b0eb13 
- Event Builder Error, SCPU_TRACER_EVENT_ID  
 
I ended the Run (202737), reset run control, cleaned up event builder again, checked L3, checked the SMs & SCPUs, and everything looked good.  We did partition-Config-Activate again, and this time it worked, with no error messages. 
 
</pre>
<!-- Author: Greg :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202738' target=_top>202738</a>) -->
<!-- Comment: Sat Aug 13 10;04;14 comment by...Greg -->
<pre>Markus (EVB expert) called and pointed out that the failure after cleaning up EVB was due to some bad words forwarded to EVB from b0cotO6.  Its not clear what solved the problem, but cleaning up EVB probably did not help in this situation.  Some details from the error log of 202737:

 Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data quality in 
 SCPU b0eb13 (forwarded by FER crate COT_06). 
 Datataking is paused: events are not being processed. 
 - HRR should help. 
[MLE]
9'44" b0l3pcom1.fnal.gov:main:Aug 13, 2005 9:09:44 AM->Host b0eb13.fnal.gov, task tRec_0
SCPU-P0-E-TracerEventId: Event 2, crate 55, channel 6 has either bad Tracer ID or bad markers around Tracer word.
 Tracer word event ID is 2, should be 2.
Tracer word = 0x00120437 (I think).
Channel data = 0x00000202 0x00120037 0x434f5044 0x00140007 0x00000002 0x000001e1 0x00000003 0x00000010 ...
... 0x0002e5a8 0x04472e22 0x00000000 0x00000000 0xaaaaaeaa 0x55555555 0x00120437 0x80000400 -->  </pre>
<!-- Date: Sat Aug 13 09:36:42.553 2005 -->
<p>I've lowered the HV on the CEM NW Gamma another 1.5 V to 
1254.0 V, and restarted the PMT readout.</p>
<!-- Author: Steve Hahn -->


<!-- Date: Sat Aug 13 09:50:48.734 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-14215&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-14215  width='300' align='top'></a></td><td>ISL B3W0l$ drawing 106mA, at 100V. Barely out of safe range is up to 105 and caution is up to 105.5. 
Asked DAQ ACE to do HRR, still at 106. Cell is now yellow.</td></tr></table>
<!-- Author: sasha -->
<!-- Comment: Sat Aug 13 09;54;30 comment by...sasha -->
That's B3W0l4, , and it has gone pink again, still @ 106mA. Will page SI.


<!-- Date: Sat Aug 13 09:57:00.515 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-14216&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-14216  width='300' align='top'></a></td><td>Still drawing around 106mA.</td></tr></table>
<!-- Author: sasha -->
<!-- Comment: Sat Aug 13 10;19;23 comment by...sasha -->
Oh yeah, not 106mA but 106 muA for microamps, fortunately we're not drawing miliAmps from a single ladder! 
<!-- Comment: Sat Aug 13 12;20;49 comment by...Ankush & Ulrich -->
We bump up the limit to 110 uA, and the ladder is green again.
<!-- Date: Sat Aug 13 10:05:37.535 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202735">202735</a>
general: <tt>Low statistics run.</tt>
<!-- Author: Consumer Operator(Ben Brau) -->



<!-- Date: Sat Aug 13 10:09:49.631 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14217&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14217' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14218&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14218' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14219&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14219' width='300'></a></center>
<!-- Author: sasha -->
<!-- Comment: Sat Aug 13 11;33;10 comment by...sasha -->
0830 to 0930
<!-- Date: Sat Aug 13 10:19:00.306 2005 -->
Flying wire scan at 10:11.
<!-- Author: Greg :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202738' target=_top>202738</a>) -->



<!-- Date: Sat Aug 13 10:21:57.098 2005 -->
<pre>Silicon Resonance SRC Fatal Error & RXPT Error 
HRR successful, details from error log: 
 
(MLE) b0dap73.fnal.gov:Thread-28:Aug 13, 2005 10:17:55 AM->FrontEnd Crate Error Condition from: 
VRB_SVX_02 
CT: 2005.08.13 10:17:59 
17'59" 1 crate/s: b0svx02(112),  in error. [RXPT] 
CT: 2005.08.13 10:18:01 
18'1" b0svx02:Messenger:Aug 13, 2005 10:18:01 AM->SRC Fatal Error:Sl 6 Resonance Detected  
 -->   
 Additional Information:  
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
 SRC Fatal Error from b0svx02: Sl 6 Resonance Detected   
 - HALT the run                    
 - record any relevant information in e-log      
 - RECOVER - RUN      
[MLE] 
(RC)   10:18:03 Halt -> HALTED 
CT: 2005.08.13 10:18:03 
18'3" 1 crate/s: b0svx02(112),  in error. [RXPT] 
(RC)   10:18:10 Recover -> RECOVERED 
(RC)   10:18:12 Run -> ACTIVE</pre>
<!-- Author: Greg :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202738' target=_top>202738</a>) -->


<!-- Date: Sat Aug 13 10:33:22.450 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-14220&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-14220  width='300' align='top'></a></td><td>L2 Warning</td></tr></table>
<!-- Author: Ben Brau -->
<!-- Comment: Sat Aug 13 10;35;55 comment by...Ben Brau -->
XMon is complaining about unprescaled triggers with 0 rate.  We looked at the history and it looks like in past runs, the rates for these triggers is 0 when lumi is above 40-45.  Should XMon really complain if the luminosity is too high for these triggers to fire?
<!-- Comment: Sat Aug 13 11;50;04 comment by...Ted -->
This L2 trigger is driven by L1_RP_&_JET5_PS200_LUMI_0_40 at L1, so it should be only enabled below luminosity of 40E30.  XMON should not complain if luminosity is above 40E30. ask Charles about this.
<!-- Comment: Sat Aug 13 16;22;35 comment by...Charles Plager -->
Charles has been asking Ted et al. to help Charles teach XMon WHICH triggers are luminosity enabled (since they won't name triggers consistently).  So, ask Ted about this.
<!-- Comment: Sat Aug 13 16;42;37 comment by...Ted -->
<pre>Just got an email from Charles, asking me to comment on his comment on my comment on Ben's good comment.   nice try, Charles. this one is even better than your "Real pizza was invented in US" comment... 

The trigger name "inconsistence" is not Ted's invention...  I am no expert in that area (and know even less now, as I learn more).  All the information about a given trigger (its dad/mom and sons/daughters) is in the database. not just the names (all the names are already so long).  My naive understanding is that XMON should/could get the information from database, not from the names... but what do I know about database? I can only understand our "write only" database...:)   cheers.</pre>
<!-- Comment: Sat Aug 13 17;00;11 comment by...Ted -->
I meant the write-only aspect of the database.   I never know how to read from database, though I do write into it once in a while.
<!-- Date: Sat Aug 13 10:34:54.435 2005 -->
<pre>SRC Fatal Error: Silicon Resonance, Details from error log: 
 
(MLE) b0xft02:Messenger:Aug 13, 2005 10:31:05 AM->Event 1045636: Bunchcounters in slot 13 (BC=80) and slot 5 (BC=106) disagree (L2B=1)  
(MLE) b0xft02:Messenger:Aug 13, 2005 10:31:05 AM->Runtime Error 3, Event 1045636: Bunch counter mismatch, mismatch count = 1</pre>
<!-- Author: Greg :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202738' target=_top>202738</a>) -->



<!-- Date: Sat Aug 13 10:39:55.627 2005 -->
<pre>
Another Silicon Resonance SRC Fatal Error.  HRR successful. 
Details: 
[MLE] 
CT: 2005.08.13 10:36:44 
36'44" 1 crate/s: b0svx02(112),  in error. [RXPT] 
(RC)   10:36:45 Halt -> HALTED 
(RC)   10:36:52 Recover -> RECOVERED 
(RC)   10:36:54 Run -> ACTIVE 
(MLE) b0xft00:Messenger:Aug 13, 2005 10:37:28 AM->Event 1167392: Bunchcounters in slot 13 (BC=0) and slot 11 (BC=15) disagree (L2B=1)  
(MLE) b0xft00:Messenger:Aug 13, 2005 10:37:28 AM->Runtime Error 2, Event 1167392: Bunch counter mismatch, mismatch count = 1 </pre>
<!-- Author: Greg :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202738' target=_top>202738</a>) -->


<!-- Date: Sat Aug 13 11:02:00.277 2005 -->
Busy Timeout b0svx06, and a popup box indicates that this may represent a problem with the EVB system. 
 
HRR attempted, but b0tsi00, b0tsi02, and evb all got stuck going to the halt state.
<!-- Author: Greg :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202738' target=_top>202738</a>) -->

<!-- Date: Sat Aug 13 11:04:08.629 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202738">202738</A>
 Terminated at 2005.08.13 11:03:39
<!-- Author: RunControl -->


<!-- Date: Sat Aug 13 11:06:00.006 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202738">202738</A>
TERMINATE: End run to clean up EVB
<!-- Author: Greg x2080 -->


<!-- Date: Sat Aug 13 11:15:10.869 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202739">202739</A>
 Activated at 2005.08.13 11:14:28
<!-- Author: RunControl -->


<!-- Date: Sat Aug 13 11:15:11.262 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202739">202739</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_06 [1,683,585] 
<!-- Author: Greg x2080 -->



<!-- Date: Sat Aug 13 11:16:02.594 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14221&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14221' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14222&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14222' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14223&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14223' width='300'></a></center>
<!-- Author: sasha -->
<!-- Comment: Sat Aug 13 11;35;09 comment by...sasha -->
0930 to 1030
<!-- Date: Sat Aug 13 11:20:53.407 2005 -->
<pre>After EVB failed to go to the Halt transition I could not Recover or End the run, so I did a Manual Overide twice to be able to end the run.  
 
The transition to "End" got stuck again, and immediately we got a Busy Timeout for b0tsi03 and a TDC_NOT_READY readout error.  I Reset run control to get everything to the start state and tryed to clean up EVB because we terminated the run in such an abrupt way.  A message appeared as the process ran that b0eb10 could not be logged into and needed to be reset manaully.   
 
Sasha ran upstairs to reset it, and when he returned I cleaned up EVB again and started a new run successfully. 
</pre>
<!-- Author: Greg :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202739' target=_top>202739</a>) -->

<!-- Date: Sat Aug 13 11:23:15.461 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202736">202736</A>
smx: <tt>CP2PadCanvas and CCRPadCanvas plots were missing from the &nbsp;web page.</tt>
<!-- Author: Consumer Operator(Ben Brau) -->





<!-- Date: Sat Aug 13 11:38:54.787 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14224&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14224' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14225&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14225' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14226&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14226' width='300'></a></center>
<!-- Author: sasha -->
<!-- Comment: Sat Aug 13 11;39;28 comment by...sasha -->
Ace plots: 1030 to 1130


<!-- Date: Sat Aug 13 11:42:21.799 2005 -->
Done Timeout b0clc00; HRR successful
<!-- Author: Greg :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202739' target=_top>202739</a>) -->


<!-- Date: Sat Aug 13 12:09:49.787 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202738">202738</A>
smx: <tt>The CP2PadCanvas and CCRPadCanvas plots are missing on the YMon web page for this run, and they also seem to be missing from the reference page. &nbsp;I checked them with the Consumer Display using the root file for this run.</tt>
<!-- Author: Consumer Operator(Ben Brau) -->



<!-- Date: Sat Aug 13 12:11:20.704 2005 -->
Collider flying wire scan
<!-- Author: Greg :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202739' target=_top>202739</a>) -->
<!-- Comment: Sat Aug 13 13;11;12 comment by...Greg -->
again
<!-- Date: Sat Aug 13 12:39:36.003 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14227&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14227' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14228&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14228' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14229&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14229' width='300'></a></center>
<!-- Author: sasha -->
<!-- Comment: Sat Aug 13 12;40;41 comment by...sasha -->
Hourly Plots: 11:30 to 12:30.


<!-- Date: Sat Aug 13 12:42:27.282 2005 -->
ProcMon RTServer Error => I ran a check and the error cleared.
<!-- Author: Greg :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202739' target=_top>202739</a>) -->


<!-- Date: Sat Aug 13 12:50:38.389 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-14230&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-14230  width='300' align='top'></a></td><td>L1 DCAS-ADMEM</td></tr></table>
<!-- Author: Ben Brau -->
<!-- Comment: Sat Aug 13 12;51;39 comment by...Ben Brau -->
There are 160+ entries in the upper right plot.  Instructions say to put plot in elog and page expert if there are more than a few entries.  We're paging...

<!-- Comment: Sat Aug 13 12;52;48 comment by...Ben Brau -->
That should be "upper left plot" - the errors plot.


<!-- Comment: Sat Aug 13 13;16;28 comment by...carla -->
<pre>The error in trigger tower eta = 18, phi = 21 is due to a difference of 1 count (.125 GeV) between the energy seen by the trigger wih respect the the value expected from the CALD bank. This results in a slight trigger over-efficency, but not enough to cause any problem in trigger rate. It will probably disappear with the next calibration/download of the Admem constants.

The band at eta = 13 is due to a single event.</pre>
<!-- Date: Sat Aug 13 12:58:46.533 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-14231&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-14231  width='300' align='top'></a></td><td>DCAS Occupancy</td></tr></table>
<!-- Author: Ben Brau -->
<!-- Comment: Sat Aug 13 13;00;00 comment by...Ben Brau -->
Error rate is > 0.001/bin in lower left plot for the same bin that had 160 entries in the admem plot.


<!-- Date: Sat Aug 13 13:01:33.802 2005 -->
Kwangzoo thinks that the TRIGMON problem with DCAS is unlikely due to failure of ADMEM, but could be related to L1 trigger. 
<!-- Author: Soo-Bong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202739' target=_top>202739</a>) -->
<!-- Comment: Sat Aug 13 13;19;13 comment by...kwangzoo -->
let me take my word back - hadn't look at the plot carefully at the time I responded to the pager, sorry.
<!-- Date: Sat Aug 13 13:18:45.993 2005 -->
According to more careful inspection by Carla and Kwangjoo, the DCAS error seemed to be related to ADMEM trouble in an event. Since the error rate is rather low, we ignore it for the time being.
<!-- Author: Soo-Bong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202739' target=_top>202739</a>) -->
<!-- Comment: Sat Aug 13 13;38;33 comment by...kwangzoo -->
purple line in the phi wedge 13(pcal03 slot18) is one event for now. this could be caused by an unstable power supply for the crate. will keep an eye on it.
<!-- Date: Sat Aug 13 13:22:55.531 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202739">202739</A>
smx: <tt>CP2PadCanvas and CCRPadCanvas plots are missing from the web page.</tt>
<!-- Author: Consumer Operator(Ben Brau) -->



<!-- Date: Sat Aug 13 13:26:41.142 2005 -->
<pre>Turned HV back on after a trip for cmx SE channels SE 10 and 11.</pre>
<!-- Author: sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202739' target=_top>202739</a>) -->


<!-- Date: Sat Aug 13 13:34:08.018 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14232&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14232' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14233&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14233' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14234&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14234' width='300'></a></center>
<!-- Author: sasha -->
<!-- Comment: Sat Aug 13 13;35;23 comment by...sasha -->
Hourly Plots: 12:30 - 13:30



<!-- Date: Sat Aug 13 13:57:27.753 2005 -->
Trigger inhibit from PEM. Problem with CAEN crate for East Plug 11.  Check Passed Status LED was off.  Tried crate reset, but LED was still off after 60s. Power cycled crate, waiting 5 mins to turn back on. After 60s, check-passed LED was still off, so power cycled again. This time will wait 10 minutes.
<!-- Author: sasha :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202739' target=_top>202739</a>) -->
<!-- Comment: Sat Aug 13 14;20;01 comment by...sasha -->
<pre>
   After a 10-minute wait, powered on crate 11. Another ~75 seconds thereafter, still no red 'check passed' LED.  Expert was paged. Just to check monitoring, the HV plug monitoring program was killed and restarted, and communication w/ crate 11 was checked as in 'recovery procedure 4' in shift crew instructions. 
There was no comunication with crate 11. Other crates responded normally to this procedure. Power was then turned off. Will wait longer to turn power on, or otherwise follow expert advice.</pre>


<!-- Comment: Sat Aug 13 14;36;30 comment by...sasha -->
<pre>     After waiting 15 mins. Turned on crate and waited ~3 mins. Then pushed crate reset button and waited another 3 minutes. No LED comes on, turned crate off at 14:35, will try 20+ minutes of down time.</pre>
<!-- Comment: Sat Aug 13 15;01;55 comment by...sasha -->
<pre>Turned on crate at 14:57 (after 22 minutes this time). 
HV monitor crashed. Started a new one, and is looks frozen. Waiting for monitor, hoping for red 'chack passed' led, still not on after 4 mins.</pre>
<!-- Comment: Sat Aug 13 15;54;17 comment by...sasha -->
<pre>This time we let ~26 minutes pass. The red led came up within 60s of powering on. Ran an individual check of crate 11. All chans communicated, but were at 0 volts. Since more than 10 were not 'ON', killed and restarted HV Monitor: everything came up to 100%.</pre>
<!-- Date: Sat Aug 13 14:22:21.613 2005 -->
D0 is going to move their Roman pot in.
<!-- Author: Soo-Bong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202739' target=_top>202739</a>) -->



<!-- Date: Sat Aug 13 14:48:57.179 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14235&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14235' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14236&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14236' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-14237&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-14237' width='300'></a></center>
<!-- Author: sasha -->
<!-- Comment: Sat Aug 13 14;49;41 comment by...sasha -->
    Hourly Plots: 13:30 - 14:30


<!-- Comment: Sat Aug 13 15;03;24 comment by...sasha -->
Monitor back up but LED for crate 11 still off.
<!-- Comment: Sat Aug 13 15;08;06 comment by...sasha -->
<pre>The most frequent error I see in the 'END PLUG HV Client' window:
CAEN-operation 0x40, Cr:11 -(Error 0003: No Slave)
Turning crate off again.</pre>
<!-- Date: Sat Aug 13 14:53:31.670 2005 -->
We have major down time (so far 1 hour) of bringing Plug HV back according to its instruction.
<!-- Author: Soo-Bong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202739' target=_top>202739</a>) -->





<!-- Date: Sat Aug 13 15:41:31.506 2005 -->
CMX trip:  SW Wedge 15  Powered it on again.  
<!-- Author: Greg :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202739' target=_top>202739</a>) -->


<!-- Date: Sat Aug 13 15:46:02.994 2005 -->
BMU Trip Wedge 276 NW, Turned back on.
<!-- Author: Greg :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202739' target=_top>202739</a>) -->


<!-- Date: Sat Aug 13 15:55:10.101 2005 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>Physics able</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L1Accepts</TH>
<TH>L2Accepts</TH>
<TH>L3Accepts</TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202735 target=_blank>202735</A> x317EF</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_3_06 [1,683,585]</FONT></TD>
<TD ALIGN=CENTER>07:32:37</TD>
<TD ALIGN=CENTER>08:43:13</TD>
<TD ALIGN=CENTER>00:29:52</TD>
<TD ALIGN=RIGHT>     43,823,530</TD>
<TD ALIGN=RIGHT>       541,629</TD>
<TD ALIGN=RIGHT>     142,978</TD>
<TD ALIGN=RIGHT>  120.161</TD>
<TD ALIGN=CENTER>0</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202736 target=_blank>202736</A> x317F0</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_3_06 [1,683,585]</FONT></TD>
<TD ALIGN=CENTER>08:45:57</TD>
<TD ALIGN=CENTER>09:01:30</TD>
<TD ALIGN=CENTER>00:13:23</TD>
<TD ALIGN=RIGHT>     18,257,183</TD>
<TD ALIGN=RIGHT>       190,925</TD>
<TD ALIGN=RIGHT>      54,680</TD>
<TD ALIGN=RIGHT>   49.042</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202738 target=_blank>202738</A> x317F2</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_3_06 [1,683,585]</FONT></TD>
<TD ALIGN=CENTER>09:20:41</TD>
<TD ALIGN=CENTER>11:03:39</TD>
<TD ALIGN=CENTER>01:30:47</TD>
<TD ALIGN=RIGHT>    133,280,587</TD>
<TD ALIGN=RIGHT>     1,580,943</TD>
<TD ALIGN=RIGHT>     414,502</TD>
<TD ALIGN=RIGHT>  301.541</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202739 target=_blank>202739</A> x317F3</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_3_06 [1,683,585]</FONT></TD>
<TD ALIGN=CENTER>11:14:28</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>02:28:46</TD>
<TD ALIGN=RIGHT>    177,957,939</TD>
<TD ALIGN=RIGHT>     2,557,587</TD>
<TD ALIGN=RIGHT>     619,154</TD>
<TD ALIGN=RIGHT>  416.392</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>15:55:01</TD>
<TD ALIGN=CENTER>04:42:50</TD>
<TD ALIGN=RIGHT>     373,319,239</TD>
<TD ALIGN=RIGHT>      4,871,084</TD>
<TD ALIGN=RIGHT>     1,231,314</TD>
<TD ALIGN=RIGHT>  887.135</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sat Aug 13 15:55:11.741 2005 -->
Plug HV finally comes alive after recycling the plug CAEN crate several times for two hours!  
<!-- Author: Soo-Bong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=202739' target=_top>202739</a>) -->


<!-- Date: Sat Aug 13 15:57:53.254 2005 -->
<font color=red>Shift Summary: </font><pre>8:00 Steve Hahn has replaced the bad power supply of CEM and is resetting HV for all of calorimeter PMTs 
8:10 Plug HV problem 
8:45 After recycling a crate power, plug HV is back 
13:00 L1 DACS-ADMEM mismatch for a single event; likely due to ADMEM problem 
13:57 Plug HV problem with CAEN crate for East Plug 11 
15:15 Large downtime (so far more than 1hr) of bringing Plug HV back according to its instruction 
15:50 Tried to recycle power of plug HV crate several times and came alive after two hours </pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   202735 - 202739
Delivered Luminosity   1396.8 nb-1  
Acquired Luminosity     772.3 nb-1  
Efficiency             55.3
   
</td></tr></table></pre>
<!-- Author: Soo-Bong -->


<!-- Date: Sat Aug 13 16:01:09.942 2005 -->
<font color=red>Shift Summary: </font><pre>Please use caution if you choose to put HTML into an entry.  Forgetting to use a close tag for each open tag will certainly cause problems. 
 
If you choose to use plain text, PLEASE USE CR's since this text window automatically wraps the text and gives the illusion of CR's.  (click to clear this message) 
</pre> 
<!-- Author: blah -->