<!-- Title: Eve shift. Wed Apr 14, 2004 -->
<!-- SciCo: Monica Tecchio -->
<!-- DAQAce: Corrinne Mills -->
<!-- MonAce: Dae Jung Kong -->
<!-- CO: Seun Hou -->
<!-- OpManager: Mary Convery -->
<!-- Notes: -->
<pre>Taking data without SI and CMU</pre>

<!-- Date: Wed Apr 14 16:05:27.258 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181654">181654</A>
si: <tt>High Volatages are off for SVX, ISL and L00</tt>
<!-- Author: Consumer Operator(John R. Smith) -->


<!-- Date: Wed Apr 14 16:05:28.776 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181654">181654</A>
central: <tt>TOF West Average Charge Plot is highlighted in red.</tt>
<!-- Author: Consumer Operator(John R. Smith) -->


<!-- Date: Wed Apr 14 16:05:29.146 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181654">181654</A>
smx: <tt>CES West Strip Occupancy per event has ~20 new dead channels...NEW. I haven't seen this before. I will insert gif file in e-log.</tt>
<!-- Author: Consumer Operator(John R. Smith) -->


<!-- Date: Wed Apr 14 16:05:30.881 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181654">181654</A>
muons: <tt>CSP Occupancy for Part NE channel 15 is high.</tt>
<!-- Author: Consumer Operator(John R. Smith) -->


<!-- Date: Wed Apr 14 16:05:31.449 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181654">181654</A>
trigger: <tt>SVT is off, but SVTD beam position monitor still produces reasonable results? Why?</tt>
<!-- Author: Consumer Operator(John R. Smith) -->



<!-- Date: Wed Apr 14 16:07:20.067 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-5120&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-5120  width='300' align='top'></a></td><td>New Dead Channels in West Strip Occupancy</td></tr></table>
<!-- Author: John R. Smith -->
<!-- Comment: Wed Apr 14 16;17;29 comment by...monica -->
Paged CES expert. Karen Byrum will look onto it.
<!-- Comment: Wed Apr 14 21;31;50 comment by...byrum -->
This hole first appeared in run 181641 (early in owl shift today).  That was the first run with good statistics following Monday morning's (april 12th) TEV quench.  The hole was not present prior to that.   Looks like one CES strip cable has  fallen off (which can be verified with DBANA).  Need an access to fix.    
<!-- Date: Wed Apr 14 16:12:11.258 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-5122&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-5122  width='300' align='top'></a></td><td>No diagonal boxes at all in XTRPMon: Track Trig plot.</td></tr></table>
<!-- Author: John R. Smith -->
<!-- Comment: Wed Apr 14 16;20;54 comment by...monica -->
Expert pages. Sonia askes if something has changed recnetly in Trigmon. Apparently there was a new release only a couple of hours ago. I am letting her know about it.
<!-- Comment: Wed Apr 14 16;28;45 comment by...Anyes -->
<Br>
Top middle plot errors showing at the bottom (trigger != simu) are to be ignored until new trigger table goes in. <BR>
The new trigSim went in over the <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=view&page=228&frame=2&anchor=10:40:36.063'>
weekend </a>
and this version knows about 7 track ORing which is not yet in action. <Br>
The new trigger table & firmware change will happen in the next few days. <Br>
The other plots are fine. <Br>
<!-- Date: Wed Apr 14 16:12:46.391 2004 -->
One (1) processor has turned red in the level3 subfarm display.  The level 3 page says that means that the level3 filter crashed and  that the relevant info would be reported to the l3 group.<br>  Should I do anything to reboot this node or do I just ignore it?  
<!-- Author: corrinne :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181655' target=_top>181655</a>) -->


<!-- Date: Wed Apr 14 16:14:36.140 2004 -->
TrigMon SVTD Monitor STILL has reasonable beam position table even though SVT is not on. Why? 
 
MuonTriggerMonitor: Low Pt XTRP for CMU canvas is reasonable even though CMU is off? Why?
<!-- Author: John R. Smith :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181854' target=_top>181854</a>) -->
<!-- Comment: Wed Apr 14 21;13;04 comment by...ps -->
xtrp refers to the track not the muon stub. cot doesn't know cmu is off.
<!-- Date: Wed Apr 14 16:21:02.947 2004 -->
<pre>We tested new cdfsoftb0 releases 5.3.1i  
 
ObjectMon was crashed, so we put back it as 5.3.1b.  
But start script(startMon.sh) did not work for this version. 
Consumer Operator should type command by hand, not use script </pre>
<!-- Author: ryo and kohei -->
<!-- Comment: Wed Apr 14 16;25;06 comment by...ryo -->
CO should type command 
% runObjectMon 0 physics b0dau32.fnal.gov cdfonprd
<!-- Date: Wed Apr 14 16:24:54.319 2004 -->
<pre>"Process Systems ALARM" not Global ALARMS on iFix 
"COT Sytems" on the Process Systems panel. 
It changed RED color and Alarm. 
It was silenced and changed to Green color.</pre>
<!-- Author: Daejung :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181655' target=_top>181655</a>) -->



<!-- Date: Wed Apr 14 16:27:26.258 2004 -->
b0mutr00 RXPT Error!  Auto-HRR fixed.  I guess they were getting this error a lot last shift, too.
<!-- Author: corrinne :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181655' target=_top>181655</a>) -->



<!-- Date: Wed Apr 14 16:38:07.436 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-5124&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-5124  width='300' align='top'></a></td><td>Lummon, CLTD plots on beam positon, run 181655</td></tr></table>
<!-- Author: suen -->


<!-- Date: Wed Apr 14 16:42:34.724 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-5125&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-5125  width='300' align='top'></a></td><td>BeamMon Cot vertex on Z, run 181655</td></tr></table>
<!-- Author: suen -->


<!-- Date: Wed Apr 14 16:44:51.412 2004 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2004&action=view&page=-5126&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2004&action=view&page=-5126  width='300' align='top'></a></td><td>Mon Triger Mon lowPt Xtrp, for CMU wedge distribution, 
Why? CMU is off!  Run 181655</td></tr></table>
<!-- Author: John R. -->
<!-- Comment: Wed Apr 14 21;14;07 comment by...ps -->
see comment above
<!-- Date: Wed Apr 14 16:58:10.466 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181634">181634</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
ISL 
L00 
</PRE>
SI not in the run
<!-- Author: cdfscico -->


<!-- Date: Wed Apr 14 16:58:52.735 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181634">181634</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
SVX 
</PRE>
SI not in the tun
<!-- Author: cdfscico -->


<!-- Date: Wed Apr 14 17:03:40.373 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181655">181655</A>
si: <tt>silicon off, for SRC fracky on vibration</tt>
<!-- Author: Consumer Operator( Suen Hou) -->


<!-- Date: Wed Apr 14 17:03:41.598 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181655">181655</A>
muons: <tt>CMU off</tt>
<!-- Author: Consumer Operator( Suen Hou) -->


<!-- Date: Wed Apr 14 17:03:42.085 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181655">181655</A>
trigger: <tt>SVT is off, beam position fit not so</tt>
<!-- Author: Consumer Operator( Suen Hou) -->
<!-- Comment: Wed Apr 14 18;31;20 comment by...Taka Maruyama -->
 TrigMon shows the beam position is OK though b0svt06 and PC does 
not fit at this run. I will look the offline data. 
<!-- Comment: Wed Apr 14 19;16;33 comment by...Taka Maruyama -->
Offline data also shows this run has good beam in 3Dbeam bank

run 181655
qual 1 errcod 0 x0(um) 1676 y0(um) -1613 dx/dz(urad) 164 dy/dz(urad) 682 (qual 1 means "beam fit is OK") 

this can be compared to yesterday's no silicon run. 
run 181634:
qual 0 errcod 0 x0(um) 1687 y0(um) -1497 dx/dz(urad) 113 dy/dz(urad) 720 (qual 0 means "fit is bad")


 We may have wired beam position information right now.
I'll send email to expert. (right now, silicon is off, so 
that it maybe OK.) 
<!-- Comment: Wed Apr 14 19;18;29 comment by...Taka  -->
<pre>Offline data also shows this run has good beam in 3Dbeam bank.

run 181655 qual 1 errcod 0 
x0(um) 1676 y0(um) -1613 dx/dz(urad) 164 dy/dz(urad) 682 
(qual 1 means "beam fit is OK") 

this can be compared to yesterday's no silicon run. 
run 181634: qual 0 errcod 0
x0(um) 1687 y0(um) -1497 dx/dz(urad) 113 dy/dz(urad) 720 
(qual 0 means "fit is bad") 

We may have wired beam position information right now. 
I'll send email to expert. (right now, silicon is
off, so that it maybe OK.) 
</pre>
<!-- Date: Wed Apr 14 17:05:30.224 2004 -->
Another RXPT error from b0mutr00.  Auto-HRR recovered.
<!-- Author: corrinne :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181655' target=_top>181655</a>) -->


<!-- Date: Wed Apr 14 17:15:07.775 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5127&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5127' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5128&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5128' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5129&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5129' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5130&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5130' width='300'></a></center>
<!-- Author: Daejung -->
<!-- Comment: Wed Apr 14 17;15;41 comment by...Daejung -->
Hourly Plots


<!-- Date: Wed Apr 14 18:15:54.437 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5131&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5131' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5132&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5132' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5133&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5133' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5134&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5134' width='300'></a></center>
<!-- Author: Daejung -->
<!-- Comment: Wed Apr 14 18;16;16 comment by...Daejung -->
Hourly Plots


<!-- Date: Wed Apr 14 18:19:37.378 2004 -->
b0mutr00 RXPD Error again.  Auto-HRR recovered.
<!-- Author: corrinne :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181655' target=_top>181655</a>) -->



<!-- Date: Wed Apr 14 19:07:35.814 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5135&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5135' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5136&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5136' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5137&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5137' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5138&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5138' width='300'></a></center>
<!-- Author: Daejung -->
<!-- Comment: Wed Apr 14 19;07;59 comment by...Daejung -->
Hourly Plots




<!-- Date: Wed Apr 14 19:39:33.723 2004 -->
SI crew is out of the dungeon....there are doing some local tests 
before integrating SI back in the run.
<!-- Author: Monica -->


<!-- Date: Wed Apr 14 19:45:29.731 2004 -->
Done Timeout from b0pcal09.  Auto-HRR recovered.  Message in ErrorLog:<br> 
<br> 
(MLE) b0pcal09:Messenger:7:42:58 PM->Runtime Error 1, Event 738787: See nonsequential event number (old 2, new 7) in L2 message<br>
<!-- Author: corrinne :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181655' target=_top>181655</a>) -->

<!-- Date: Wed Apr 14 19:56:47.356 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181655">181655</A>
si: <tt>Silicon off for SRC work</tt>
<!-- Author: Consumer Operator( Suen Hou) -->



<!-- Date: Wed Apr 14 20:20:23.991 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5139&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5139' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5140&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5140' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5141&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5141' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5142&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5142' width='300'></a></center>
<!-- Author: Daejung -->
<!-- Comment: Wed Apr 14 20;20;48 comment by...Daejung -->
Hourly Plots


<!-- Date: Wed Apr 14 20:21:55.898 2004 -->
L00, SVX and ISL testing by svx expert
<!-- Author: Daejung -->

<!-- Date: Wed Apr 14 20:27:31.269 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181655">181655</A>
 Terminated at 2004.04.14 20:27:03
<!-- Author: RunControl -->


<!-- Date: Wed Apr 14 20:27:32.246 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181655">181655</A>
TERMINATE: ending to ramp up silicon
<!-- Author: corrinne x2080 -->



<!-- Date: Wed Apr 14 20:29:38.309 2004 -->
L00, SVX and ISL HV on by requst of svx expert
<!-- Author: Daejung -->


<!-- Date: Wed Apr 14 20:48:56.170 2004 -->
<pre>Trouble ending run -- l3 errors, and l3 failed the state transition.  
 
(MLE) b0l3pcom2.fnal.gov:main:8:28:06 PM-&gt;Error on L3 node b0l3pcom2.fnal.gov (partition 0) Wed Apr 14 20:27:56 2004 l3proxy 
  in End (L3ProxyPartition.cc:2230) 
  @L3_TRANS_FAIL - transition actions failed:      b0l3123 
 
(MLE) b0l3pcom2.fnal.gov:main:8:34:58 PM-&gt;Error on L3 node b0l3pcom2.fnal.gov (partition 0) Wed Apr 14 20:34:58 2004 l3proxy 
  in End (L3ProxyPartition.cc:2248) 
  @L3_TRANS_FAIL - partition End timeout:     b0l3123 
 
I'm trying to abort the run now.  l3 transitions are taking a 
very long time.  We're paging l3. 
</pre>
<!-- Author: corrinne :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181655' target=_top>181655</a>) -->

<!-- Date: Wed Apr 14 20:52:14.679 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181655">181655</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
CMU 
SVX 
ISL 
L00 
</PRE>
CMU and SI HV off. 
For CCAL, there was hole in CES strip occupancy. Will ask the expert
<!-- Author: cdfscico -->



<!-- Date: Wed Apr 14 20:52:45.429 2004 -->
<table border=0><td><tr><td bgcolor=#e0e030><h3>Silicon status</h3> 
The silicon is checked out, and has been given 
to the aces. Please include silicon for the rest of this store, and in the next as per usual instructions. If any SRC errors are <b>not 
cleared by HRR</b>, please page Sal (722-7599) to reset them. 
As usual, page the main silicon pager for any incidents 
not related to un-clearable SRC errors. 
Details on the silicon checkout are available 
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=191&frame=2&anchor=20:47:11'> 
here</a>.  
<br><br> 
<b>Please be careful when walking in the counting room!</b> 
Do not cause unneccesary vibrations, as they can cause silicon 
trips due to bad grounding in the clock fanout crate (among others). 
We are working to resolve this problem.</td></tr></td></table>
<!-- Author: Sal, Rong -->


<!-- Date: Wed Apr 14 21:05:19.466 2004 -->
SI experts asks to be called when final protons loading is done for the next shot
<!-- Author: monica -->


<!-- Date: Wed Apr 14 21:07:00.977 2004 -->
In response to the above l3 problem, we stopped and restarted the l3 proxy (although it did seem to be running) and are cleaning up the level3 farm in teh ace control panel.  In the text output from the cleanup, it does say "Connection Refused: b0l3123".  This is the same node that reported "faiiled transition" earlier.  l3 expert is on his way here.  
<!-- Author: corrinne -->



<!-- Date: Wed Apr 14 21:13:43.789 2004 -->
<pre>Here is the extensive error list we got upon trying to reset and then partition the run: 
 
10'2" b0l3pcom2.fnal.gov:main:9:10:02 PM-&gt;Error on L3 node b0l3pcom2.fnal.gov (partition 0) Wed Apr 14 21:10:02 2004 l3proxy 
  in Initialize (L3ProxyPartition.cc:2870) 
  @L3_RELAY_FAILURE Level3 Relay failure detected! 
The following subfarms need to be removed from the next runs: 
      None. You can use all the same subfarms. 
The following Level3 converter nodes are marked bad for now: 
      None. 
The full list of Level3 nodes giving Relay trouble is printed below 
 
 b0l3123 
 
 ************************************************ 
 These Level3 nodes will be marked OFFLINE!!! 
 Abort/End/Reset and Partition again!!! 
 ************************************************ 
 
 --&gt; 
 Additional Information: 
 
 Attention!!!. L3_RELAY_FAILURE Error !!!. 
 Level3 internal communications failure. One or more L3 nodes is likely 
 to have died. 
   Do not panic! Level3 will try to repair itself. We recommend: 
     - do End/Abort+Reset ON ALL DEFINED PARTITIONS 
     - exclude all affected subfarms from your run configuration 
       (find out list of bad subfarms from Error Logger: 
          if None, exclude nothing!) 
     - start another run and page L3 farm pager at closest day/evening shift 
     - if &gt;2 subfarms affected page L3 farm IMMEDIATELY 
[MLE] 
</pre>
<!-- Author: corrinne -->
<!-- Comment: Wed Apr 14 21;18;52 comment by...corrinne -->
Expert came in and manually removed the troubled node.  The node should have been removed automatically but for some reason it wasn't.

<!-- Date: Wed Apr 14 21:23:04.986 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181673">181673</A>
 Activated at 2004.04.14 21:22:38
<!-- Author: RunControl -->



<!-- Date: Wed Apr 14 21:23:05.958 2004 -->
<pre>Answering the page.  
 
Node 123 failed and was put offline as it should.  
L3proxy failed to pick up a new configuration twice.  
I had to kill l3proxy manually and restart it.   
Cleanup Level 3 and start a new run.  
</pre>
<!-- Author: Boris.  -->

<!-- Date: Wed Apr 14 21:23:27.711 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181673">181673</A>
ACTIVATE: quick run to test including silicon before they drop the store
<!-- Author: corrinne x2080 -->


<!-- Date: Wed Apr 14 21:26:45.719 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181673">181673</A>
 Terminated at 2004.04.14 21:25:53
<!-- Author: RunControl -->


<!-- Date: Wed Apr 14 21:26:46.525 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181673">181673</A>
TERMINATE: ending run since they are switching to beam studies before dropping the store.
<!-- Author: corrinne x2080 -->
<!-- Comment: Wed Apr 14 21;33;58 comment by...Daejung -->
All HV is normal state before running
<!-- Date: Wed Apr 14 21:32:05.704 2004 -->
MCR calls to say they are ready to start collimator studies at 9.30. We managed to integrate Si in the run just in time! 
We ramp dowm all HV's and call MCR back to say we are ready.
<!-- Author: monica -->



<!-- Date: Wed Apr 14 21:35:49.905 2004 -->
<pre>PSM Alarm 1RR18E 1, 
Expert is working</pre>
<!-- Author: Daejung -->


<!-- Date: Wed Apr 14 21:36:59.968 2004 -->
And Bill Badgett appear in B0 just as we are ending the run... 
He needs to replace CMP00 VME controller. And get his wish granted.
<!-- Author: monica -->

<!-- Date: Wed Apr 14 21:39:03.282 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181655">181655</A>
si: <tt>silicon off for SRC work</tt>
<!-- Author: Consumer Operator( Suen Hou) -->



<!-- Date: Wed Apr 14 21:44:27.302 2004 -->
<center><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5143&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5143' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5144&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5144' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5145&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5145' width='300'></a><a href='/cgi/elog/elog.pl?nb=2004&action=view&page=-5146&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2004&action=view&page=-5146' width='300'></a></center>
<!-- Author: Daejung -->
<!-- Comment: Wed Apr 14 21;45;16 comment by...Daejung -->
Hourly Plots by end of run


<!-- Date: Wed Apr 14 21:56:02.472 2004 -->
<pre>Have replaced the MVME2301 in crate b0cmp00 CMP_00  
with a new loaner processor MVME5500;  tested with SEVB  
and readout proceeded normally.   Now will try a  
L2 torture run while in spy readout mode. 
</pre>
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181676' target=_top>181676</a>) -->

<!-- Date: Wed Apr 14 21:58:13.581 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181676">181676</A>
ACTIVATE: l2 torture run for W Badgett
<!-- Author: corrinne x2080 -->



<!-- Date: Wed Apr 14 22:11:55.969 2004 -->
<pre>Turned off spy-mode readout in b0cmp00, as the crate  
was balking.   Now in non-spy mode readout, things are  
working AOK. 
</pre>
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181677' target=_top>181677</a>) -->


<!-- Date: Wed Apr 14 22:17:29.573 2004 -->
During L2 torture, got two consecutive done TO's from b0ccal00.  Shepherded the crate, and it seems to be happy now, although it spat out an "RXPT Error LOOP" during the recover,run transition.  
<!-- Author: corrinne :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181677' target=_top>181677</a>) -->


<!-- Date: Wed Apr 14 22:27:06.589 2004 -->
For the last few hours of todays store we were reading out the both pulsar boards downstream of the Muon fiber splitters. 
The splitting worked without error for both, the pulsar in b0l2de00 and the pulsar b0l2pu00. 
Now I swapped the pulsar Muon board in the pulsar crate, so that we can do some further development for the board we was in the beam. 
<!-- Author: Burkard -->


<!-- Date: Wed Apr 14 22:27:58.481 2004 -->
<pre>Important consideration when rebooting the MVME5500 
in crate b0cmp00: 
 
  =&gt; The boot time for this MVME5500 is siginficantly longer  
     than a typical MVME2301, due to the fact that the VxWorks  
     kernel has not yet been loaded into flashRAM, and must be  
     copied over the network. 
 
  =&gt; b0cmp00 takes about 1:45 minutes to boot, compared  
     to about 0:25 for a MVME2301, so be patient if you have  
     to reboot and shepherd the crate.  This does not include  
     transition times. 
 
If you have serious problems with b0cmp00, gib me a call. 
 
</pre>
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181678' target=_top>181678</a>) -->

<!-- Date: Wed Apr 14 22:28:29.116 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181678">181678</A>
ACTIVATE: Another L2 Torture with MVME5500 in b0cmp00
<!-- Author: Corrinne x2080 -->



<!-- Date: Wed Apr 14 22:36:12.226 2004 -->
MCR calls. They are going to dump sptre 3399.
<!-- Author: monica -->
<!-- Comment: Wed Apr 14 22;39;40 comment by...monica -->
interesting way to mispell "store", isn't it? 
Also they are setting up for the new shot right away, i.e. no quiet time for calibrations.

<!-- Date: Wed Apr 14 22:42:30.421 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181678">181678</A>
TERMINATE: ending to take calibrations
<!-- Author: Corrinne x2080 -->


<!-- Date: Wed Apr 14 22:44:29.277 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181680">181680</A>
ACTIVATE: cot calibs
<!-- Author: Corrinne x2080 -->


<!-- Date: Wed Apr 14 22:45:12.746 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181680">181680</A>
TERMINATE: end cot calibs
<!-- Author: Corrinne x2080 -->


<!-- Date: Wed Apr 14 22:47:19.448 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181682">181682</A>
ACTIVATE: muon calibrations
<!-- Author: Corrinne x2080 -->


<!-- Date: Wed Apr 14 22:47:20.831 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181682">181682</A>
TERMINATE: end muon calibrations
<!-- Author: Corrinne x2080 -->


<!-- Date: Wed Apr 14 22:49:14.259 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181684">181684</A>
ACTIVATE: tof tac calibrations
<!-- Author: Corrinne x2080 -->


<!-- Date: Wed Apr 14 22:51:03.329 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181684">181684</A>
TERMINATE: end calibs
<!-- Author: Corrinne x2080 -->


<!-- Date: Wed Apr 14 22:57:24.119 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181685">181685</A>
 Activated at 2004.04.14 22:57:13
<!-- Author: RunControl -->


<!-- Date: Wed Apr 14 22:57:43.919 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181685">181685</A>
ACTIVATE: pmt spike run
<!-- Author: Corrinne x2080 -->


<!-- Date: Wed Apr 14 23:07:12.013 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181685">181685</A>
 Terminated at 2004.04.14 23:06:28
<!-- Author: RunControl -->


<!-- Date: Wed Apr 14 23:08:27.548 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181685">181685</A>
TERMINATE: end PMT spike run.  L1 accept rate (avg over run) was 55 Hz
<!-- Author: Corrinne x2080 -->


<!-- Date: Wed Apr 14 23:13:41.603 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181686">181686</A>
 Activated at 2004.04.14 23:13:15
<!-- Author: RunControl -->


<!-- Date: Wed Apr 14 23:13:58.249 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181686">181686</A>
ACTIVATE: re-starting PMT spike run, Ymon, trigmon running now
<!-- Author: Corrinne x2080 -->


<!-- Date: Wed Apr 14 23:22:47.649 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181686">181686</A>
 Terminated at 2004.04.14 23:22:40
<!-- Author: RunControl -->


<!-- Date: Wed Apr 14 23:23:57.711 2004 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181686">181686</A>
TERMINATE: Ending PMT spike run.  L1 accept rate for the run is 55 Hz.  
Ymon and TrigMon look okay as far as we can tell.
<!-- Author: Corrinne x2080 -->



<!-- Date: Wed Apr 14 23:35:17.019 2004 -->
MCR says they are in shot setup
<!-- Author: monica -->


<!-- Date: Wed Apr 14 23:44:02.265 2004 -->
<pre>More L3 errors:  In trying to Partition to do an L2 torture run,  I got the following error: 
 
27'36" b0l3pcom2.fnal.gov:main:11:27:36 PM-&gt;Error on L3 node b0l3pcom2.fnal.gov (partition 0) Wed Apr 14 23:27:36 2004 l3proxy 
  in Initialize (L3ProxyPartition.cc:2870) 
  @L3_RELAY_FAILURE Level3 Relay failure detected! 
The following subfarms need to be removed from the next runs: 
      5 
The following Level3 converter nodes are marked bad for now: 
      b0l3c05 
The full list of Level3 nodes giving Relay trouble is printed below 
 
 b0l3c05 b0l3067 b0l3069 b0l3071 b0l3073 b0l3075 
      b0l3077 b0l3079 b0l3081 b0l3068 b0l3070 b0l3072 
      b0l3074 b0l3076 b0l3078 b0l3080 b0l3082 
 
 --&gt; 
 Additional Information: 
 
 Attention!!!. L3_RELAY_FAILURE Error !!!. 
 Level3 internal communications failure. One or more L3 nodes is likely 
 to have died. 
 
----------------------------------- 
 
So, I reset in RC and removed the L3 subfarm 5 from the run  
using "Edit Run Settings" (you uncheck the box),and paged the  
expert, as the error message prompts you to do.  The partition 
transition failed again (l3 box in RC partition display turned 
red). 
 
Said expert (Boris) is upstairs trying to rescusitate a node. 
</pre>
<!-- Author: corrinne :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181686' target=_top>181686</a>) -->


<!-- Date: Wed Apr 14 23:53:06.506 2004 -->
<font color=red>Shift Summary: </font><pre>16:00   Smooth physics data taking with SI and CMU out 
 
20:30   Run 181655 is ended to integrate SI 
 
21:30   SI is integrated in the run just in time before we have to 
        bring all HV's down for start of collimator studies 
 
22:30   Store 3399 dumped.  
 
23:35   Starting shot setup</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   181655
Delivered Luminosity   230  
Acquired Luminosity    192  
Efficiency             83.6
   
</td></tr></table></pre>
<!-- Author: monica -->

<!-- Date: Wed Apr 14 23:55:24.731 2004 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181655>
181655</A> 
x2C597</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_2_03 [2,456,449]</TD>                                       
<TD ALIGN=CENTER>15:15:37</TD>                                          
<TD ALIGN=CENTER>20:27:03</TD>                                          
<TD ALIGN=CENTER>05:08:59</TD>                                          
<TD ALIGN=RIGHT>    182,314,065</TD>                                    
<TD ALIGN=RIGHT>       854,301</TD>                                     
<TD ALIGN=RIGHT>     256,384</TD>                                       
<TD ALIGN=RIGHT>  223.796</TD>                                          
<TD ALIGN=CENTER>0</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=181673>
181673</A> 
x2C5A9</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_2_03 [2,456,449]</TD>                                       
<TD ALIGN=CENTER>21:22:38</TD>                                          
<TD ALIGN=CENTER>21:25:53</TD>                                          
<TD ALIGN=CENTER>00:03:09</TD>                                          
<TD ALIGN=RIGHT>        526,481</TD>                                    
<TD ALIGN=RIGHT>         8,821</TD>                                     
<TD ALIGN=RIGHT>       2,823</TD>                                       
<TD ALIGN=RIGHT>    1.972</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>0</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>23:55:03</TD>                                                       
<TD ALIGN=CENTER>05:12:08</TD>                                          
<TD ALIGN=RIGHT>     182,840,546</TD>                                   
<TD ALIGN=RIGHT>        863,122</TD>                                    
<TD ALIGN=RIGHT>       259,207</TD>                                     
<TD ALIGN=RIGHT>  225.768</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Wed Apr 14 23:58:05.018 2004 -->
<pre>c05  died - rebooted.  
eb10 was not responding - reset,,  
Cleanup EVB. Trying to start a new run 
</pre>
<!-- Author: Boris  -->