<!-- Title: Owl shift. Thu Jul 13, 2006 -->
<!-- SciCo: virgil barnes -->
<!-- DAQAce: emily nurse -->
<!-- MonAce: alice abulencia -->
<!-- CO: sam hewamanage -->
<!-- OpManager: mark mattson -->
<!-- Notes: -->
<pre>protons are in, Stash 106 E10</pre>


<!-- Date: Thu Jul 13 00:15:25.268 2006 -->
Run 219659 red in DQMon due to W->munu cross section. The instructions say to call the Physmon expert, which I assume is  Collin Wolfe since his e-mail is on the web page. We will try and contact him.
<!-- Author: Emily -->
<!-- Comment: Thu Jul 13 00;26;24 comment by...m mattson -->
<br>How old is this run?  Contact by e-mail?  Although the instructions say "call", I assume that you don't want to page someone on the owl shift about conditions during a run from the previous store.
<!-- Comment: Thu Jul 13 00;27;49 comment by...Emily -->
Hi Mark, yes I e-mailed him.
<!-- Comment: Thu Jul 13 00;35;25 comment by...m mattson -->
Sorry for the paranoia.  I still need to catch up on my zzzz's.
<!-- Comment: Thu Jul 13 03;25;52 comment by...Emily -->
Run 219612 also bad. Following Mark's advice I won't call.
<!-- Date: Thu Jul 13 00:22:55.410 2006 -->
When I talked to the evening Sci-Co (after being paged for protons during shot setup), Jim mentioned that the last entry in the Run Coordinator log was still from about 6pm. The current plan  is *no* access planned for today.  If D0 encounters power supply problems during this owl shift, then the access will most likely be back on.
<!-- Author: m mattson (ops manager) -->



<!-- Date: Thu Jul 13 00:27:02.966 2006 -->
<pre>pbars are in, preparing to ramp</pre>
<!-- Author: virgil barnes -->


<!-- Date: Thu Jul 13 00:30:54.039 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219787>219787</A>
TERMINATE: Getting ready for data
<!-- Author: Emily  x2080 -->
<!-- Comment: Thu Jul 13 00;44;21 comment by...Emily -->
More info:

RC waiting for b0tsi{00-02}, csl, evb2b and l2ctrlb. Everything else but errlog (which is green) waiting for RC. 
<!-- Date: Thu Jul 13 00:39:35.715 2006 -->
RC taking ages to get through Partition transition. I will try to reset and start again.
<!-- Author: Emily -->
<!-- Comment: Thu Jul 13 00;41;00 comment by...Emily -->
When I tried again I got: 
<pre>
2 crate/s: b0svt09(116), b0svt08(117),  in error. [RXPT]
</pre>

in the Error Handler.

<!-- Comment: Thu Jul 13 00;43;02 comment by...Emily -->
Partition taking ages again. We have paged DAQ pager.



<!-- Date: Thu Jul 13 00:50:36.705 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6297&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6297' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6298&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6298' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6299&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6299' width='300'></a></center>
<!-- Author: Scraping Si plots -->


<!-- Date: Thu Jul 13 00:55:36.204 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-6300&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-6300  width='300' align='top'></a></td><td>Here is a more recent ace plot 1.  Jennifer gave me expert permission to bias the silicon so I am turning it on now in hopes RC problems will resolve</td></tr></table>
<!-- Author: Alice -->


<!-- Date: Thu Jul 13 01:04:05.562 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6301&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6302' ></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6303&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6304' ></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6305&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6306' ></a></center>
<!-- Author: Alice -->
<!-- Comment: Thu Jul 13 01;04;26 comment by...Alice -->
<pre>I just got 3 bias pinkies that went green after unmarking them</pre>


<!-- Comment: Thu Jul 13 01;10;37 comment by...Jen -->
Probably unhappy because they haven't been taken through config yet.  If it looks like the DAQ will be a while to resolve, you might want try just taking the silicon through a coldstart.  Or you can just wait; they might go pink but they shouldn't trip.
<!-- Comment: Thu Jul 13 01;16;27 comment by...Alice -->
<pre>Right now it looks like the silicon crates are being held hostage ( we are still stuck in a partition state) so I'm going to hope they don't trip - if they become free I'll try to coldstart.</pre>
<!-- Comment: Thu Jul 13 01;22;12 comment by...Alice -->
<pre>Did the coldstart</pre>
<!-- Comment: Thu Jul 13 01;23;40 comment by...Jen -->
Fabulous! :)

<!-- Date: Thu Jul 13 01:04:50.510 2006 -->
<pre>B0lum  64 E30.  problems starting run</pre>
<!-- Author: virgil barnes -->



<!-- Date: Thu Jul 13 01:21:42.220 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219788>219788</A>
TERMINATE: Coldstart
<!-- Author: Alice X2080 -->





<!-- Date: Thu Jul 13 01:30:37.260 2006 -->
I have been on the phone for a while with the DAQ expert trying to diagnose what is wrong with RC. It seems there is a problem with the EVB. I have cleaned it up twice but it is still not running. I spoke to the L3 expert who is contacting another L3 expert and they will get back to us. I will try cleaning up once more.
<!-- Author: Emily -->
<!-- Comment: Thu Jul 13 01;34;26 comment by...Emily -->
I tried cleaning up EVB, the check says ok. I try partition, doesn't work. EVB check now says not running on partition 0.

<!-- Comment: Thu Jul 13 01;37;10 comment by...Jeff Miles -->
<pre>Yes, the EVB proxy crashes on every partition transition. There was a version change made ~30 hours and we are probably going to switch it back, although this is an unforseen effect.  Work is ongoing</pre>


<!-- Date: Thu Jul 13 01:37:13.039 2006 -->
<pre>The rc problem is that during partition  
- most crates are light yellow 
- a few are dark yellow 
- error log has "svt08 and svt09 in error" 
by watching on ccal01, I could see this crate 
was not getting the transition message so we suspected 
the messages were not being passed.  We restarted 
rc and rtserver and the problem persisted. 
We found all crates could receive partition message  
and would turn green if partition message was sent by hand. 
This means rtserver is working.  We shepherded svt08 and 09 
and that didn't help.  Sent parition to svt08 and 09 by hand 
and they went green but that didn't help. 
Cleaned up evb and that didn't help (twice after first time 
the check said proxy was not running). 
Advised to call Bill. 
 
</pre>
<!-- Author: Ray Culbertson -->
<!-- Comment: Thu Jul 13 01;38;58 comment by...Alice -->
<pre>Can't get a hold of Bill via cell so we are waiting for L3 guys to see if they can fix this before we call him at home...</pre>


<!-- Date: Thu Jul 13 01:59:02.862 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6307&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6307' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6308&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6308' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6309&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6309' width='300'></a></center>
<!-- Author: Alice Hourly Plots -->

<!-- Date: Thu Jul 13 02:00:27.832 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219789>219789</A>
 Activated at 2006.07.13 02:00:22
<!-- Author: RunControl -->


<!-- Date: Thu Jul 13 02:01:12.312 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219789>219789</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_00 [3,798,638] 
<!-- Author: Emily x2080 -->
<!-- Comment: Thu Jul 13 02;05;18 comment by...Emily -->
We eventually started the run. The b0 int. lumi in RC was 262.8 when we started.
<!-- Date: Thu Jul 13 02:02:00.366 2006 -->
Done TO for b0imu00, auto HRR fixed it
<!-- Author: Emily :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219789' target=_top>219789</a>) -->
<!-- Comment: Thu Jul 13 02;03;28 comment by...Emily -->
And again


<!-- Comment: Thu Jul 13 02;08;32 comment by...Emily -->
An again, shepherded crate which failed. So I did it again. It worked.


<!-- Comment: Thu Jul 13 02;12;23 comment by...Emily -->
And twice more. Have shepherded 3 times. Am calling Muon TDC expert.
<!-- Comment: Thu Jul 13 02;13;34 comment by...Emily -->
In the middle of all this I got an EVB error:
<pre>
Attention !!!. FERML_TM2_HEVB2B_EVENTS_PENDING_TIMEOUT !!! 
 - AUTO HRR will be issued. 
 
 - If problem persists, you may have to clean up the Event Builder. 
</pre>
Fixed by HRR
<!-- Comment: Thu Jul 13 02;52;00 comment by...Emily -->
I spoke to Eric James who asked me to power cycle then shepherd imu00. Unfortunately I forgot to do the vxshutdown, luckily I havn't got any RTserver errors.

Anyway, the power cycle didn't work as I got a Done TO again.
Apparantly earlier today some muon crates (including imu00) were put into SPY-MODE read-out. Eric talked me through changing the read-out mode back to NON SPY-MODE using the Hardware database selector. Now we are running fine.  
<!-- Comment: Thu Jul 13 02;55;14 comment by...Emily -->
The following crates had their read-out mode changed:
<pre>
CMU_01, IMU_01, IMU_00, MUON_SCINT_00
</pre>
If I get problems with these crates during my shift I will do the same thing to them. I still have the Hardware database selector CRATES editor open, logged in as Eric so I will be able to do this.
The experts will look into the problem in the morning.
<!-- Comment: Thu Jul 13 03;01;03 comment by...Emily -->
<table border=0><td><tr><td bgcolor=#e0e030>Other ACES, if you get repeated Done TO in these crates that shepherding doesn't fix, page the DAQ pager.</td></tr></td></table>
<!-- Date: Thu Jul 13 02:08:41.331 2006 -->
<pre>At 2006/07/13-00:32:56 the EVB proxy for partition 0 failed to unpack the partition message and failed ever since. I loaded an older version of the EVB software, which handle the communication in a different way and which works fine. I will investigate in the morning what really went wrong.</pre>
<!-- Author: markus -->




<!-- Date: Thu Jul 13 02:26:49.388 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-6310&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-6310  width='300' align='top'></a></td><td>A "small tune change" caused lostpb to spike... no tevmon warnings those and the spike is kind of small (less than 2 kHz)</td></tr></table>
<!-- Author: Alice -->
<!-- Comment: Thu Jul 13 03;05;44 comment by...Alice -->
<pre>Same behavior again....</pre>
<!-- Date: Thu Jul 13 03:02:40.857 2006 -->
I got the following error in the middle of the imu00 problems: 
<pre> 
Attention!!!. FERML_HIGH_DEADTIME !!!  
 - Trigger/DAQ Deadtime is too high (at 75.72073 %).  
 - Look for source of hot triggers.  
 - Contact a trigger expert for further assistance.  
</pre> 
But the deadtime went back down again.
<!-- Author: Emily. :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219789' target=_top>219789</a>) -->
<!-- Comment: Thu Jul 13 07;17;03 comment by...Donatella T. -->
ScalerMon does wait 60 secs to check deadtime after a HALT. Most of the time the deadtime goes back to normal by then.I'll see if we should increase this time window a little..
<!-- Date: Thu Jul 13 03:13:00.142 2006 -->
I had trouble getting SVXMon Display to run. Since I started the Displays after 20 minutes of data taking, all the other Mons started the slide shows automatically except SVXMon Display. I tried to open the connection manually using the gui did not work.  killed the display and monitor and restarted SVXMon. Same result. Suddenly, it connected! 
<!-- Author: sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219789' target=_top>219789</a>) -->
<!-- Comment: Thu Jul 13 03;24;15 comment by...Emily -->
I got the following error a couple of times:
<pre>
SVXMon not connected to ConsumerErrorReceiver (aka ConsumerMonitor)
  since 2006-07-13 02:49:45.0
ConsumerErrorReceiver forwards errors and HRR requests from SVXMon
  and these processes are very important to data taking!
Restart SVXMon immediately!
</pre>
<!-- Date: Thu Jul 13 03:16:59.930 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-6315&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-6316  width='300' align='top'></a></td><td>SVXMon SVX Chop Status</td></tr></table>
<!-- Author: Emily -->


<!-- Date: Thu Jul 13 03:17:04.997 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6311&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6311' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6312&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6312' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6314&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6314' width='300'></a></center>
<!-- Author: Alice Hourly Plots -->


<!-- Date: Thu Jul 13 03:19:24.300 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-6317&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-6318  width='300' align='top'></a></td><td>SVXMon L00 & ISL Chip status!</td></tr></table>
<!-- Author: sam -->




<!-- Date: Thu Jul 13 03:26:43.971 2006 -->
<pre>Paged Jennifer about SVXMon plots freaking out.  She can't find anything wrong & iMon looks fine so we restart SVXMon and see how that goes.</pre>
<!-- Author: Alice :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219789' target=_top>219789</a>) -->


<!-- Date: Thu Jul 13 03:41:59.740 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6321&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6322' ></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6319&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6320' ></a></center>
<!-- Author: sam -->
<!-- Comment: Thu Jul 13 03;47;52 comment by...Alice -->
<pre>Apparently even Si pager carriers require a bit of sleep - this is not pressing and Jen will look at it in the morning.</pre>

<!-- Comment: Thu Jul 13 03;48;37 comment by...sam -->
On expert advice restarted the SVXMon. Initially there were lot of yellows(no data). After couple of minutes of running yellows dissapeared and lot of reds (failures) appeared in LOO & ISL  and SVX chip status maps. paged the experts again.

<!-- Comment: Thu Jul 13 03;58;04 comment by...sam -->
Jen will look at it in the morning.


<!-- Date: Thu Jul 13 03:58:26.904 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6323&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6323' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6324&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6324' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6325&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6325' width='300'></a></center>
<!-- Author: Alice Hourly Plots -->


<!-- Date: Thu Jul 13 04:15:41.652 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-6326&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-6327  width='300' align='top'></a></td><td>I got a WHA error from DQMon. From searching through the elog this is a known problem concerning low stats. Since it instructs us to "page the relevant expert now" I would really appreciate some clarification on how to react to this message.  Will it be changed so that it doesn't give us an error when stats are low? Its hard for ACEs to remember all the exceptions to the rules. Could we have a post-it note on this console telling us when not to worry about this message and when to react? Thanks :)  </td></tr></table>
<!-- Author: Emily -->
<!-- Comment: Thu Jul 13 05;12;00 comment by...Emily -->
At last, WHA in DQMon is green again.
<!-- Date: Thu Jul 13 04:17:57.379 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-6328&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-6329  width='300' align='top'></a></td><td>TrigMon XTRDMonitor: XTRP Summary Canvas: 
CMU and CMX is showing large number of mismathces(erros) with the sim. None of the triggers seems to be really hot though. </td></tr></table>
<!-- Author: sam -->
<!-- Comment: Thu Jul 13 05;22;04 comment by...KPitts -->
This is a known problem that intermittently shows up.   Its affect on the muon triggers is really tiny, so it isn't worth the downtime to try to fix it.  It will probably disappear again soon.  (It would be much easier to fix if it would just stay broken.)
<!-- Date: Thu Jul 13 04:45:22.232 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219789>219789</A>
trigger: <tt>High errors in TrigMon: XTRP Summary Canvas: CMU and CMX is worrying. IBut everthing else seems to ok. See early comments in the elog. </tt>
<!-- Author: Consumer Operator(samantha) -->


<!-- Date: Thu Jul 13 04:45:23.113 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219789>219789</A>
general: <tt>SVXMon is still showing lot of Red cells in L00 ISL and SVX chip staus.</tt>
<!-- Author: Consumer Operator(samantha) -->



<!-- Date: Thu Jul 13 05:00:00.161 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6330&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6330' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6331&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6331' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6332&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6332' width='300'></a></center>
<!-- Author: Alice Hourly Plots -->



<!-- Date: Thu Jul 13 05:20:26.932 2006 -->
<pre>Water alarm "Back-up Pump Warning Indicator" went off to make sure we are awake.  Doesn't impact us operationally</pre>
<!-- Author: Alice :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219789' target=_top>219789</a>) -->



<!-- Date: Thu Jul 13 06:01:21.233 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6333&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6333' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6334&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6334' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6335&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6335' width='300'></a></center>
<!-- Author: Alice Hourly Plots -->


<!-- Date: Thu Jul 13 06:15:23.524 2006 -->
I got an explosion of RXPT errors beginning with 
<pre> 
RXPT Error for crate/s:  
    b0svx02    
</pre> 
I did a Halt Recover Run and we are running ok now. The full ErrorHandler output is below for interested parties: 
<pre> 
1 crate/s: b0svx02(112),  in error. [RXPT] 
(MLE) b0control21.fnal.gov:Thread-26:Jul 13, 2006 06:08:33 AM->FrontEnd Error: VRB_SVX_02  
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
L1 Done Timeout.[RXPT] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
(MLE) b0svx00:Messenger:Jul 13, 2006 06:09:02 AM->Silicon Resonator: Sl 16 Ch 9 -> e074: Bytes: 00146 
 
(MLE) b0svx08:Messenger:Jul 13, 2006 06:09:02 AM->Silicon Resonator: Sl 16 Ch 3 -> f843: Bytes: 00780 
 
(MLE) b0svx02:Messenger:Jul 13, 2006 06:09:02 AM->Silicon Resonator: Sl 20 Ch 9 -> e2b4: Bytes: 00082 
 
b0svx02:Messenger:Jul 13, 2006 06:09:02 AM->SRC Fatal Error:Sl 6 Resonance Detected  
 -->   
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error from b0svx02: Sl 6 Resonance Detected  
  
 - HALT the run                    
 - record any relevant information in e-log      
 - RECOVER - RUN      
[MLE] 
(MLE) b0svx07:Messenger:Jul 13, 2006 06:09:02 AM->Silicon Resonator: Sl 13 Ch 3 -> fa23: Bytes: 00780 
 
(MLE) b0felix.fnal.gov:TEDCpp:Jul 13, 2006 06:09:03 AM->{ 6:9:3, 7/13/2006 }: L2-Pulsar-Linux: Halted while  waiting for SVT data 
RXPT Error LOOP !!! Disabling AUTOMATIC HRRs.[EH Info] 
1 crate/s: b0svx02(112),  in error. [RXPT] 
(RC)   6:09:19 Halt -> HALTED 
(RC)   6:09:49 Recover -> RECOVERED 
(RC)   6:10:2 Run -> ACTIVE 
(MLE) b0xft02:Messenger:Jul 13, 2006 06:10:03 AM->Event 4153075: Bunchcounters in slot 11 (BC=18) and slot 9 (BC=118) disagree (L2B=1)  
(MLE) b0xft02:Messenger:Jul 13, 2006 06:10:04 AM->Runtime Error 4, Event 4153075: Bunch counter mismatch, mismatch count = 1 
Resuming AUTOMATIC HRRs.[EH Info] 
</pre>
<!-- Author: Emily :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219789' target=_top>219789</a>) -->
<!-- Comment: Thu Jul 13 06;36;18 comment by...Ankush -->
<table border=0><td><tr><td bgcolor=#e0e030><p>Page the calorimter & L1 people NOW. We have proved this is a problem with the JET5 trigger and not a silicon problem. 
</p></td></tr></td></table>


<!-- Date: Thu Jul 13 06:49:58.186 2006 -->
<h3>Suggestions to fix the SVXMon problems</h3> 
 
<p>I just looked at the silicon raw data and it seems to be OK.   
So let me just ask the following questions</p> 
 
<ul> 
<li>Are consumers in PHYSICS mode ?</li> 
<li>Is SVXMon updating...ie: look at the monitor above the extra-wide moinitor.  Does the window for SVXMon have messages which are changing?</li> 
<li>Check the CSL display....is SVXMon actually receiving data ?</li> 
<li>Is the Cosumer Histogram display set to auto-update ?</li> 
</ul> 
 
<p>Another useful thing would be to look at the SVXMon error-logger window.  If SVXMon is reporting correctly, all the ladders should be reporting errors.  Put a copy of some of the errors in to the e-log. </p>
<!-- Author: Ankush :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219789' target=_top>219789</a>) -->
<!-- Comment: Thu Jul 13 07;23;53 comment by...sam -->
Consumers are in Physics
SVXMon is updating and receiving data
Consumer histos are in auto update mode

<!-- Date: Thu Jul 13 07:19:38.532 2006 -->
Done TO for PCAL04, auto HRR didn't fix, shepherd did.
<!-- Author: Emily :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219789' target=_top>219789</a>) -->


<!-- Date: Thu Jul 13 07:21:02.215 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-6336&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-6337  width='300' align='top'></a></td><td>SVXErrorLogger Stats</td></tr></table>
<!-- Author: sam -->
<!-- Comment: Thu Jul 13 07;45;28 comment by...Jen & Satyajit -->
SVXMon says it's having heartbeat errors.  We're investigating the logfile, looks like the error receiver didn't connect the way it should have when SVXMon was started.
<!-- Date: Thu Jul 13 07:22:34.229 2006 -->
We had HB alarms for CLC, CPR, BSC etc and a COT trip. Alice is downstairs trying to restart the clc PC. Spoke to Bob Wagner - COT tripped because of CLC problems, we will turn it back on when CLC is fixed. ALso got CLC error in DQMon. CLC expert on his way in.
<!-- Author: Emily :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219789' target=_top>219789</a>) -->
<!-- Comment: Thu Jul 13 07;46;42 comment by...convery, Alice, Sasha -->
I think what happened is that the CLC CAEN hung.  Then Alice had trouble restarting the pc (it hung).  I will update the instructions to say not to bother trying "restart" but to use the power button (and that you have to press it for a really long time!).
<br>Then when she and Sasha tried to restart the CAEN, it had trouble booting because the communication cable was still connected and some prior communication attempt was confusing the processor.  Disconnecting the cable until the CAEN rebooted solved the problem.
<!-- Comment: Thu Jul 13 07;47;30 comment by...Alice -->
<pre>First: the PC had trouble restarting:
NB: if you hit restart Windows will not reboot on its own unless you hold down the computer on/off switch for a good 30 seconds until the monitor goes blank
Then the monitoring could not start because the CLC CAEN crate was bad.  By then experts had shown up and showed me how you fix this in the future - you have to pull out the input wire and turn the key on and off.
Once CLC was up and going we restarted the COT and turned the Si back on (I had turned it to standby in the midst of all this).
We are running now</pre>
<!-- Comment: Thu Jul 13 07;58;41 comment by...convery -->
Note that the instructions for the CAEN communication hangups are explicit, but probably not so easy to follow at the end of an owl shift!
<!-- Date: Thu Jul 13 07:46:03.295 2006 -->
Got the error below. Looks ok now. 
<pre> 
Attention!!!. FERML_HIGH_DEADTIME !!!  
 - Trigger/DAQ Deadtime is too high (at 44.751446 %).  
 - Look for source of hot triggers.  
 - Contact a trigger expert for further assistance.  
</pre>
<!-- Author: Emily :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219789' target=_top>219789</a>) -->




<!-- Date: Thu Jul 13 07:47:32.232 2006 -->
During all the CLC problems we got about 6 of the following errors: 
<pre> 
SVXMon not sending heart beat to ConsumerErrorReceiver (aka ConsumerMonitor) 
  since 2006-07-13 06:53:40.0 
ConsumerErrorReceiver forwards errors and HRR requests from SVXMon 
  and these processes are very important to data taking! 
Restart SVXMon immediately! 
</pre> 
Experts are looking into it now.
<!-- Author: Emily :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219789' target=_top>219789</a>) -->


<!-- Date: Thu Jul 13 07:51:41.493 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6338&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6338' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6339&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6339' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-6340&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-6340' width='300'></a></center>
<!-- Author: Alice Hourly Plots -->
<!-- Comment: Thu Jul 13 07;52;23 comment by...Alice -->
<pre>Not to worry Si folks - the Si was in standby pretty much from the time the CLC tripped until we got it back up</pre>

<!-- Date: Thu Jul 13 07:56:08.768 2006 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=219789 target=_runSummary>219789</A> x35A8D</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [3,798,638]</FONT></TD>
<TD ALIGN=CENTER>02:00:22</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>04:26:07</TD>
<TD ALIGN=RIGHT>   1,195,020</TD>   <TD ALIGN=RIGHT>    811.44</TD>
<TD ALIGN=RIGHT>    680.26</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>07:55:01</TD>
<TD ALIGN=CENTER>04:26:07</TD>
<TD ALIGN=RIGHT>     1,195,020</TD> <TD ALIGN=RIGHT>    811.44</TD>
<TD ALIGN=RIGHT>    680.26</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->




<!-- Date: Thu Jul 13 08:03:51.669 2006 -->
<font color=red>Shift Summary: </font><pre>shot setup successful, initial lumi 0.64 E32 
lots of trouble starting run, due to bad latest version of EVB code, fixed by going back to previous version, lost 260 nb-1 or more. 
COT and CLC trips 7:20, but now back running 
 
Totals 
Date:	2006.07.13 
Shift:	owl 
Delivered luminosity:	1079.1 nb-1 
Acquired luminosity:	687.4 nb-1 
Efficiency:	63.7</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   219789
Delivered Luminosity   1079.1  
Acquired Luminosity    687.4  
Efficiency             63.7
   
</td></tr></table></pre>
<!-- Author: virgil barnes -->

<!-- Date: Thu Jul 13 08:10:00.553 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219789>219789</A>
smx: <tt>ShowerMax/CCRPadCanvas shows new deads! probably due to low stats.</tt>
<!-- Author: Consumer Operator(samantha) -->


<!-- Date: Thu Jul 13 08:10:01.739 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=219789>219789</A>
monitors: <tt>Couple of triggers seems to be abnormal. see plots.</tt>
<!-- Author: Consumer Operator(samantha) -->
<!-- Comment: Thu Jul 13 17;58;46 comment by...sam -->
<table border=0><td><tr><td bgcolor=#e0e030>Sorry. My mistake. I posted the plots in the Evening shift Wed Jul 12, 2006. Here are the links for the plots.

http://www-cdfonline.fnal.gov/elog/notebooks/2006/Thu_Jul_13_2006_Day.081047.ps
http://www-cdfonline.fnal.gov/elog/notebooks/2006/Thu_Jul_13_2006_Day.081028.ps

Or just go back one shift. look at the bottom of the page.</td></tr></td></table>

<!-- Comment: Thu Jul 13 18;06;02 comment by...sam -->
<table border=0><td><tr><td bgcolor=#e0e030>Lets try again.

<a href= 'http://www-cdfonline.fnal.gov/elog/notebooks/2006/Thu_Jul_13_2006_Day.081047.ps'> Plot 1</a>
</td></tr></td></table>

<!-- Comment: Thu Jul 13 18;07;53 comment by...sam -->
<table border=0><td><tr><td bgcolor=#e0e030><a href = 'http://www-cdfonline.fnal.gov/elog/notebooks/2006/Thu_Jul_13_2006_Day.081028.ps'>Plot 2 </a>
</td></tr></td></table>