<!-- Title: Owl shift. Wed Aug 30, 2006 -->
<!-- SciCo: Donatella Lucchesi -->
<!-- DAQAce: Alice Abulencia -->
<!-- MonAce: Daryl Hare -->
<!-- CO: ByeongRok Ko -->
<!-- OpManager: Tom Wright -->
<!-- Notes: -->
<pre>Start shift with shot setup in progress. 
Plan is to take data with the default trigger table and without the L2 pulsar crate</pre>


<!-- Date: Wed Aug 30 00:12:24.166 2006 -->
<table border=0><td><tr><td bgcolor=#e0e030>h3>SB1W4L1 returning bad data </h3> <p> Expect to see an extra hole, or an under present cell in SVT occupancy. </p></td></tr></td></table>
<!-- Author: Tuula and Ankush -->

<!-- Date: Wed Aug 30 00:12:58.255 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222351>222351</A>
 Terminated at 2006.08.30 00:12:32
<!-- Author: RunControl -->


<!-- Date: Wed Aug 30 00:12:59.220 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222351>222351</A>
TERMINATE: switching to shotsetup
<!-- Author: Alice -->


<!-- Date: Wed Aug 30 00:16:33.180 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222352>222352</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_4_00 [4,820,642] 

no l2pulsar02
<!-- Author: Alice -->
<!-- Comment: Wed Aug 30 00;17;07 comment by...Alice -->
<pre>Final protons are almost all in</pre>


<!-- Date: Wed Aug 30 00:26:37.207 2006 -->
<pre>TrigMon and XMon didn't seem to be getting any events after I asked the CO to switch to physics mode so I stopped and restarted the monitors again. 
 
They are loading anitprotons now.</pre>
<!-- Author: Alice :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222352' target=_top>222352</a>) -->
<!-- Comment: Wed Aug 30 00;27;29 comment by...Alice -->
TrigMon and XMon looks happy  now

<!-- Date: Wed Aug 30 00:51:12.600 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222352>222352</A>
TERMINATE: Accelerating... prepare to take data
<!-- Author: Alice -->



<!-- Date: Wed Aug 30 00:51:20.878 2006 -->
All antiprotons have been loaded
<!-- Author: Donatella :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222352' target=_top>222352</a>) -->


<!-- Date: Wed Aug 30 01:10:09.666 2006 -->
Store 4930 Scraping complete
<!-- Author: DonatellaStore  :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222353' target=_top>222353</a>) -->

<!-- Date: Wed Aug 30 01:15:10.242 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222353>222353</A>
 Activated at 2006.08.30 01:14:47
<!-- Author: RunControl -->


<!-- Date: Wed Aug 30 01:15:11.526 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222353>222353</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_00 [4,820,642] 
<!-- Author: Alice -->
<!-- Comment: Wed Aug 30 01;19;45 comment by...Alice -->
<pre>l2pulsar02 is out

I got impatient and reset the luminosity by hand.</pre>
<!-- Comment: Wed Aug 30 01;20;45 comment by...Alice -->
<pre>I cleared the clock errors</pre>
<!-- Date: Wed Aug 30 01:15:16.978 2006 -->
Store 4930: 
TevPR:9914.08E09 
TevPB:1784.24E09 
CDF lumi: 164E30
<!-- Author: Donatella :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222353' target=_top>222353</a>) -->



<!-- Date: Wed Aug 30 01:20:02.578 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10066&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10066' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10067&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10067' width='300'></a></center>
<!-- Author: Bruno Silicon shot setup plots -->



<!-- Date: Wed Aug 30 01:32:20.075 2006 -->
CMX trip NW wedges 22 and 23
<!-- Author: Daryl :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222353' target=_top>222353</a>) -->

<!-- Date: Wed Aug 30 01:47:02.123 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222353>222353</A>
 Terminated at 2006.08.30 01:46:26
<!-- Author: RunControl -->


<!-- Date: Wed Aug 30 01:47:03.950 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222353>222353</A>
TERMINATE: I have stopped taking events - going to try to see if new run helps
<!-- Author: Alice -->
<!-- Comment: Wed Aug 30 01;55;36 comment by...Alice -->
got a l3 relay failure while ending the run
<!-- Comment: Wed Aug 30 02;18;46 comment by...Alice -->
details of the l3 relay failure are below
<!-- Comment: Wed Aug 30 02;36;01 comment by...Alice -->
<pre>l3 did repair itself fine - it didn't get hung up on the l3 relay failure so I didn't see the need to cleanup l3 or the evb.</pre>
<!-- Date: Wed Aug 30 01:52:22.725 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222354>222354</A>
 Activated at 2006.08.30 01:52:08
<!-- Author: RunControl -->


<!-- Date: Wed Aug 30 01:52:23.756 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222354>222354</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_00 [4,820,642] 
<!-- Author: Alice -->
<!-- Comment: Wed Aug 30 01;54;56 comment by...Alice -->
<pre>I really don't know what just happened. We were having troubles getting SvxMon running and when I came back over from the CO desk we were no longer taking any events.  I got error messages that L2 and L3 rates had dropped to 0 Hz but nothing else looked weird and no strange messages in the error logger (ProcMon OK too).  I tried a couple of HRRs but that didn't help so I tried starting a new run.  That seems to have done the trick - maybe someone on the daq side can figure out what really happened later.</pre>


<!-- Comment: Wed Aug 30 02;02;01 comment by...Sasha Golossanov -->
<pre>
DAQ was paged.  Will take a look...

</pre>

<!-- Comment: Wed Aug 30 02;04;21 comment by...Alice -->
<pre>Comment on the SvxMon troubles - I think that concon may have been lying to us again and we weren't actually in physics mode.  So I stopped everything by hand and then sourced the physics tcl by hand and we were finally able to get SvxMon to run.  But that's just a guess.</pre>

<!-- Comment: Wed Aug 30 02;14;14 comment by...sasha golossanov -->
<pre>
donno about concon, but as far as the 0 Hz goes:

......
(MLE) b0svx07:Messenger:Aug 30, 2006 01:46:44 AM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; fa03: Bytes: 00780

(RUN Comment) rundb.RunComment@1e520d
  RunNumber   222353
  CreateTime  2006-08-30 01:46:44.0
  TextComment I have stopped taking events - going to try to see if new run helps
  Name        Alice
  State       TERMINATE
  ELog        true
CT: 2006.08.30 01:48:42
48'42" b0eproxy02.fnal.gov:main:Aug 30, 2006 01:48:42 AM-&gt;Error on L3 node b0eproxy02.fnal.gov (partition 0) Wed Aug 30 01:48:42 2006 l3proxy 
  in End (L3ProxyPartition.cc:3081)
  @L3_RELAY_FAILURE Level3 Relay failure detected
This message is mainly for your information - Level3 should be able to recover by itself
Just continue with your normal operations
The full list of Level3 nodes giving Relay trouble is:
 b0l3s08p15
These nodes will be marked OFFLINE in the database
Level3 will now update its configuration to be in sync with the database

 --&gt;  
 Additional Information: 

 Attention!!!. L3_RELAY_FAILURE Error !!!. 

 One or more L3 nodes is likely to have died.
 Do not panic - this message is mainly for your information.
 Level3 should be able to recover by itself.
 Please read the error log for more specific instructions.
[MLE]
(MLE) b0eproxy02.fnal.gov:main:Aug 30, 2006 01:48:42 AM-&gt;Error on L3 node b0eproxy02.fnal.gov (partition 0) Wed Aug 30 01:48:42 2006 l3proxy 
  in End (L3ProxyPartition.cc:2266)
  @L3_TRANS_FAIL - transition actions failed:      b0l3s08p15(Unrecognized response type for request)

(RC)   1:49:07 End -&gt; IDLE
(EOR) End of Run report
......

</pre>


<!-- Comment: Wed Aug 30 02;33;39 comment by...sasha golossanov -->
<pre>
we've been taking data for about 45 min and things look normal. so seems like ACE SOLVED THE PROBLEM.
done for now.

  



 </pre>


<!-- Comment: Wed Aug 30 02;40;45 comment by...ByeongRok Ko -->
For SVXMon,
I think the problem is on concon.
Just restarting concon resolved the problem 
(https://www-cdfonline.fnal.gov/internal/mon/consumer/howto.html).


<!-- Comment: Wed Aug 30 02;48;11 comment by...sasha golossanov -->
<pre>more e-log polution:

.........
(MLE) b0xft04:Messenger:Aug 30, 2006 01:38:41 AM-&gt;Event 814382: Bunchcounters in slot 11 (BC=74) and slot 9 (BC=65) disagree (L2B=1) Failure at read of XFTF flashram contents 
(MLE) b0xft04:Messenger:Aug 30, 2006 01:38:41 AM-&gt;Event 814382: Bunchcounters in slot 11 (BC=74) and slot 10 (BC=65) disagree (L2B=1) Failure at read of XFTF flashram contents 
(MLE) b0xft04:Messenger:Aug 30, 2006 01:38:41 AM-&gt;Runtime Error 1, Event 814382: Bunch counter mismatch, mismatch count = 15
(MLE) b0ccal01:Messenger:Aug 30, 2006 01:41:02 AM-&gt;Event 906789: Bunchcounters in slot 16 (BC=3) and slot 19 (BC=83) disagree (L2B=2) 
(MLE) b0ccal01:Messenger:Aug 30, 2006 01:41:02 AM-&gt;Runtime Error 1, Event 906789: Bunch counter mismatch, mismatch count = 1
(MLE) b0server03.fnal.gov:ConsumerErrorRe:Aug 30, 2006 01:41:04 AM-&gt;DAQErrMon:DataSyncError: Error 1 Event 906789 Run 222353 TMPD BC mismatchSlotMask 0x00080000 Block 1 Address 81 (b0ccal01.fnal.gov) 
(MLE) b0svx03:Messenger:Aug 30, 2006 01:43:49 AM-&gt;Silicon Resonator: Sl 18 Ch 9 -&gt; e394: Bytes: 00114

(MLE) b0svx08:Messenger:Aug 30, 2006 01:43:50 AM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; f843: Bytes: 00780

(MLE) b0svx07:Messenger:Aug 30, 2006 01:43:50 AM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; fa03: Bytes: 00780

(MLE) b0felix.fnal.gov:TEDCpp:Aug 30, 2006 01:43:51 AM-&gt;{ 1:43:51, 8/30/2006 }: L2-Pulsar-Linux: Halted while  waiting for ClusterFinder data
(RC)   1:43:58 Halt -&gt; HALTED
(RC)   1:44:05 Recover -&gt; RECOVERED
(RC)   1:44:13 Run -&gt; ACTIVE
(TS)   1:44:17 ********* L2 trigger rate has dropped to 0 Hz! Events are not being logged !
(CSL)  1:44:24 ********* L3 trigger rate has dropped to 0 Hz! Events are not being logged !
(MLE) b0svx03:Messenger:Aug 30, 2006 01:44:32 AM-&gt;Silicon Resonator: Sl 18 Ch 9 -&gt; e394: Bytes: 00114

(MLE) b0felix.fnal.gov:TEDCpp:Aug 30, 2006 01:44:36 AM-&gt;{ 1:44:36, 8/30/2006 }: L2-Pulsar-Linux: Halted while  waiting for ClusterFinder data
(MLE) b0svx08:Messenger:Aug 30, 2006 01:44:41 AM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; f843: Bytes: 00780

(MLE) b0svx07:Messenger:Aug 30, 2006 01:44:43 AM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; fa03: Bytes: 00780

(RC)   1:44:43 Halt -&gt; HALTED
(RC)   1:44:59 Recover -&gt; RECOVERED
(RC)   1:45:13 Run -&gt; ACTIVE
(MLE) b0svx03:Messenger:Aug 30, 2006 01:45:52 AM-&gt;Silicon Resonator: Sl 18 Ch 9 -&gt; e394: Bytes: 00114

(MLE) b0svx08:Messenger:Aug 30, 2006 01:45:52 AM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; f843: Bytes: 00780

(MLE) b0svx07:Messenger:Aug 30, 2006 01:45:52 AM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; fa03: Bytes: 00780

(MLE) b0felix.fnal.gov:TEDCpp:Aug 30, 2006 01:45:53 AM-&gt;{ 1:45:53, 8/30/2006 }: L2-Pulsar-Linux: Halted while  waiting for ClusterFinder data
(RC)   1:45:59 Halt -&gt; HALTED
(MLE) b0svx03:Messenger:Aug 30, 2006 01:46:38 AM-&gt;Silicon Resonator: Sl 18 Ch 9 -&gt; e394: Bytes: 00114

(MLE) b0svx08:Messenger:Aug 30, 2006 01:46:42 AM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; f843: Bytes: 00780

(MLE) b0svx07:Messenger:Aug 30, 2006 01:46:44 AM-&gt;Silicon Resonator: Sl 16 Ch 3 -&gt; fa03: Bytes: 00780

(RUN Comment) rundb.RunComment@1e520d
  RunNumber   222353
  CreateTime  2006-08-30 01:46:44.0
  TextComment I have stopped taking events - going to try to see if new run helps
.........

</pre>
<!-- Comment: Wed Aug 30 05;34;07 comment by...Alice -->
<pre>more elog abuse...

None of the xft errors are unfamiliar to me and I've never seen them mess up a run like this.

The L3 relay failure on the end transition was also not that abnormal and L3 repaired itself just fine.</pre>
<!-- Date: Wed Aug 30 02:40:56.770 2006 -->
ISL occasionally is flashing yellow in IFIX.  When checking the alarm details, it seems The LT-AIRSEP value is occasionally dropping to 16.9.
<!-- Author: Daryl :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222354' target=_top>222354</a>) -->
<!-- Comment: Wed Aug 30 03;04;46 comment by...Alice -->
<pre>Following the directions on the Si instructions for monitoring aces that states : 
"Note: A non-severe cooling problem can be signaled by a red (or yellow) tweety iFix alarm that is not accompanied by a sono-alarm or a Process Systems alarm. These are rare enough that we would like you to page the on-call cooling/interlock expert at 218-8626 so we can understand the problem. "
We are paging.  It has happened at least 4 times since our shift started.</pre>
<!-- Comment: Wed Aug 30 03;14;45 comment by...Ulrich -->
This warning tells you that the coolant level in the ISL is on the low side. The cryo techs will top off the coolant level tomorrow morning. Please keep an eye on the coolant level and page me, should any big jump (say by 1 unit) occur.
<!-- Comment: Wed Aug 30 03;14;54 comment by...Daryl -->
Silicon is aware.  Really nothing we can do.  Ulrich said they will try to add cooling to the system tomorrow sometime.  Just watch this and make sure it does not take any major drops.
<!-- Date: Wed Aug 30 02:48:04.208 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10068&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10068' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10069&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10069' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10070&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10070' width='300'></a></center>
<!-- Author: Daryl Hourly Silicon Plots -->
<!-- Comment: Wed Aug 30 02;48;47 comment by...Daryl -->
Sorry for the delay.  I also pushed these back 15 minutes to 1:15 to pick up from the start of store (around where the startup plots left off).



<!-- Date: Wed Aug 30 03:13:57.640 2006 -->
<pre>Got a procmon error for fcdfsgi1 tape.  I don't have to take immediate action on this.</pre>
<!-- Author: Alice :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222354' target=_top>222354</a>) -->
<!-- Comment: Wed Aug 30 07;50;44 comment by...Alice -->
<pre>I just checked it again and here is what the error says:

2006.08.30 07:49:37 Status: CHECKING
2006.08.30 07:49:43 Status: CHECK FAILED 
Wed Aug 30 07:49:39 CDT 2006   procAction  fcdfsgi1   check   b0control12
Checking fcdfsgi1
check file date: Wed Aug 30 07:10:01  (should be within 2 hours) 
Found 3 entries in 'encp status' (0 expected)
fcdfsgi1 check failed</pre>
<!-- Date: Wed Aug 30 03:36:42.902 2006 -->
<pre>Many exciting things have just happened: 
 
1)Got a done TO for boc0t03.  It didn't halt fast enough so I shepherded it. 
2)We got a trip on the TSU.  I did not get an inhibit.  Initally I halted but I figured it was OK to keep running while Daryl tries to fix it as there was no inhibit.  Daryl is downstairs. 
3)Got a done TO for pcal04.  One HRR didn't fix the problem and I had to shepherd the crate.</pre>
<!-- Author: Alice :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222354' target=_top>222354</a>) -->


<!-- Date: Wed Aug 30 03:40:22.304 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10071&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10071' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10072&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10072' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10073&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10073' width='300'></a></center>
<!-- Author: Alice Hourly Plots -->


<!-- Date: Wed Aug 30 04:04:17.764 2006 -->
IFix alarmed for a TSU trip.  Problem in SW-ISU-B.  Many channels flashing red.  Went downstairs to check the muon computer.  When I clicked on SW-ISU-B, a message came up saying that it was unreachable "please email expert and he will reset CCU."  I then tried the step 2 instructions and turned HV off, and then back on for the SW side.  I was then able to see SW-ISU-B.  Red channels continued to flash, so I tried the step 1 instructions of updating the CCU channels for SW-ISU-B.  I then came back upstairs.  The alarm still appears, but both bars are back to 100% on ifix, and Alice pointed out that it said an alarm can take half an hour to clear.  At this point, I presume everything is alright, since bars are back up (bar in TSU_hv_main.grf in IFix is also back to normal level).  Will wait for alarm to clear.
<!-- Author: Daryl :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222354' target=_top>222354</a>) -->
<!-- Comment: Wed Aug 30 05;22;00 comment by...Daryl -->
Problem has not seemed to resolve.  I went downstairs and tried again to update the individual flashing channels.  They complain of  "Wrong Threshold! High Voltage Deviation on this Channel is more than 30V!"
<p>
Sent an email to Oleg.  Perhaps I am wrong, but I do not think this is page worthy at 5 am.
<!-- Comment: Wed Aug 30 08;01;44 comment by...Dale Stentz -->
<br>
I had Oleg paged thinking that this was a real problem - of course I based this assumtion off of the TSU plot.  I had conviently forgotten that this was the gap where the TSU counters were removed.  <br><br>My intial guess/insintic is the same as my conclusion is now that the CCU was just having a read-back problem and than everything looks fine for ISU (TSU+BSU).  Went down and did a slighty differnt set of tricks then what Daryl attempted.  TSU is now in the green (no alarms state) with the min bar equal to ~100%.  This was an improvement on the summary page but I don't think anything changed on the CCU side.
<!-- Date: Wed Aug 30 04:06:00.505 2006 -->
<pre>Got  heap corrupt for b0pcal10 -&gt;shepherd</pre>
<!-- Author: Alice :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222354' target=_top>222354</a>) -->


<!-- Date: Wed Aug 30 04:34:18.891 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10074&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10074' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10076&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10076' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10075&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10075' width='300'></a></center>
<!-- Author: Daryl Hourly Silicon Plots -->


<!-- Date: Wed Aug 30 04:51:35.169 2006 -->
<pre>Done TO for b0pcal05 that needs a shepherd to fix</pre>
<!-- Author: Alice :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222354' target=_top>222354</a>) -->

<!-- Date: Wed Aug 30 04:57:55.832 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222354>222354</A>
 Terminated at 2006.08.30 04:56:34
<!-- Author: RunControl -->


<!-- Date: Wed Aug 30 04:57:56.256 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222354>222354</A>
TERMINATE: SVX daq problems - ending the run
<!-- Author: Alice -->
<!-- Comment: Wed Aug 30 05;01;48 comment by...Alice -->
<pre>b0fib05 would not go to halt on an hrr.  Last time this happened to me it starting a new run fixed it</pre>
<!-- Comment: Wed Aug 30 05;16;43 comment by...Alice -->
b0fib05 never turned red in RC... it just would not halt.
<!-- Date: Wed Aug 30 05:00:35.872 2006 -->
<table><tr><td><a href=/elog/notebooks/2006/Wed_Aug_30_2006_Owl.050033.ps align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-10078  align='top'></a></td><td> Several consecutive Red ladders</td></tr></table>
<!-- Author: ByeongRok Ko -->



<!-- Date: Wed Aug 30 05:04:36.732 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10079&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10079' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10080&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10080' width='300'></a></center>
<!-- Author: Bruno -->
<!-- Comment: Wed Aug 30 05;07;07 comment by...Bruno -->
Seen many pinkies in AVDD on L00 and ISL, they look all like plots above
<!-- Date: Wed Aug 30 05:06:44.528 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222355>222355</A>
TERMINATE: l3 wouldn't config
<!-- Author: Alice -->




<!-- Date: Wed Aug 30 05:09:45.361 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-10081&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-10081  width='300' align='top'></a></td><td>Snapshot of iMon in crisis</td></tr></table>
<!-- Author: Alice -->

<!-- Date: Wed Aug 30 05:10:58.239 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222354>222354</A>
trigger: <tt>	ReformatErrors:	304 ( 0.00%) [Hz] &nbsp;</tt>
<!-- Author: Consumer Operator( ByeongRok Ko) -->


<!-- Date: Wed Aug 30 05:11:56.212 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222356>222356</A>
 Activated at 2006.08.30 05:11:43
<!-- Author: RunControl -->


<!-- Date: Wed Aug 30 05:11:57.041 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222356>222356</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_00 [4,820,642] 
<!-- Author: Alice -->



<!-- Date: Wed Aug 30 05:14:17.894 2006 -->
<pre>We are running again (took a long time because L3 was greatly abused by the many manual overides I had to do to get out of the fib problem - I cleaned up l3 and the evb).  All of the pinkies went away on a global unmark.  Still waiting on SvxMon.</pre>
<!-- Author: Alice :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222356' target=_top>222356</a>) -->
<!-- Comment: Wed Aug 30 05;26;08 comment by...Alice -->
<pre>The big block of red cells is gone now in SvxMon.  Starting a new run seems to fix everything tonight.</pre>


<!-- Comment: Wed Aug 30 05;39;56 comment by...Tuula -->
Good job, Alice, on spotting that all the problems were FIB wide! Next time you could try if the problematic fib crate goes through config before stopping the run.
<!-- Comment: Wed Aug 30 06;23;41 comment by...Alice -->
<pre>Ahh...so the thing to do is to try to issue transitions to that fib crate in it's local client window?  i.e. you would do a partition, config, activate, halt and then recover and run with everything else?  I assume you don't need to reboot in this case.

Let's hope there is no next time for me as I have one shift left, but this has happened enough that it's probably worth clarifying what to do.</pre>
<!-- Date: Wed Aug 30 05:27:54.667 2006 -->
<table><tr><td><a href=/elog/notebooks/2006/Wed_Aug_30_2006_Owl.052749.ps align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-10083  align='top'></a></td><td>Update of the bad SvxMon slide</td></tr></table>
<!-- Author: Alice -->
<!-- Comment: Wed Aug 30 05;29;30 comment by...Alice -->
I'm worred about that yellow ladder in isl1


<!-- Comment: Wed Aug 30 05;36;54 comment by...Tuula -->
It could be that there are not yet enough statistics for this run. Please keep an eye on the yellow cell, and if it doesn't go away within half an hour, please page me.

<!-- Comment: Wed Aug 30 05;39;25 comment by...Alice -->
Actually it's gone now.


<!-- Date: Wed Aug 30 05:39:48.241 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10084&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10084' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10085&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10085' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10086&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10086' width='300'></a></center>
<!-- Author: Daryl Hourly Silicon Plots -->
<!-- Comment: Wed Aug 30 05;40;45 comment by...Daryl -->
Odd blip in B0PAGC coincided with small rise in AGIGI2.  Seems to be nothing.  The value has settled back down to where it was before.


<!-- Date: Wed Aug 30 05:48:56.259 2006 -->
<table><tr><td><a href=/elog/notebooks/2006/Wed_Aug_30_2006_Owl.054851.ps align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-10088  align='top'></a></td><td>One more missing wedge in SVT. 
SHOULD let expert know this. 
</td></tr></table>
<!-- Author: ByeongRok Ko -->
<!-- Comment: Wed Aug 30 06;02;45 comment by...ByeongRok Ko -->
But, too small statistics.
waiting for more data.


<!-- Date: Wed Aug 30 06:21:27.162 2006 -->
<table><tr><td><a href=/elog/notebooks/2006/Wed_Aug_30_2006_Owl.062125.ps align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-10092  align='top'></a></td><td>The b0W12 white block is gone now with more statistics</td></tr></table>
<!-- Author: Alice -->


<!-- Date: Wed Aug 30 06:41:54.118 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222356>222356</A>
 Terminated at 2006.08.30 06:41:45
<!-- Author: RunControl -->


<!-- Date: Wed Aug 30 06:42:17.737 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222356>222356</A>
TERMINATE: Run control problems again... let's try starting a new run (again)
<!-- Author: Alice -->



<!-- Date: Wed Aug 30 06:47:39.419 2006 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2006&action=view&page=-10095&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2006&action=view&page=-10095  width='300' align='top'></a></td><td>Bad things are happening in daq again.  We have no alarms on the monitoring side</td></tr></table>
<!-- Author: Alice -->
<!-- Comment: Wed Aug 30 06;48;07 comment by...Alice -->
<pre>Needless to say we have paged daq.  I have no idea what to do.</pre>


<!-- Comment: Wed Aug 30 07;00;45 comment by...Alice -->
<pre>I started shepherding everything that I could and got up to the cmx crates when everything just turned green as suddenly as it went bad.  Maybe daq pager did some magic?  </pre>
<!-- Comment: Wed Aug 30 07;16;11 comment by...Alice -->
<pre>I don't think this was the same problem as earlier because the first warning I got was about a calorimeter crate becoming unresponsive before everything died whereas earlier this morning the first warning I got was "L2 trigger rate has dropped to 0 Hz"

</pre>
<!-- Comment: Wed Aug 30 07;56;04 comment by...sasha golossanov -->
<pre>
attempted to reset the "red" crates quickly, but most of them
turned green by themselves.  appears that system fixed itself.
not sure what the problem was.

</pre>
<!-- Date: Wed Aug 30 06:51:50.248 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10099&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10099' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10100&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10100' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10101&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10101' width='300'></a></center>
<!-- Author: Daryl Hourly Silicon Plots -->

<!-- Date: Wed Aug 30 07:00:09.999 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222357>222357</A>
 Activated at 2006.08.30 06:59:59
<!-- Author: RunControl -->


<!-- Date: Wed Aug 30 07:00:40.039 2006 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222357>222357</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_4_00 [4,820,642] 
<!-- Author: Alice -->




<!-- Date: Wed Aug 30 07:05:56.912 2006 -->
<pre>Our L1A is not normal! Right now it is about 6 kHz.  Waiting for a prescale to kick in and fix everything.</pre>
<!-- Author: Alice :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222357' target=_top>222357</a>) -->
<!-- Comment: Wed Aug 30 07;07;18 comment by...Alice -->
<pre>Prescale just changed right now our rates are 15kHz for L1A and ~217 for L2A.</pre>



<!-- Date: Wed Aug 30 07:19:31.076 2006 -->
<pre>I got at daq: 
 
Trigger Inhibit is set from the components: 
 PCAL12 Trip  		[027] Plug Calorimeter Trip 11 
 
None of the LEDs were on upstairs.  We cycled the plug between standby and on and now we are running again.</pre>
<!-- Author: Alice :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=222357' target=_top>222357</a>) -->


<!-- Date: Wed Aug 30 07:48:50.408 2006 -->
<center><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10102&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10102' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10103&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10103' width='300'></a><a href='/cgi/elog/elog.pl?nb=2006&action=view&page=-10104&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2006&action=view&page=-10104' width='300'></a></center>
<!-- Author: Daryl Hourly Silicon Plots -->


<!-- Date: Wed Aug 30 07:55:19.201 2006 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=222353 target=_runSummary>222353</A> x36491</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [4,820,642]</FONT></TD>
<TD ALIGN=CENTER>01:14:47</TD>
<TD ALIGN=CENTER>01:46:26</TD>
<TD ALIGN=CENTER>00:24:36</TD>
<TD ALIGN=RIGHT>     174,227</TD>   <TD ALIGN=RIGHT>    294.36</TD>
<TD ALIGN=RIGHT>    227.64</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=222354 target=_runSummary>222354</A> x36492</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [4,820,642]</FONT></TD>
<TD ALIGN=CENTER>01:52:08</TD>
<TD ALIGN=CENTER>04:56:34</TD>
<TD ALIGN=CENTER>02:40:49</TD>
<TD ALIGN=RIGHT>   1,121,182</TD>   <TD ALIGN=RIGHT>  1,314.27</TD>
<TD ALIGN=RIGHT>  1,143.96</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=222356 target=_runSummary>222356</A> x36494</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [4,820,642]</FONT></TD>
<TD ALIGN=CENTER>05:11:43</TD>
<TD ALIGN=CENTER>06:41:45</TD>
<TD ALIGN=CENTER>01:24:17</TD>
<TD ALIGN=RIGHT>     493,657</TD>   <TD ALIGN=RIGHT>    490.19</TD>
<TD ALIGN=RIGHT>    455.08</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=222357 target=_runSummary>222357</A> x36495</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_4_00 [4,820,642]</FONT></TD>
<TD ALIGN=CENTER>06:59:59</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>00:44:44</TD>
<TD ALIGN=RIGHT>     241,432</TD>   <TD ALIGN=RIGHT>    261.42</TD>
<TD ALIGN=RIGHT>    216.70</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>07:55:01</TD>
<TD ALIGN=CENTER>05:14:27</TD>
<TD ALIGN=RIGHT>     2,030,498</TD> <TD ALIGN=RIGHT>  2,360.23</TD>
<TD ALIGN=RIGHT>  2,043.39</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->




<!-- Date: Wed Aug 30 07:56:58.780 2006 -->
<font color=red>Shift Summary: </font><pre>Owl shift summary: 
================== 
Shift started just before shot setup. 
 
Store 4930 
0:16 Proton loaded 
0:51 Antiproton Loaded 
1:10 Scraping completed 
1:15 start taking data, CDF lumi: 164E30 
  
Quite busy night.  
Several trips and several DAQ problems. DAQ pager called. 
Problems solved starting new run.</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   222353,222354,222356,222357
Delivered Luminosity   2564.3 nb-1  
Acquired Luminosity    2016.6 nb-1  
Efficiency              78.6
   
</td></tr></table></pre>
<!-- Author: Donatella -->