<!-- Title: Eve shift. Thu Nov 22, 2007 -->
<!-- SciCo: Allan Clark -->
<!-- DAQAce: Donatella Torretta +Sam Hewamanage -->
<!-- MonAce: &nbsp; -->
<!-- CO: Yuri Oksuzian -->
<!-- OpManager: Massimo Casarsa -->
<!-- Notes: -->
<pre>Inherit shift with stable beams, store 5743 run 254121 
TeVPR= 6060E09 TeVPB= 2430E09  LUMB0= 146.39E30 
LostPR= 9KHZ LostPB=5 KHZ</pre>

<!-- Date: Thu Nov 22 16:14:20.466 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254086>254086</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
L1T 
RSF 
XFT needs rechecking 
</PRE>
<!-- Author: cdfscico -->


<!-- Date: Thu Nov 22 16:17:14.053 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254038>254038</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
L1T 
RSF 
XFT needs checking
</PRE>
<!-- Author: cdfscico -->


<!-- Date: Thu Nov 22 16:18:24.390 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254040>254040</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
L1T 
RSF 
XFT needs checking
</PRE>
<!-- Author: cdfscico -->


<!-- Date: Thu Nov 22 16:18:58.358 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254044>254044</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
L1T 
RSF 
XFT needs checking
</PRE>
<!-- Author: cdfscico -->



<!-- Date: Thu Nov 22 16:19:35.878 2007 -->
Diagnose timeout from EVB. auto hrr fixed it.
<!-- Author: Sam & D :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->
<!-- Comment: Thu Nov 22 16;20;03 comment by...Sam -->
i meant a busy time out...

<!-- Comment: Thu Nov 22 16;21;36 comment by...Sam -->
again

<!-- Comment: Thu Nov 22 16;24;39 comment by...Sam & D -->
again. we did a diagnose in timeouts from EVB monitor. it did not report any load or byte errors!
<!-- Comment: Thu Nov 22 16;25;48 comment by...Sam & D -->
again. no errors reported by the monitor
<!-- Comment: Thu Nov 22 16;37;18 comment by...Sam -->
we are gettting these busy errros almost every minute now. like yesterday. but no errors shown in the EVB control. can some expert comment on this... so we dont have to constantly worry about it...
<!-- Comment: Thu Nov 22 16;58;46 comment by...Sam -->
paged Georgios. he is looking in to it.
<br>
we also checked with Allison if the new L2 config got anything to do with this. She does not think so.
<!-- Comment: Thu Nov 22 18;28;59 comment by...Georgios & Si -->
We see this when L2 accept rate exceeds 600Hz.  It's not probably a coincidence that we have only 12 subfarms in the run. 12/18*900=600.  So, probably L3 has hit its capacity with 12 subfarms.  The problem goes away the moment L2 accept gets below 600Hz.

We will prepare two more subfarms (s17, s18) for inclusion, and they should be included next time you restart the run.  Now, luminosity got low so we are out of the woods, but when next store comes, we may hit the ceiling again.  With 14 subfarms, the ceiling will be at 700Hz.
<!-- Date: Thu Nov 22 16:24:07.031 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254047>254047</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
L1T 
COT 
SVX 
ISL 
L00 
SVT 
RSF 
COT SL4 Quadraaant 
No SVX
Need to check XFT
</PRE>
<!-- Author: cdfscico -->


<!-- Date: Thu Nov 22 16:24:39.636 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254087>254087</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
L1T 
RSF 
Check XFT
</PRE>
<!-- Author: cdfscico -->



<!-- Date: Thu Nov 22 16:25:45.701 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254088>254088</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
L1T 
RSF 
Check XFT
</PRE>
<!-- Author: cdfscico -->



<!-- Date: Thu Nov 22 16:27:01.585 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254089>254089</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
L1T 
RSF 
Check XFT
</PRE>
<!-- Author: cdfscico -->



<!-- Date: Thu Nov 22 16:28:01.836 2007 -->
L2-decision time out. hrr fixed it.
<!-- Author: Sam & D :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->

<!-- Date: Thu Nov 22 16:29:11.671 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254090>254090</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
L1T 
COT 
SVX 
ISL 
L00 
SVT 
RSF 
COT SL4 quadrant off
SVX off
Check XFT
</PRE>
<!-- Author: cdfscico -->


<!-- Date: Thu Nov 22 16:31:20.633 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254120>254120</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
L1T 
COT 
SVX 
ISL 
L00 
SVT 
RSF 
COT SL4 quadrant off
SVX off
Check XFT
</PRE>
<!-- Author: cdfscico -->



<!-- Date: Thu Nov 22 16:41:41.831 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121>254121</A>
trigger: <tt>XFT needs to be checked</tt>
<!-- Author: Consumer Operator( Yuri Oksuzian) -->



<!-- Date: Thu Nov 22 16:46:20.347 2007 -->
 
 Attention!!!. FERML_HIGH_DEADTIME !!!  
 - Trigger/DAQ Deadtime is too high (at 55.842846 %). 
<!-- Author: Sam & D. :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->


<!-- Date: Thu Nov 22 16:47:31.138 2007 -->
  L3 trigger rate has dropped to 0 Hz!!!  
     Events are not being logged !!!
<!-- Author: Sam & D. :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->


<!-- Date: Thu Nov 22 16:50:08.719 2007 -->
did manual HRR . Deatime down to 8%
<!-- Author: Sam & D. :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->



<!-- Date: Thu Nov 22 17:05:50.553 2007 -->
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  
  
 Stuck Cellid I/B3/W0/L0/C0-3 .  
 AUTO HRR will be issued  -- and ok.
<!-- Author: Sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->


<!-- Date: Thu Nov 22 17:07:28.589 2007 -->
L2 decision time out. hrr fixed it.
<!-- Author: Sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->


<!-- Date: Thu Nov 22 17:08:17.804 2007 -->
Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!!  
 - Timeout on global TDC Done  for crate b0cot06    
 <br> 
and 
<br> 
 Done TO. for crate/s:  
    b0cot06    
<br>auto hrr fixed it.
<!-- Author: Sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->


<!-- Date: Thu Nov 22 17:26:13.298 2007 -->
*** Node b0fib06 in Error *** 
Last data more than 120s ago!  Node is not responsive 
<br> 
we halt the run. 
ping did not wor. reset/shepherd it. 
no luck. FER_smartInitMS: Could not connect to RTserver! 
 
 
so we paged the silicon experts.
<!-- Author: Sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->
<!-- Comment: Thu Nov 22 17;37;16 comment by...Sam & D -->
paged expert. we did a manual reboot and then another shepherd put it back on line.


<!-- Comment: Thu Nov 22 18;11;22 comment by...SashaS -->
Good job, Donatella and Sam! Thanks!
<!-- Date: Thu Nov 22 17:55:17.839 2007 -->
L2 decision time out. auto hrr to save.
<!-- Author: Sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->



<!-- Date: Thu Nov 22 18:12:09.188 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-7014&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-7014  width='300' align='top'></a></td><td>Pinkie bias in S_BIAS_B4W9L4</td></tr></table>
<!-- Author: Yuri and Allan -->



<!-- Date: Thu Nov 22 18:29:00.197 2007 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2007&action=view&page=-7015&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2007&action=view&page=-7015  width='300' align='top'></a></td><td>Another pinkie (bias)  - I_VBIAS_B4W1L4</td></tr></table>
<!-- Author: Yuri and Allan -->


<!-- Date: Thu Nov 22 18:39:17.297 2007 -->
L2 Decision Timeout ERROR !!. auto hrr recover.
<!-- Author: Sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->


<!-- Date: Thu Nov 22 19:05:24.136 2007 -->
Trigger Inhibit is set from the components: 
 PLUG HV  		[138] Plug High Voltage from iFix 
 
 
<br>  
then 
PEM and PSH tripped. 
 
we followed instruction and power cycled. we are back online.
<!-- Author: Sam & D :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->


<!-- Date: Thu Nov 22 19:12:46.221 2007 -->
Sebastian calls. 
He is unable to access the CDF e-log from outside Fermilab. 
Yuri calls Sasha to see if he has the same problem.  
Sasha is able to log in. So we do njothing for now.  
  
<!-- Author: Scico :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->
<!-- Comment: Thu Nov 22 19;41;45 comment by...sebastian -->
thanks, now it's working for me too.
<!-- Date: Thu Nov 22 19:22:28.537 2007 -->
 TOF01 HV Trip . power cycled. and we are up.
<!-- Author: Sam & D :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->


<!-- Date: Thu Nov 22 19:34:43.696 2007 -->
Done TO. for crate/s:  
    b0pcal02    
<br> hrr recovered.
<!-- Author: Sam & D :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->


<!-- Date: Thu Nov 22 19:40:29.824 2007 -->
Done TO. for crate/s:  
    b0pcal01   --- auto hrr recovered.
<!-- Author: Sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->
<!-- Comment: Thu Nov 22 19;44;31 comment by...Sam & D -->
spoke too soon. it did not respond to after auto hrr. we had reshephered it to recover.


<!-- Date: Thu Nov 22 19:54:42.190 2007 -->
L2 Decision Timeout ERROR !!. -- auto hrr recovery.
<!-- Author: Sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->


<!-- Date: Thu Nov 22 19:56:11.280 2007 -->
Happy Thanksgiving!! :-)
<!-- Author: Jennifer :-) -->


<!-- Date: Thu Nov 22 20:04:33.119 2007 -->
TOMCAT was not responding. restared it.
<!-- Author: Sam & D :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->


<!-- Date: Thu Nov 22 21:19:19.510 2007 -->
Done TO. for crate/s:  
    b0pcal04    
--did not respond during auto hrr. needed two shepherdings to bring it back up.
<!-- Author: Sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->


<!-- Date: Thu Nov 22 21:23:12.752 2007 -->
ICICLE hb. we killed it and restarted.
<!-- Author: Sam  :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->
<!-- Comment: Thu Nov 22 21;54;09 comment by...Sam & D. -->
Icicle heartbeat again. And Procmon error. Killed/restarted Icicle.
Procmon cleared but hb still there...
<!-- Comment: Thu Nov 22 22;00;49 comment by...Sam & D. -->
finally hb cleared
<!-- Comment: Thu Nov 22 22;07;52 comment by...Sam & D. -->
Another Icicle error. For the records: it looks like it can't get pass "read SVX data".
<!-- Date: Thu Nov 22 21:36:22.901 2007 -->
L2 Decision Timeout ERROR !!.  
 -- auto hrr recovered
<!-- Author: Sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->
<!-- Comment: Thu Nov 22 21;54;58 comment by...Sam & D. -->
L2 Decision Timeot ERROR!! -- auto-hrr recovered 




<!-- Date: Thu Nov 22 22:20:34.057 2007 -->
SVX & ISL ps10 tripped. auto hock did not work. we recovered slowly following the instruction on the desk. and we are back online. we also got few pinkies which we are looking into now. 
<!-- Author: Sam & D :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->
<!-- Comment: Thu Nov 22 22;24;32 comment by...Scico -->
The pinkies were associated to the bias, and probably the result of the trip


<!-- Date: Thu Nov 22 22:26:32.962 2007 -->
Ops Manager (Massimo) called 
Tom Wright has requested a test with beam of the trigger.  
Both Massimo and Sacha (silicon LRP) are coming in for this. 
<!-- Author: Scico :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->


<!-- Date: Thu Nov 22 22:29:51.450 2007 -->
Call from JJ. 
a) He asked Scico to transfer shift info to the new Scico 
   (Joe Incandela) - will do this 
b) He informed us that the CO for OWL shift is expected to  
   be calling in from Pisa. Yuri and myself are asked to 
   familiarise ourselves with the Polycom etc. - we will  
   do this. 
<!-- Author: Scico :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->
<!-- Comment: Thu Nov 22 22;39;35 comment by...jj -->
Short version to get Polycom setup:<br>
<ul>
<li> Move window for ConCon to another part of CO console screen.
<li> Use buttons on top monitor to enable PIP (picture in picture)
     with svideo input.
<li> Flip little button switch on the Polycom V500 to open
     the lens cover.
<li> Pick up the remote for the Polycom and see if display
     appears in the PIP window.
<li> Find the "mute" button on the remote control and unmute
     the microphone.
<li> Wait for Remote CO to call you on the Polycom video system.
</ul>
<!-- Comment: Thu Nov 22 22;44;16 comment by...jj -->
<img src="http://www-cdfonline.fnal.gov/ops/cohelp/remote_co/v500_overview.jpg"><br>
<br><br>
<img src="http://www-cdfonline.fnal.gov/ops/cohelp/remote_co/v500_details.jpg">
<!-- Comment: Thu Nov 22 22;49;12 comment by...jj -->
<img src="http://www-cdfonline.fnal.gov/ops/cohelp/remote_co/dell_2001fp.jpg"><br>
<br>
Buttons 3,4,5 are the one to use for working thru menus to enable PIP.
<!-- Comment: Thu Nov 22 22;49;58 comment by...oops -->
"ones to use"
<!-- Date: Thu Nov 22 22:34:31.343 2007 -->
L2 Decision Timeout ERROR !!. -- auto hrr recovered.
<!-- Author: Sam & D :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121' target=_top>254121</a>) -->


<!-- Date: Thu Nov 22 22:43:55.886 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121>254121</A>
 Terminated at 2007.11.22 22:42:18
<!-- Author: RunControl -->


<!-- Date: Thu Nov 22 22:43:56.008 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121>254121</A>
TERMINATE: End run to test new physics table.
<!-- Author: franceso/tino -->



<!-- Date: Thu Nov 22 22:48:17.737 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254122>254122</A>
TERMINATE: need to exclude a subfarm 16.
<!-- Author: franceso/tino -->




<!-- Date: Thu Nov 22 22:52:07.251 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254123>254123</A>
 Activated at 2007.11.22 22:51:05
<!-- Author: RunControl -->


<!-- Date: Thu Nov 22 22:52:09.977 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254123>254123</A>
ACTIVATE: RunSet: AAA_NOSILICON  Trigger: PHYSICS_TEST_5_01 [7,1006,712] 
<!-- Author: franceso/tino -->


<!-- Date: Thu Nov 22 22:54:54.192 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121>254121</A>
trigger: <tt>XFT needs recheking</tt>
<!-- Author: Consumer Operator( Yuri Oksuzian) -->



<!-- Date: Thu Nov 22 22:55:08.431 2007 -->
L2 time outs. several hrr's did not fix it. we are getting 100% dead time.
<!-- Author: Sam & D :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254123' target=_top>254123</a>) -->

<!-- Date: Thu Nov 22 22:56:56.463 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254123>254123</A>
 Terminated at 2007.11.22 22:56:13
<!-- Author: RunControl -->


<!-- Date: Thu Nov 22 22:56:57.847 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254123>254123</A>
TERMINATE: too many L2 Decision Timeout.
<!-- Author: franceso/tino -->
<!-- Comment: Thu Nov 22 22;58;05 comment by...Sam & D. -->
<pre>here's the error message:(
MLE) b0felix.fnal.gov:TEDCpp:Nov 22, 2007 10:52:58 PM-&gt;{ 22:52:58, 11/22/2007 }: L2-Pulsar-Linux: Control node is unable to activate decision node #0 at address 192.168.0.10
</pre>

<!-- Date: Thu Nov 22 22:59:24.147 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254125>254125</A>
 Activated at 2007.11.22 22:59:13
<!-- Author: RunControl -->


<!-- Date: Thu Nov 22 22:59:54.087 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254125>254125</A>
ACTIVATE: RunSet: AAA_NOSILICON  Trigger: PHYSICS_TEST_5_01 [7,1006,712] testing new trigger table
<!-- Author: franceso/tino -->


<!-- Date: Thu Nov 22 22:59:55.998 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254125>254125</A>
 Terminated at 2007.11.22 22:59:52
<!-- Author: RunControl -->


<!-- Date: Thu Nov 22 23:00:11.241 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254125>254125</A>
TERMINATE: too many l2 tos
<!-- Author: franceso/tino -->


<!-- Date: Thu Nov 22 23:01:31.299 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121>254121</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
L1T 
COT 
RSF 
Please check XFT
</PRE>
<!-- Author: cdfscico -->


<!-- Date: Thu Nov 22 23:03:35.684 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254126>254126</A>
 Activated at 2007.11.22 23:03:02
<!-- Author: RunControl -->


<!-- Date: Thu Nov 22 23:03:36.546 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254126>254126</A>
ACTIVATE: RunSet: AAA_NOSILICON  Trigger: PHYSICS_5_01 [2,1002,710] testing new physics table. 
<!-- Author: franceso/tino -->
<!-- Comment: Thu Nov 22 23;04;54 comment by...Sam & D. -->
correction: we are back to the default table to see if L2 is OK. It is. No L2 Decision Timeouts right now.

<!-- Date: Thu Nov 22 23:10:12.500 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254121>254121</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
RSF 
</PRE>
<!-- Author: cdfscico -->



<!-- Date: Thu Nov 22 23:16:54.937 2007 -->
L2 Decision Timeout ERROR. auto hrr recovered,
<!-- Author: Sam :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254126' target=_top>254126</a>) -->

<!-- Date: Thu Nov 22 23:21:07.155 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254126>254126</A>
 Terminated at 2007.11.22 23:20:42
<!-- Author: RunControl -->


<!-- Date: Thu Nov 22 23:21:55.036 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254126>254126</A>
TERMINATE: going back to old cofig. 
<!-- Author: franceso/tino -->
<!-- Comment: Thu Nov 22 23;23;58 comment by...Sam & D. -->
Andrews checked the L2 e-logs and said the new L2 exe crashed.

<!-- Comment: Thu Nov 22 23;26;48 comment by...Sam & D. -->
so the test of the new physics table was un-successful.
<!-- Comment: Thu Nov 22 23;29;38 comment by...twright -->
OK, we'll have to debug it next week when everyone is back in town.

<!-- Date: Thu Nov 22 23:25:02.445 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254127>254127</A>
 Activated at 2007.11.22 23:24:57
<!-- Author: RunControl -->


<!-- Date: Thu Nov 22 23:26:14.144 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254127>254127</A>
ACTIVATE: RunSet: AAA_CURRENT_L2UP  Trigger: PHYSICS_5_01 [2,1002,710] 
<!-- Author: Sam & Donatella -->





<!-- Date: Thu Nov 22 23:30:49.510 2007 -->
<pre>Whenever we restart the run. TrigMon gets stuck with the error message: 
*** glibc detected *** double free or corruption (out): 0x88f35008 *** 
XFTSys::~XFTSys 
DisplayServer::pollClients(): request = --- (probably socket not valid). 
%ERLOG?? : 
 
Any ideas why?</pre>
<!-- Author: Yuri :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254127' target=_top>254127</a>) -->
<!-- Comment: Thu Nov 22 23;34;56 comment by...Yuri -->
It takes about 5 mins to restart it. It would be nice to fix that


<!-- Date: Thu Nov 22 23:53:21.456 2007 -->
RXPT Error for crate/s:  
    b0svx02    
<br> 
we did a hrr then <br> 
 
Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatality Error for b0svx02: Sl 6 Resonance Detected  
  
 - HALT the run      
 - record any relevant information in e-log      
 - RECOVER - RUN      
  
<!-- Author: Sam & D :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254127' target=_top>254127</a>) -->

<!-- Date: Thu Nov 22 23:54:06.542 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254127>254127</A>
 Terminated at 2007.11.22 23:53:39
<!-- Author: RunControl -->


<!-- Date: Thu Nov 22 23:54:07.343 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254127>254127</A>
TERMINATE: b0svx02 errors.
<!-- Author: Sam/Donatella -->


<!-- Date: Thu Nov 22 23:55:13.964 2007 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=254121 target=_runSummary>254121</A> x3E0A9</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [2,1002,710]</FONT></TD>
<TD ALIGN=CENTER>15:53:04</TD>
<TD ALIGN=CENTER>22:42:18</TD>
<TD ALIGN=CENTER>05:25:44</TD>
<TD ALIGN=RIGHT>   2,445,075</TD>   <TD ALIGN=RIGHT>  2,490.91</TD>
<TD ALIGN=RIGHT>  1,961.58</TD>
<TD ALIGN=CENTER>0</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=254126 target=_runSummary>254126</A> x3E0AE</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [2,1002,710]</FONT></TD>
<TD ALIGN=CENTER>23:03:02</TD>
<TD ALIGN=CENTER>23:20:42</TD>
<TD ALIGN=CENTER>00:16:58</TD>
<TD ALIGN=RIGHT>      53,123</TD>   <TD ALIGN=RIGHT>     76.42</TD>
<TD ALIGN=RIGHT>     71.94</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=254127 target=_runSummary>254127</A> x3E0AF</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [2,1002,710]</FONT></TD>
<TD ALIGN=CENTER>23:24:57</TD>
<TD ALIGN=CENTER>23:53:39</TD>
<TD ALIGN=CENTER>00:24:46</TD>
<TD ALIGN=RIGHT>     113,276</TD>   <TD ALIGN=RIGHT>    119.49</TD>
<TD ALIGN=RIGHT>    101.75</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>23:55:01</TD>
<TD ALIGN=CENTER>06:07:28</TD>
<TD ALIGN=RIGHT>     2,611,474</TD> <TD ALIGN=RIGHT>  2,686.81</TD>
<TD ALIGN=RIGHT>  2,135.26</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->


<!-- Date: Thu Nov 22 23:57:12.218 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254128>254128</A>
 Activated at 2007.11.22 23:57:02
<!-- Author: RunControl -->


<!-- Date: Thu Nov 22 23:57:36.799 2007 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254128>254128</A>
ACTIVATE: RunSet: AAA_CURRENT_L2UP  Trigger: PHYSICS_5_01 [2,1002,710] 
<!-- Author: Sam/Donatella -->



<!-- Date: Thu Nov 22 23:57:44.056 2007 -->
<font color=red>Shift Summary: </font><pre>Inherited stable store 5743, run 254121 
 
Generally calm 
- L3 problems when L2 rate exceeded 600 Hz. Thought to be 
  related to L3 capacity. Not a concern for the luminosity 
  of this store, but should use more sub-farms next store. 
 
- Several silicon bias pinkies, logged. 
 
- Had a silicon ISL and SVX HV trip. Restored OK 
  LRP from silicon informed 
 
- Special trigger being tested (Massimo, Sasha, Tom Wright) 
 
- Informed by JJ that OWL CO will be remote (Pisa) 
  Pisa have now logged in and transfer of CO 
 
- Trigger table study - not successful. Return to test of 
  XFT stereo (Alison) 
 
- Note foreseen controlled access tomorrow at 6am, store ends 
  at 05:30. 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   254121, 254126, 254127
Delivered Luminosity   2727.8  
Acquired Luminosity    2077.2  
Efficiency             76.1
   
</td></tr></table></pre>
<!-- Author: Allan Clark -->


<!-- Date: Fri Nov 23 00:06:08.872 2007 -->
Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  
  
 Pipeline out of synch in 75 silicon readout chips .  
 HRR is STRONGLY RECOMMENDED.  
 
Auto HRR was disabled. we did HRR and let Auto HRR abled.
<!-- Author: Eunsin / Joachim :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=254128' target=_top>254128</a>) -->