<!-- Title: Eve shift. Tue Jan  8, 2008 -->
<!-- SciCo: Chris Neu -->
<!-- DAQAce: Pedro TTito -->
<!-- MonAce: &nbsp; -->
<!-- CO: Georgios Choudalakis -->
<!-- OpManager: Farrukh Azfar -->
<!-- Notes: -->
<pre>In shot setup for store 5835, close to final protons 
 
Run long and hard...</pre>


<!-- Date: Tue Jan 8 16:12:33.291 2008 -->
<pre>Final protons loaded.  </pre>
<!-- Author: neu -->

<!-- Date: Tue Jan 8 16:19:27.561 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255926>255926</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_5_01 [3,1016,713] 
<!-- Author: Si Xie -->



<!-- Date: Tue Jan 8 16:20:45.646 2008 -->
SVX and PSM are red in Global alarms
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255926' target=_top>255926</a>) -->
<!-- Comment: Tue Jan  8 16;48;29 comment by...Pedro -->
BOth are green now
<!-- Date: Tue Jan 8 16:27:14.074 2008 -->
<P> 
It turns out that the TDCs in b0clc00 CLC_00 were  
considered to be special in the L3DaqErrorFilter code,  
and the TMPD bunch counter check was skipped, while  
the CLTD bank check subtracted one crossing from the bunch  
counter before doing the check. 
<P> 
When we try to run the CLTD bank in the new crate b0clc02  
CLC_02, the TMPD check was no longer disabled, so it appeared  
as if the TMPD and CLTD bunch counters differred by one,  
but this difference was entirely artificial.   
<P> 
The L3DaqErrorFilter runs at Level 3 (as well as in the  
DAQErrMon consumer) so it is difficult to change.   So I  
propose to temporarily disable the TMPD bunch counter check  
by putting a flag in the TMPD bank for b0clc02 CLC_02. 
Then once the switch has been completed, and the L3 code  
is updated, the check can be re&euml;nabled in the front  
end code.    Since the new crate has only TDCs, and no  
ADMEMs, there is no reason to do the special kludged checking. 
<P> 
 
  
<!-- Author: badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255926' target=_top>255926</a>) -->


<!-- Date: Tue Jan 8 16:30:38.278 2008 -->
As an attempt to fix the EVB Busy TO issue, we have rebooted scpu11 with the slow module driver loaded. This will cause higher deadtime at high luminosity, but will be more reliable and as a result we will not get the EVB event fragment corruption, and thus we will not get BUSY Timeouts (we hope). If for some reason this slow module does not work, or we get unacceptably high deadtime, we can switch back to the fast module by simply rebooting scpu 11. The device drivers are set up such that you can pick up the old default one just by rebooting that scpu. 
<!-- Author: Si :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255926' target=_top>255926</a>) -->


<!-- Date: Tue Jan 8 16:41:57.143 2008 -->
<pre>Antiprotons are loaded.</pre>
<!-- Author: neu -->


<!-- Date: Tue Jan 8 16:45:43.707 2008 -->
Silicon integrated dose alarm.
<!-- Author: neu -->



<!-- Date: Tue Jan 8 17:00:53.513 2008 -->
<pre>Store 5835  
TEV PR    8718.69E09 
TEV PB    3183.51E09 
C:LOSTP   12054 Hz 
C:LOSTPB  9100 Hz 
C:B0ILUM  267E30 
</pre>
<!-- Author: neu :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255931' target=_top>255931</a>) -->

<!-- Date: Tue Jan 8 17:03:55.547 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255931>255931</A>
 Activated at 2008.01.08 17:03:34
<!-- Author: RunControl -->


<!-- Date: Tue Jan 8 17:04:49.112 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255931>255931</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [3,1016,713] 
<!-- Author: Pedro -->



<!-- Date: Tue Jan 8 17:07:09.214 2008 -->
Done TO for crate b0clc00. Auto HRR recovered it
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255931' target=_top>255931</a>) -->
<!-- Comment: Tue Jan  8 17;07;58 comment by...Pedro -->
Again


<!-- Comment: Tue Jan  8 17;12;22 comment by...Pedro -->
Again
<!-- Comment: Tue Jan  8 17;14;30 comment by...Pedro -->
Again
<!-- Comment: Tue Jan  8 17;29;58 comment by...Pedro -->
Again
<!-- Comment: Tue Jan  8 17;31;14 comment by...Pedro -->
Again
<!-- Comment: Tue Jan  8 17;41;13 comment by...Pedro -->
Again
<!-- Comment: Tue Jan  8 17;41;38 comment by...Pedro -->
Again
<!-- Comment: Tue Jan  8 17;53;28 comment by...Pedro -->
Again
<!-- Comment: Tue Jan  8 17;53;48 comment by...Pedro -->
Again
<!-- Comment: Tue Jan  8 17;54;08 comment by...Pedro -->
Again
<!-- Comment: Tue Jan  8 17;54;58 comment by...Pedro -->
Again
<!-- Comment: Tue Jan  8 18;22;14 comment by...Pedro -->
This was happening many times, so we shepherded the crate and it seem to be fine now.
<!-- Date: Tue Jan 8 17:10:53.986 2008 -->
L2 Decision Timeout. Auto HRR fixed it.
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255931' target=_top>255931</a>) -->
<!-- Comment: Tue Jan  8 17;14;02 comment by...Pedro -->
Again
<!-- Comment: Tue Jan  8 17;54;29 comment by...Pedro -->
Again
<!-- Comment: Tue Jan  8 18;23;29 comment by...Pedro -->
Again
<!-- Comment: Tue Jan  8 19;53;04 comment by...Pedro -->
Again
<!-- Comment: Tue Jan  8 19;58;51 comment by...Pedro -->
Again
<!-- Comment: Tue Jan  8 21;35;48 comment by...Pedro -->
Again
<!-- Comment: Tue Jan  8 21;54;43 comment by...Pedro -->
Again
<!-- Comment: Tue Jan  8 23;24;12 comment by...Pedro -->
Again
<!-- Date: Tue Jan 8 17:13:35.394 2008 -->
RXPT Error for crate b0svt00. Auto HRR recovered it
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255931' target=_top>255931</a>) -->
<!-- Comment: Tue Jan  8 17;29;25 comment by...Pedro -->
<pre>We got a chain of problems:

  Done TO. for crate/s: 
    b0svx02    Done TO. 1 crate/s: b0svx02  

  RXPT Error for crate/s: 
    b0svx02    Done TO. 1 crate/s: b0svx02 

 Done TO. for crate/s: 
    b0svx02    Done TO. 1 crate/s: b0svx02   

 RXPT Error for crate/s: 
   L1 Done Timeout.[RXPT] 

 

  Attention!!!.   RXPT Errors LOOP!!!. 
  
  &gt;&gt; Please Check the crate/s in the following list: &lt;&lt; 
  
   b0svx02    Done TO. 1 crate/s: b0svx02    
    
  &gt;&gt;        and take the appropriate actions.        &lt;&lt;

  &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;
  &gt;&gt;  NO MORE AUTOMATIC HRRs WILL BE ISSUED.                &gt;&gt;
  &gt;&gt;  ACES ......  YOU HAVE CONTROL of RC NOW.                    &gt;&gt;
  &gt;&gt;  AUTOMATIC HRRs WILL RESUME AFTER 3 MIN.              &gt;&gt;
  &gt;&gt;  OF SMOOTH RUNNING                                                     &gt;&gt;
  &gt;&gt;  (ie. EVENTS INCREMENTING and NO TIMEOUTS)            &gt;&gt;
  &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;
 




</pre>
<!-- Date: Tue Jan 8 17:17:03.217 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255931>255931</A>
 Terminated at 2008.01.08 17:16:49
<!-- Author: RunControl -->
<!-- Comment: Tue Jan  8 17;22;14 comment by...neu -->
We stopped because we had a trip of the complete SVX detector. 
<!-- Date: Tue Jan 8 17:17:41.586 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255931>255931</A>
TERMINATE: end the run to start whitout silicon
<!-- Author: Pedro -->


<!-- Date: Tue Jan 8 17:21:30.812 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255932>255932</A>
 Activated at 2008.01.08 17:20:23
<!-- Author: RunControl -->


<!-- Date: Tue Jan 8 17:21:31.115 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255932>255932</A>
ACTIVATE: RunSet: AAA_NOSILICON  Trigger: PHYSICS_5_01 [3,1016,713] 
<!-- Author: Pedro -->




<!-- Date: Tue Jan 8 17:24:25.542 2008 -->
DO for crate b0cot14. HRR fixed it
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255932' target=_top>255932</a>) -->


<!-- Date: Tue Jan 8 17:24:44.572 2008 -->
TrigMon died in the middle of run, and I started it again.
<!-- Author: Georgios :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255932' target=_top>255932</a>) -->
<!-- Comment: Tue Jan  8 22;54;52 comment by...Georgios -->
Again
<!-- Date: Tue Jan 8 17:31:35.582 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255932>255932</A>
 Terminated at 2008.01.08 17:31:27
<!-- Author: RunControl -->


<!-- Date: Tue Jan 8 17:31:46.773 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255932>255932</A>
TERMINATE: end to put silicon
<!-- Author: Pedro -->


<!-- Date: Tue Jan 8 17:38:27.566 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255934>255934</A>
 Activated at 2008.01.08 17:38:23
<!-- Author: RunControl -->


<!-- Date: Tue Jan 8 17:38:34.112 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255934>255934</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [3,1016,713] 
<!-- Author: Pedro -->





<!-- Date: Tue Jan 8 17:47:46.688 2008 -->
Readout task suspended on b0l2pu01.  Needed to shepherd.  Also, found that after SVX trip, auto HRR was disabled.  Re-enabled it.
<!-- Author: jdl :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255934' target=_top>255934</a>) -->







<!-- Date: Tue Jan 8 18:07:33.554 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-269&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-269  width='300' align='top'></a></td><td>SVXRAD during shot setup for store 5835</td></tr></table>
<!-- Author: Pedro -->


<!-- Date: Tue Jan 8 18:09:24.378 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-270&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-270  width='300' align='top'></a></td><td>SVXRAD during shot setup for store 5835</td></tr></table>
<!-- Author: Pedro -->
<!-- Comment: Tue Jan  8 18;10;32 comment by...Pedro -->
I Mean Losses during shot setup for store 5835


<!-- Date: Tue Jan 8 18:12:51.371 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-271&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-271  width='300' align='top'></a></td><td>L2_Table_beam</td></tr></table>
<!-- Author: Georgios -->
<!-- Comment: Tue Jan  8 18;15;40 comment by...neu -->
The beamline yz slope change discussed 
<a href=https://www-cdfonline.fnal.gov/cgi/internal/elog/elog.pl?nb=2008&action=copyentry&page=23&time=10:26:03.529> here</a> appears to have succeeded.  The yz slope was ~215 murad, now it is about 110.
<!-- Comment: Tue Jan  8 20;03;38 comment by...Phil -->
They only went for a 100 microrad change so the hit it bang on. Thanks to the shift crew for posting it.
<!-- Date: Tue Jan 8 18:14:17.983 2008 -->
Done TO for crate b0imu01. HRR recovered it
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255934' target=_top>255934</a>) -->





<!-- Date: Tue Jan 8 18:43:21.842 2008 -->
we have SVX and ISL alarms
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255934' target=_top>255934</a>) -->
<!-- Comment: Tue Jan  8 18;59;10 comment by...Pedro -->
We had trip in:

SVX Crate 10:
B3W3,B3W2,B3W0,B4W3,B4W2,B4W1,B5W3 and B5W2

ISL: B5W0

we followed the instruccions and we recovered all.
<!-- Date: Tue Jan 8 18:48:17.453 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-272&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-272  width='300' align='top'></a></td><td>To let the TDC experts know.</td></tr></table>
<!-- Author: Georgios -->
<!-- Comment: Tue Jan  8 20;04;47 comment by...Phil -->
There isn't any chamber to match with this scintillator so there isn't any practical consequence. It should be fixed though.
<!-- Date: Tue Jan 8 19:04:18.026 2008 -->
<center><a href='/cgi/elog/elog.pl?nb=2008&action=view&page=-273&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2008&action=view&page=-273' width='300'></a><a href='/cgi/elog/elog.pl?nb=2008&action=view&page=-274&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2008&action=view&page=-275' ></a></center>
<!-- Author: neu -->
<!-- Comment: Tue Jan  8 19;05;37 comment by...neu -->
<pre>Two pinkies, unmarked and recovered: 
  SVX AVDD B3 W2 L0 - big jump was from auto-hockering
  ISL DVDD B5 W1 L4 - seems to be stabilizing</pre>


<!-- Date: Tue Jan 8 19:26:01.224 2008 -->
<center><a href='/cgi/elog/elog.pl?nb=2008&action=view&page=-276&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2008&action=view&page=-277' ></a><a href='/cgi/elog/elog.pl?nb=2008&action=view&page=-278&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2008&action=view&page=-279' ></a><a href='/cgi/elog/elog.pl?nb=2008&action=view&page=-280&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2008&action=view&page=-281' ></a></center>
<!-- Author: neu -->
<!-- Comment: Tue Jan  8 19;27;13 comment by...neu -->
<pre>Three more pinkes, unmarked and recovered.  Each shows evidence of another recent auto-hockerization:
  L00 AVDD B1 W4 L0
  L00 DOIM B1 W5 L5
  ISL AVDD B5 W2 L0

</pre>


<!-- Date: Tue Jan 8 19:50:14.053 2008 -->
<pre>Summary was happening so far: 
 
 
TevMon went to red in ProcMon. I start TevMon, that did not worked, so I start ACNETY Monitor, then TevMon now is updating. 
 
We got again ISL and L00 alarms. 
 
ISL: B3W2,B5W4 
L00 East W4 
L00 East W5 
 
We Followed again Automatic Hockerization and it fixed the problem. 
 
We shepherded the crate b0pcal02 twice 
</pre>
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255934' target=_top>255934</a>) -->


<!-- Date: Tue Jan 8 19:52:01.570 2008 -->
ENCP TO TAPE is red in ProcMon.
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255934' target=_top>255934</a>) -->
<!-- Comment: Tue Jan  8 20;01;34 comment by...Pedro -->
ENCP TO TAPE is green again in ProcMon.




<!-- Date: Tue Jan 8 20:05:42.954 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-282&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-283  width='300' align='top'></a></td><td>Pinky unmarked and recovered: ISL DVDD B5 W1 L2 </td></tr></table>
<!-- Author: neu -->


<!-- Date: Tue Jan 8 20:09:02.894 2008 -->
The odd behavior of stores plot <br> 
<a href="http://www-cdfonline.fnal.gov/daq/stores/store5835.html">http://www-cdfonline.fnal.gov/daq/stores/store5835.html</a><br> 
for this run may have been related to problems with AcnetMonitor. (I would have guessed TOMCAT but I often get confused about data 
moving back and forth between AD, various databases, servlets, etc..)<br> 
Bottom line: Funky stuff in STORES plot is usually worth investigating.
<!-- Author: JJ -->

<!-- Date: Tue Jan 8 20:17:23.050 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255931>255931</A>
<HTML>Run <A HREF=http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN=255931 target=_runSummary>255931</A> has not yet been checked for goodrun status; please do so.</HTML>
<!-- Author: RunControl -->



<!-- Date: Tue Jan 8 20:21:39.711 2008 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2008&action=view&page=-284&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2008&action=view&page=-285  width='300' align='top'></a></td><td>Pinky, familiar channel ISL DVDD B5 W1 L4.  Unmarked and recovered.</td></tr></table>
<!-- Author: neu -->

<!-- Date: Tue Jan 8 20:31:52.734 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255932>255932</A>
<HTML>Run <A HREF=http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN=255932 target=_runSummary>255932</A> has not yet been checked for goodrun status; please do so.</HTML>
<!-- Author: RunControl -->



<!-- Date: Tue Jan 8 20:31:52.238 2008 -->
we got BMU alarm and a few seconds after b0svx02 is red in replies 
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255934' target=_top>255934</a>) -->
<!-- Comment: Tue Jan  8 20;48;21 comment by...Pedro -->
We recovered BMU twice, now is fine.
<!-- Date: Tue Jan 8 20:32:53.456 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255934>255934</A>
 Terminated at 2008.01.08 20:32:46
<!-- Author: RunControl -->


<!-- Date: Tue Jan 8 20:42:34.125 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935>255935</A>
 Activated at 2008.01.08 20:42:08
<!-- Author: RunControl -->


<!-- Date: Tue Jan 8 20:43:05.073 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935>255935</A>
ACTIVATE: RunSet: AAA_NOSILICON  Trigger: PHYSICS_5_01 [3,1016,713] 
<!-- Author: Pedro -->
<!-- Comment: Tue Jan  8 20;57;08 comment by...Pedro -->
We tried to reboot svx02 twice but that did not work. Silicon expert said the stop the run ans start without silicon. Now they are investigating
<!-- Comment: Tue Jan  8 21;39;44 comment by...SashaS -->
It looks like b0svx02 VME CPU problem: CPU is un-ping-able. I did hard reset (pressing reset button on the CPU front pannel) It became pingable and I was able to do PARTIOTION in COLDSTART rc, but CONFIG transiotion stuck in SVX02 crate and crate became unresponsive again. I did hard reset with the same result. It looks like we need to swap b0svx02 vme CPU
<!-- Comment: Tue Jan  8 21;58;37 comment by...Sebastian -->
This is unusual. If you want to try to swap the mvme you can take the one in the teststand (tstsvx1), I think they are both "Motorola MVME2300 - MPC 604r". I am not sure if you can hot swap the cpu, check with Bill B or other expert.
<!-- Comment: Tue Jan  8 21;59;56 comment by...Sebastian -->
You could try hard-reseting the CPU and then talk to the crate using svxdaq and see if the commands get to the SRC.
<!-- Date: Tue Jan 8 20:43:26.741 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255931>255931</A>
general: <tt>It was a short run.</tt>
<!-- Author: Consumer Operator( Georgios Choudalakis) -->




<!-- Date: Tue Jan 8 20:49:30.160 2008 -->
RXPT Error for crate b0imu01, it is fixed
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935' target=_top>255935</a>) -->


<!-- Date: Tue Jan 8 20:53:46.998 2008 -->
Done TO for crate b0cot07. HRR recovered it
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935' target=_top>255935</a>) -->



<!-- Date: Tue Jan 8 21:14:04.749 2008 -->
Done TO for crate b0cot01. HRR recovered it
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935' target=_top>255935</a>) -->

<!-- Date: Tue Jan 8 21:18:03.956 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255932>255932</A>
general: <tt>Very short run.</tt>
<!-- Author: Consumer Operator( Georgios Choudalakis) -->



<!-- Date: Tue Jan 8 21:18:03.618 2008 -->
BMU alarm ,144 East
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935' target=_top>255935</a>) -->
<!-- Comment: Tue Jan  8 21;19;24 comment by...Pedro -->
Now is working fine again
<!-- Date: Tue Jan 8 21:18:34.334 2008 -->
RXPT Error for crate/s:  
    b0cmx01    Done TO. 1 crate/s: b0cmx01   
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935' target=_top>255935</a>) -->
<!-- Comment: Tue Jan  8 21;21;32 comment by...Pedro -->
<pre>Also happened in the same time:

Done TO. for crate/s: 
    b0cmx01    Done TO. 1 crate/s: b0cmx01 </pre>


<!-- Date: Tue Jan 8 21:23:03.856 2008 -->
IMON went to red a few seconds in ProcMOn, after check it is green again
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935' target=_top>255935</a>) -->


<!-- Date: Tue Jan 8 21:32:04.893 2008 -->
<pre> Busy Timeout 
 
If problem persists, please use the DiagnoseTO button  
 on the EVB Ace Control Panel. </pre>
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935' target=_top>255935</a>) -->
<!-- Comment: Tue Jan  8 21;32;40 comment by...Pedro -->
Again


<!-- Date: Tue Jan 8 21:35:07.922 2008 -->
RTSERVER went to red a few seconds in ProcMon, after check it is ok
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935' target=_top>255935</a>) -->
<!-- Comment: Tue Jan  8 21;48;43 comment by...Pedro -->
RTSERVER is red again

<!-- Comment: Tue Jan  8 21;51;46 comment by...Pedro -->
after check twice is ok again


<!-- Date: Tue Jan 8 21:54:23.395 2008 -->
Real troubles with b0svx02.  Si SPL Sasha Sukhanov is here investigating, 
b0svx02 will not partition and becomes unresponsive quickly after 
reboot.  Power cycling this crate is a last resort so other means 
for bringing it back to life are being investigated.  Now considering hot swap of crate controller CPU.
<!-- Author: neu :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935' target=_top>255935</a>) -->
<!-- Comment: Tue Jan  8 22;00;26 comment by...neu -->
<pre>Per DAQ pager carrier, hot swap of crate CPU not ideal.  We are going to bite the bullet and take the voltages down on SVX, power down b0svx02, swap, power up the crate and recover. </pre>
<!-- Date: Tue Jan 8 22:00:16.968 2008 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>DAQ was paged with respect to swapping b0svx02 CPU.  there should be mvme CPU spares on the second floor.  i think the crate must be powered down to swap.  it may depend on the mvme model.  please consult Bill Badgett if you want to do hot swap.</pre></td></tr></td></table>
<!-- Author: Sasha Golossanov :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935' target=_top>255935</a>) -->


<!-- Date: Tue Jan 8 22:00:21.209 2008 -->
ICICLE red in ProcMon
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935' target=_top>255935</a>) -->
<!-- Comment: Tue Jan  8 22;16;42 comment by...Pedro -->
now is fine and it is working
<!-- Date: Tue Jan 8 22:07:59.500 2008 -->
PSM and BMU(144 east)alarms
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935' target=_top>255935</a>) -->
<!-- Comment: Tue Jan  8 22;10;02 comment by...Pedro -->
Now is fine


<!-- Date: Tue Jan 8 22:10:35.449 2008 -->
PEM and PHA alarms
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935' target=_top>255935</a>) -->
<!-- Comment: Tue Jan  8 22;43;43 comment by...Pedro -->
we tried to fix following the instrccions but it not responding. So we Go to the plug HV PC in rack 2RR08C and kill and restart the plug HV monitoring PROGRAM. Now is working again
<!-- Date: Tue Jan 8 22:28:24.013 2008 -->
Done TO for crate b0imu01. HRR fixed it
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935' target=_top>255935</a>) -->

<!-- Date: Tue Jan 8 22:31:56.535 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935>255935</A>
 Terminated at 2008.01.08 22:31:47
<!-- Author: RunControl -->


<!-- Date: Tue Jan 8 22:32:15.102 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255935>255935</A>
TERMINATE: end to put silicon in the run
<!-- Author: Pedro -->



<!-- Date: Tue Jan 8 22:49:51.962 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255940>255940</A>
 Activated at 2008.01.08 22:49:28
<!-- Author: RunControl -->


<!-- Date: Tue Jan 8 22:49:52.898 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255940>255940</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [3,1016,713] 
<!-- Author: Pedro -->
<!-- Comment: Tue Jan  8 23;01;09 comment by...Pedro -->
<pre>we got:

Trigger Inhibit is set from the components:
 L00  		[150] Silicon Layer 00 from iFix


Trigger Inhibit is set from the components:
 ISL02 Trip  		[010] Intermediate Silicon Trip 02

Now they are fixed and we are taking data again</pre>
<!-- Date: Tue Jan 8 22:57:43.935 2008 -->
<pre>Problems unpacking smartsockets message 
 
 Error unpacking smartsockets Status Message from crate: b0cot03 
 - Please reboot the offending crate: b0cot03 
 
 *IF* the ErrorHandler is yellow in Reply/Acknowledgements window:  
 - Stop/restart current ErrorHandler GUI  
 - Shepherd Special needs Client ErrorHandler 
 - Recover/Run 
 
 If still problems, page DAQ.</pre>
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255940' target=_top>255940</a>) -->



<!-- Date: Tue Jan 8 23:06:22.858 2008 -->
Done TO for crate b0pcal04. It was shepherded twice, now is fine
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255940' target=_top>255940</a>) -->

<!-- Date: Tue Jan 8 23:10:25.868 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255880>255880</A>
RUNSTATUS: <PRE>Marked Bad, explanation:
RSF 
</PRE>
<!-- Author: cdfscico (Ben Kilminster) -->




<!-- Date: Tue Jan 8 23:26:26.571 2008 -->
<h2>SVX02 crate problem</h2> 
We had problem with the b0svx02 crate which became unresponsive. See e-log entries <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2008&action=view&page=24&frame=2&anchor=20:31:52.238'> 
here </a>, 
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2008&action=view&page=24&frame=2&anchor=20:43:05.073'> 
here </a>, 
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2008&action=view&page=24&frame=2&anchor=21:54:23.395'> 
here </a> and 
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2008&action=view&page=24&frame=2&anchor=22:00:16.968'> 
here </a>. 
I had to turn OFF SVX detector and swapped this cpu with the spare one.  
The spare cpu did not work - I could not ping it. So I put back the original b0svx02 cpu. I have also removed the "spy" connector from the SRC which has been installed earlier today to spy on the SRC input clock signals. After that the b0svx02 came back and I turned SVX to ON. Sebastian checked SRC with the svxdaq and SRC seems to be fine. It is still not clear what had happened and where the source of the problem: is it the flaky cpu (DAQ problem) or the SRC somehow screw up the vme bus in the crate and halt the cpu. At least part of the SRC which sends "keep alive" signals was operational: not s single SVX ladder tripped while the cpu was at halt. We will try to reproduce the problem in the test stand.
<!-- Author: SashaS/SebastianG -->
<!-- Comment: Wed Jan  9 17;39;03 comment by...SebastianG -->
The spare mvme would not work because of the wrong boot settings, but in any case a power cycle was all that was needed. The mvme setting are now updated and the cpu is ready to be use as spare (it has a lable saying "for SVX or SVT" and it's in the mvme spares crate). BTW, hot swapping mvmes is a bad idea.
<!-- Date: Tue Jan 8 23:35:35.563 2008 -->
<pre>consumer error 
 
Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  
  
 Pipeline out of synch in 464 silicon readout chips .  
 AUTO HRR will be issued  </pre>
<!-- Author: Pedro :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255940' target=_top>255940</a>) -->

<!-- Date: Tue Jan 8 23:45:38.351 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255940>255940</A>
 Terminated at 2008.01.08 23:45:11
<!-- Author: RunControl -->


<!-- Date: Tue Jan 8 23:49:08.422 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255941>255941</A>
 Activated at 2008.01.08 23:48:53
<!-- Author: RunControl -->


<!-- Date: Tue Jan 8 23:50:04.059 2008 -->
Run <A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=255941>255941</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_5_01 [3,1016,713] 
<!-- Author: Pedro -->


<!-- Date: Tue Jan 8 23:55:14.003 2008 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>PhysicsTable</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L3Accepts</TH>
<TH>DelivLumi nb<SUP>-1</SUP></TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=255930 target=_runSummary>255930</A> x3E7BA</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [3,1016,713]</FONT></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>16:53:30</TD>
<TD ALIGN=CENTER>00:00:00</TD>
<TD ALIGN=RIGHT></TD>               <TD ALIGN=RIGHT></TD>
<TD ALIGN=RIGHT></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>0</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=255931 target=_runSummary>255931</A> x3E7BB</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [3,1016,713]</FONT></TD>
<TD ALIGN=CENTER>17:03:34</TD>
<TD ALIGN=CENTER>17:16:49</TD>
<TD ALIGN=CENTER>00:05:40</TD>
<TD ALIGN=RIGHT>      79,679</TD>   <TD ALIGN=RIGHT>    198.21</TD>
<TD ALIGN=RIGHT>     83.38</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=255932 target=_runSummary>255932</A> x3E7BC</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [3,1016,713]</FONT></TD>
<TD ALIGN=CENTER>17:20:23</TD>
<TD ALIGN=CENTER>17:31:27</TD>
<TD ALIGN=CENTER>00:05:35</TD>
<TD ALIGN=RIGHT>      59,733</TD>   <TD ALIGN=RIGHT>    157.05</TD>
<TD ALIGN=RIGHT>     75.06</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=255934 target=_runSummary>255934</A> x3E7BE</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [3,1016,713]</FONT></TD>
<TD ALIGN=CENTER>17:38:23</TD>
<TD ALIGN=CENTER>20:32:46</TD>
<TD ALIGN=CENTER>02:13:36</TD>
<TD ALIGN=RIGHT>   1,514,446</TD>   <TD ALIGN=RIGHT>  1,939.21</TD>
<TD ALIGN=RIGHT>  1,473.18</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=255935 target=_runSummary>255935</A> x3E7BF</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [3,1016,713]</FONT></TD>
<TD ALIGN=CENTER>20:42:08</TD>
<TD ALIGN=CENTER>22:31:47</TD>
<TD ALIGN=CENTER>01:25:04</TD>
<TD ALIGN=RIGHT>     808,065</TD>   <TD ALIGN=RIGHT>    941.65</TD>
<TD ALIGN=RIGHT>    729.63</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=255940 target=_runSummary>255940</A> x3E7C4</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [3,1016,713]</FONT></TD>
<TD ALIGN=CENTER>22:49:28</TD>
<TD ALIGN=CENTER>23:45:11</TD>
<TD ALIGN=CENTER>00:45:09</TD>
<TD ALIGN=RIGHT>     387,233</TD>   <TD ALIGN=RIGHT>    418.78</TD>
<TD ALIGN=RIGHT>    334.89</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN=255941 target=_runSummary>255941</A> x3E7C5</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_5_01 [3,1016,713]</FONT></TD>
<TD ALIGN=CENTER>23:48:53</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>00:03:56</TD>
<TD ALIGN=RIGHT>      21,705</TD>   <TD ALIGN=RIGHT>     30.73</TD>
<TD ALIGN=RIGHT>     27.16</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>23:55:01</TD>
<TD ALIGN=CENTER>04:39:02</TD>
<TD ALIGN=RIGHT>     2,870,861</TD> <TD ALIGN=RIGHT>  3,685.62</TD>
<TD ALIGN=RIGHT>  2,723.29</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Tue Jan 8 23:55:14.665 2008 -->
<font color=red>Shift Summary: </font><pre> 
 
=========================================== 
 
Eve 1/8/2008 
 
 
Start of shift in shot setup. 
 
Silicon guys rapidly finished up their work and relinquished 
silicon resources right before HEP.   
 
17:00 Store came in at 267. 
 
17:15 Complete trip of SVX. Sasha+Sebastian recovered it. 
Took nosilicon data for 15 min.  Back to silicon data immediately 
thereafter.  Issues: 
 - need to reinforce to aces not to complete the HRR transition 
unless the system is ready 
 - in RC of the "Auto HRR" functionality turned off but did not 
resume in the new run. 
 
Lots of DTOs for clc00 - shepherded crate and things improved. 
Problem is not solved.  BB+JDL will work on this tomorrow. 
 
EVB problems of recent days solved by using slow readout on the 
offending SCPU 11, aka crate b0evme11.  This allows readout of large 
event packets w/o TO.  But this slow readout could introduce large 
deadtime potentially at large Linst.  Can revert to default readout 
for this CPU by rebooting. 
 
 
b0svx02 problem really hurt us.  Could not make transition to HALT 
during an HRR from auto-hockerization.  Led to lots of data taken in  
nosilicon configuration.  Eventually came back after rotating out  
and back in the b0svx02 crate controller and disconnecting the LA 
from the SRC.  Unsatisfying resolution - nothing was learned from this. 
Seems stable now we'll see for how long. 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   25593[1,2,4,5], 25594[0,1]
Delivered Luminosity   4141.4 nb-1  
Acquired Luminosity    2731.6 nb-1  
Efficiency             66.0
   
</td></tr></table></pre>
<!-- Author: neu -->