<!-- Title: Eve shift. Thu Oct 13, 2005 -->
<!-- SciCo: Tom LeCompte -->
<!-- DAQAce: Elena Vataga -->
<!-- MonAce: Gian Piero DiGiovanni -->
<!-- CO: Javier Cuevas -->
<!-- OpManager: Aron Soha/Greg Feild -->
<!-- Notes: -->
<pre>Shot setup for 4444 in progress.</pre>


<!-- Date: Thu Oct 13 16:14:26.527 2005 -->
TM2 events pending TO
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205539' target=_top>205539</a>) -->

<!-- Date: Thu Oct 13 16:19:30.145 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205539">205539</a>
 Terminated at 2005.10.13 16:18:57
<!-- Author: RunControl -->


<!-- Date: Thu Oct 13 16:19:31.374 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205539">205539</a>
TERMINATE: Ending run to restart CSL
<!-- Author: Elena x2080 -->


<!-- Date: Thu Oct 13 16:29:50.868 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205540">205540</a>
 Activated at 2005.10.13 16:29:17
<!-- Author: RunControl -->


<!-- Date: Thu Oct 13 16:29:51.184 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205540">205540</a>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,716,538] 
<!-- Author: Elena x2080 -->


<!-- Date: Thu Oct 13 16:31:39.540 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205540">205540</a>
 Terminated at 2005.10.13 16:31:15
<!-- Author: RunControl -->


<!-- Date: Thu Oct 13 16:31:41.663 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205540">205540</a>
TERMINATE: L3 proxy test ...
<!-- Author: Elena x2080 -->



<!-- Date: Thu Oct 13 16:43:16.277 2005 -->
Shot setup for 444<b>5</b> in progress.
<!-- Author: TJL -->

<!-- Date: Thu Oct 13 16:54:05.177 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205541">205541</a>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_3_07 [2,713,594] 
<!-- Author: Elena x2080 -->


<!-- Date: Thu Oct 13 16:55:55.225 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205541">205541</a>
TERMINATE: test l3 proxy - switch to cosmic
<!-- Author: Elena x2080 -->



<!-- Date: Thu Oct 13 17:16:55.389 2005 -->
<pre>We did another test for the l3proxy and included a few nodes which were marked offline before.</pre>
<!-- Author: markus -->


<!-- Date: Thu Oct 13 17:21:24.389 2005 -->
Firus PC making a horrible noise - something is grinding.  Maybe a fan?
<!-- Author: TJL -->
<!-- Comment: Thu Oct 13 17;36;24 comment by...TJL & Willis -->
Actually, it's the Fenwal PC.  Steve Hahn says to power cycle it, and we do, and it never comes back on.  Steve says to just leave it, it's not safety critical, and he'll work on it tomorrow.
<!-- Date: Thu Oct 13 17:25:00.971 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205542">205542</a>
 Activated at 2005.10.13 17:24:16
<!-- Author: RunControl -->


<!-- Date: Thu Oct 13 17:25:01.190 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205542">205542</a>
ACTIVATE: RunSet: ACE_COSMICS_ALL  Trigger: COSMICS [15,716,538] 
<!-- Author: Elena x2080 -->



<!-- Date: Thu Oct 13 18:09:07.026 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205542">205542</a>
 Terminated at 2005.10.13 18:08:48
<!-- Author: RunControl -->


<!-- Date: Thu Oct 13 18:09:45.868 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205542">205542</a>
TERMINATE: L2 problems
<!-- Author: Elena x2080 -->



<!-- Date: Thu Oct 13 18:10:44.149 2005 -->
Have L2 TO - with RC "Active" window red and no error message. 
Later come message from L3 - "rate has dropped to 0Hz".
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205542' target=_top>205542</a>) -->


<!-- Date: Thu Oct 13 18:15:58.279 2005 -->
L3 does not pass config. CSL as always checks OK.. 
<!-- Author: elena -->
<!-- Comment: Thu Oct 13 18;51;12 comment by...Conor (x5045) -->
I presume when L3 failed the config, some or all subfarms were green? This indicates a CSL error. As you later saw...
<!-- Comment: Thu Oct 13 20;23;30 comment by...elena -->
yes, it was green indeed
<!-- Date: Thu Oct 13 18:21:57.061 2005 -->
Finally! After 3d attempt to restart the run guilty party steps out: "CSL error" during config. Need to switch to short setup anyhow
<!-- Author: elena -->


<!-- Date: Thu Oct 13 18:23:06.252 2005 -->
Final protons injected.
<!-- Author: TJL -->


<!-- Date: Thu Oct 13 18:24:43.561 2005 -->
<TABLE bgcolor=#cdc9a5> 
<TR><TH>Date</th><th>Time</th><th>BLM</th><th colspan=2>Dose</th></tr> 
<tr><td>2005.10.13</td><td>18:22:12</td><td>W Inner BLM</td><td>0.07</td><td>RADS</td></tr> 
<tr><td>2005.10.13</td><td>18:22:12</td><td>W Outer BLM</td><td>0.00</td><td>RADS</td></tr> 
<tr><td>2005.10.13</td><td>18:22:12</td><td>E Inner BLM</td><td>0.31</td><td>RADS</td></tr> 
<tr><td>2005.10.13</td><td>18:22:12</td><td>E Outer BLM</td><td>0.67</td><td>RADS</td></tr> 
</table> 
<strong>Integrated dosage</strong>
<!-- Author: Gian Piero -->

<!-- Date: Thu Oct 13 18:28:44.034 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205545">205545</a>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_3_07 [2,713,594] 
<!-- Author: Elena x2080 -->
<!-- Comment: Thu Oct 13 18;32;32 comment by...markus -->
<pre>Can you please cleanup EVB before you start AAA_CURRENT. This gives me a new set of log-files. Thanks!</pre>
<!-- Date: Thu Oct 13 18:30:38.652 2005 -->
Busy TO from Level2_Pulsar_01. Which blames EVB system.
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205545' target=_top>205545</a>) -->
<!-- Comment: Thu Oct 13 18;39;01 comment by...markus -->
<pre>which blames CCAL10 ...</pre>
<!-- Date: Thu Oct 13 18:38:18.696 2005 -->
<P> 
I have finally gotten store  
<A HREF=/internal/daq/stores/store4442.html>4442</a> downtime entries  
and store summary numbers all corrected & approximatelly  
right.  Had to introduce a new rundb package version v2_3u  
to make the downtime accounting during the CLC-off period  
work. 
<P>
<!-- Author: W.Badgett -->


<!-- Date: Thu Oct 13 18:47:40.540 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205545">205545</a>
TERMINATE: anti protons almost here
<!-- Author: Elena x2080 -->



<!-- Date: Thu Oct 13 18:48:23.305 2005 -->
Final pbars injected.
<!-- Author: TJL -->



<!-- Date: Thu Oct 13 18:52:28.972 2005 -->
TeV at 980 GeV
<!-- Author: TJL -->


<!-- Date: Thu Oct 13 19:06:47.561 2005 -->
Scraping complete.
<!-- Author: TJL -->

<!-- Date: Thu Oct 13 19:08:30.628 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205546">205546</a>
TERMINATE: need to change table
<!-- Author: Elena x2080 -->



<!-- Date: Thu Oct 13 19:08:39.711 2005 -->
Luminosity is 151.3 x 10^33.  Switching to high lum table.
<!-- Author: TJL -->
<!-- Comment: Thu Oct 13 19;17;39 comment by...TJL -->
Oops.  10^30 (thanks Kevin).  The shot was good, but not <b>that</b> good!
<!-- Comment: Thu Oct 13 20;16;27 comment by...TJL -->
<b> Official number: 152.41</b>
<!-- Date: Thu Oct 13 19:16:35.668 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205547">205547</a>
 Activated at 2005.10.13 19:16:21
<!-- Author: RunControl -->


<!-- Date: Thu Oct 13 19:16:36.853 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205547">205547</a>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_HIGHLUM_3_06 [1,726,584] 
<!-- Author: Elena x2080 -->



<!-- Date: Thu Oct 13 19:18:09.753 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205547">205547</a>
 Terminated at 2005.10.13 19:17:12
<!-- Author: RunControl -->


<!-- Date: Thu Oct 13 19:18:10.078 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205547">205547</a>
TERMINATE: l3 error
<!-- Author: Elena x2080 -->
<!-- Comment: Thu Oct 13 19;21;32 comment by...markus -->
<pre>node b0l3106 gave up on activate. needed to reset and start over again.</pre>
<!-- Date: Thu Oct 13 19:18:14.548 2005 -->
CMP HV "yellow".   Will investigate once we start running.
<!-- Author: TJL -->
<!-- Comment: Thu Oct 13 19;19;40 comment by...TJL -->
Back to green.  


<!-- Comment: Thu Oct 13 20;18;20 comment by...Phil -->
Software current limit needs to be adjusted upwards. I admit I thought it was good to 175. Maybe something is different in this store. I'll ping Xiaojian to check it.
<!-- Comment: Thu Oct 13 21;03;15 comment by...PHil -->
Geez, I can't even quote myself correctly. CMP went yellow <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=9&frame=2&anchor=14:47:10'>
right on target. Oct. 6: "I'd say we could get some software overcurrents any time in CMP/X, but the hardware limits are fine."</a>
<!-- Date: Thu Oct 13 19:22:20.065 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205548">205548</a>
 Activated at 2005.10.13 19:21:53
<!-- Author: RunControl -->


<!-- Date: Thu Oct 13 19:22:21.003 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205548">205548</a>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_HIGHLUM_3_06 [1,726,584] 
<!-- Author: Elena x2080 -->



<!-- Date: Thu Oct 13 19:24:43.342 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18428&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18428' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18429&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18429' width='300'></a></center>
<!-- Author: Gian Piero -->
<!-- Comment: Thu Oct 13 19;25;26 comment by...Gian Piero -->
LOSSES and SVRAD during the shot
<!-- Date: Thu Oct 13 19:24:44.683 2005 -->
Getting a lot of RTserver errors. Re-check is OK
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205548' target=_top>205548</a>) -->



<!-- Date: Thu Oct 13 19:29:14.407 2005 -->
Got several Busy TO from cot16. Shepherding crate
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205548' target=_top>205548</a>) -->
<!-- Comment: Thu Oct 13 19;48;13 comment by...TJL -->
Paged expert.
<!-- Date: Thu Oct 13 19:32:32.740 2005 -->
Acnet data are not updated
<!-- Author: elena -->
<!-- Comment: Thu Oct 13 19;47;56 comment by...TJL -->
Steve Hahn gives Ace instructions on resetting it.  Work in progress.
<!-- Date: Thu Oct 13 19:35:51.391 2005 -->
again busy TO cot16
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205548' target=_top>205548</a>) -->




<!-- Date: Thu Oct 13 19:49:16.841 2005 -->
after many errors from cot16 got pcal11. Auto HRR was issured but crate did not followed. Shepherding helped
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205548' target=_top>205548</a>) -->


<!-- Date: Thu Oct 13 19:49:16.148 2005 -->
One Level 3 subfarm turns magenta.  Page Markus.
<!-- Author: TJL -->


<!-- Date: Thu Oct 13 19:49:46.219 2005 -->
heap corrupt from b0svt08
<!-- Author: elena  -->

<!-- Date: Thu Oct 13 19:56:30.308 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205548">205548</a>
 Terminated at 2005.10.13 19:56:14
<!-- Author: RunControl -->


<!-- Date: Thu Oct 13 19:56:43.240 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205548">205548</a>
TERMINATE: changing table
<!-- Author: Elena x2080 -->



<!-- Date: Thu Oct 13 19:59:20.965 2005 -->
<h4>SVT SPYMon processes are not working </h4> 
 
 
<p>As part of our checks, Silicon look at some SVT plots to check the Silicon is working.</p> 
 
<p>The plots are generated from the web page: 
<a href="http://www-cdfonline.fnal.gov/cgi/ops/cdf_svt/spycheck.cgi"> 
http://www-cdfonline.fnal.gov/~cdf_svt/cgi-bin/spycheck.cgi</a> 
 
and 
<a href="http://www-cdfonline.fnal.gov/internal/ops/svt/data/svtspymon/ps_files/SVTSPYMON_current.ps"> 
http://www-cdfonline.fnal.gov/svtdaq/svtspymon/ps_files/SVTSPYMON_current.ps</a> 
</p> 
 
<p>These plots are not updating at the moment.  I suspect the problem lies with the yesterday's file trasfer.</p>  
  
<!-- Author: Ankush -->


<!-- Date: Thu Oct 13 20:01:06.552 2005 -->
<pre>The AcnetMonitor process on b0dap85 hung up around 19:11  
this evening;   I cannot kill the old processes completely,  
even with the "kill -9" command which indicates something  
is very wrong with the machine.   The "reboot" command  
also does not work.   Will call control room to ask them  
to reboot from the console.</pre>
<!-- Author: W.Badgett -->
<!-- Comment: Thu Oct 13 20;05;55 comment by...W.Badgett -->
I have started AcnetMonitor on node b0dap86;   I did manage 
to start the b0dap85 reboot...
<!-- Comment: Thu Oct 13 21;31;14 comment by...Stephan Lammel -->
The system log on b0dap85 shows no problem prior to the reboot.
A "kill -9" will not succeed on processes with open devices.
I changed the acnet NFS mounts to accept interrupts, in case that
was causing the hang. It will become effective at the next reboot.
Hardware problem, cosic, or application?
<!-- Date: Thu Oct 13 20:03:32.446 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549">205549</a>
 Activated at 2005.10.13 20:03:19
<!-- Author: RunControl -->


<!-- Date: Thu Oct 13 20:04:01.220 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549">205549</a>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_07 [2,713,594] 
<!-- Author: Elena x2080 -->



<!-- Date: Thu Oct 13 20:04:55.730 2005 -->
busy TO from b0cot16
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->



<!-- Date: Thu Oct 13 20:06:36.243 2005 -->
Switched to low lumi table. In between restarted CSL - it cleared  
all red spots in L3
<!-- Author: elena -->


<!-- Date: Thu Oct 13 20:07:03.844 2005 -->
Flying wire scan
<!-- Author: elena -->
<!-- Comment: Thu Oct 13 21;07;10 comment by...elena -->
no, it's not Busy TO. it's only wire scan...
<!-- Date: Thu Oct 13 20:07:35.692 2005 -->
some time ago - Si resonance
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->
<!-- Comment: Thu Oct 13 20;09;14 comment by...elena -->
<pre>7'55" b0svx02:Messenger:Oct 13, 2005 8:07:56 PM->SRC Fatal Error:Sl 6 Resonance Detected </pre>


<!-- Date: Thu Oct 13 20:10:21.253 2005 -->
<pre>Have moved AcnetMonitor back to b0dap85 and it seems to be  
running OK now.     However, DAQ Ace will have to restart  
the SVT SPYMON process</pre>
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->
<!-- Comment: Thu Oct 13 20;12;08 comment by...W.Badgett -->
That is, AcnetMonitor is running OK after the b0dap85 reboot...

<!-- Date: Thu Oct 13 20:10:33.347 2005 -->
Errors from ProcMon: AcNet + SVT SPY Mon. First AcNet error since begining of problems with TevMon (19:11)
<!-- Author: elena -->



<!-- Date: Thu Oct 13 20:12:18.924 2005 -->
Again busy TO from cot16. Auto HRR
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->
<!-- Comment: Thu Oct 13 20;22;17 comment by...elena -->
<pre>one more time. Expert on the phone told:
1. HRR
2. Shepherd
3. Clean up EVB......</pre>
<!-- Date: Thu Oct 13 20:14:25.102 2005 -->
One more Si resonance
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->
<!-- Comment: Thu Oct 13 20;15;29 comment by...elena -->
<pre>13'9" b0svx02:Messenger:Oct 13, 2005 8:13:10 PM->SRC Fatal Error:Sl 6 Resonance Detected </pre>




<!-- Date: Thu Oct 13 20:21:42.692 2005 -->
Following the istruction of the expert, SVTSPYMON and SpyManager restarted on b0dap85.
<!-- Author: Gian Piero :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->
<!-- Comment: Thu Oct 13 20;25;40 comment by...elena -->
ProcMon is finally happy! 


<!-- Date: Thu Oct 13 20:27:38.490 2005 -->
Busy TO from b0l2pu01. Auto HRR
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->


<!-- Date: Thu Oct 13 20:36:18.844 2005 -->
Busy TO from b0cot00, b0cot01, b0cot16
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->


<!-- Date: Thu Oct 13 20:38:19.517 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18430&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18430' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18431&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18431' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18432&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18432' width='300'></a></center>
<!-- Author: Gian Piero: 19.30 to 20.30 -->


<!-- Date: Thu Oct 13 20:39:04.218 2005 -->
could people learn to clean after themself? I removed 4 glasses (half-full of water) from DAQ ace table since beginning of this shift. Otherwise ProcMon needs another button: wet keyboard.
<!-- Author: elena -->


<!-- Date: Thu Oct 13 20:44:04.292 2005 -->
<h3>Mid shift summary</h3> 
 
Lots happened at the beginning of this store, and if you try and follow what has happened by reading the log entries in chronological order, it's confusing.  A number of problems hit us all in parallel - at one time, we managed to have every phone line in use!<p> 
 
<li> Problem 1A: SVT SpyMon stopped working </li> 
<li> Problem 1B: TevMon complains that ACNET has stopped updating </li> 
<li> Problem 1C: Store 4445 didn't show up on the "Stores" page, and the live luminosity app wasn't updating. </li><p> 
 
These seemingly separate problems were all solved when b0dap85 was rebooted.  There is still some discussion among the experts about whether or not we needed to reboot this and why it happened in the first place. <p> 
 
<li> Problem 2: A Level 3 subfarm turned red.  </li><p> 
 
When we switched to the Low Lum table, we restarted the CSL and this fixed 
things.<p> 
 
<li> Problem 3: A seemingly endless cascade of crate cot16 timeouts.<p> 
 
We got the expert on the phone, but she wasn't able to give any advice other than the standard procedures: HRR, shepherding, etc.  This problem is still happening, but the rate is substantially reduced.  (Don't know why this should be so, but there it is)<p> 
 
We're now running more or less smoothly on the Low Lum table. 
 
  
<!-- Author: TJL -->


<!-- Date: Thu Oct 13 20:44:17.965 2005 -->
DQMon was stuck. Restarted.
<!-- Author: Gian Piero :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->
<!-- Comment: Thu Oct 13 20;49;04 comment by...elena -->
CEM complains about dead channel. Our CO is checking this
<!-- Date: Thu Oct 13 20:45:43.822 2005 -->
svx02 - si resonances
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->
<!-- Comment: Thu Oct 13 20;47;20 comment by...elena -->
<pre>44'48" b0svx02:Messenger:Oct 13, 2005 8:44:49 PM->SRC Fatal Error:Sl 6 Resonance Detected </pre>



<!-- Date: Thu Oct 13 20:50:15.437 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-18433&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-18433  width='300' align='top'></a></td><td>already reported possible problem: the current YMon plot shows the possibly dead CEM 11,+2</td></tr></table>
<!-- Author: cuevas -->
<!-- Comment: Thu Oct 13 21;20;24 comment by...Larry   -->
"Put a fork in it"
<!-- Date: Thu Oct 13 20:55:01.571 2005 -->
Busy TO from cot16. Followed by Busy TO cot16, tsi03, l2pu01. Auto HRR(s)
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->


<!-- Date: Thu Oct 13 20:58:46.423 2005 -->
This is new one: "L2 decision TO". Those people we did non paged so far...
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->


<!-- Date: Thu Oct 13 21:00:38.337 2005 -->
Busy TO from b0cot16. Auto HRR
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->
<!-- Comment: Thu Oct 13 21;11;32 comment by...elena -->
one more time
<!-- Date: Thu Oct 13 21:04:45.610 2005 -->
Busy TO from b0svx02
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->



<!-- Date: Thu Oct 13 21:10:40.052 2005 -->
Busy TO from b0l2pu01
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->
<!-- Comment: Thu Oct 13 21;32;37 comment by...elena -->
again
<!-- Comment: Thu Oct 13 21;36;44 comment by...elena -->
again
<!-- Comment: Thu Oct 13 21;51;12 comment by...elena -->
again
<!-- Comment: Thu Oct 13 21;53;48 comment by...elena -->
again
<!-- Comment: Thu Oct 13 21;59;59 comment by...elena -->
and again
<!-- Comment: Thu Oct 13 22;25;05 comment by...elena -->
we are back to Active after rebooting COT PC and here it is...
<!-- Comment: Thu Oct 13 22;28;43 comment by...elena -->
again
<!-- Comment: Thu Oct 13 22;32;51 comment by...elena -->
again. It indicates EVB....
<!-- Date: Thu Oct 13 21:20:55.387 2005 -->
Seems to be a dead TSU counter at 47E.
<!-- Author: TJL :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205548' target=_top>205548</a>) -->


<!-- Date: Thu Oct 13 21:22:50.362 2005 -->
Here too.
<!-- Author: TJL :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->


<!-- Date: Thu Oct 13 21:30:01.578 2005 -->
RXPT from b0svx02
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->
<!-- Comment: Thu Oct 13 21;30;40 comment by...elena -->
<pre>28'34" b0svx02:Messenger:Oct 13, 2005 9:28:35 PM->SRC Fatal Error:Sl 6 Resonance Detected </pre>

<!-- Comment: Thu Oct 13 21;44;11 comment by...elena -->
<pre>again: 
43'41" b0svx02:Messenger:Oct 13, 2005 9:43:41 PM->SRC Fatal Error:Sl 6 Resonance Detected </pre>
<!-- Date: Thu Oct 13 21:31:14.973 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549">205549</a>
cal: <tt>the current YMon plot shows the possibly dead CEM 11,+2</tt>
<!-- Author: Consumer Operator( javier cuevas) -->






<!-- Date: Thu Oct 13 21:40:57.493 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18434&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18434' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18435&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18435' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18436&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18436' width='300'></a></center>
<!-- Author: Gian Piero: 20.30 to 21.30 -->




<!-- Date: Thu Oct 13 21:51:34.164 2005 -->
COT HV PC loses connection with IFIX.  I page Kevin by mistake.  I then call Ting's cell but get no answer.
<!-- Author: TJL -->
<!-- Comment: Thu Oct 13 21;58;40 comment by...TJL -->
Fortunately, Morris has been spotted, so we Shanghai him into working on the problem.  He's investigating the best way to fix this.
<!-- Date: Thu Oct 13 21:58:12.473 2005 -->
COT got an heartbeat and looking at the details all the superlayer are red, but we think that the problem is related to the comunication with iFix. The RC is still running. Paged the expert.
<!-- Author: Gian Piero :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->
<!-- Comment: Thu Oct 13 22;18;29 comment by...Gian Piero -->
The expert suggested us to halt the run while he stopped and restarted the HV. During the restarting we got a trip for the superlayer 2. The expert solved the problem.
<!-- Date: Thu Oct 13 21:59:19.147 2005 -->
Busy TO from cot09
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->



<!-- Date: Thu Oct 13 22:05:06.216 2005 -->
Busy TO for crate b0l2pu01. Auto HRR issued.
<!-- Author: Gian Piero :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->


<!-- Date: Thu Oct 13 22:11:22.197 2005 -->
L3 converter 15 processor 249 (1) is red
<!-- Author: elena -->


<!-- Date: Thu Oct 13 22:12:09.898 2005 -->
We halted the run to reboot cot pc
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->
<!-- Comment: Thu Oct 13 22;17;46 comment by...TJL -->
Halted means "paused" here, not "ended".



<!-- Date: Thu Oct 13 22:23:31.069 2005 -->
Resonances
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->
<!-- Comment: Thu Oct 13 22;24;26 comment by...elena -->
<pre>b0svx00:Messenger:Oct 13, 2005 10:23:41 PM-&gt;Silicon Resonator: Sl 16 Ch 7 -&gt; e072: Bytes: 00036

</pre>



<!-- Date: Thu Oct 13 22:25:59.425 2005 -->
TevMon is not updating again
<!-- Author: elena -->


<!-- Date: Thu Oct 13 22:26:46.674 2005 -->
Had to reboot COT2 HV PC (first floor) because of an IFIX problem.  Consulted with Aseet first.  Then restarted the MCS program on COT3 (control room).
<!-- Author: Morris -->




<!-- Date: Thu Oct 13 22:36:40.132 2005 -->
Done TO from b0pcal05
<!-- Author: elena :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549' target=_top>205549</a>) -->
<!-- Comment: Thu Oct 13 22;39;45 comment by...elena -->
Tried to shepherd. Stak with: could not connect to RTserver

<!-- Date: Thu Oct 13 22:42:05.398 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549">205549</A>
 Terminated at 2005.10.13 22:41:50
<!-- Author: RunControl -->


<!-- Date: Thu Oct 13 22:43:04.098 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549">205549</A>
TERMINATE: RTserver is stuck
<!-- Author: Elena x2080 -->



<!-- Date: Thu Oct 13 22:44:25.732 2005 -->
MCR calls.  They have lost a proton abort kicker.  Should there be an abort, it is likely that there will be high losses into the silicon.  I've set the silicon to standby, and paged silicon.<p><p> 
 
Ankush calls, and he agrees with this decision.  When the kicker is fixed, page silicon before turning it on.
<!-- Author: TJL -->
<!-- Comment: Thu Oct 13 22;49;36 comment by...TJL -->
Contact COT, and Dave Ambrose says turn the COT off.


<!-- Comment: Thu Oct 13 22;55;40 comment by...TJL -->
Of course, with COT off, might as well bring everything back down to standby.
<!-- Comment: Thu Oct 13 23;03;46 comment by...Gian Piero -->
Turn HV to "End of the Store" position.
<!-- Date: Thu Oct 13 22:50:49.088 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18437&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18437' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18438&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18438' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18439&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18439' width='300'></a></center>
<!-- Author: Gian Piero: 21.30 to 22.30 -->




<!-- Date: Thu Oct 13 23:09:04.422 2005 -->
<table border=0><td><tr><td bgcolor=#e0e030>Before turning silicon on, page the silicon pager. <br> 
At the end of this store, page the silicon pager.</td></tr></td></table>
<!-- Author: TJL -->
<!-- Comment: Thu Oct 13 23;42;46 comment by...TJL -->
<table border=0><td><tr><td bgcolor=#e0e030>No longer necessary.</td></tr></td></table>
<!-- Date: Thu Oct 13 23:10:04.396 2005 -->
Ace documentation for RTserver is extremely confusing. Besides, 
link from DAQ page SmartSockets/RTServer is different from ProcMon checks of Rtserver. 
<!-- Author: elena -->
<!-- Comment: Thu Oct 13 23;10;29 comment by...elena -->
<pre>b0dap85a:149:~&gt; ~cdfdaq/rtserver/rtserver_checkpc.csh

Thu Oct 13 23:02:13 CDT 2005
This script will check for bad rtserver connections
 
 
****Call daq expert if questions/problems 
 
*****************Here is your connection info**********************8
total connections =  2 number of bad connections =  1
tcp        0      0 b0dap85.fnal.gov:32796  b0dap33.fnal.gov:5101   CLOSE_WAIT  cdfdaq     4387    </pre>


<!-- Date: Thu Oct 13 23:32:58.919 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18440&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18440' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18441&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18441' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18442&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18442' width='300'></a></center>
<!-- Author: Gian Piero: 22.30 to 23.30 -->


<!-- Date: Thu Oct 13 23:33:29.694 2005 -->
MCR called.  Problem fixed.
<!-- Author: TJL -->
<!-- Comment: Thu Oct 13 23;40;59 comment by...TJL -->
<table border=0><td><tr><td bgcolor=#e0e030>SVX has given us permission to turn on. <b>
Still fighting b0dap85 problems and Plug HV. <b>
COT goes to standby, and can go to On at 2400. <b></td></tr></td></table>
<!-- Date: Thu Oct 13 23:37:45.192 2005 -->
<pre>The busy TOs this evening were mainly caused by bad data from WCAL00 ( 25 times ). There was single bad events from cmx01 and ccal08. </pre>
<!-- Author: markus -->


<!-- Date: Thu Oct 13 23:38:53.514 2005 -->
Dynamic Prescaling function already in use and owned by parti <B>  -1 </B>
<!-- Author: elena -->



<!-- Date: Thu Oct 13 23:41:52.535 2005 -->
b0fcal00 unresponsive
<!-- Author: elena -->
<!-- Comment: Thu Oct 13 23;49;59 comment by...elena -->
Shepherd was successful

<!-- Date: Thu Oct 13 23:55:50.486 2005 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>Physics able</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L1Accepts</TH>
<TH>L2Accepts</TH>
<TH>L3Accepts</TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205548 target=_blank>205548</A> x322EC</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_HIGHLUM_3_06 [1,726,584]</FONT></TD>
<TD ALIGN=CENTER>19:21:53</TD>
<TD ALIGN=CENTER>19:56:14</TD>
<TD ALIGN=CENTER>00:22:11</TD>
<TD ALIGN=RIGHT>     12,158,259</TD>
<TD ALIGN=RIGHT>       597,233</TD>
<TD ALIGN=RIGHT>     119,697</TD>
<TD ALIGN=RIGHT>  179.863</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205549 target=_blank>205549</A> x322ED</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_3_07 [2,713,594]</FONT></TD>
<TD ALIGN=CENTER>20:03:19</TD>
<TD ALIGN=CENTER>22:41:50</TD>
<TD ALIGN=CENTER>01:59:15</TD>
<TD ALIGN=RIGHT>    187,588,781</TD>
<TD ALIGN=RIGHT>     2,996,827</TD>
<TD ALIGN=RIGHT>     705,592</TD>
<TD ALIGN=RIGHT>  707.987</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>23:55:01</TD>
<TD ALIGN=CENTER>02:21:27</TD>
<TD ALIGN=RIGHT>     199,747,040</TD>
<TD ALIGN=RIGHT>      3,594,060</TD>
<TD ALIGN=RIGHT>       825,289</TD>
<TD ALIGN=RIGHT>  887.849</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Thu Oct 13 23:59:42.537 2005 -->
<font color=red>Shift Summary: </font><pre>Shift summary 
 
 * Store 4445 went in with initial luminosity 152.41 x 10^30. 
 * Problems with b0dap85 showed up in several places. (see Midshift Summary for the 
gory details)  These were fixed when the machine was rebooted.  The problem 
occurred twice, 
  * COT PC looses communication with IFix.  Morris fixes it.   
  * Beams loses a proton abort kicker - silicon set to standby, COT 
off,everything else to standby.  Recovery at 2330. 
  * Plug HV goes red, loses communication  
 
 
Known problems: 
 
 * YMON calorimeter plots turn red at the very highest luminosities 
 * Many crate cot16 busy timeouts 
 * Still a dead CEM phototube 
 * XFT still has a section masked on - this also shows up in BMU trigger rates 
 * There's apparently a dead TSU counter 
 * Plug Calorimeter HV</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   205548-205549
Delivered Luminosity   1341+  
Acquired Luminosity    885  
Efficiency             66%
   
</td></tr></table></pre>
<!-- Author: TJL -->