<!-- Title: Eve shift. Sat Oct 15, 2005 -->
<!-- SciCo: Bill Ashmanskas -->
<!-- DAQAce: Lester Pinera -->
<!-- MonAce: Sacha Baroiant -->
<!-- CO: Anna Sfyrla -->
<!-- OpManager: Steve Hahn -->
<!-- Notes: -->
<pre>Shot setup ....  loading antiprotons! 
</pre>


<!-- Date: Sat Oct 15 16:23:07.878 2005 -->
<pre>Can you please reset/shepherd COT10 and CCAL10. 
</pre>
<!-- Author: markus -->
<!-- Comment: Sat Oct 15 16;31;35 comment by...Lester -->
Done


<!-- Date: Sat Oct 15 16:36:23.267 2005 -->
Shepherded Cot10 and CCal10 as Markus instructed. But when I issued the recover ccal05 went red. So I shepherded that too.
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205618' target=_top>205618</a>) -->


<!-- Date: Sat Oct 15 16:38:52.753 2005 -->
TM2_HevB2B_EVENTS_PENDING_TO.  Auto HRR issued.  What now Markus?
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205618' target=_top>205618</a>) -->
<!-- Comment: Sat Oct 15 16;46;04 comment by...Lester -->
Again.  


<!-- Date: Sat Oct 15 16:46:22.463 2005 -->
Antiproton load complete.  
<!-- Author: Bill -->


<!-- Date: Sat Oct 15 16:46:23.393 2005 -->
Ramping COT to standby, all pbars in tevatron
<!-- Author: Teresa :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205618' target=_top>205618</a>) -->


<!-- Date: Sat Oct 15 16:48:27.165 2005 -->
<pre>The CLCQIE calibration uses the same Consumer definition  
that the Calorimeter QIE calibration uses  
(CONSUMER.NAME='QIEPED', CONSUMER.CHILDNAME='QIE'), and  
RunControl prohibits multiples instances of the same  
name to exist at one time for a given partition.   Therefore,  
if CLC_QIE is done immediately after Calorimeter_QIE, then  
there is a chance that the Calorimeter_QIE QIEPED consumer  
is still running.   This is probably what happened during the  
late day shift when shiftcrew did not find CLC QIE in  
the database.     When Teresa ran the CLC QIE again later,  
the calibration worked.  This has been a long-standing  
issue;  we should create a new and separate Consumer  
definition just for CLC QIE. 
</pre>
<!-- Author: W.Badgett :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205614' target=_top>205614</a>) -->

<!-- Date: Sat Oct 15 16:49:06.713 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205618">205618</A>
TERMINATE: Markus needs to do some quick work 
<!-- Author: Teresa x2080 -->
<!-- Comment: Sat Oct 15 16;55;14 comment by...Lester -->
At Markus' request I reset crates ccal10 & cot10.
<!-- Date: Sat Oct 15 16:50:28.012 2005 -->
We are contacting Bill Badgett, at EVB expert's request, to get our hands on a spare VRB (COT, cal, XFT combo).  
<!-- Author: Bill -->


<!-- Date: Sat Oct 15 17:01:31.692 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205619">205619</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_3_07 [2,713,594] 
<!-- Author: Lester x2080 -->


<!-- Date: Sat Oct 15 17:02:59.652 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205619">205619</A>
TERMINATE: Markus' plan didn't work
<!-- Author: Lester x2080 -->



<!-- Date: Sat Oct 15 17:07:13.653 2005 -->
<pre>Replace a DAQ VRB in crate 12 slot 10 ( only one VRB in the crate ) 
The potentially bad boars as serial number 551011, the new board is 551052. Let's see if this improves data taking. </pre>
<!-- Author: markus -->
<!-- Comment: Sat Oct 15 17;27;25 comment by...Bill A -->
Let the record show that neither Steve Hahn nor I had the sense to insist that this was a bad thing to do during shot setup.

<!-- Date: Sat Oct 15 17:10:19.237 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205620">205620</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_3_07 [2,713,594] 
<!-- Author: Lester x2080 -->


<!-- Date: Sat Oct 15 17:10:51.733 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205620">205620</A>
TERMINATE: nope
<!-- Author: Lester x2080 -->



<!-- Date: Sat Oct 15 17:17:38.383 2005 -->
MCR says scraping is done.  Waiting for CLC HV. 
We're still putzing around with event builder ....  
<!-- Author: Bill -->
<!-- Comment: Sat Oct 15 17;24;01 comment by...Bill -->
Currently L=89E30.

<!-- Date: Sat Oct 15 17:18:03.595 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205621">205621</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_3_07 [2,713,594] 
<!-- Author: Lester x2080 -->


<!-- Date: Sat Oct 15 17:20:25.789 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205621">205621</A>
TERMINATE: now the old vrb isn't working.
<!-- Author: Lester x2080 -->





<!-- Date: Sat Oct 15 17:32:30.273 2005 -->
In desperation, we're about to call Bill Badgett again.  
<!-- Author: Bill -->
<!-- Comment: Sat Oct 15 17;44;37 comment by...Bill -->
Bill is on his way in to help us to run some crate diagnostics.

<!-- Date: Sat Oct 15 17:32:53.622 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205622">205622</A>
ACTIVATE: RunSet: AAA_SHOTSETUP  Trigger: PHYSICS_3_07 [2,713,594] 
<!-- Author: Lester x2080 -->



<!-- Date: Sat Oct 15 17:42:18.015 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18569&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18569' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18571&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18571' width='300'></a></center>
<!-- Author: sasha -->
<!-- Comment: Sat Oct 15 17;42;51 comment by...sasha -->
During shot-setup


<!-- Date: Sat Oct 15 17:48:15.887 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205623">205623</A>
TERMINATE: it worked
<!-- Author: Lester x2080 -->


<!-- Date: Sat Oct 15 17:51:45.663 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624">205624</A>
 Activated at 2005.10.15 17:51:13
<!-- Author: RunControl -->



<!-- Date: Sat Oct 15 17:57:06.753 2005 -->
OK, we're up and running.  The most recent change was to swap the 6U to 9U adapter card for the scanner CPU crate b0evme12.  Now we're working on getting the consumers up and running, and we're seeing occasional trigger inhibits from the CLC PC.  
<!-- Author: Bill -->


<!-- Date: Sat Oct 15 17:57:21.248 2005 -->
We're up and running! We got 3 CLC PC inhibits. The first 2 cleared immediately but it looks like the 3rd is sticking around.  Sasha wnt downstairs a 2nd time to check on it
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624' target=_top>205624</a>) -->
<!-- Comment: Sat Oct 15 18;02;09 comment by...Lester -->
masked clc: PC inhibit
<!-- Comment: Sat Oct 15 18;05;11 comment by...sasha -->
<pre>The CLC HV is an unwavering 100% as reported by the clc pc.  The usual 0.1Hz checks are being executed, as can be seen in the console. However, I can see that the message that comes up evert 10 seconds, once in a while claims that it returns inhibit code 30 w/status 0, while code 31 is the 'all is well' code. The code 30 is not acomponied by any explanation.</pre>
<!-- Comment: Sat Oct 15 18;18;02 comment by...convery -->
All code 30 means is that there is an inhibit for clc.  However I don't think there is ever a case where an inhibit should be generated without an associated alarm.  Possibly I screwed something up when I kludged a bigger tollerance for ch 5 of slot 3???
<!-- Comment: Sat Oct 15 18;32;31 comment by...Lester -->
Enabled the CLC PC inhibit to see if it had cleared. . .it hadn't. Sasha went downstairs and checked the PC again--he says it's in the same state. We'll keep this inhibit masked fow now. 
<!-- Date: Sat Oct 15 17:58:11.936 2005 -->
<pre>Bad idea to change VRBs at the beginning of the store. We (I) have been confused by a bad adapter card for the SCPU, which caused VME transfer errors. The card likely went bad when we power cycled the VME crate to insert a new VRB. </pre>
<!-- Author: markus -->



<!-- Date: Sat Oct 15 18:04:13.093 2005 -->
CLC00 Done TO.  Shepherded the crate.  Maybe I jumped the gun on this, but it didn't hurt
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624' target=_top>205624</a>) -->



<!-- Date: Sat Oct 15 18:07:58.936 2005 -->
b0l2pu01 Busy TO. manually issued HRR. 
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624' target=_top>205624</a>) -->


<!-- Date: Sat Oct 15 18:16:51.968 2005 -->
Flying wire
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624' target=_top>205624</a>) -->
<!-- Comment: Sat Oct 15 19;19;22 comment by...Lester -->
Again
<!-- Comment: Sat Oct 15 20;17;48 comment by...Lester -->
Again
<!-- Comment: Sat Oct 15 21;17;45 comment by...Lester -->
Again
<!-- Date: Sat Oct 15 18:42:14.387 2005 -->
Silicon expert stopped by to inform us that in fact no ladders were lost during yesterday's L1 trigger resonance episode.  
<!-- Author: Bill -->

<!-- Date: Sat Oct 15 18:46:56.691 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624">205624</A>
smx: <tt>YMon:ShowerMax/CP2PadCanvas--> White area (low event number)</tt>
<!-- Author: Consumer Operator(Anna Sfyrla) -->


<!-- Date: Sat Oct 15 18:46:57.953 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624">205624</A>
muons: <tt>YMon:Slides/SlideMuon --> still too few events</tt>
<!-- Author: Consumer Operator(Anna Sfyrla) -->


<!-- Date: Sat Oct 15 18:46:59.211 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624">205624</A>
trigger: <tt>* L1 TFRD --> 3rd plot not zero (bit 38 -- PreFerd)<br>
 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(bits 38 & 39 -- unprescaled)<br>
* L1 Dirac Occupancy --> Lower plots have 'known' errors...</tt>
<!-- Author: Consumer Operator(Anna Sfyrla) -->


<!-- Date: Sat Oct 15 18:47:00.460 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624">205624</A>
monitors: <tt>ObjectMon Slideshow --> Looks ok, more data needed.<br>
XMon --> Some outside the error range... (already from other runs though).</tt>
<!-- Author: Consumer Operator(Anna Sfyrla) -->



<!-- Date: Sat Oct 15 18:57:15.663 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18572&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18572' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18573&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18573' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18574&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18574' width='300'></a></center>
<!-- Author: sasha -->


<!-- Date: Sat Oct 15 19:03:03.328 2005 -->
EVB experts are going to change momentarily the prescale on L2 bit 102, feeding TEST_EVB_JET_70, to increase the L2 accept rate, in order to stress the event builder.  As long as the dead time stays under control, this will have neligible impact on the data we take. 
I am told that the Trigger Working Group authorized this test.  
<!-- Author: Bill -->
<!-- Comment: Sat Oct 15 19;10;35 comment by...Bill -->
Because of what you see in the entry below, we have not yet made this prescale change.  Silicon expert is here anyway.  We're going to wait 10 minutes for smooth running before making this change.

<!-- Comment: Sat Oct 15 19;26;54 comment by...Bill -->
OK, here we go ...

<!-- Date: Sat Oct 15 19:03:20.924 2005 -->
Bunch of errors all at once: Busy TO b0cot08,09 x 3 (all at once) and a TM2_HEVB2B_EVENTS_PENDING TO thrown in for good measure.  This was the same error we were getting before with Markus.  Issued HRR and fortunately all was well. 
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624' target=_top>205624</a>) -->



<!-- Date: Sat Oct 15 19:11:54.776 2005 -->
RXPT Error b0svx02 FERML_SRC_Fatality Error.  The Trigger rates looked fine.  I manually did an HRR.  
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624' target=_top>205624</a>) -->
<!-- Comment: Sat Oct 15 19;13;26 comment by...Lester -->
<pre>(MLE) b0svx00:Messenger:Oct 15, 2005 7:03:39 PM-&gt;Silicon Resonator: Sl 16 Ch 7 -&gt; e072: Bytes: 00042

(MLE) b0svx03:Messenger:Oct 15, 2005 7:03:39 PM-&gt;Silicon Resonator: Sl 10 Ch 9 -&gt; e334: Bytes: 00068
e380 16:e360 12:e340

CT: 2005.10.15 19:03:38
3'38" b0svx02:Messenger:Oct 15, 2005 7:03:38 PM-&gt;SRC Fatal Error:Sl 6 Resonance Detected 
 --&gt;  
 Additional Information: 

 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatal Error from b0svx02: Sl 6 Resonance Detected </pre>


<!-- Comment: Sat Oct 15 19;18;53 comment by...Lester -->
<pre>AGAIN

(MLE) b0svx00:Messenger:Oct 15, 2005 7:17:14 PM-&gt;Silicon Resonator: Sl 18 Ch 9 -&gt; e094: Bytes: 00090

(MLE) b0svx03:Messenger:Oct 15, 2005 7:17:14 PM-&gt;Silicon Resonator: Sl 10 Ch 9 -&gt; e334: Bytes: 00050
e380 16:e360 12:e340

CT: 2005.10.15 19:17:14
17'14" b0svx02:Messenger:Oct 15, 2005 7:17:13 PM-&gt;SRC Fatal Error:Sl 6 Resonance Detected 
 --&gt;  
 Additional Information: 

 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatal Error from b0svx02: Sl 6 Resonance Detected 
 </pre>
<!-- Comment: Sat Oct 15 21;31;34 comment by...Lester -->
<pre>Again

(MLW) b0xtrp00:Messenger:Oct 15, 2005 9:28:57 PM-&gt;No new event ...
(MLE) b0svx03:Messenger:Oct 15, 2005 9:29:00 PM-&gt;Silicon Resonator: Sl 08 Ch 8 -&gt; e313: Bytes: 00098
e380 16:e360 12:e340

(MLE) b0svx00:Messenger:Oct 15, 2005 9:29:00 PM-&gt;Silicon Resonator: Sl 16 Ch 7 -&gt; e072: Bytes: 00066

CT: 2005.10.15 21:29:00
29'0" b0svx02:Messenger:Oct 15, 2005 9:28:59 PM-&gt;SRC Fatal Error:Sl 6 Resonance Detected 
 --&gt;  
 Additional Information: 

 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatal Error from b0svx02: Sl 6 Resonance Detected </pre>
<!-- Date: Sat Oct 15 19:17:40.528 2005 -->
Busy TO cot08. Auto HRR issued
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624' target=_top>205624</a>) -->
<!-- Comment: Sat Oct 15 21;51;01 comment by...Lester -->
Busy TO cot08 & 09 -- Auto HRR 
<!-- Date: Sat Oct 15 19:24:28.199 2005 -->
Restarted DQMon-- All the monitors and subsystem buttons were white .
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624' target=_top>205624</a>) -->



<!-- Date: Sat Oct 15 19:41:56.680 2005 -->
<pre>We did a short rate test for the current version of the EVB SCPU code. We reached 630Hz with 5% deadtime. More optimization is needed.</pre>
<!-- Author: khaldoun and markus -->


<!-- Date: Sat Oct 15 19:42:19.128 2005 -->
Mary did some work on the CLC PC downstairs. Caused a very brief inhibit up here and some iFix alarms went off
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624' target=_top>205624</a>) -->
<!-- Comment: Sat Oct 15 19;44;07 comment by...Bill -->
I think she said she was backing out the CLC PC software to an earlier version.

<!-- Date: Sat Oct 15 19:42:46.221 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18576&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18576' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18578&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18578' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18579&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18579' width='300'></a></center>
<!-- Author: sasha -->


<!-- Date: Sat Oct 15 19:43:04.467 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-18580&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-18580  width='300' align='top'></a></td><td>Woo hoo!</td></tr></table>
<!-- Author: Bill -->


<!-- Date: Sat Oct 15 19:43:57.969 2005 -->
Busy TO b0cot01,02 -- Auto HRR issued 
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624' target=_top>205624</a>) -->


<!-- Date: Sat Oct 15 19:50:28.578 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624">205624</A>
central: <tt>YMon:Slides/SlideCOT1, Slides/SlideCOT2 &nbsp;--> All plots with red frame, there does not seem to be any problem in the plots...</tt>
<!-- Author: Consumer Operator(Anna Sfyrla) -->


<!-- Date: Sat Oct 15 19:50:29.191 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624">205624</A>
smx: <tt>YMon:ShowerMax/CCRPadCanvas &nbsp;&nbsp;--> white area smaller now, seems ok.</tt>
<!-- Author: Consumer Operator(Anna Sfyrla) -->


<!-- Date: Sat Oct 15 19:50:30.245 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624">205624</A>
muons: <tt>YMon:Slides/SlideMuon --> few new noisy channels</tt>
<!-- Author: Consumer Operator(Anna Sfyrla) -->


<!-- Date: Sat Oct 15 19:50:31.748 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624">205624</A>
monitors: <tt>DAQErrMon: 2 'AnyError' & 2 'DataSyncErr'</tt>
<!-- Author: Consumer Operator(Anna Sfyrla) -->



<!-- Date: Sat Oct 15 19:50:53.735 2005 -->
Busy TO cot08 - Auto HRR issued
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624' target=_top>205624</a>) -->


<!-- Date: Sat Oct 15 19:57:35.345 2005 -->
It appears that the HV for CLC channel 10 of slot 0 is fluctuating in and out of tollerance: ~1003V when it should be 1024V.  I guess the inhibit should stay masked pending advice from the CLC guys.  The alarm (but not the inhibit) was incorrectly masked by changes I made to kludge the tollerance for another channel.  Since that channel is effectively disabled now, I reverted to an old version of the monitoring code.
<!-- Author: convery -->
<!-- Comment: Sat Oct 15 20;31;59 comment by...Roberto Rossin (CLC pager) -->
I lowered the HV setting for this channel from 1024 to 1015V. The read out value is still around 1005V. This should prevent the channel from going out of tolerance again. The impact on the readout luminosity is negligible. Data should be marked as good.
<!-- Date: Sat Oct 15 20:07:05.117 2005 -->
Got a pretty weird Run Control complaint.  It appears that so many error messages were issued on behalf of so many different TDCs that the H/R/R took a really long time to complete.  (We thought Run Control was hung up for a moment.)  Here's a sample: 
 
<pre> 
 
(MLE) b0cot01:Messenger:Oct 15, 2005 7:57:09 PM->TDC Invalid SRAM Header evt 2056326 slot 14: exp/fnd mod 131/231 l2b 1/1 73D5A917 
(MLE) b0htdc00:Messenger:Oct 15, 2005 7:57:09 PM->TDC evt 2056326 slot 17 static/fifo: ModID 32/147 L2B 0/0 Hits 2/469 BC 56/76 
(MLE) b0musc00:Messenger:Oct 15, 2005 7:57:09 PM->TDC Invalid SRAM Header evt 2056326 slot 14: exp/fnd mod 17/17 l2b 1/0 0x08800438 
(MLE) b0cot12:Messenger:Oct 15, 2005 7:57:10 PM->TDC Invalid SRAM Header evt 2056326 slot 15: exp/fnd mod 198/128 l2b 1/1 4035719D 
(MLE) b0cot17:Messenger:Oct 15, 2005 7:57:10 PM->TDC Invalid SRAM Header evt 2056326 slot 14: exp/fnd mod 491/491 l2b 1/0 F5801938 
(MLE) b0cot02:Messenger:Oct 15, 2005 7:57:09 PM->TDC Invalid SRAM Header evt 2056326 slot 14: exp/fnd mod 133/161 l2b 1/3 50CDC96B 
(MLE) b0cot11:Messenger:Oct 15, 2005 7:57:09 PM->TDC Invalid SRAM Header evt 2056326 slot 16: exp/fnd mod 194/175 l2b 1/1 57C4B80A 
(MLE) b0cot10:Messenger:Oct 15, 2005 7:57:10 PM->TDC Invalid SRAM Header evt 2056326 slot 19: exp/fnd mod 0/206 l2b 1/2 674AB2F9 
(MLE) b0cot03:Messenger:Oct 15, 2005 7:57:09 PM->TDC Invalid SRAM Header evt 2056326 slot 18: exp/fnd mod 401/401 l2b 1/0 C8800138 
(MLE) b0cot00:Messenger:Oct 15, 2005 7:57:09 PM->TDC Invalid SRAM Header evt 2056326 slot 13: exp/fnd mod 157/221 l2b 1/0 6ED2B07C 
(MLE) b0cot07:Messenger:Oct 15, 2005 7:57:09 PM->TDC Invalid SRAM Header evt 2056326 slot 16: exp/fnd mod 424/424 l2b 1/0 D4000038 
(MLE) b0cot13:Messenger:Oct 15, 2005 7:57:10 PM->TDC Invalid SRAM Header evt 2056326 slot 17: exp/fnd mod 464/464 l2b 1/0 E8000038 
(MLE) b0cot05:Messenger:Oct 15, 2005 7:57:09 PM->TDC Invalid SRAM Header evt 2056326 slot 19: exp/fnd mod 413/413 l2b 1/0 CE800038 
(MLE) b0cot09:Messenger:Oct 15, 2005 7:57:10 PM->TDC Invalid SRAM Header evt 2056326 slot 14: exp/fnd mod 154/299 l2b 1/1 95F56D21 
(MLE) b0cot19:Messenger:Oct 15, 2005 7:57:09 PM->TDC Invalid SRAM Header evt 2056326 slot 19: exp/fnd mod 19/133 l2b 1/1 42E695D2 
(MLE) b0cot06:Messenger:Oct 15, 2005 7:57:10 PM->TDC Invalid SRAM Header evt 2056326 slot 20: exp/fnd mod 415/415 l2b 1/0 CF800038 
(MLE) b0cot18:Messenger:Oct 15, 2005 7:57:09 PM->TDC Invalid SRAM Header evt 2056326 slot 16: exp/fnd mod 220/231 l2b 1/2 73B97F61 
(MLE) b0cot15:Messenger:Oct 15, 2005 7:57:09 PM->TDC Header Mismatch evt 2056326 slot 15: static/fifo 0xED801647/0xED800038 [0B8,0B4] 
(MLE) b0imu00:Messenger:Oct 15, 2005 7:57:09 PM->TDC Header Mismatch evt 2056326 slot 17: static/fifo 0x06000C38/0x0042255D [0F4,1F8] 
(MLE) b0cot04:Messenger:Oct 15, 2005 7:57:10 PM->TDC Header Mismatch evt 2056326 slot 20: static/fifo 0xCA802047/0xCA801338 [0B8,0B4] 
(MLE) b0cot08:Messenger:Oct 15, 2005 7:57:09 PM->TDC Header Mismatch evt 2056326 slot 13: static/fifo 0x4C802247/0x9E278A33 [0B8,0B4] 
(MLE) b0cot16:Messenger:Oct 15, 2005 7:57:09 PM->TDC Header Mismatch evt 2056326 slot 14: static/fifo 0xF1800E47/0x98511FAB [0B8,0B4] 
(MLE) b0cot14:Messenger:Oct 15, 2005 7:57:10 PM->TDC Header Mismatch evt 2056326 slot 17: static/fifo 0xEA802047/0xEA800038 [0B8,0B4] 
(MLE) b0cot01:Messenger:Oct 15, 2005 7:57:10 PM->TDC Header Mismatch evt 2056326 slot 14: static/fifo 0x41802E47/0x73D5A917 [0B8,0B4] 
(MLE) b0htdc00:Messenger:Oct 15, 2005 7:57:10 PM->Event 2056326: Bunchcounters in slot 6 (BC=114) and slot 7 (BC=116) disagree (L2B=1)  
(MLE) b0musc00:Messenger:Oct 15, 2005 7:57:09 PM->TDC Header Mismatch evt 2056326 slot 14: static/fifo 0x08800438/0x10808BD3 [044,10C] 
(MLE) b0cot12:Messenger:Oct 15, 2005 7:57:10 PM->TDC Header Mismatch evt 2056326 slot 15: static/fifo 0x63002C47/0x4035719D [0B8,0B4] 
(MLE 
</pre>  
<!-- Author: Bill -->
<!-- Comment: Sat Oct 15 20;15;15 comment by...Lester  -->
These TDC Errors were accompanied by a silicon resonance and a Done TO for all the following crates imu00, imu01,cmu00,cmp00,musc00,cmx00,cmx01,clc00,htdc00,etdc00, and svx02. Does anybody know what happened here?  As far as the resonance is concerned the L1A rate was normal (~24K). 


<!-- Comment: Sat Oct 15 20;22;22 comment by...Lester -->
<pre>(MLE) b0svx01:Messenger:Oct 15, 2005 7:57:20 PM-&gt;Silicon Resonator: Sl 10 Ch 2 -&gt; e122: Bytes: 00070

(MLE) b0svx03:Messenger:Oct 15, 2005 7:57:20 PM-&gt;Silicon Resonator: Sl 08 Ch 7 -&gt; e312: Bytes: 00068
e380 16:e360 12:e340

CT: 2005.10.15 19:57:58
57'58" b0svx02:Messenger:Oct 15, 2005 7:57:20 PM-&gt;SRC Fatal Error:Sl 6 Resonance Detected 
 --&gt;  
 Additional Information: 

 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatal Error from b0svx02: Sl 6 Resonance Detected </pre>



<!-- Date: Sat Oct 15 20:36:17.228 2005 -->
RXPT error crate  svx02. Followed by silicon resonance.  Manually issued HRR
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624' target=_top>205624</a>) -->
<!-- Comment: Sat Oct 15 20;38;02 comment by...Lester -->
<pre>(MLE) b0svx00:Messenger:Oct 15, 2005 8:33:09 PM-&gt;Silicon Resonator: Sl 16 Ch 7 -&gt; e072: Bytes:00046

(MLE) b0svx05:Messenger:Oct 15, 2005 8:33:09 PM-&gt;Silicon Resonator: Sl 08 Ch 3 -&gt; e503: Bytes:00084

(MLE) b0svx02:Messenger:Oct 15, 2005 8:33:08 PM-&gt;Silicon Resonator: Sl 13 Ch 9 -&gt; e254: Bytes:00068
e200

CT: 2005.10.15 20:33:09
33'9" b0svx02:Messenger:Oct 15, 2005 8:33:08 PM-&gt;SRC Fatal Error:Sl 6 Resonance Detected 
 --&gt;  
 Additional Information: 

 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatal Error from b0svx02: Sl 6 Resonance Detected 
 </pre>


<!-- Date: Sat Oct 15 21:02:31.934 2005 -->
<pre>Silicon Resonance:  
 
MLE) b0dap73.fnal.gov:Thread-25:Oct 15, 2005 9:00:29 PM-&gt;FrontEnd Error: VRB_SVX_02  
(MLW) b0xtrp00:Messenger:Oct 15, 2005 9:00:40 PM-&gt;No new event ... 
(MLE) b0svx00:Messenger:Oct 15, 2005 9:00:43 PM-&gt;Silicon Resonator: Sl 12 Ch 4 -&gt; e044: Bytes: 00080 
 
(MLE) b0svx03:Messenger:Oct 15, 2005 9:00:43 PM-&gt;Silicon Resonator: Sl 08 Ch 9 -&gt; e314: Bytes: 00248 
e380 16:e360 12:e340 
 
CT: 2005.10.15 21:00:43 
0'43" b0svx02:Messenger:Oct 15, 2005 9:00:43 PM-&gt;SRC Fatal Error:Sl 6 Resonance Detected  
 --&gt;   
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error from b0svx02: Sl 6 Resonance Detected </pre>
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624' target=_top>205624</a>) -->


<!-- Date: Sat Oct 15 21:03:31.366 2005 -->
<center><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18581&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18581' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18582&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18582' width='300'></a><a href='/cgi/elog/elog.pl?nb=2005&action=view&page=-18583&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2005&action=view&page=-18583' width='300'></a></center>
<!-- Author: sasha -->


<!-- Date: Sat Oct 15 21:14:23.836 2005 -->
Busy TO for crates b0cot00,01, Auto HRR issued
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624' target=_top>205624</a>) -->



<!-- Date: Sat Oct 15 21:24:34.968 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-18584&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-18584  width='300' align='top'></a></td><td>ISL bias pinkie. Currently at 171muA, it goes pink at 170.5.</td></tr></table>
<!-- Author: sasha -->



<!-- Date: Sat Oct 15 21:46:09.124 2005 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2005&action=view&page=-18585&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2005&action=view&page=-18585  width='300' align='top'></a></td><td>Well, it's not improving. Better page Si before bedtime.</td></tr></table>
<!-- Author: sasha -->
<!-- Comment: Sat Oct 15 21;53;14 comment by...sasha -->
Just got off the phone w/Ankush.  No cause for alarm. The current limits may be changed tomorrow.

<!-- Date: Sat Oct 15 22:12:29.219 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624">205624</A>
 Terminated at 2005.10.15 22:11:53
<!-- Author: RunControl -->


<!-- Date: Sat Oct 15 22:12:30.046 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624">205624</A>
TERMINATE: L3/CSL flow problem
<!-- Author: Lester x2080 -->


<!-- Date: Sat Oct 15 22:19:56.926 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205625">205625</A>
TERMINATE: no good
<!-- Author: Lester x2080 -->


<!-- Date: Sat Oct 15 22:26:14.402 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205626">205626</A>
TERMINATE: l3 problems
<!-- Author: Lester x2080 -->



<!-- Date: Sat Oct 15 22:44:15.996 2005 -->
We're having a lot of trouble getting CSL stopped and restarted, which we did in response to run 205624 ending with all L3 nodes becoming full and never draining.  
<!-- Author: Bill -->
<!-- Comment: Sat Oct 15 23;39;06 comment by...Bill -->
CSL expert is here now, and Jedong Lee has been on the phone with us for some time.  Jedong guided Lester through shutting down and powering down CSL computer and RAID disks.  Expert powered them back up, says we're now ready to run.  We're in coldstart now.



<!-- Comment: Sat Oct 15 23;40;50 comment by...Bill -->
It appears that we're running ....

<!-- Date: Sat Oct 15 23:40:48.653 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205627">205627</A>
 Activated at 2005.10.15 23:39:43
<!-- Author: RunControl -->


<!-- Date: Sat Oct 15 23:40:50.179 2005 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205627">205627</A>
ACTIVATE: RunSet: AAA_CURRENT  Trigger: PHYSICS_3_07 [2,713,594] 
<!-- Author: Lester x2080 -->



<!-- Date: Sat Oct 15 23:55:35.625 2005 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>RunNumber</TH>
<TH>DataType</TH>
<TH>Physics able</TH>
<TH>Begin</TH>
<TH>End</TH>
<TH>LiveTime</TH>
<TH>L1Accepts</TH>
<TH>L2Accepts</TH>
<TH>L3Accepts</TH>
<TH>LiveLumi nb<SUP>-1</SUP></TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624 target=_blank>205624</A> x32338</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_3_07 [2,713,594]</FONT></TD>
<TD ALIGN=CENTER>17:51:13</TD>
<TD ALIGN=CENTER>22:11:53</TD>
<TD ALIGN=CENTER>03:49:16</TD>
<TD ALIGN=RIGHT>    334,398,728</TD>
<TD ALIGN=RIGHT>     4,422,375</TD>
<TD ALIGN=RIGHT>   1,134,948</TD>
<TD ALIGN=RIGHT>  862.425</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205627 target=_blank>205627</A> x3233B</TD>
<TD>BEAM</TD>
<TD><FONT SIZE=-1>PHYSICS_3_07 [2,713,594]</FONT></TD>
<TD ALIGN=CENTER>23:39:43</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>00:13:30</TD>
<TD ALIGN=RIGHT>     12,661,426</TD>
<TD ALIGN=RIGHT>       129,155</TD>
<TD ALIGN=RIGHT>      46,196</TD>
<TD ALIGN=RIGHT>   34.410</TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER></TD>
<TD ALIGN=CENTER>1</TD></TR>

<TR><TD>Totals</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD>
<TD>23:55:01</TD>
<TD ALIGN=CENTER>04:02:46</TD>
<TD ALIGN=RIGHT>     347,060,154</TD>
<TD ALIGN=RIGHT>      4,551,530</TD>
<TD ALIGN=RIGHT>     1,181,144</TD>
<TD ALIGN=RIGHT>  896.835</TD>
<TD> </TD>
<TD> </TD>
<TD> </TD></TR>

</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sun Oct 16 00:00:37.201 2005 -->
<pre>The EVB downtime at the beginning of the shift ( store ) does not show up in the down time logger. Just a remark, no complain!</pre>
<!-- Author: markus -->


<!-- Date: Sun Oct 16 00:08:30.083 2005 -->
<pre>Shift summary: 
 
Shift began with shot setup well underway.  Shot was in and 
scraping complete at 17:17, with initial luminosity 90.8E30. 
 
EVB expert tried a VRB swap as a solution to frequent busy 
timeouts seen recently, originating in EVB crate 12, slot 10. 
Quite a number of reconfigurations was tried before we were 
able to start a run; what eventually seemed to work (at least 
was the last change made) was to swap the 6U-to-9U adapter 
card for this crate's scanner CPU.  Unfortunately, this fix 
was done toward the end of shot setup, and since a few  
iterations were required, it took us until 17:51 to get data 
taking underway. 
 
We had a sequence of trigger inhibits from the CLC PC.  We 
probably would have had even more of them, but apparently the 
CLC PC software had been suppressing some of the high voltage 
messages, until software was updated (downdated?) tonight. 
CLC expert reports that one PMT HV was drooping by a few 
percent.  He lowered the nominal setting for this tube, noted 
that the occupancy didn't change, and the complaints from the 
HV PC went away. 
 
A parasitic test of EVB throughput (required changing one 
prescale for a few minutes) was successful:  600 Hz L2A was 
achieved.  Expert did this, apparently with TWG approval. 
 
Around 22:15, CSL stopped accepting events from L3.  HRR didn't 
fix it, nor did restarting the run.  R_C recommended a CSL 
stop/restart, which didn't complete cleanly.  After lots of 
discussion with CSL experts, ace shut down and powered down CSL, 
and expert arrived to revive it.  We were running again at 
23:40. 
 
</pre>
<!-- Author: Bill -->


<!-- Date: Sun Oct 16 00:08:30.878 2005 -->
All Hell broke loose.   
 
First we got a TM2 HEVB2B events pending TO Followed by a message warning L3trigger rates had dropped to zero.  The entire L3 display went green (I figured it was a CSL error).  While reading the CSL procedure we decided to end the run and start a new one (why not?).  As we were going through the config state of the new run. I got a L3 relay failure error (1 or more L3 nodes likely to have died).  The message prompted me to do End/Abort+Reset on all partitions.  L3 cleared and we then started a new run.   
 
Then immediatley after got CSL Receiver max error. CSL rejected connection.  The message prompted us to reset and restart CSL.  Tried doing that: logged into b0dau32 and did a check, stop, cleanup and when I tried start got the message: some csl processes are still running.start aborted.  Called CSL expert and  we went through this same process a few more times.  Expert was coming in, but it would take a few minutes so we called Jedong and he wanted me to restart b0dau32.  The restart did not go smoothly.  We got I/O error and bunch of others. Then the CSL expert arrived.
<!-- Author: Lester :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=205624' target=_top>205624</a>) -->
<!-- Comment: Sun Oct 16 00;46;18 comment by...Geumbong -->
<pre>Thank you Lester for following rebooting processes so we saved time and data! I took over from Lester and finished up the processes. But, unfortunately, we found disk4 was missing after rebooting b0dau32. We decided to take data without disk4 just for now. The stream a, which used disk4, is sending information to backup disk. Now everything works fine. After the end of this store, we will reboot b0dau32 again to recover the connection to disk4.</pre>
<!-- Date: Sun Oct 16 00:09:55.379 2005 -->
<font color=red>Shift Summary: </font><pre>See above. 
</pre> 
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   205624, 205627
Delivered Luminosity   1478  
Acquired Luminosity    937  
Efficiency             63%
   
</td></tr></table></pre>
<!-- Author: Bill Ashmanskas -->