<!-- Title: Eve shift. Sat Aug 17, 2002 -->
<!-- SciCo: Bill Ashmanskas -->
<!-- DAQAce: Jedong Lee -->
<!-- MonAce: Mario Spezziga -->
<!-- CO: Arnaldo Stefanini -->
<!-- OpManager: Eric James -->
<!-- Notes: -->
 
 
<p> <b>Status at 16:00, start of evening shift.</b> Store in since 
about 14:00.  L=1.7E31.  Stack at 19mA, stacking at 9mA/hr.  Inherit 
run 150079 from day shift.  L1/L2/L3 rates 7500/190/37 Hz, DT=0.8%, 
rfmterr=0.72%.  Trigger table PHYSICS_1_02[2,131,312].  Experts 
working on L00, SVXII trips; diagnosing unexpected trigger inhibit 
behavior.  <p> 
 
 
 
 
 
  

<!-- Date: Sat Aug 17 16:16:57 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080">150080</A>
 Activated at 2002.08.17 16:17:11
<!-- Author: RunControl -->


<!-- Date: Sat Aug 17 16:27:48 2002 -->
Run <A HREF="/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080">150080</A>
ACTIVATE: physics_1_02 [2,131,312]
<!-- Author: ace 2080 -->



<!-- Date: Sat Aug 17 16:29:20 2002 -->
<p>Previous shift reported unexpected trigger inhibit behavior from 
the silicon.  Trips were being reported from the silicon GUI to 
the silicon iFix PC (as evidenced by the fact that the offending 
ladders showed up on the Trigger Inhibit Status page), but apparently the iFix inhibit signal was not being propagated 
to CDF_S2 (as evidenced by the fact that the "LED" on the status page 
remained green).  I suspected there was a communication problem 
between the silicon iFix PC and the CDF_S2 PC, but I checked that 
and it was fine.  Then Gino and I induced a L00 trip (of the same 
nature and from the same wedge as the one that started all this). 
The trigger inhibit system behaved normally.</p> 
<p> 
Conclusion: Problem not reproducible.  Could have been just a 
temporary communication failure, I guess -- I think there's a 
way I can trap this in the future.</p> 
<p> 
See silicon e-log for a few more details.</p>  
<!-- Author: Andy H. -->
<!-- Comment: Sat Aug 17 16;37;53 comment by...Andy H. -->
<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=120&frame=2&anchor=16:36:49'>
Link to the silicon e-log entry</a> 


<!-- Date: Sat Aug 17 16:42:04 2002 -->
<center><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6231&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6231' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6232&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6232' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6233&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6233' width='300'></a></center>
<!-- Author: Kostas/Mario -->


<!-- Date: Sat Aug 17 16:43:48 2002 -->
Done timeout in PCAL_02. HRR fixed the problem.
<!-- Author: Jedong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sat Aug 17 16:50:54 2002 -->
<pre>WInner BLM 100.5 
WOuter BLM 0 
EInner BLM 0 
EOuter BLM 28.5</pre>
<!-- Author: Kostas -->


<!-- Date: Sat Aug 17 16:56:59 2002 -->
<pre>Mod done mask read from crate controller: 0x000008c1 
Clearing L2 Fatal error counters. 
Clearing L2 warning counters for all four buffers. 
L1Mon: saw 0 L1 DMA transfers, expect 1 (buffer number 0) 
FaultmManager: Fatal error, sending L2 timeout directly 
Stop or Recover stateStop or Recover stateFirst Event Started 
</pre>
<!-- Author: ace -->
<!-- Comment: Sat Aug 17 16;57;27 comment by...ace -->
L2 decision timeout: run 150080


<!-- Date: Sat Aug 17 17:00:49 2002 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2002&action=view&page=-6234&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-6234  width='300' align='top'></td><td>Integrated radiation seen by the BLMs in the last hour.</a></td></tr></table>
<!-- Author: Kostas/Mario -->


<!-- Date: Sat Aug 17 17:41:32 2002 -->
<center><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6235&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6235' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6236&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6236' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6237&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6237' width='300'></a></center>
<!-- Author: Kostas/Mario -->


<!-- Date: Sat Aug 17 17:50:33 2002 -->
<table><tr><td><a href=/elog/notebooks/2002/Sat_Aug_17_2002_Eve.175028.ps align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-6239  align='top'></td><td>Hot channels on SL8</a></td></tr></table>
<!-- Author: a stefanini -->


<!-- Date: Sat Aug 17 18:03:38 2002 -->
<table><tr><td><a href=/elog/notebooks/2002/Sat_Aug_17_2002_Eve.180318.ps align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-6241  align='top'></td><td>Warning on Shower max</a></td></tr></table>
<!-- Author: a stefanini -->


<!-- Date: Sat Aug 17 18:21:28 2002 -->
<pre>Mod done mask read from crate controller: 0x000008c1 
Clearing L2 Fatal error counters. 
Clearing L2 warning counters for all four buffers. 
Error: Number of svt words is 970  
   Setting Number of svt words to 290  
SvtMon: SVT buffno 0 != event buffno 1 
FaultmManager: Fatal error, sending L2 timeout directly 
Stop or Recover stateStop or Recover stateFirst Event Started 
</pre>
<!-- Author: ace :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->
<!-- Comment: Sat Aug 17 18;21;55 comment by...ace -->
L2 decision timeout.  Recovered with H/R/R.



<!-- Date: Sat Aug 17 18:27:29 2002 -->
<center><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6242&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6242' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6243&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6243' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6244&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6244' width='300'></a></center>
<!-- Author: Kostas/Mario -->


<!-- Date: Sat Aug 17 18:30:26 2002 -->
Mod done mask read from crate controller: 0x000008c1 
Clearing L2 Fatal error counters. 
Clearing L2 warning counters for all four buffers. 
 
 
In stopped state. 3061984 L1A since last stop 
Bits0c 0x4007ff7f, Bits18 0x100,  Bits14 0x4fff  
Buffer number 3, load buffer 3 
First Event Started  
<!-- Author: ace :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->
<!-- Comment: Sat Aug 17 18;31;04 comment by...ace -->
Done timeout CCAL_09. Recovered with HRR.


<!-- Comment: Sat Aug 17 22;46;57 comment by...BA -->
oops, this was a done timeout, not a decision timeout, so we
didn't need to draw any attention to L2 on this one ...
<!-- Date: Sat Aug 17 18:41:49 2002 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2002&action=view&page=-6245&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-6245  width='300' align='top'></td><td>No data channels on SVX L4</a></td></tr></table>
<!-- Author: a stefanini -->


<!-- Date: Sat Aug 17 18:43:45 2002 -->
<table><tr><td><a href=/elog/notebooks/2002/Sat_Aug_17_2002_Eve.184339.ps align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-6247  align='top'></td><td>No data channels on SVX L4</a></td></tr></table>
<!-- Author: a stefanini -->


<!-- Date: Sat Aug 17 19:04:57 2002 -->
<table><tr><td><a href=/elog/notebooks/2002/Sat_Aug_17_2002_Eve.190453.ps align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-6249  align='top'></td><td>Warning on Tau Mon</a></td></tr></table>
<!-- Author: a stefanini -->


<!-- Date: Sat Aug 17 19:17:50 2002 -->
<pre>Mod done mask read from crate controller: 0x000008c1 
Clearing L2 Fatal error counters. 
Clearing L2 warning counters for all four buffers. 
Error: Number of svt words is 352  
   Setting Number of svt words to 290  
Error: Number of svt words is 502  
   Setting Number of svt words to 290  
SvtMon: SVT buffno 0 != event buffno 1 
FaultmManager: Fatal error, sending L2 timeout directly 
Stop or Recover stateStop or Recover stateFirst Event Started 
 
</pre>
<!-- Author: eusebi :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->
<!-- Comment: Sat Aug 17 19;18;26 comment by...eusebi -->
<pre>Level 2 Decision timeout.. Recover by HRR.
</pre>


<!-- Date: Sat Aug 17 19:26:04 2002 -->
<center><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6250&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6250' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6251&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6251' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6252&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6252' width='300'></a></center>
<!-- Author: Kostas -->


<!-- Date: Sat Aug 17 19:33:37 2002 -->
Mod done mask read from crate controller: 0x000008c1 
Clearing L2 Fatal error counters. 
Clearing L2 warning counters for all four buffers. 
Error: Number of svt words is 308  
   Setting Number of svt words to 290  
SvtMon: SVT buffno 0 != event buffno 1 
FaultmManager: Fatal error, sending L2 timeout directly 
Stop or Recover stateStop or Recover stateFirst Event Started 
  
<!-- Author: eusebi :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->
<!-- Comment: Sat Aug 17 19;34;27 comment by...eusebi -->
<pre>l2 timeout. HRR recovery.
</pre>


<!-- Date: Sat Aug 17 19:35:17 2002 -->
Mod done mask read from crate controller: 0x000008c1 
Clearing L2 Fatal error counters. 
Clearing L2 warning counters for all four buffers. 
 
 
In stopped state. 379948 L1A since last stop 
Bits0c 0x4007ff7f, Bits18 0x100,  Bits14 0x4fff  
Buffer number 3, load buffer 3 
First Event Started 
  
<!-- Author: eusebi :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->
<!-- Comment: Sat Aug 17 19;35;48 comment by...eusebi -->
<pre>Busy TimeOut.HRR recovery.
</pre>


<!-- Comment: Sat Aug 17 22;52;51 comment by...scico -->
oops, this one was EVB, not L2.  still, information 
never hurts.

<!-- Date: Sat Aug 17 19:49:36 2002 -->
<table><tr><td><a href=/elog/notebooks/2002/Sat_Aug_17_2002_Eve.194930.ps align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-6254  align='top'></td><td>I do not know if these dead channels are really new. It would help to have more recent reference plates.</a></td></tr></table>
<!-- Author: a stefanini -->


<!-- Date: Sat Aug 17 19:50:26 2002 -->
<table><tr><td><a href=/elog/notebooks/2002/Sat_Aug_17_2002_Eve.195021.ps align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-6256  align='top'></td><td>same as above</a></td></tr></table>
<!-- Author: a stefanini -->


<!-- Date: Sat Aug 17 20:02:15 2002 -->
<pre> Trigger Inhibit from SVX (B2W3L0) and ISL (B0W1L0) shown 
on the "Trigger Inhibit Status" window. 
 
 The PS GUI shows trips (red wedges) in the SVX and the ISL; 
SVX: B0W0, B0W2, B0W3, B1W0, B1W2, B1W3, B2W2, B2W3 
ISL: B0W1 
 
 The strange thing is that iFIX does now show the silicon in 
an alarming state! The only sign of problem is that the bars 
that show the lower voltage value for the SVX and the ISL  
are missing. The bar that shows the upper HV is up and green... 
 
  Silicon expert on the phone with Ricardo to resolve the mystery. 
</pre>
<!-- Author: Kostas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->
<!-- Comment: Sat Aug 17 21;13;44 comment by...Kostas -->
<pre>One more strange thing: There was no alarm/warning reported 
by CAEN in the white window of the "Silicon Power Supply Control" monitor.</pre>
<!-- Date: Sat Aug 17 20:02:48 2002 -->
<table><tr><td><a href=/elog/notebooks/2002/Sat_Aug_17_2002_Eve.200241.ps align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-6258  align='top'></td><td>ISL</a></td></tr></table>
<!-- Author: a stefanini -->


<!-- Date: Sat Aug 17 20:05:11 2002 -->
<pre>CMX tripped: SE 8 and 9 
High Voltage turned on again. Worked fine. 
 
Note: this trip was shown by iFIX, unlike the silicon trip, see         above. 
</pre>
<!-- Author: Kostas/Mario :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->
<!-- Comment: Sat Aug 17 20;11;46 comment by...Kostas/Mario -->
<pre>LOSTPB (p-bar losses) where spiky at the moment (reching up to
~12kHZ). 
Subsequently LOSTP (proton losses) went a bit up (to ~ 9 kHZ) and then down to the usual ~8 kHz.</pre>

<!-- Comment: Sat Aug 17 20;15;36 comment by...Kostas -->
<pre>Ooops, wrong reading of the LOSTPB scale in the comment above:
LOSTPB reached ~8 kHZ, from the "usual" ~3 kHz with moderate spikes. </pre>


<!-- Date: Sat Aug 17 20:21:28 2002 -->
<center><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6259&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6259' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6260&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6260' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6261&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6261' width='300'></a></center>
<!-- Author: Kostas -->


<!-- Date: Sat Aug 17 20:27:03 2002 -->
At 20:10 we had two Trigger Inhibits one for SVX and ISL as seen in the run Supervisor terminal. 
The Global Alarm page didn't reflect the problems in the SVX and ISL boxes.But the High Voltage Summary page drop the low end bar to zero. 
The Trigger status page did show the inhibits. 
Following the instructions of the Silicon person we restart the tripped ladders in the GUI. 
The ladders were  
For SVX : B0 W3,W2,W0 
          B1 W3 W2 and W0 
          B2 W2 and W3 
For ISL : B0 W1 
After turning the tripped ladders ON the inhibits cleared. 
A HHR recovery was done and continue with the data adquisition. 
 
We don't know why the global alarm page didn't show the trips. 
 
 
 
<!-- Author: eusebi :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->
<!-- Comment: Mon Aug 19 07;23;36 comment by...SCN -->
Hmm, these guys are all in the same corner, same Fib crate and possibly same CAEN crate (or at least near proximity).  When
do we get our titanium curtains?

<!-- Date: Sat Aug 17 20:29:52 2002 -->
<pre>Mod done mask read from crate controller: 0x000008c1 
Clearing L2 Fatal error counters. 
Clearing L2 warning counters for all four buffers. 
 
 
In stopped state. 76837 L1A since last stop 
Bits0c 0x4006ff7f, Bits18 0x100,  Bits14 0x4fff  
Buffer number 3, load buffer 3 
First Event Started 
</pre>
<!-- Author: Jedong :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->
<!-- Comment: Sat Aug 17 20;30;23 comment by...Jedong -->
Done timeout with FCAL_00. Recovered with HRR.

<!-- Comment: Sat Aug 17 20;44;44 comment by...scico -->
Halt transition didn't succeed for FCAL00, so we logged into
crate.  Saw no suspended tasks.  Tried single-crate HALT 
transition, it didn't respond.  Tried single-crate RECOVER,
it didn't respond.  Tried single-crate RESET transition, it
worked.  Followed it up with PARTITION, CONFIGURE, ACTIVATE.
Finally gave the whole run a H/R/R.



<!-- Date: Sat Aug 17 20:45:36 2002 -->
<pre>Mod done mask read from crate controller: 0x000008c1 
Clearing L2 Fatal error counters. 
Clearing L2 warning counters for all four buffers. 
Error: Number of svt words is 2326  
   Setting Number of svt words to 290  
SvtMon: SVT buffno 0 != event buffno 2 
FaultmManager: Fatal error, sending L2 timeout directly 
Stop or Recover stateStop or Recover stateFirst Event Started 
 
</pre>
<!-- Author: ace :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->
<!-- Comment: Sat Aug 17 20;46;53 comment by...ace -->
L2 Timeout. HRR recover.



<!-- Date: Sat Aug 17 21:05:44 2002 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2002&action=view&page=-6262&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-6262  width='300' align='top'></td><td>for my amusement</a></td></tr></table>
<!-- Author: SCICO -->


<!-- Date: Sat Aug 17 21:10:27 2002 -->
<pre>L2 Timeout . HRR recovery 
</pre>
<!-- Author: ace :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->
<!-- Comment: Sat Aug 17 21;10;59 comment by...ace -->
<pre>Mod done mask read from crate controller: 0x000008c1
Clearing L2 Fatal error counters.
Clearing L2 warning counters for all four buffers.
Error: Number of svt words is 1848 
   Setting Number of svt words to 290 
Error: Number of svt words is 440 
   Setting Number of svt words to 290 
SvtMon: SVT buffno 0 != event buffno 1
FaultmManager: Fatal error, sending L2 timeout directly
Stop or Recover stateStop or Recover stateFirst Event Started

</pre>



<!-- Date: Sat Aug 17 21:22:25 2002 -->
<pre>Trigger Inhibit by the CMX. 
When it cleared, we kept taking data. 
 
 
</pre>
<!-- Author: ace :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sat Aug 17 21:29:37 2002 -->
<center><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6263&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6263' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6264&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6264' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6265&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6265' width='300'></a></center>
<!-- Author: Kostas -->
<!-- Comment: Sat Aug 17 21;31;35 comment by...Kostas -->
<pre>CMX triped when LOSTPB spike went up to ~25 kHz around 21:15</pre>


<!-- Comment: Sat Aug 17 21;38;46 comment by...Kostas -->
<pre>Thanks Jason. The LOSTPB spike went up to only ~17 kHZ, not 25 kHz as written above (anyway, the plots says it all when read correctly!). The 17 kHz were enough to trip the CMX though.</pre>
<!-- Comment: Sat Aug 17 21;53;36 comment by...Kostas/Mario -->
Correction to the correction: the LOSTPB spike DID ACTUALLY GO to 25kHz as said above. We saw this in the fast time plots. The Lumberjack plotter however does not have the same resolution, so it looks like the spike is shorter.
<!-- Date: Sat Aug 17 21:37:48 2002 -->
<table><tr><td><a href=/elog/notebooks/2002/Sat_Aug_17_2002_Eve.213740.ps align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-6267  align='top'></td><td>XTRD dirac does not agree with Sim dirac.</a></td></tr></table>
<!-- Author: a stefanini -->
<!-- Comment: Sun Aug 18 00;14;25 comment by...spiropulu -->
Please check the XTRD monitor from a previous run (but after
Thursday aug 15 when some XTRP firmware changed). The errors may be related to a change in teh XTRP firmware (see eve shift
aug 15)
<!-- Date: Sat Aug 17 21:51:39 2002 -->
CMX trigger inhibits stopped the  run.   
<!-- Author: ace :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sat Aug 17 21:52:21 2002 -->
Called MCR to report spikes in antiproton losses.  They will 
look into it.  
<!-- Author: Ashmanskas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->



<!-- Date: Sat Aug 17 21:54:45 2002 -->
tripped channels in the CMx were reset and we resume the run.  
<!-- Author: ace :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sat Aug 17 21:57:25 2002 -->
<pre>LOSTPB in the Fast Time Plot spiked up to 25 kHz, again, and tripped CMX, again (SE 8-13, and NE 0-4).  
High Voltage brought up succesfully. 
 
Interesting observation: When plotting LOSTPB in the 1 hour interval, the spike shows up to go only up to 18 kHz. </pre>
<!-- Author: Kostas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sat Aug 17 21:58:06 2002 -->
<pre>Mod done mask read from crate controller: 0x000008c1 
Clearing L2 Fatal error counters. 
Clearing L2 warning counters for all four buffers. 
SvtMon: SVT buffno 3 != event buffno 0 
FaultmManager: Fatal error, sending L2 timeout directly 
Stop or Recover stateStop or Recover stateFirst Event Started 
 
</pre>
<!-- Author: ace :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->
<!-- Comment: Sat Aug 17 21;59;02 comment by...ace -->
<pre>L2 decision timeout.HRR recovery succesful.
</pre>
<!-- Date: Sat Aug 17 21:58:57 2002 -->
MCR called to say they lowered horizontal tunes to try to 
control spikes in losses.
<!-- Author: Ashmanskas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->



<!-- Date: Sat Aug 17 22:02:07 2002 -->
<pre>Mod done mask read from crate controller: 0x000008c1 
Clearing L2 Fatal error counters. 
Clearing L2 warning counters for all four buffers. 
 
 
In stopped state. 896407 L1A since last stop 
Bits0c 0x4007ff7f, Bits18 0x100,  Bits14 0x4fff  
Buffer number 2, load buffer 2 
First Event Started 
</pre>
<!-- Author: ace :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->
<!-- Comment: Sat Aug 17 22;03;02 comment by...ace -->
Dome timeout detected in COt_02



<!-- Date: Sat Aug 17 22:07:53 2002 -->
<pre>CMX tripped again (the usual upper SE part 8-11). 
High Voltage brought up succesfully. 
LOSTPB in the "Fast Time Plots" went at that moment up to ~22 kHz 
       (~14 kHz in the "Lumberjack" plot).</pre>
<!-- Author: Kostas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sat Aug 17 22:10:01 2002 -->
<pre>Mod done mask read from crate controller: 0x000008c1 
Clearing L2 Fatal error counters. 
Clearing L2 warning counters for all four buffers. 
 
 
In stopped state. 327979 L1A since last stop 
Bits0c 0x4007ff7f, Bits18 0x100,  Bits14 0x4fff  
Buffer number 3, load buffer 3 
First Event Started 
 
 
Done timeout fcal01. We reset the crate and that's it. 
 
</pre>
<!-- Author: ace :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sat Aug 17 22:19:56 2002 -->
LOSTPB spikes to 23 kHz, CMX trips again.  
<!-- Author: Ashmanskas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sat Aug 17 22:22:53 2002 -->
<pre>CMX upper SE (8-13) and upper NE (0-4) triped again,  
in coincidence with a LOSTPB spike at ~22 kHz.  
High Voltage brought up fine.  </pre>
<!-- Author: Kostas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sat Aug 17 22:24:03 2002 -->
<pre>CMX triger inhibit again. 
Cleared and taking data again. 
</pre>
<!-- Author: ace :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=15008' target=_top>15008</a>) -->


<!-- Date: Sat Aug 17 22:30:17 2002 -->
22kHz spike in LOSTP.  Called MCR.  They said it was an 
unannounced flying wire.  They're working with their TeV 
experts on some diagnostic measurements.  They see a 
periodic instability on the Schottky plates about every 
8 minutes, which seems correlated with spikes in losses.  
<!-- Author: Ashmanskas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sat Aug 17 22:31:12 2002 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2002&action=view&page=-6268&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-6268  width='300' align='top'></td><td>.</a></td></tr></table>
<!-- Author: Kostas/Mario -->


<!-- Date: Sat Aug 17 22:39:22 2002 -->
<pre>The victims of the spike in the proton losses (~22.5 kHz) were:  
 CMX:  SE 8-11, NE 2-3, SW 9-10 and NW 0-3. 
 CMP: Top, and almost half of South and North walls. 
 
</pre>
<!-- Author: Kostas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sat Aug 17 22:40:30 2002 -->
We try a H/R/R to see if ISL chip status red spots 
go away.  
<!-- Author: scico :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->
<!-- Comment: Sat Aug 17 22;49;02 comment by...scico -->
Arnoldo says that fixed 2 of the 4.

<!-- Date: Sat Aug 17 22:47:58 2002 -->
<center><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6271&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6271' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6272&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6272' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6273&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6273' width='300'></a></center>
<!-- Author: Kostas -->




<!-- Date: Sat Aug 17 23:02:43 2002 -->
One of the L3 output node b0l3u02 crashed. 
  
<!-- Author: ace :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sat Aug 17 23:33:44 2002 -->
<center><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6274&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6274' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6275&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6275' width='300'></a><a href='/cgi/elog/elog.pl?nb=2002&action=view&page=-6276&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/elog.pl?nb=2002&action=view&page=-6276' width='300'></a></center>
<!-- Author: Kostas -->


<!-- Date: Sat Aug 17 23:37:12 2002 -->
<table><tr><td><a href=/elog/notebooks/2002/Sat_Aug_17_2002_Eve.233703.ps align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-6278  align='top'></td><td>After the HRR the situation of ISL and SVX seems to deteriorate. See the four following plots.</a></td></tr></table>
<!-- Author: a stefanini -->


<!-- Date: Sat Aug 17 23:39:56 2002 -->
<table><tr><td><a href=/elog/notebooks/2002/Sat_Aug_17_2002_Eve.233950.ps align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-6280  align='top'></td><td>see above</a></td></tr></table>
<!-- Author: a stefanini -->


<!-- Date: Sat Aug 17 23:40:43 2002 -->
<pre>Busy timeout due to crate VRB_SVX_02 
HRR to recover. 
</pre>
<!-- Author: ace :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sat Aug 17 23:40:45 2002 -->
<table><tr><td><a href=/elog/notebooks/2002/Sat_Aug_17_2002_Eve.234037.ps align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-6282  align='top'></td><td>see above</a></td></tr></table>
<!-- Author: a stefanini -->


<!-- Date: Sat Aug 17 23:41:42 2002 -->
<table><tr><td><a href=/elog/notebooks/2002/Sat_Aug_17_2002_Eve.234135.ps align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-6284  align='top'></td><td>the last of ISL & SVX saga</a></td></tr></table>
<!-- Author: a stefanini -->


<!-- Date: Sat Aug 17 23:44:52 2002 -->
<pre>CMX trigger inhibits. 
Cleared and taking data again. 
</pre>
<!-- Author: ace :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sat Aug 17 23:47:38 2002 -->
<pre>CMX upper SE (8-11) and upper NE (2-3) tripped again. 
High voltage brought upper succesfully. </pre>
<!-- Author: Kostas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sat Aug 17 23:50:54 2002 -->
<pre>Busy timeout. HRR recovery 
</pre>
<!-- Author: ace :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sat Aug 17 23:52:46 2002 -->
<pre>Output node 02 of the Level3 farm should be OK now 
It lost the network fro some reason at about 18:30 this evening  
 
So , aces should  clean up Level3 after they end the run to be able to use all of Level3 farms </pre>
<!-- Author: Arkadiy -->

<!-- Date: Sat Aug 17 23:55:34 2002 -->
<TABLE BORDER=2 CELLSPACING=1 CELLPADDING=1 BGCOLOR=WHITE>
<TR BGCOLOR=YELLOW><TH>Run Number</TH>
<TH>Data Type</TH>
<TH>Physics Table</TH>
<TH>Begin Time</TH>
<TH>End Time</TH>
<TH>Live Time</TH>
<TH>L1 Accepts</TH>
<TH>L2 Accepts</TH>
<TH>L3 Accepts</TH>
<TH>Live Lumi, nb-1</TH>
<TH>GR</TH>
<TH>SC</TH>
<TH>RC</TH></TR>

<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150079>
150079</A> 
x24A3F</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_02 [2,131,312]</TD>                                       
<TD ALIGN=CENTER>14:07:09</TD>                                          
<TD ALIGN=CENTER>16:09:36</TD>                                          
<TD ALIGN=CENTER>01:23:53</TD>                                          
<TD ALIGN=RIGHT>     42,434,973</TD>                                    
<TD ALIGN=RIGHT>     1,080,229</TD>                                     
<TD ALIGN=RIGHT>     209,349</TD>                                       
<TD ALIGN=RIGHT>   96.218</TD>                                          
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD>                                                 
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD><A HREF=/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080>
150080</A> 
x24A40</TD>                                                             
<TD>BEAM</TD>                                                           
<TD>PHYSICS_1_02 [2,131,312]</TD>                                       
<TD ALIGN=CENTER>16:17:11</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>06:27:03</TD>                                          
<TD ALIGN=RIGHT>    126,175,289</TD>                                    
<TD ALIGN=RIGHT>     3,202,923</TD>                                     
<TD ALIGN=RIGHT>     679,454</TD>                                       
<TD ALIGN=RIGHT>  300.610</TD>                                          
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER></TD>                                                  
<TD ALIGN=CENTER>1</TD></TR>                                            
                                                                        
<TR><TD>Totals</TD>                                                     
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD>23:55:01</TD>                                                       
<TD ALIGN=CENTER>07:50:57</TD>                                          
<TD ALIGN=RIGHT>     168,610,262</TD>                                   
<TD ALIGN=RIGHT>      4,283,152</TD>                                    
<TD ALIGN=RIGHT>       888,803</TD>                                     
<TD ALIGN=RIGHT>  396.828</TD>                                          
<TD> </TD>                                                              
<TD> </TD>                                                              
<TD> </TD></TR>                                                         
                                                                        
</TABLE>
<!-- Author: End of Shift Report -->



<!-- Date: Sun Aug 18 00:00:34 2002 -->
CMX trigger inhibits. 
Cleared and taking data again. 
End of our shift. 
 
  
<!-- Author: ace :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sun Aug 18 00:01:48 2002 -->
<pre>CMX tripped again: SE 8-13 and NE 2-3. 
Again due to spikes in pbar losses (LOSTPB in the Fast Time Plot). 
Brought High Voltage back up fine.</pre>
<!-- Author: Kostas :: (run <a href='/java/cdfdb/servlet/RunSummary?RUN_NUMBER=150080' target=_top>150080</a>) -->


<!-- Date: Sun Aug 18 00:05:33 2002 -->
<table><tr><td><a href=/cgi/elog/elog.pl?nb=2002&action=view&page=-6285&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/elog.pl?nb=2002&action=view&page=-6285  width='300' align='top'></td><td>Luminosity summary</a></td></tr></table>
<!-- Author: Kostas/Mario -->
<!-- Comment: Sun Aug 18 00;07;38 comment by...b -->
<pre>
<!-- Date: Sun Aug 18 00:07:01 2002 -->
<font color=red>Shift Summary: </font>	 
Evening shift summary, 2002-08-17. 
 
Status at 16:00: store 1668 in since about 14:00.  L=1.7E31.  Stack at 
19mA, stacking at 9mA/hr.  Inherit run 150079 from day shift. 
L1/L2/L3 rates 7500/190/37Hz, DT=0.8%, rfmterr=0.72%.  Trigger table 
PHYSICS_1_02[2,131,312].  Experts working on L00, SVXII trips; 
diagnosing unexpected trigger inhibit behavior. 
 
New Aces on overlap shift: Kostas Kordas (Monitoring) and Ricardo 
Eusebi (DAQ). 
 
At 16:09, run 150079 was ended at SVX experts' request, after a series 
of trigger inhibit tests. 
 
At 16:17, run 150080 began.  XFT02 DTO at 16:17, PCAL DTO at 16:40, 
high (1.85%) instantaneous reformatter error rate warning at 16:42, 
L2TO (saw 0 L1 DMA transfers) at 16:56, 1.01% rfmtr warning at 17:08. 
 
As of 17:13, L1/L2/L3 rates 6600/165/31Hz, DT=0.7%, refmterr=0.7%. 
 
At 19:49, many SVX+ISL wedges spontaneously powered down: SB0W0, 
SB0W2, SB0W3, SB1W0, SB1W2, SB1W3, SB2W2, SB2W3, IB0W1.  We detected 
this as a trigger inhibit, not a trip; there was no HV alarm.  IFIX 
indicated that some HV was off, but treated this event as a deliberate 
power-down, not a trip, so there was no alarm.  Andy Hocker said that 
IFIX itself had behaved according to specification.  He thinks the 
power-down was the result of one crate getting confused; the cause of 
this needs investigation.  Gino thinks that at least the SVX PS GUI 
should have treated this as a trip; he would like this to be 
understood. 
 
At 20:00, while we investigate SVX trips, CMX SE 8,9 trips.  We become 
wary of spikes in the pbar losses 
 
Around 21:30, SVX pager (Jason) reaffirmed that he was comfortable 
leaving SVX in the run for now, and said he would continue to monitor 
the hourly loss plots in the e-log from home. 
 
At 21:45, CMX tripped.  Seems correlated with a spike in LostPB to 
about 27kHz.  Spoke with Abraham Gallas; he suggested we ask MCR to 
improve the losses.  Called MCR.  They will look into it.  CMX back up 
at 21:53.  At 21:58, MCR called to say they lowered horizontal tunes 
and expect this to improve the pbar losses. 
 
At 22:04, CMX (SEU 8-11) tripped again.  22kHz spike in LOSTPB. 
Abraham wants a picture of LOSTPB, B0PAGC every time CMX trips. 
 
22:15 Gino Bolla is concerned that we had lost HV in several wedges without 
CAEN GUI reporting a trip.  Wants us to make sure this gets 
investigated.  Suggests we call CAEN GUI pager (again). 
 
22:19 LOSTPB spike to 23kHz, CMX SEU and NEU trip.   
 
22:26 CMP and CMX trip, due to an unannounced flying wire measurement. 
 
22:40 Try H/R/R to see if red spots in ISL chip status map go away. 
It works for 2 out of 4 of them?!?! 
 
22:45 Dave in MCR calls to say they've implemented a number of changes 
suggested by the on-call TeV expert to try to get the losses down. 
 
23:02 One L3 output node dies.  We can still run, but want to find out 
whether expert intervention is needed; if so, we should deal with it 
before owl shift.  Called expert; he wants to come in. 
 
23:16 CO notes that mean Zvertex position seems to have moved away 
from zero (is now about 1.5 cm) since yesterday 
 
--- 
 
Run 150080 has been going continuously from near the start of this 
shift, with only a few DAQ problems: 8 L2 decision timeouts, 7 done 
timeouts, 2 busy timeouts. 
 
Throughout shift, losses not high, but show significant spikes. 
Worked with MCR to get this under control, after several muon HV 
trips.  HV trips and loss spikes were biggest source of trouble this 
shift.   
 
CMX expert (Abraham) would like us to put a plot of C:LOSTPB and 
C:B0PAGC into the e-log when CMX trips. 
 
Gino Bolla asked us to contact SVX PS GUI expert about GUI not 
reporting unpowered ladders as trips.  We tried the pager 3 times over 
the course of the shift and got no response. 
 
Kathy Copic is stranded in Cleveland with a flat tire.  She says 
Nathan Goldschmidt will fill in for her tomorrow as Day shift Ace. 
 
b0fcal00 and b0fcal01 (once each) did not H/R/R normally and required 
a reboot.  (b0fcal00 was rebooted at 20:26, and b0fcal01 at 22:08.) 
 
TauMon plot shows some funny features: scico e-mailed Pasha Murat. 
 
BMU occupancy plots have some features noted by CO: scico e-mailed 
Hyunsoo Kim. 
 
SvxMon features noted by CO: paged silicon.  Jason asked us to note it 
in the logbook so that experts can diagnose.  Scico also made a note 
in the SVX logbook to this effect. 
 
TrigMon L1 DIRAC data and simulation disagreements noted by CO: we 
called L1 calorimeter trigger pager.  Expert (smaria) suspects that this is 
related to XTRP firmware change on the evening of August 15, for which 
a comment was made that "updated TrigMon is not ready yet."  Maria 
is on a slow dial-in connection and would like someone to check 
whether this disagreement seems to have suddenly appeared on the 15th. 
 
CPR wedge occupancy plot and COT cell occupancy plot noted by CO have 
not yet been followed up with pages or e-mails.  Next shift may want 
to try to e-mail some relevant contact people so that these plots are 
looked at by experts. 
 
--- 
 
possible work other than data taking: 
 
  * COT experts may come by to test COT HV trip trigger. 
 
  * Kaori may come by at end of store to test new consumer code. 
 
--- 
 
Dumb question: Is there a way to monitor losses vs time from SciCo's 
terminal? 
   
<br><b>End of Shift Numbers</b>
<table><tr><td width=60% align=left>
<b>CDF Run II <p></b>                       
<pre>
Runs                   150080
Delivered Luminosity   372.9  
Acquired Luminosity    308.8  
Efficiency             82.8
   
</td></tr></table></pre>
<!-- Author: Bill Ashmanskas -->
<!-- Comment: Sun Aug 18 00;08;12 comment by...b -->
</pre>