<!-- Title: 11:07:40  Mon May 12 2003 -->
<!-- Crewchief: Gino  -->
<!-- Op1: Reiner as a shadow -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>New week: 
Store 2538 ongoing 
Run 162837 ongoing</pre>


<!-- Date: Mon May 12 11:09:54 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1955&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1955  width='300' align='top'></td><td>Things are looking good now (run # 162837).</a></td></tr></table>
<!-- Author: Gino -->

<!-- Date: Mon May 12 12:16:55 2003 -->
Entered in Silicon database: <ul>
<li>Ladder: IB3W0/1All 
<li>problem n: 995 
<li>Status: fixed
<li>Type: Unknown
<li>Action: Monitoring
<li>entered by: tompkins 
<li>description is: no new problems, only low rates of error on individual ladder basis, not wedge/board wide problem.  buffed ftm worked, fixed. 
<li>See <a href="http://nkek15.fnal.gov/cgi-bin/newproblemdb/DBreport.pl?detsys=ISL&wedge=B3W0/1&layer=All">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Mon May 12 14:40:57 2003 -->
<p><font size=+1> D-mode Calibration Job Finished </font>
<p><pre>
This is a computer-generated silicon D-mode calibration status report.
Silicon D-mode calibration job 260 FAILED.

Job info
--------
Calibration type : standard
DPS-off run      : 162823
DPS-on run       : 162821
Requested by     : Jason Nielsen
Submitted        : 2003-05-12 07:06:18
Completed        : 2003-05-12 12:40:49
Status           : FAILURE
Comment          : resubmit with disk space low

Last processing stage reported : comparing to the reference calibration
Failure diagnostics :
The calibration failed the following reference comparison criteria:

           Cut Name    Sensor Set   Cut Value   Data Value
-------------------    ----------   ---------   ----------
dps_off_ped abs max    SVX_PHI      0.5         0.99997   
 dps_on_ped abs max    SVX_PHI      0.2         0.371782  
 dps_on_ped abs max    ISL_PHI      0.25        0.278914  
 dps_on_ped abs max    ISL_Z        0.25        0.259791  
   bad_chan rel max    SVX_Z        0.25        0.25829   

Reference calibration runs are 159187 (DPS off) and 159186 (DPS on)
</pre>
<!-- Author: D-mode script run_calib_analysis.tcl -->
<!-- Comment: Wed May 14 12;37;49 comment by...Jason -->
This calibration has been checked and submitted to the database.
<!-- Date: Mon May 12 17:09:53 2003 -->
<pre>Just for you all to know this week Reiner will carry my personal pager (2189520) while I (Gino) have the general silicon pager (2188227).</pre>
<!-- Author: Gino -->


<!-- Date: Tue May 13 01:32:50 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1956&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1956  width='300' align='top'></td><td>Store 2540 ongoing. Run 162856 ongoing. Few minutes with the TEL OFF. A close eye on the Abort Gap losses from both CDF-CR and MCR.</a></td></tr></table>
<!-- Author: gino -->


<!-- Date: Tue May 13 12:39:17 2003 -->
<h1> News </H1> 
As of this morning at the 8:30 am ACEs meeting the Decision to raise the Silicon HV is up to TeVMon and the ACEs. 
<p>I will try to monitor the performance during my week of carrying the pager
<!-- Author: Gino -->
<!-- Comment: Tue May 13 12;41;06 comment by...Gino -->
<a href='http://www-cdfonline.fnal.gov/internal/ops/svxii/runii/ace_mon.html'> modified instruction for the ACEs </a>.


<!-- Date: Tue May 13 17:41:37 2003 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1957&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1958' ></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1959&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1960' ></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1961&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1962' ></a></center>
<!-- Author: Gino and Andy -->
<!-- Comment: Tue May 13 17;47;34 comment by...Gino and Andy -->
Investigating <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=153&frame=2&anchor=14:46:05'>
spontaneous resets of CAEN crates </a>.
Following the assumption that the resets signal can be spuriously induced by the reset buttons on the first floor Andy and I decided to power down (umplug the power cord) for the 5V power supply that allows you to reset a CAEN crate. 

So if there is a need to reset a CAEN crate we should first plug the power cord back into the power strip and only after that reset the selcted crate.
</p>

The IDEA here is that if we get another failure with this power supply OFF we will have to <b>dig</b> in some other direction.



<!-- Date: Tue May 13 23:43:54 2003 -->
<p>Store 2542 ended due to a quench during scraping in the F sector. 
<p>Silicon Passed coldstart with flying colors (GREEN). 
<p>There is a PINK in IMON on AVDD for SB0W7L4. 
<p>This ladder already has a broken jumper.  
<p>From <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2003&action=view&page=399&frame=2&anchor=23:38:51.390'> 
the IMON short term current plot </a> the ladder was already misbehaving before the quench. So I guess not a big deal. 
<p>I will dig deeper tomorrow (there is NOT going to be collisions in a while.
<!-- Author: Gino -->
<!-- Comment: Wed May 14 00;02;29 comment by...Gino -->
<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2003&action=view&page=399&frame=2&anchor=23:54:23.402'>
A link to the output of the BLM buffer </a>. The abort was clean but before the beam was aborted (due to the quench) the losses were steadily growing. Shortely this is not the worse kind of quench we can get. 

<!-- Comment: Wed May 14 00;16;45 comment by...Gino -->
Checked the readback of the initialization bitstream for SB0W7L4 
from the log file and even post quench it is still reading back some garbled bits (as most of the ladders with a broken jumper) so the AVDD2 failure can be ruled out. I think there is nothing new wrong with this ladder but I'll check deeper tomorrow.
I lower the IMON to 930 mA.
</p>
<b> things look pretty much the same as before the quench </b>

<!-- Date: Wed May 14 00:31:22 2003 -->
Entered in Silicon database: <ul>
<li>Ladder: IB1W5L4 
<li>problem n: 657 
<li>Status: needs_observation
<li>Type: RX
<li>Action: Monitoring
<li>entered by: tompkins 
<li>description is: Bit 5 low problem seen again(162825-5.5%, 162820-.2%).  Low rate, not much to worry about now, but should be observed to see if rate increases. 
<li>See <a href="http://nkek15.fnal.gov/cgi-bin/newproblemdb/DBreport.pl?detsys=ISL&wedge=B1W5&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Wed May 14 11:59:16 2003 -->
<center><a href='/elog/notebooks/silicon/Wed_May_14_2003_115834.ps'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1964' ></a><a href='/elog/notebooks/silicon/Wed_May_14_2003_115850.ps'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1966' ></a></center>
<!-- Author: SCN -->
<!-- Comment: Wed May 14 12;05;09 comment by...SCN, Lester, Greg -->
<h3>Testing using SRC in tracer mode</h3>
Above you have the online histograms for 4 HDIs in the diagnostic crate (the 5th, on L0, gives CAP ID errors, so I left it out)
in two modes <ul>
<li> Left: Normal "Laser" run using SRC in normal mode
<li> Right: Cosmics/Beam Run using TS in Calibration Continuous
Mode at very low rate (so the svxdaq readout machinery can keep up)
</ul>
The only noticable difference is that the Noise in standalone mode is considerably higher, but this could just be due to the rate (this was DPS OFF).  Most importantly there were no DAQ errors of any type.  We have discovered a few operational issues
that will need to be worked around, however...experts thinking about it now.


<!-- Comment: Wed May 14 17;13;29 comment by...SCN -->
This SRC and an SRCTM now are installed in b0svx06.
Soon (but not during a store) we can work on hooking them up
at least to test configuration etc.
<!-- Date: Wed May 14 16:53:46 2003 -->
Entered in Silicon database: <ul>
<li>Ladder: IB5W0L4 
<li>problem n: 888 
<li>Status: needs_observation
<li>Type: DRO_00F3
<li>Action: Monitoring
<li>entered by: tompkins 
<li>description is: Rate is slowly increasing, now about 1%-2.5% in almost 100% of runs.  Should keep under observation. 
<li>See <a href="http://nkek15.fnal.gov/cgi-bin/newproblemdb/DBreport.pl?detsys=ISL&wedge=B5W0&layer=L4">report</a></ul>
<!-- Author: Problem DB -->




<!-- Date: Wed May 14 19:11:54 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1967&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1967  width='300' align='top'></td><td>Store 2546 ongoing, Run 162937 ongoing, SVT coverage as usual.</a></td></tr></table>
<!-- Author: Gino -->


<!-- Date: Wed May 14 21:44:26 2003 -->
<pre>Paged for PINKs in TeVMon (spiky losses). Beam seems to be OK. 
</pre>
<!-- Author: Gino -->
<!-- Comment: Thu May 15 02;57;55 comment by...Gino -->
<pre>Again around midnight. Considering they lost the store I now have second thoguhts.</pre>
<!-- Date: Thu May 15 02:51:50 2003 -->
Store 2546 ended in a quench in the F sector (again). 
<li> Silicon passed COLD-START (CONFIG now) (one smal hick-up with b0svx07 solved by trying again) 
<li> No pinkies in IMON 
<li> From the <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2003&action=view&page=403&frame=2&anchor=02:43:52.200'> 
BLM buffer PLOT </a> it looks like the abort was very clean. 
</p> 
Looks like we are in the same status as before.  
<!-- Author: Gino -->



<!-- Date: Thu May 15 11:05:40 2003 -->
The following channels have had their init readback unmasked:<p> 
IB4W4L2<br> 
SB2W4L4<br> 
SB2W9L1<br> 
SB2W9L4<p> 
All have not had init failures for some time now.
<!-- Author: D. Clark -->


<!-- Date: Thu May 15 14:22:50 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1968&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1968  width='300' align='top'></td><td>Store #2549 ongoing. Run # 162982 ongoing. SVT looking as expected.</a></td></tr></table>
<!-- Author: Gino -->


<!-- Date: Thu May 15 14:33:56 2003 -->
<pre>TevMON Green: Event = 341709 Time:  15-May-03 14:07:52  
TEL DOWN:     Event = 342189 Time:  15-May-03 14:23:57 
TEL BACK UP:  Event = 342292 Time:  15-May-03 14:27:24  
 
This is for the RECORD they were doing studies with the TEL.</pre>
<!-- Author: Gino -->


<!-- Date: Fri May 16 16:59:49 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1969&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1969  width='300' align='top'></td><td>Store 2551 ongoing, Run 163012 ongoing, SVT coverage as expected. Hopefully at the end of this store we will be running with 2 SRCs. </a></td></tr></table>
<!-- Author: Gino -->


<!-- Date: Sat May 17 01:36:35 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1970&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1970  width='300' align='top'></td><td>LB1W3L0 DVDD current from IMON. To be noticed that it goes all the way to 15 A when the PS can only supply 1/2 A.</a></td></tr></table>
<!-- Author: Gino -->
<!-- Comment: Sat May 17 01;49;07 comment by...Gino and Rainer -->
<pre>The Power supply on DVDD of L0 of LB1W3 started to read the 
absurd value at this time stamp:
New time 85551 Fri May 16 23:45:51 2003
Here are the number for 05/16/03
 174  2 -32765     5.21 14978.00 
 174  2 -32765     5.20 14939.00 
 174  2 -32765     5.22 14970.00 
 174  1      3     0.57 14877.00 
 174  1      3     0.11 14878.00 
 174  1      3     0.11 14879.00 
 174  1      3     0.11 14878.00 
 174  1      3     0.11 14879.00 
 174  1      3     0.11 14878.00 
 174  1      3     0.11 14879.00 
 174  2 -32765     5.20 15156.00 
 174  2 -32765     5.20 14940.00 
 174  2 -32765     5.20 14939.00 
 174  1      3     0.54 14877.00 
 174  1      3     0.11 14879.00 
 174  1      3     0.11 14878.00 
 174  1      3     0.11 14879.00 
 174  1      3     0.51 14877.00 
 174  1      3     0.52 14877.00 
 174  1      3     0.11 14879.00 
Similar number are available for 05/17. 
The problem was solved by power cycling the RACK (SETOP or SVSET).
After power cycling the RACK we got all VRB crates yellow and so 
I reboot them all (maybe excessive).
Later on b0fib05 the FER messenger died twice.

What is questionable is the following:
1. The HIGH current value was always there indenpendently from 
   the status of the PS.
2. As a consequence even when the whole wedge was out of the 
   HDWDB it was still tripping on software protection trip.
3. in these conditions the Trigger Inhibit was still red.

Basically I was in a position where for a problem in a single 
ladder I WAS NOT ABLE TO TAKE ANY DATA WITH the whole L00 (and 
probably ISL as well I need to ask Andy). We might wanna review 
the logic of the trigger inhibit with respect to this failure 
mode.</pre>


<!-- Comment: Sat May 17 13;54;39 comment by...SCN -->
There is a problem deep deep in the code where when the partition number is -1 (i.e. the crates are not in a partition) and a FFO
sends the "resynch please" (which it sends to all the VRB crates)
the VRB crates hang.  I attempted to find this last week, but
to no avail (I believe something somewhere is accessing array element -1).  If you know you are going to unsynch a FFO (by power cycling a crate) it is best to put the VRB crates in a partition beforehand.  WE'll try to find this bug this week (and
the "instruction 0" one which makes the Fib crates hang sometimes)
<!-- Date: Sat May 17 07:44:21 2003 -->
<h3>Two SRC testing</h3> 
Came in early this morning to test running two SRCs. 
Procedure was<ul> 
<li> Take <a href='http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163026'>Reference Run 163026</a> in partition 7 
<li> Power DOWN ISL and L00, and move the FFO and VRB cables 
to the new SRC.  This seemed to cause various DVDD trips b0fib02. 
It is possible that this is due to the attempt at auto-synching the Glinks. 
<li> Install new RC code in the TSI crate, and turn online 
the SRC_01 and TRACER_124 card in VRB_ISL_06 via the cardEditor 
<li> Attempted runs <a href='http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163031'>163031</a> and <a href='http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163032'>163032</a> in "Two SRC Mode"<P> 
Basically, the SCPU complained every few hundred events about <pre>(MLE) b0l3gate1.fnal.gov:main:5:35:31 AM->Host b0eb25.fnal.gov, task tRec_0 
SCPU-P7-E-EventIdMismatch: VRB ID for event 129 is 0 instead of 1 (slot 8).</pre> 
for each of the 21 VRBs used by the second SRC.  Basically, it happened 16 times in 2184 L2A in 163031, 8 times in the 681 L2As 
in 163032.  All but once it was "0 instead of 1", one time it 
was "0 instead of 2".  This error (I think) relates to the  
Event ID coming from the TS through the SRC to the VRBs with 
the L2A message.  I am not quite sure how it gets this 
with "express L2As" so I will ask the experts.  Still, this is the lowest 4 bits of the actual event number, so it should basically go 0->f for each L2A and then roll, and it did not error on every single event.  I suspect this could be due to<ul> 
<li>Flaky connection between SRC and VRBs 
<li>Unforseen timing intricacy in the special L2A message (not sure how the SRC receives the Event Number) 
<li> Tweaking the Taxi connection for the new SRC-TS 
</ul> 
<li> In addition, these runs were so short that SVXMon didn't get a chance to look at things.  We need to run SVXMon on these runs offline to see if there is any trouble. 
<li> The run control side of things looks ok, although there 
were a few hiccups there to that we had to deal with... 
</ul> 
Pending the results of SVXMon and the hypotheses regarding the  
VRB ID problem, we will probably hold off until study time for 
our next shot at gettting this in.
<!-- Author: SCN -->
<!-- Comment: Sat May 17 09;28;48 comment by...SCN -->
I attempted to look with SVXMon offline, did not get too far,
but what is fairly shining <ul>
<li> VRBs out of synch error (I don't really know what this means) maybe every event?  If this were the SCPU error above, I wouldn't expect to see these in the data.  I believe the TSL1A word only gets checked at the reformatter, and only inside a 
bank-since our SRC split is also a split between SIXD and ISLD,
probably this is why the RF doesn't pick it up but perhaps 
SVXMon does
<li> BXING out of synch every event.  ISL and L00 are only
getting BXING == 0.  Since BXING gets sent at READOUT, perhaps this is not non-sensical.  Not sure what we thought it should be.
<li> Pipeline out of Synch: All ISL and L00 are internally synched but not synched with SVXII.
</ul>
I can probably do some tests with the 2nd SRC to see what it
sends, also should ask the experts, and someone ought to run SVXMon on these runs other than me to see what they think.
I copied every file I could find in the look area for these two
runs to /cdf/data05/s1/hardware/svxii/workarea on fcdfsgi2.<P>
I need a rest.  The Si is back in nominal configuration for the
next shot.



<!-- Date: Sat May 17 15:15:48 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1971&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1971  width='300' align='top'></td><td>Store 2555 ongoing, Run 163064 ongoing, SVT coverage as expected. </a></td></tr></table>
<!-- Author: Gino -->


<!-- Date: Sat May 17 17:02:27 2003 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1972&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1972' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1973&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1973' width='300'></a></center>
<!-- Author: Gino -->
<!-- Comment: Sat May 17 17;05;48 comment by...Gino -->
SVT coverage for the 2 runs taken last night with <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=154&frame=2&anchor=07:44:21'>
 2 SRCs.</a>  No big surprises (nothing should have changed in SVXII. Good to know that on this side there is nothing wrong. The real question is how ISL and L00 where behaving and that might take a little longer. The 2 runs have extremely low statistic.


<!-- Date: Sun May 18 13:54:50 2003 -->
Beam was lost during scraping. Passed cold start, no pinkies in IMON and the BLMs buffer shows a <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2003&action=view&page=413&frame=2&anchor=13:05:38.044'> 
CLEAN ABORT </a> 
<!-- Author: Gino -->


<!-- Date: Mon May 19 06:48:13 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1974&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1974  width='300' align='top'></td><td>Store 2562 ongoing, Run 163113 ongoing, new hole on SVT coverage. SB2W9 is empty. I'm gonna try to dig more into this.</a></td></tr></table>
<!-- Author: Gino -->


<!-- Date: Mon May 19 06:55:22 2003 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1975&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1975  width='300' align='top'></td><td>SB2W9L1 is responsible for the new hole in SVT</a></td></tr></table>
<!-- Author: Gino -->
<!-- Comment: Mon May 19 07;01;09 comment by...GIno -->
<pre>No much data is coming out of it:
5420740 670f 6801 a511 a411 a311 a211 a111 a011
5420760 7f01 c1c1 0000 e290 09fc 8311 00f3 2af3
5421000 2b09 2cf3 3407 3514 36f3 3730 380b 3903
5421020 8211 7f02 c1c1 &lt;b&gt;e291 09fc c2c2 0000&lt;/b&gt; e292
5421040 09fc 8911 0002 6104 620f 6307 8811 8711
5421060 8611 21f3 2226 2304 8511 a411 7205 730e
5421100 74f3 a311 a211 a111 2307 2416 250f 2603
5421120 a011 7ff3 c1c1 e293 09fc 8911 00f3 8811
5421140 8711 8611 8511 1a03 1b0f 1c09 1df3 3e07
5421160 3f26 4008 4107 4209 4307 8411 a311 a211
</pre>


<!-- Comment: Mon May 19 07;48;53 comment by...Gino -->
<pre>We will investigate this problem at the end of the store and if necessary request an access.</pre>
<!-- Comment: Mon May 19 07;49;33 comment by...SCN -->
Also it fails to init, starting this morning at 4:04
<pre>
b0dap30.fnal.gov% grep -e " e291" 20030518.si_daq ; grep -e " e291 " 20030519.si_daq
05-18-2003 00:15:16 CDT : 1053234916 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 00:21:59 CDT : 1053235319 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 00:23:20 CDT : 1053235400 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 00:45:51 CDT : 1053236751 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 00:55:21 CDT : 1053237321 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 00:56:52 CDT : 1053237412 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 01:34:25 CDT : 1053239665 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 01:37:50 CDT : 1053239870 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 01:55:43 CDT : 1053240943 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 02:04:04 CDT : 1053241444 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 02:29:18 CDT : 1053242958 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 02:51:10 CDT : 1053244270 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 02:53:25 CDT : 1053244405 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 02:58:18 CDT : 1053244698 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 03:43:19 CDT : 1053247399 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 04:49:27 CDT : 1053251367 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 04:56:30 CDT : 1053251790 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 05:15:52 CDT : 1053252952 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 05:16:09 CDT : 1053252969 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 05:17:46 CDT : 1053253066 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 05:56:44 CDT : 1053255404 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 05:58:21 CDT : 1053255501 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 06:39:28 CDT : 1053257968 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 12:33:05 CDT : 1053279185 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-18-2003 13:12:24 CDT : 1053281544 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-19-2003 03:44:41 CDT : 1053333881 Init Fib e291 hdi 16 SUCCESS for 6 chips
05-19-2003 04:04:04 CDT : 1053335044 Init Fib e291 hdi 16 ERRORS: tried 6 chips:
05-19-2003 04:04:58 CDT : 1053335098 Init Fib e291 hdi 16 ERRORS: tried 6 chips:
05-19-2003 04:08:33 CDT : 1053335313 Init Fib e291 hdi 16 ERRORS: tried 6 chips:
05-19-2003 04:11:59 CDT : 1053335519 Init Fib e291 hdi 16 ERRORS: tried 6 chips:
05-19-2003 04:21:43 CDT : 1053336103 Init Fib e291 hdi 16 ERRORS: tried 6 chips:
05-19-2003 05:34:40 CDT : 1053340480 Init Fib e291 hdi 16 ERRORS: tried 6 chips:
05-19-2003 05:49:26 CDT : 1053341366 Init Fib e291 hdi 16 ERRORS: tried 6 chips:
05-19-2003 05:54:05 CDT : 1053341645 Init Fib e291 hdi 16 ERRORS: tried 6 chips:
05-19-2003 06:09:42 CDT : 1053342582 Init Fib e291 hdi 16 ERRORS: tried 6 chips:
05-19-2003 06:43:57 CDT : 1053344637 Init Fib e291 hdi 16 ERRORS: tried 6 chips:
05-19-2003 07:14:30 CDT : 1053346470 Init Fib e291 hdi 16 ERRORS: tried 6 chips:
05-19-2003 07:17:14 CDT : 1053346634 Init Fib e291 hdi 16 ERRORS: tried 6 chips:
</pre>


<!-- Date: Mon May 19 07:15:01 2003 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1976&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1976' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1977&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-1977' width='300'></a></center>
<!-- Author: Gino -->
<!-- Comment: Mon May 19 07;25;05 comment by...Gino -->
Lots of RED CROSSES in the error logger. If you look at the two pictures above they are all in <b>b0fib02</b>. By looking deeper this fib crate went south at a certain point and then recovered. The errors are STUCK LADDER and 00f3. <b> This is pretty bad </b>
All the errors are between event # 137326 and event # 144451. I hope Steve can get more diagnostic by looking at errors in the fib crates (I do not really know if this is possible).

<!-- Comment: Mon May 19 08;16;58 comment by...SCN -->
Currently, last timestamp in the Boot Logger (which also logs
Glink Synch requests) is from Saturday, but this may be just
because it hasn't flushed its output queue (it buffers output
until it feels it has enough to warrant actually writing it).
I can add a line to flush immediately, but this won't help now.
I suggest someone reboot the diagnostic crate 12 times or so...
that should be enough to flush it out.  File you want to look at is  <i> /data1/hmon/log/runSVXBootLoader.log</i> <P>
In addition, in <a href='http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=163113'>Run 163113</a> there was a RUN command at
event 103586 at 04:07:52 and a HALT at 145811 04:11:08,  RECOVER at 146625 04:11:11 due to Stuck ladder in the same crate <pre>
(RC)   4:7:45 Halt -> HALTED
(RC)   4:7:52 Recover -> RECOVERED
(RC)   4:7:55 Run -> ACTIVE
9'54" b0dau32.fnal.gov:csl_mon_send:4:10:38 AM->CSL received a "BAD EVENT" in partition 0.[MLW]
(MLE) b0l3gate2:main:4:10:38 AM->Error on L3 node b0l3067 (partition 0) Mon May 19 04:10:38 2003 l3_node 
  in exe_exchange (l3_ana_exe.c:567)
  @L3_EXE_TERM:  Filter 0 died

11'6" b0dap63.fnal.gov:ConsumerErrorRe:4:11:51 AM->Runtime Error 2, Event 55, RunNum 163113: SvxMon Halt Recover Run: Stuck Cellid in 5 events in Silicon/S/B0/W4/L0/C0-1 . -->  
 Additional Information: 

 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
 
 Stuck Cellid S/B0/W4/L0/C0-1 . 
 AUTO HRR will be issued 
[MLE]
(MLE) b0dap73.fnal.gov:Start Error HandlerThread[Thread-66,6,main]:4:11:07 AM->Requested Halt-Recover-Run issued [errmon]
(RC)   4:11:11 Halt -> HALTED
(RC)   4:11:17 Recover -> RECOVERED
(RC)   4:11:21 Run -> ACTIVE
</pre>
Note there is some clock skew on the order of 40 secs (SVXMon caused the HRR 40 s AFTER it was done...) so from what know right now I'd say it is likely there was some problem either with the FFO or the backplane on this crate.  Not yet clear if it was
a Glink problem.  Previously we had <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=149&frame=2&anchor=16:26:57'>suspected</a> the FFO and in fact have already  <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=149&frame=2&anchor=15:26:49'>swapped it</a>. 




<!-- Date: Mon May 19 07:48:04 2003 -->
Entered in Silicon database: <ul>
<li>Ladder: SB2W9L1 
<li>problem n: 1036 
<li>Status: needs_work
<li>Type: RX
<li>Action: Incident
<li>entered by: bolla 
<li>description is: SB2W9L1 is responsible for a new hole in SVT <a href= 'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=154&frame=2&anchor=06:55:22'> the data look like c2c2 only </a>  I suggest at the end of the store to check if the problem is reproducible with svxdaq.  
<li>See <a href="http://nkek15.fnal.gov/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B2W9&layer=L1">report</a></ul>
<!-- Author: Problem DB -->






<!-- Date: Mon May 19 11:08:09 2003 -->
End of Week report: 
Paged 5 times total: 
Tuesday QUENCH 
Wednesday TeVMON warnings (spiky lossess) 
Thursday QUENCH 
Saturday LB1W3L0 Power supply went south 
Sunday QUENCH 
Other importatn things: 
MOnday morning new hole in SVT (SB2W9) that is being addresses at the end of this store. 
 
As far as the QUENCHES go I really would like the ACE to do the post-quench checkout and page us only if problems show up. 
 
The proposed instructions for the ACE are available at this <a href=  
'http://www-cdfonline.fnal.gov/internal/ops/svxii/runii/quench.html'> LINK </a> that you all should review. 
 
  
<!-- Author: Gino -->