<!-- Title: 09:24:56  Mon Mar  4 2002 -->
<!-- Crewchief: SiliPeople -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Run Silicon </pre>

<!-- Date: Mon Mar 4 12:22:43 2002 -->
<p><font size=+1> D-mode Calibration Job Started </font>
<p><pre>
Job id            : 92
Calibration type  : standard
Run period tag    : Mar_2002
DPS-off run       : 140261
DPS-on run        : 140259
Requested by      : muge karagoz unel
Main process id   : 16334578
</pre>
<!-- Author: CDF silicon D-mode calibration job server v1.3 -->


<!-- Date: Mon Mar 4 12:22:44 2002 -->
<p><font size=+1> D-mode Calibration Job Started </font>
<p><pre>
Job id            : 92
Calibration type  : standard
Run period tag    : Mar_2002
DPS-off run       : 140261
DPS-on run        : 140259
Requested by      : muge karagoz unel
Main process id   : 16334578
</pre>
<!-- Author: CDF silicon D-mode calibration job server v1.3 -->



<!-- Date: Mon Mar 4 12:45:12 2002 -->
<pre>pink fib01 during run 140258 coldstart (silicon calib)&lt;br&gt; 
(MLE) b0fib01:Messenger:12:06:13 PM-&gt;Error Initializing HDI Slot 9 Chan 4: ISL B0W2 
&lt;br&gt; and also fib00 for 260&lt;br&gt; 
(MLE) b0fib01:Messenger:12:15:59 PM-&gt;Error Initializing HDI Slot 9 Chan 4: ISL B0W2 
(MLE) b0fib00:Messenger:12:16:07 PM-&gt;Error Initializing HDI Slot 19 Chan 6: SVX B2Wb 
 
&lt;br&gt; and also for 140262: 
&lt;br&gt;(MLE) b0fib01:Messenger:12:28:27 PM-&gt;Error Initializing HDI Slot 9 Chan 4: ISL B0W2&lt;br&gt; 
they all passed at 2nd trial, runs 259, 261 and 263 should be ok.. 
</pre>
<!-- Author: muge karagoz unel -->


<!-- Date: Mon Mar 4 15:10:22 2002 -->
<pre>Made a series of (temporaary) changes to L00 bitsream motivated 
by recent data problems. 
 
1) Made sure all DPS off, quite a few were DPS ON 
2) Made sure all READALL 
3) Made sure all BIAS RATIO = 1 
4) Set all ISEL to L00 default per Tony's note 
5) Set all RAMP PED = 11 
 
Please do not change any of these settings without talking to me.</pre>
<!-- Author: Chris Hill -->

<!-- Date: Mon Mar 4 16:35:27 2002 -->
<p><font size=+1> D-mode Calibration Job Started </font>
<p><pre>
Job id            : 93
Calibration type  : standard
Run period tag    : Mar_2002
DPS-off run       : 140289
DPS-on run        : 140287
Requested by      : Your mom
Main process id   : 17718866
</pre>
<!-- Author: CDF silicon D-mode calibration job server v1.3 -->



<!-- Date: Mon Mar 4 17:57:13 2002 -->
<pre>IB4W3L4 bias voltage changed from 2.0 to 3.5V 
IB5W2L4 bias voltage changed from 2.0 to 3.5V 
 
These should now be in "high standby."</pre>
<!-- Author: Chris Hill -->

<!-- Date: Mon Mar 4 20:29:59 2002 -->
<p><font size=+1> D-mode Calibration Job Finished </font>
<p><pre>
This is a computer-generated silicon D-mode calibration status report.
Silicon D-mode calibration job 93 FAILED.

Job info
--------
Calibration type : standard
DPS-off run      : 140289
DPS-on run       : 140287
Requested by     : Your mom
Submitted        : 2002-03-04 14:35:18
Completed        : 2002-03-04 18:29:56
Status           : FAILURE

Last processing stage reported : disk cleanup
Failure diagnostics :
The calibration failed the following reference comparison criteria:

           Cut Name    Sensor Set   Cut Value   Data Value
-------------------    ----------   ---------   ----------
dps_off_ped abs max    SVX_PHI      0.5         0.859112  
 dps_on_ped abs max    SVX_PHI      0.2         0.395259  

Reference calibration runs are 139904 (DPS off) and 139902 (DPS on)
</pre>
<!-- Author: D-mode script run_calib_analysis.tcl -->



<!-- Date: Mon Mar 4 22:42:47 2002 -->
<pre>Well, I was still around when the time to turn On the HV came so  
I paged the main silicon pager (Saverio today) and he let me do it. 
Well there were some excitement: 
 
I played by the rules and I asked the monitoring ACE to put the 
5 minutes plot of the losses on the CDF e-log. 
He did so and everything looked fine. 
So: HV to ON from IFIX. 
While waiting the LOSTP started to increase vs time with a scary  
slope. 
In agreement with Andy Hocker (the fanatstic DAQ ACE) we decided  
not to take any risk and bring the HV back to STDBY and wait a  
little longer and see the losses behavior. 
So we did and we ended up turning HV ON through IFIX safely. 
 
Both time we turned the silicon to ON LB0W3L3 tripped on  
EXTERNAL TRIP. This ladder was already set back from 135 to 130 VOlts. 
I turned it to 130 by hands a step at the time: 
 
Mon Mar 04 22:13:41 CST 2002     success: Channel3 L00 WEST W3 to STANDBY 
Mon Mar 04 22:13:54 CST 2002     success: Voltage0 changed for LB0W3 L3 to 50.0 V 
Mon Mar 04 22:14:42 CST 2002     success: Voltage0 changed for LB0W3 L3 to 80.0 V 
Mon Mar 04 22:15:14 CST 2002     success: Voltage0 changed for LB0W3 L3 to 110.0 V 
Mon Mar 04 22:15:55 CST 2002     success: Voltage1 changed for LB0W3 L3 to 70.0 V 
Mon Mar 04 22:16:10 CST 2002     success: Voltage0 changed for LB0W3 L3 to 125.0 V 
Mon Mar 04 22:16:39 CST 2002     success: Voltage0 changed for LB0W3 L3 to 135.0 V 
Mon Mar 04 22:16:46 CST 2002     success: Voltage1 changed for LB0W3 L3 to 95.0 V 
Mon Mar 04 22:17:19 CST 2002     success: Voltage1 changed for LB0W3 L3 to 115.0 V 
Mon Mar 04 22:18:25 CST 2002     success: Voltage1 changed for LB0W3 L3 to 125.0 V 
Mon Mar 04 22:21:52 CST 2002     success: Voltage1 changed for LB0W3 L3 to 130.0 V 
 
Now the crew is taking data. 
 
I'm going to fill a "soft" incident report. 
 
 
 
 
 
</pre>
<!-- Author: Gino -->
<!-- Comment: Tue Mar  5 00;20;21 comment by...Saverio -->
<pre>ch 56  133.30 V    11.64 uA LB0W3L3 Vbias1
ch 59  134.60 V     4.60 uA LB0W3L3 Vbias0


</pre>


<!-- Comment: Thu Mar  7 10;03;20 comment by...Andy H. -->
The L00 trips were not external trips, but rather overcurrent trips on VBIAS1:
<pre>
03/04/02 09:54:33 PM - L00 B0W3 layer 3 DVDD   has TRIPPED: 0000 0010 0000 0011
03/04/02 09:54:33 PM - L00 B0W3 layer 3 AVDD   has TRIPPED: 0000 0010 0000 0011
03/04/02 09:54:33 PM - L00 B0W3 layer 3 VBIAS  has TRIPPED: 0010 0010 0000 0011
03/04/02 09:54:33 PM - L00 B0W3 layer 3 VBIAS1 has TRIPPED: 0011 0000 0000 0011
03/04/02 09:59:55 PM - L00 B0W3 layer 3 DVDD   has CLEARED: 1000 0000 0000 0011
03/04/02 09:59:55 PM - L00 B0W3 layer 3 AVDD   has CLEARED: 1000 0000 0000 0011
03/04/02 09:59:55 PM - L00 B0W3 layer 3 VBIAS  has CLEARED: 1000 0000 0000 0011
03/04/02 09:59:55 PM - L00 B0W3 layer 3 VBIAS1 has CLEARED: 1000 0000 0000 0011
03/04/02 10:09:46 PM - L00 B0W3 layer 3 DVDD   has TRIPPED: 0000 0010 0000 0011
03/04/02 10:09:46 PM - L00 B0W3 layer 3 AVDD   has TRIPPED: 0000 0010 0000 0011
03/04/02 10:09:46 PM - L00 B0W3 layer 3 VBIAS  has TRIPPED: 0010 0010 0000 0011
03/04/02 10:09:46 PM - L00 B0W3 layer 3 VBIAS1 has TRIPPED: 0011 0000 0000 0011
03/04/02 10:13:46 PM - L00 B0W3 layer 3 DVDD   has CLEARED: 1000 0000 0000 0011
03/04/02 10:13:46 PM - L00 B0W3 layer 3 AVDD   has CLEARED: 1000 0000 0000 0011
03/04/02 10:13:46 PM - L00 B0W3 layer 3 VBIAS  has CLEARED: 1100 0000 0000 0011
03/04/02 10:13:46 PM - L00 B0W3 layer 3 VBIAS1 has CLEARED: 1100 0000 0000 0011
</pre>
<!-- Date: Tue Mar 5 10:24:40 2002 -->
<h3>Fib Sequence Check</h3> 
All Fib Sequences check out OK.  Current Checksum is 0x19c82.
<!-- Author: SCN -->


<!-- Date: Tue Mar 5 11:44:54 2002 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>&lt;a href= '/cgi/elog/elog.pl?nb=2002&action=copyentry&page=191&time=11:04:44'&gt; Someone killed the scripts (PSMon, AlmMon, TempOut) running on the first floor machine ? 
&lt;/a&gt; 
 
</pre></td></tr></td></table>
<!-- Author: Mircea C -->
<!-- Comment: Tue Mar  5 11;46;23 comment by...Mircea C -->
<pre>&lt;a href= 
'/cgi/internal/elog/elog.pl?nb=2002&action=xe&page=191&time=11:04:44'&gt; Sorry! The link is here ! &lt;/a&gt;.</pre>

<!-- Comment: Tue Mar  5 11;47;14 comment by...Mircea C -->
<a href= 
'/cgi/internal/elog/elog.pl?nb=2002&action=xe&page=191&time=11:04:44'> Sorry! The link is here ! </a>.




<!-- Date: Tue Mar 5 12:11:41 2002 -->
Andrew and I tried to track down why the L00 bar is 120% in 
STBY.  We had to use an "everything is 3V" CAENvolts.dat file 
since it is not quiet time.  Unfortunately this was enough 
of a perturbation to render the problem unreproducible.  We 
will have to try again when there is quiet time (as far as I 
can tell, the problem is triggered when we turn ON (to full volts) and then back to STBY).
<!-- Author: Andy H. -->


<!-- Date: Tue Mar 5 13:12:41 2002 -->
<h3>New Chip Editor Feature</h3> 
I added the ability of the Chip Editor to read from files. 
In principle if someone wants to mess up the hdwdb chip settings 
and restore them, you can<ul> 
<li> Bring up the chip editor in the normal way 
<li> In the "directory" text field, give the name 
of the directory where you want the files to be stored.  Please 
do not use the default.  (you can also put the full path of this 
directory on the command line when you start the program) 
<li> Select the crates, read from the HDWDB in the normal way 
<li> Click on "write files".  This will take the selected crates and write the entire configuration to files. 
<li> Mess with the chips in any way you like... 
<li> When ready, click on "read files".  This reloads the chips 
with the parameters as they were when you wrote the files. 
<li> To impose this state on the database, select the 
Chips you've been mucking with in the Chip Editor on the multichip selection panel (The Chip editor frame  will 
disappear, you'll need to click the button again) and 
click on "Write Selected Chips", put in your username and password and you should be all set.  You can even exit 
the Chip Editor and come back, go straight to "Read Files" and 
you'll be all set (you get some java errors having to do with FFOs,but they are harmless). 
</ul> 
I've tested it to my satisfaction, but you might want me around 
when you try it the first time.  Please try to make this not 
after 12 midnight...
<!-- Author: SCN -->
<!-- Comment: Thu Mar  7 18;06;58 comment by...Jason -->
<pre>Aaron and I tested this new feature by mucking with the channel
masks and reading back the originals.  It works.

(It works inasmuch as I couldn't find any "Read Files" button.
I used the "Read Selected HDIs" button, which seemed to work
as expected when the MultiChip selection was set to all.)
</pre>
<!-- Comment: Fri Mar  8 10;21;41 comment by...Jason -->
<pre>Actually I realize I was not testing the readback of the config
files, but rather the read from the hdwdb.  There really was no
"Read Files" button!</pre>
<!-- Date: Tue Mar 5 17:24:01 2002 -->
<h2><a href= 
'/cgi/elog/elog.pl?nb=2002&action=view&page=192&frame=2&anchor=17:08:34'> 
We lost another VME crate</a>  
</h2>
<!-- Author: Andy H. -->


<!-- Date: Tue Mar 5 19:04:16 2002 -->
Since Doug and Andrew have sped up the GUI considerably, I 
decreased the iFix time-out from 5 min to 3 min (could probably 
be even less).
<!-- Author: Andy H. -->


<!-- Date: Tue Mar 5 19:32:17 2002 -->
NEW PROBLEM LADDERS USING DEM <br> 
(compare to <a href = '/internal/ops/svxii/detector/detector.html'> list </a> dated Feb 24) <br> 
These ladders were added to the <a href = 'http://nkek15.fnal.gov/cgi-bin/newproblemdb/DBmain.pl'> problem  
database </a> <br> 
<body> 
&nbsp; 
<table BORDER COLS=4 WIDTH="100%" NOSAVE > 
<tr> 
<td>ladder</td> 
 
<td>problem</td> 
 
<td>% errors</td> 
 
<td>run range</td> 
</tr> 
 
<tr> 
<td>e0b4&nbsp;</td> 
 
<td>&nbsp;chip 8a: 
<br>NN missing. 
<br>f3</td> 
 
<td>up and down 
<br>now around 2-3% 
<br>(was up to 100%)</td> 
 
<td>since ~2200d</td> 
</tr> 
 
<tr> 
<td>e142</td> 
 
<td>DRO,00f3</td> 
 
<td>100%</td> 
 
<td>since ~20003</td> 
</tr> 
 
<tr> 
<td>e203&nbsp;</td> 
 
<td>DRO,00f3</td> 
 
<td>15%&nbsp;</td> 
 
<td>2235f</td> 
</tr> 
 
<tr> 
<td>e2b3&nbsp;</td> 
 
<td>Bad capid 
<br>87, 86, 84 
<br>Sometimes&nbsp; 
<br>DRO, 00f3 
<br>&nbsp;</td> 
 
<td>100%</td> 
 
<td>since ~21bf9 
<br>(on and off)</td> 
</tr> 
 
<tr> 
<td>e384</td> 
 
<td>chip 8d: 
<br>bit 7 stuck high 
<br>or ff</td> 
 
<td>&lt; 1 %&nbsp;</td> 
 
<td>since ~21925</td> 
</tr> 
 
<tr> 
<td>e4b4&nbsp;</td> 
 
<td>DRO,c2c2&nbsp;</td> 
 
<td>100%</td> 
 
<td>since ~220a1</td> 
</tr> 
</table> 
 
</body>
<!-- Author: Eva -->


<!-- Date: Tue Mar 5 23:12:44 2002 -->
When I started my ace shift at 16:00 LB0W3 L3 and LB1W2 L3 were both 
tripped off due to VBIAS1 overcurrents.  I thought someone 
would walk in and claim responsibility, but no one did. 
Therefore I took it upon myself to return these guys to STBY.
<!-- Author: Andy H. -->
<!-- Comment: Tue Mar  5 23;45;51 comment by...CH -->
<pre>David's PS monitor shows LB0W3L3 going to 0V at 16:30:55
(during your shift).  

Did you not get any IFIX alarm to alert 
you to the trip?

If there is a trip alarm we should page main silicon pager, not 
wait until someone walks in and claims responsibility.
If there was no trip alarm that is also a problem.  </pre>

<!-- Comment: Tue Mar  5 23;57;05 comment by...Andy H. -->
I admit I dropped the ball on this one....



<!-- Comment: Thu Mar  7 08;46;44 comment by...Saverio -->
<pre>It looks like it was not bias overcurrent, but EXTERNAL TRIP</pre>
<!-- Comment: Thu Mar  7 09;28;45 comment by...Andy H. -->
External trip was reported in the GUI window (and logfile), but
I remember that the full status window showed overcurrent on
VBIAS1 and ext trip on AVDD, DVDD, and VBIAS.  I can check the
PSMon logfile to be sure.
<!-- Comment: Thu Mar  7 09;54;43 comment by...Andy H. -->
Yup, it was definitely a VBIAS1 overcurrent trip (status bit 12):
<pre>
03/05/02 04:31:04 PM - L00 B0W3 layer 3 DVDD   has TRIPPED: 0000 0010 0000 0011
03/05/02 04:31:04 PM - L00 B0W3 layer 3 AVDD   has TRIPPED: 0000 0010 0000 0011
03/05/02 04:31:04 PM - L00 B0W3 layer 3 VBIAS  has TRIPPED: 0000 0010 0000 0011
03/05/02 04:31:04 PM - L00 B0W3 layer 3 VBIAS1 has TRIPPED: 0001 0000 0000 0011
03/05/02 04:31:45 PM - L00 B1W2 layer 3 DVDD   has TRIPPED: 0000 0010 0000 0011
03/05/02 04:31:45 PM - L00 B1W2 layer 3 AVDD   has TRIPPED: 0000 0010 0000 0011
03/05/02 04:31:45 PM - L00 B1W2 layer 3 VBIAS  has TRIPPED: 0000 0010 0000 0011
03/05/02 04:31:45 PM - L00 B1W2 layer 3 VBIAS1 has TRIPPED: 0001 0000 0000 0011
03/05/02 09:49:07 PM - L00 B0W3 layer 3 DVDD   has CLEARED: 1000 0000 0000 0011
03/05/02 09:49:07 PM - L00 B0W3 layer 3 AVDD   has CLEARED: 1000 0000 0000 0011
03/05/02 09:49:07 PM - L00 B0W3 layer 3 VBIAS  has CLEARED: 1000 0000 0000 0011
03/05/02 09:49:07 PM - L00 B0W3 layer 3 VBIAS1 has CLEARED: 1000 0000 0000 0011
03/05/02 09:49:45 PM - L00 B1W2 layer 3 DVDD   has CLEARED: 1000 0000 0000 0011
03/05/02 09:49:45 PM - L00 B1W2 layer 3 AVDD   has CLEARED: 1000 0000 0000 0011
03/05/02 09:49:45 PM - L00 B1W2 layer 3 VBIAS  has CLEARED: 1000 0000 0000 0011
03/05/02 09:49:45 PM - L00 B1W2 layer 3 VBIAS1 has CLEARED: 1000 0000 0000 0011
</pre>

<!-- Date: Tue Mar 5 23:58:34 2002 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-390&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-390  width='300' align='top'></td><td>The L00 trips at 16:30 might very well have been due to losses -- Note that LOSTP and LOSTPB hit their rails at 16:30</a></td></tr></table>
<!-- Author: Andy H. -->


<!-- Date: Wed Mar 6 10:25:32 2002 -->
<pre>Reacting to b0fib03 VME PS failure: 
changed in HDWDB the ONLINE FLAG of b0fib03 from 1 to 0 
generated a new CAENvolts_hdwdb.dat 
this file was corrupted and I fixed it by hands (Eva has been contacted). 
Checked the new file and copied it over. 
Loaded it into the CAEN GUI and the following wedges: 
 
LB0W0,Lb0W1,LB0W4,LB0W5 
IB0W0,IB0W1,IB0W4,IB0W5,IB1W1 
 
are now OFF. 
 
</pre>
<!-- Author: Gino -->
<!-- Comment: Wed Mar  6 12;22;49 comment by...Andy H. -->
Generating a new powerset.dat would probably be a good idea.
Does cratematch do this?  I forget.
<!-- Comment: Wed Mar  6 19;47;54 comment by...Eva -->
I don't understand why we have this error now.  This is the EXACT same script we have been using for over a month now.  I talked with Bill Badgett and he doesn't understand the problem either.  I will fix it and the new SQL script will be the default soon.
<!-- Comment: Thu Mar  7 17;15;42 comment by...Eva -->
This is now fixed.  Let me (evah@fnal.gov) know if there are
any problem.
<!-- Date: Wed Mar 6 12:04:53 2002 -->
<pre>FROM main CDF logbook: 
(MLE) b0fib01:Messenger:9:29:47 AM-&gt;Error Initializing HDI Slot 9 Chan 4: ISL B0W2 
same error some time later: 
(MLE) b0fib01:Messenger:10:14:24 AM-&gt;Error Initializing HDI Slot 9 Chan 4: ISL B0W2).  
passed after issuing abort.  
RUN 140415  
DB entry done.</pre>
<!-- Author: Saverio -->


<!-- Date: Wed Mar 6 12:07:05 2002 -->
<a href= 
'/cgi/elog/elog.pl?nb=2002&action=view&page=194&frame=2&anchor=11:35:46'> 
It happened again at 11:33:07</a><br> 
11:33:07 AM->Error Initializing HDI Slot 9 Chan 4: ISL B0W2<br>
<!-- Author: saverio -->


<!-- Date: Wed Mar 6 12:52:30 2002 -->
<p><font size=+1> D-mode Calibration Job Started </font>
<p><pre>
Job id            : 94
Calibration type  : standard
Run period tag    : Mar_2002
DPS-off run       : 140246
DPS-on run        : 140244
Requested by      : Igor
Main process id   : 23235791
</pre>
<!-- Author: CDF silicon D-mode calibration job server v1.3 -->


<!-- Date: Wed Mar 6 17:17:34 2002 -->
<p><font size=+1> D-mode Calibration Job Finished </font>
<p><pre>
This is a computer-generated silicon D-mode calibration status report.
Silicon D-mode calibration job 94 finished successfully.

Job info
--------
Calibration type : standard
DPS-off run      : 140246
DPS-on run       : 140244
Requested by     : Igor
Submitted        : 2002-03-06 10:51:49
Completed        : 2002-03-06 15:17:11
Status           : SUCCESS
</pre>
<!-- Author: D-mode script run_calib_analysis.tcl -->



<!-- Date: Wed Mar 6 18:10:42 2002 -->
DIAGNOSTIC REPORT FOR LB0W3L3 and LB1W2L3 trip.  
<br> 
<br> 
See   <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=101&frame=2&anchor=23:12:44'> 
entry above</a>    
<br> 
Previous elog entries <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=101&frame=2&anchor=22:42:47'> 
here </a>  
<br> 
<br> 
GUI logfile viewer reported: 
<br> 
Tue Mar 05 16:30:43 CST 2002 TRIP: L00 WEST W3 External Trip  
<br> 
Tue Mar 05 16:31:21 CST 2002 TRIP: L00 EAST W2 External Trip 
<br> 
<br> 
Current plotter (from Auke's web-page) output files are located 
on b0 cluster in: 
<br> 
~svxii/evah/currents_030602_LB0W3L3.ps 
<br> 
~svxii/evah/currents_030602_LB1W2.ps 
<br> 
<br> 
DAQ Error log:  
<br> 
Run # at the time was 140374, a Time of Flight Calib run.  Not really 
useful info. 
<br> 
<br> 
DEM: 
<br> 
Looked at DEM errors for LB0W3 L3 and LB1W2 L3.   
These have 0% errors for a long time. 
<br> 
<br> 
CONCLUSION: 
<br> 
Most probable cause of the trips was losses.  See <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=101&frame=2&anchor=23:58:34'> 
entry and plots</a>   
<!-- Author: Eva -->



<!-- Date: Wed Mar 6 23:56:31 2002 -->
<pre>b0fib01 failed the Cold Start 4 times due to the same HDI:  
(see errorfiles): 
 /cdf/code-common/cdfonline/errmon/log/errorfile140463.log:(MLE) b0fib01:Messenger:6:33:20 
PM-&gt;Error Initializing HDI Slot 9 Chan 4: ISL B0W2  
 
/cdf/code-common/cdfonline/errmon/log/errorfile140470.log:(MLE) b0fib01:Messenger:8:13:25 
PM-&gt;Error Initializing HDI Slot 9 Chan 4: ISL B0W2  
 
/cdf/code-common/cdfonline/errmon/log/errorfile140472.log:(MLE) b0fib01:Messenger:8:42:27 
PM-&gt;Error Initializing HDI Slot 9 Chan 4: ISL B0W2  
 
/cdf/code-common/cdfonline/errmon/log/errorfile140474.log:(MLE) b0fib01:Messenger:11:46:28 
PM-&gt;Error Initializing HDI Slot 9 Chan 4: ISL B0W2  
 
A simple cure was to try Cold Start one more time, instead of going through Reset and Partition.</pre>
<!-- Author: Mircea C -->

<!-- Date: Thu Mar 7 06:49:30 2002 -->
<p><font size=+1> D-mode Calibration Job Started </font>
<p><pre>
Job id            : 95
Calibration type  : nonstandard
Run period tag    : Mar_2002
DPS-off run       : 140488
DPS-on run        : 140487
Requested by      : Keyoung
Main process id   : 31562802
</pre>
<!-- Author: CDF silicon D-mode calibration job server v1.3 -->



<!-- Date: Thu Mar 7 08:35:07 2002 -->
<br><a href= 
'/cgi/elog/elog.pl?nb=2002&action=view&page=196&frame=2&anchor=06:44:44'> 
Various Silicon calibrations </a> were taken with Solenoid OFF<br>
<!-- Author: Sav -->



<!-- Date: Thu Mar 7 09:08:26 2002 -->
<pre>LB0W3L3 after crate power up 
Thu Mar 07 08:50:06 CST 2002 ERROR: Power Supply L00 EAST W3 Handshake - FIB initialization failed 
Thu Mar 07 08:50:13 CST 2002     TRIP: L00  EAST  W3 DVDD3 OverVoltage 
</pre>
<!-- Author: Saverio -->


<!-- Date: Thu Mar 7 09:11:34 2002 -->
<pre>Turned OFF IB4W0. 
IB4W1 was already OFF.</pre>
<!-- Author: Gino -->
<!-- Comment: Mon Mar 11 17;26;30 comment by...Saverio -->
<pre>IB4W0L0, L2, L4 re-integrated on 03/07/02 at 10:23 by S. Nahn</pre>
<!-- Date: Thu Mar 7 09:15:01 2002 -->
<pre>After consulting with DAURIA LESTER and SCN we decided to turn the POWER OFF for the silicon during the access. 
This should make our life easier. 
</pre>
<!-- Author: Gino -->



<!-- Date: Thu Mar 7 09:38:23 2002 -->
<pre>I would suggest to check again the Power Supply for  
LB0W3L3 if we have time now. It has a history of external trips and now overvoltage on dvdd.</pre>
<!-- Author: Saverio -->
<!-- Comment: Thu Mar  7 10;07;11 comment by...Andy H. -->
Not external trips -- see my comments
<a href=
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=101&frame=2&anchor=22:42:47'>
here</a> and
<a href=
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=101&frame=2&anchor=23:12:44'>
here.</a> 


<!-- Comment: Thu Mar  7 10;14;37 comment by...Andy H. -->
I just had Andrew take a look at the GUI code -- he saw that for
L00 the logic for deciding whether to report a trip as external
or overcurrent was messed up.  He and Doug will fix this later today.  One less thing to confuse us!


<!-- Comment: Thu Mar  7 10;22;56 comment by...Andrew -->
<pre>In case of a trip at L00 L3 BIAS1, the GUI was mistakenly
reporting status of trip at L3 BIAS0, which would always be
External. I fixed and will submit the corrected version later
today.</pre>


<!-- Date: Thu Mar 7 12:33:31 2002 -->
<H3>Prep for Rx work</h3> 
We tested SB4W10/11 SB5W10/11 via the High Speed Test in preperation for Lester swapping out FTMs.  We saw problems on<ul> 
<li> SB4W10 L3 
<li> SB4W11 L2 
<li> <FONT COLOR = FF0000>SB4W11 L4- Not in Lester's list </FONT> 
<li> SB5W11 L0 
</ul> 
We did not see problems in the following (included in Lester's list) <ul> 
<li> SB5W10 L3 
<li> SB5W11 L1 
</ul> 
We also noticed that SB4W10 L0 returns a garbled initialization-looks shifted by one bit:<pre> 
Initing DDR: 0 Fib Pipe: 0 in   Crate 9, Slot 15  e4a0	 -----> Error <----- 
 TRIED TO WRITE:  
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 2d 36 90 a 28 78 f8 d 0 0 0 0  
0 0 0 0 0 0 0 0 0 0 0 c0 a2 c5 16 12 1 5 f bf 1 0 0 0 0 0 0 0 0 0  
0 0 0 0 0 0 54 b4 d8 5 22 a0 e0 e1 37 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
0 80 8a 16 fb 40 4 14 3c fc 6  
 READBACK:  
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2c 5a 6c 20 15 50 f0 f0 1b 0 0 0  
0 0 0 0 0 0 0 0 0 0 0 0 80 45 8b 2d 24 2 a 1e 7e 3 0 0 0 0 0 0 0  
0 0 0 0 0 0 0 0 a8 68 b1 b 44 40 c1 c3 6f 0 0 0 0 0 0 0 0 0 0 0 0  
0 0 0 0 15 2d f6 81 8 28 78 f8 d  
</pre>  
<!-- Author: SCN, Gino -->
<!-- Comment: Thu Mar  7 18;36;10 comment by...Eva -->
I recently added SB4W11L4 to the silicon problem database.
Using DEM I found DRO,c2c 100%, since run # 220a1 (139425).
<!-- Date: Thu Mar 7 17:49:15 2002 -->
DIAGNOSTIC REPORT FOR IB0W2L4: Init Errors 
<br> 
Run, time <br> 
140412, 9:29 am  <br> 
140415, 10:14 am <br> 
140418, 11:33 am <br> 
140463, 6:33 pm <br> 
140470, 8:13 pm <br> 
140472, 8:42 pm <br> 
140474, 11:46 pm <br> 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=101&frame=2&anchor=12:04:53'> 
First Entry </a>  
<br> 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=101&frame=2&anchor=23:56:31'> 
Later Entry </a>  
<br> 
This ladder/wedge has had a history of problems in December with high 
currents:<br> 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=83&frame=2&anchor=22:31:13'> 
Dec 10: DVDD currents</a>  
<br> 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=84&frame=2&anchor=06:22:32'> 
Dec. 15: AVDD pink in IMON. Follow up by Naho </a> 
<br> 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=84&frame=2&anchor=11:24:37'> 
Dec. 17: Many reformatter errors </a>  
<br> 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=84&frame=2&anchor=14:59:46'> 
Dec. 18: IAVDD up and down </a>  
<br> 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=84&frame=2&anchor=09:51:11'> 
Dec. 19: More RF errors </a>  
<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=90&frame=2&anchor=08:39:36'> 
Jan 13: Fib01 PS dies </a>  
<br> 
<a href = '/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=96&frame=2&anchor=11:58:36'> 
Feb 5: Supervised Access </a> 
<br> 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=101&frame=2&anchor=12:45:12'> 
March 6: Another init error </a> 
<br> 
<br> 
DEM: Many bits stuck low beginning with run number ~138436 (21cc4), Feb 5, the supervised access! The errors in DEM were 0% before this access and then went up to 30%->100%.  
<br> 
<br> 
DAQ Initilization Reports: 
<a href = '/svx-data1/datalog/daq/list_init.html'> (here) </a> 
<br> 
Going through runs listed above, starts off with one error (sent: c6, read:c0) and then increases to more errors every time. 
(Hard to add this in elog, so you can see for yourself by going to link above or ~svxii/evah/diag_030602_IB0W2L4.txt). 
<br> 
<br> 
Currents:<br> 
Checked the currents in PS plots web-page (Auke's).  Noticed that the DOIM currents go high just as the run starts and only come 
back down when the next one begins. Checked to see if the same is true for the corresponding wedge in that FIB (IB0W3) and the  
same thing happens!  Since I don't know how to add plots to the elog you can see these plots in: ~svxii/evah/currents_030602_IB0W*ps. 
<br> 
<br> 
Diagnosis/Summary:<br> 
DEM shows 100% errors since Feb 5 access. <br> 
DOIM currents go up during these init failures for both 
IB0W2 and IB0W3.  Seems to be a Fib-wide problem.  Therefore, 
problem is either with (1) the sequence, (2) the Fib hardware, (3) the OFTM hardware.
<!-- Author: Eva -->
<!-- Comment: Sat Mar  9 11;21;52 comment by...Eva -->
<pre>I should be more clear:
Run #    time of init error   next run   next run begins
------   ------------------   --------   ----------------
140412      9:29 am            140413     9:34  am       
140415      10:14 am           abort      10:16 am
140418      11:33 am           140419     11:36 am
140463      6:33 pm            140464     6:40 pm
140470      8:13 pm            140471     8:22 pm
140472      8:42 pm            140473     8:44 pm
140474      11:46 pm           140475     23:48 pm

I get the init error times from HRR records.  For example:
140412 (0x2247C) HardEvb 0 null 2002.03.06 null 09:29:27 0,0 Physics DAQ Testing COMSICS_NOTRACKS
[3,233] 445 Silicon was integrated
b0fib01:Messenger:9:29:47 AM-&gt;Error Initializing HDI Slot 9 Chan 4: ISL B0W2 
b0fib01:Messenger:9:29:47 AM-&gt;Statechange failed 
end time 

Run Summary web page gives time stamp of next run:
140413 (0x2247D) HardEvb 0 1054 2002.03.06 09:34:24 09:54:17 0,0 Physics DAQ Testing
COMSICS_NOTRACKS [3,233] 445 Silicon was integrated
</pre>
<!-- Comment: Sat Mar  9 11;29;00 comment by...Eva -->
<pre>EXAMPLE:

&lt;a href=
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=101&frame=2&anchor=11:25:05'&gt;
Here &lt;/a&gt; is a zoomed in plot of the DOIM currents for IB0W2
at around 11am on 3/6/02.  

Compare to these times:

Run #    time of init error   next run   next run begins
------   ------------------   --------   ----------------
140418      11:33 am           140419     11:36 am
</pre>
<!-- Comment: Sat Mar  9 11;30;25 comment by...Eva -->
Sorry.  
<a href=
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=101&frame=2&anchor=11:25:05'>
Here is the plot </a> 
<!-- Date: Thu Mar 7 18:28:18 2002 -->
<pre>Installed CAEN crate interlocks.  There is now a switch which 
can be used to turn off each crate separately. The switches 
are in the same rack as the 5V enable power supplies. 
 
  The interlock cable for SE Bot Bot failed.  This crate is 
not yet interlocked. 
 
 
  Here's the switch panel information.  Note: Sys, corner, 
rack and crate forms the label that we would like to have on the switch. Database Line and number are the text label on the cable. They all go to IRR34C in the first floor counting room. 
 
Sys  Corner    Rack   Crate        Database Line          Database # 
SVX   NW        Top    Top         ISL NW B 5VCAEN_EN_4   86400 
SVX   NW        Top    Bot         ISL NW B 5VCAEN_EN_5   86401 
ISL   NW        Bot    Top         ISL NW B 5VCAEN_EN_6   86402 
ISL   NW        Bot    Bot         L00 NW B 5VCAEN_EN_2   86404 
ISL   SW        Top    Top         ISL SW T 5VCAEN_EN_4   86392 
ISL   SW        Top    Bot         ISL SW T 5VCAEN_EN_5   86393 
SVX   SW        Bot    Top         ISL SW T 5VCAEN_EN_6   86394 
SVX   SW        Bot    Bot         L00 SW T 5VCAEN_EN_2   86396 
SVX   NE        Top    Top         ISL NE B 5VCAEN_EN_4   86384 
SVX   NE        Top    Bot         ISL NE B 5VCAEN_EN_5   86385 
ISL   NE        Bot    Top         ISL NE B 5VCAEN_EN_6   86386 
ISL   NE        Bot    Bot         SVX NE T 5VCAEN_EN_4   86364 
ISL   SE        Top    Top         ISL SE T 5VCAEN_EN_4   86376 
ISL   SE        Top    Bot         ISL SE T 5VCAEN_EN_5   86377 
SVX   SE        Bot    Top         ISL SE T 5VCAEN_EN_6   86378 
SVX   SE        Bot    Bot         SVX SE B 5VCAEN_EN_4   86360 
</pre>
<!-- Author: herndon -->



<!-- Date: Thu Mar 7 21:54:27 2002 -->
With the added heat load of the newly-powered IB1W0, the 
temperature of ISL_TE_CCW1L6_2OUT has crept above its warning 
limit of 28 deg C.  Gino and I increased the ISL blocked CC 
temperature trip limit from 30 to 31 deg C and re-set the 
iFix alarm/warning limits (trip limit - 1 and trip limit - 2, 
respectively).
<!-- Author: Andy H. -->
<!-- Comment: Thu Mar  7 21;55;21 comment by...Andy H. -->
Read that as "trip limit minus one" and "trip limit minus two"


<!-- Comment: Fri Mar  8 17;07;02 comment by...Andy H. -->
31 deg C wasn't quite enough, it turns out -- raised it to 33 deg C.
<!-- Date: Fri Mar 8 02:04:59 2002 -->
<H2>Layer 00 Gain Scans</H2> 
These gain scans cover a range of bandwidth and charge injection DAC settings.  All runs are approx. 1000 events read in DPS-off, Read-All mode with charge injected on every 16th strip. 
(I did not change any RampPed settings.) 
<P> 
We did not manage to turn on LB1W1 due to DVDD overcurrents, so this wedge was excluded from the hardware database for these runs.  Data-taking was smooth except for some failed ColdStarts from b0fib01. 
 
<UL> 
<LI>Run 140563 -- BW=7, DAC=0</LI> 
<LI>Run 140566 -- BW=7, DAC=5</LI> 
<LI>Run 140567 -- BW=7, DAC=6</LI> 
<LI>Run 140568 -- BW=7, DAC=7</LI> 
<LI>Run 140575 -- BW=7, DAC=8</LI> 
<LI>Run 140580 -- BW=3, DAC=5</LI> 
<LI>Run 140581 -- BW=3, DAC=6</LI> 
<LI>Run 140583 -- BW=3, DAC=7</LI> 
<LI>Run 140584 -- BW=3, DAC=8</LI> 
<LI>Run 140587 -- BW=5, DAC=5</LI> 
<LI>Run 140589 -- BW=5, DAC=6</LI> 
<LI>Run 140591 -- BW=5, DAC=7</LI> 
<LI>Run 140593 -- BW=5, DAC=8</LI> 
<LI>Run 140595 -- BW=2, DAC=5</LI> 
<LI>Run 140600 -- BW=2, DAC=6</LI> 
<LI>Run 140602 -- BW=2, DAC=7</LI> 
<LI>Run 140604 -- BW=2, DAC=8</LI> 
</UL> 
<P> 
Doing my best to return the system to the previous state...  
<!-- Author: Jason -->


<!-- Date: Fri Mar 8 03:22:16 2002 -->
I have tried to load the configuration files back from 
/cdf/people2/svxii/chipConfigs/preL00scan without any luck. 
<P> 
Also, the cards which I thought had been put offline in the card editor are now back online. 
<P> 
I will have to come back in a few hours and fix these things up with Steve or another real expert. 
<P> 
Turning silicon to standby now...  
<!-- Author: Jason -->
<!-- Comment: Fri Mar  8 10;24;14 comment by...Jason -->
<pre>Now loaded the preL00scan config files.  I needed the new 
version of the chipEditor which Steve provided.

The mysterious online ISL cards had been changed from offline to
online without my knowledge while I was performing the scans.
</pre>
<!-- Date: Fri Mar 8 07:42:42 2002 -->
<pre>f020 we masked the check on readback of the initialization bit stream</pre>
<!-- Author: Saverio -->
<!-- Comment: Fri Mar  8 07;52;44 comment by...Saverio -->
The reason for this is due to failure of initialization: see
<a href=
'/cgi/elog/elog.pl?nb=2002&action=view&page=199&frame=2&anchor=03:55:35'>
This Logbook entry </a> 


<!-- Comment: Fri Mar  8 07;55;10 comment by...Saverio -->
<pre>This laddeer was initializing successfully the second time.
See Eva's entry above. So it deteriorated with time.</pre>


<!-- Date: Fri Mar 8 07:58:20 2002 -->
<pre>SB2W11L1 failed init</pre>
<!-- Author: Saverio -->
<!-- Comment: Fri Mar  8 08;00;38 comment by...Saverio -->
<pre>It inited correctly at the secon attempt
It looks like the previous one (ISL)! Worrying.</pre>


<!-- Date: Fri Mar 8 08:01:22 2002 -->
<pre>No beam, Silicon biased on SiCo's request.</pre>
<!-- Author: Saverio -->


<!-- Date: Fri Mar 8 08:48:54 2002 -->
A lot of  
<a href="/svx-data1/datalog/PSGUI/log/20020307.log"> ACTIVITY</a> took place last night,mainly 
concerning L00:<br>We had several trips on LB1W1L3<br><br> 
Thu Mar 07 21:08:52 CST 2002     success: Power Supply L00 EAST W1 ON<br> 
Thu Mar 07 21:09:00 CST 2002     TRIP: L00  EAST  W1 DVDD3 OverCurrent<br><br> 
No entry in the logbook, no paging, what was going on ?  
<!-- Author: Saverio (pager carrier) -->
<!-- Comment: Fri Mar  8 10;48;29 comment by...CH -->
<pre>There were failures (DVDD trips) in post-access OFF-&gt;STANDBY
transition. GUI experts were paged.

Both SPLs were present and in charge of the situation.

Procedural clarification:

During an access STANDBY-&gt;OFF OFF-&gt;STANDBY are to be authorized
by an SPL only.

When system is back in STANDBY and SPLs give it back to CDF,
SiExpert MUST be paged for STANDBY-&gt;ON even if no beam,
i.e. access is still going on.</pre>
<!-- Date: Fri Mar 8 09:22:51 2002 -->
<pre>FTM swapped yesterday during the access: 
SB4W10-11 
                OUT             IN 
PREP ID         562711          565890 
Serial #        100             135 
RX-ch-0         258             134 
RX-ch-1         638             146 
RX-ch-2         720             131 
RX-ch-3         311             178 
RX-ch-4         779             145 
RX-ch-5         525             1096 
RX-ch-6         422             993 
RX-ch-7         505             1108 
RX-ch-8         701             1078 
RX-ch-9         629             1106 
 
SB5W10-11 
 
                OUT             IN 
PREP ID         564054          565897 
Serial #        103             129 
RX-ch-0         335             1143 
RX-ch-1         641             1113 
RX-ch-2         373             1127 
RX-ch-3         347             1141 
RX-ch-4         319             1120 
RX-ch-5         963             1118 
RX-ch-6         827             1142 
RX-ch-7         221             1022 
RX-ch-8         627             1024 
RX-ch-9         291             1123 
 
Now the 2 FTMs that came out are in the Silicon Office ready for  
SUEN and Auke-Peter to pick them up and understand the bit  
failures. 
 
</pre>
<!-- Author: Gino -->
<!-- Comment: Fri Mar  8 15;35;35 comment by...gino -->
<pre>Given the 2 faulty boards to SUE at 3:30 pm today.
</pre>
<!-- Comment: Mon Mar 11 09;17;50 comment by...Lester -->
<pre>Retested boards found only remaining problem is
SB4W11L2 DRO

So fixed problems are
SB4W11L2 bit 1 low
SB4W11L4 DRO
SB5W11L0 bit 0 high
SB5W11L1 bit 3 high rare
</pre>
<!-- Date: Fri Mar 8 09:38:50 2002 -->
<p><font size=+1> D-mode Calibration Job Started </font>
<p><pre>
Job id            : 96
Calibration type  : nonstandard
Run period tag    : Mar_2002
DPS-off run       : 140488
DPS-on run        : 140487
Requested by      : Igor
Main process id   : 2819957
</pre>
<!-- Author: CDF silicon D-mode calibration job server v1.3 -->


<!-- Date: Fri Mar 8 09:41:01 2002 -->
<p><font size=+1> D-mode Calibration Job Started </font>
<p><pre>
Job id            : 97
Calibration type  : standard
Run period tag    : Mar_2002
DPS-off run       : 140650
DPS-on run        : 140649
Requested by      : kim giolo
Main process id   : 3196969
</pre>
<!-- Author: CDF silicon D-mode calibration job server v1.3 -->


<!-- Date: Fri Mar 8 09:42:08 2002 -->
<p><font size=+1> D-mode Calibration Job Finished </font>
<p><pre>
This is a computer-generated silicon D-mode calibration status report.
Silicon D-mode calibration job 97 FAILED.

Job info
--------
Calibration type : standard
DPS-off run      : 140650
DPS-on run       : 140649
Requested by     : kim giolo
Submitted        : 2002-03-08 07:40:41
Completed        : 2002-03-08 07:42:00
Status           : FAILURE

Last processing stage reported : checking silicon configuration
Failure diagnostics :
error {{} {The Power Supply configuration at start time is not available, comparison with DAQ list is not possible &lt;/b&gt;} {&lt;br&gt;&lt;br&gt;&lt;b&gt;Check the GUI spy file ! ! &lt;/b&gt;&lt;/font&gt;} Success}
Exiting.
</pre>
<!-- Author: D-mode script run_calib_analysis.tcl -->


<!-- Date: Fri Mar 8 09:54:12 2002 -->
<p><font size=+1> D-mode Calibration Job Started </font>
<p><pre>
Job id            : 98
Calibration type  : standard
Run period tag    : Mar_2002
DPS-off run       : 140650
DPS-on run        : 140649
Requested by      : Igor
Main process id   : 3236788
</pre>
<!-- Author: CDF silicon D-mode calibration job server v1.3 -->






<!-- Date: Fri Mar 8 10:58:47 2002 -->
<pre>The Ace called me 'cause Global Alarm page wasn't getting  
updated. So it was requested to restart the IFix on all the local  
node, instead of restarting the IFix on node1 ! I obeyed  
the rule and did it. I restarted all the Silicon scripts 
and also Inchworm2IFix one. 
Still I am not sure if there are no more inchworm related  
scripts which are not running! So an expert might have a  
look at this. 
 
  </pre>
<!-- Author: Mircea C -->


<!-- Date: Fri Mar 8 11:58:48 2002 -->
<pre>b0fib01 on-line flag needs to be re-set to 1 
This crate is presently off-line.</pre>
<!-- Author: Saverio -->
<!-- Comment: Fri Mar  8 12;28;58 comment by...Saverio -->
I was fouled by some PS work going on.
As Gino points out the flag was changed (and it was b0fib03).
cratematch needs upgrade to have access to this flag.
 

<!-- Date: Fri Mar 8 13:18:30 2002 -->
<p><font size=+1> D-mode Calibration Job Finished </font>
<p><pre>
This is a computer-generated silicon D-mode calibration status report.
Silicon D-mode calibration job 96 finished successfully.

Job info
--------
Calibration type : nonstandard
DPS-off run      : 140488
DPS-on run       : 140487
Requested by     : Igor
Submitted        : 2002-03-08 07:38:13
Completed        : 2002-03-08 11:18:29
Status           : SUCCESS
</pre>
<!-- Author: D-mode script run_calib_analysis.tcl -->



<!-- Date: Fri Mar 8 13:18:51 2002 -->
<pre>SB2W10. 
 
Had many trips on DVDD overcurrent. 
Had this problem in the past many times. 
Took it out of the HDWDB and switched it OFF.</pre>
<!-- Author: Gino -->

<!-- Date: Fri Mar 8 13:45:51 2002 -->
<p><font size=+1> D-mode Calibration Job Finished </font>
<p><pre>
This is a computer-generated silicon D-mode calibration status report.
Silicon D-mode calibration job 98 FAILED.

Job info
--------
Calibration type : standard
DPS-off run      : 140650
DPS-on run       : 140649
Requested by     : Igor
Submitted        : 2002-03-08 07:54:03
Completed        : 2002-03-08 11:45:31
Status           : FAILURE

Last processing stage reported : comparing to the reference calibration
Failure diagnostics :
The calibration failed the following reference comparison criteria:

           Cut Name    Sensor Set   Cut Value   Data Value
-------------------    ----------   ---------   ----------
dps_off_ped abs max    SVX_PHI      0.5         0.870822  
 dps_on_ped abs max    SVX_PHI      0.2         0.418766  

Reference calibration runs are 139904 (DPS off) and 139902 (DPS on)
</pre>
<!-- Author: D-mode script run_calib_analysis.tcl -->



<!-- Date: Fri Mar 8 13:55:19 2002 -->
<pre>1. The problem, related to DVDD current trips during OFF-&gt;STBY  
transition, which happened last night, is fixed. 
It was a bug in the code, we incorporated a few days ago to speed 
up the GUI. Because of wrong cast from float to ushort we 
occassionally had DVDD trip limit set to 0.0 
2. We commented out broadcast of "spontaneous" STBY&lt;-&gt;ON  
transitions, since as we investigated they are not related to  
a CAEN problem, but rather appear during global power  
transition because Biases are ramping up/down slowly. 
3. During our test we had many ON&lt;-&gt;STBY transitions to high bias 
voltages and we not able to reproduce the problem with L00 power 
bar on iFix.</pre>
<!-- Author: Andrew  -->



<!-- Date: Fri Mar 8 16:27:46 2002 -->
DIAGNOSTIC REPORT FOR IB0W2L4: Init Errors <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=101&frame=2&anchor=17:49:15'> 
(again) </a>  
<br> 
<br> 
Run, time <br> 
140601, 12:58 am <br> 
140603, 1:11 am <br> 
(plus many other times - not silicon runs though)<br> 
Responding to Saverio's incident report <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=101&frame=2&anchor=07:42:42'> 
here </a>  
<br> 
<br> 
The DAQ Initialization reports show many errors for this ladder 
and they have been increasing all week!  <a href='/svx-data1/datalog/daq/20020308.si_daq'> See the report here </a> 
<br> 
<br> 
Again the DOIM currents increase with the beginning of the bad 
run and come back down at the beginning of the next run.  Same 
is true for IB0W3, as before.  Plots of the currents/voltages  
are in ~svxii/evah/currents_030802_IB0W2*ps. 
<br> 
<br> 
Again, it is a fib-wide problem.   
Recall that DEM shows 100% errors since  
the access on Feb 5.  Someone should check the sequence  
and/or the Fib/OFTM hardware. 
<br>
<!-- Author: Eva -->


<!-- Date: Fri Mar 8 16:55:02 2002 -->
<table><tr><td><a href=/elog/notebooks/silicon/Fri_Mar__8_2002_165423.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-395  align='top'></td><td>currents_030802_IB0W2.ps</a></td></tr></table>
<!-- Author: Sav -->



<!-- Date: Fri Mar 8 19:47:34 2002 -->
DIAGNOSTIC REPORT FOR SB2W11L1: Init Errors  <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=101&frame=2&anchor=07:58:20'> 
(see entry) </a> 
<br> 
<br> 
Found init errors using HRR search tool: 
<br> 
Run, Time: 
<br> 
140610, 3:45am <br> 
140612, 4:35am  <br> 
140641, 7:09am <br> 
140645, 7:58am <br> 
140646, 8:04am <br> 
140647, 8:06am <br> 
<br> 
 
Also see errors in the DAQ init. report <a href = '/svx-data1/datalog/daq/20020308.si_daq'> here </a> 
<br> 
<br>  
This ladder has similar behavior as IB0W2L4, described above. 
Again I find a correlation between the DOIM currents and 
when these init failures happen.  Also, we see the strage 
behavior of the DOIM currents in the wedge that is the FIB partner, SB2W10.  Plots located in <br> ~svxii/evah/currents_030802_SB2W11.ps <br> 
~svxii/evah/currents_030802_SB2W10.ps <br> 
but I will also try adding them below. 
<br> 
<br> 
DEM:  DEM shows on and off CAPID ERRORS at ~100% since around 
run # 138233 (jan 31). 
<br> 
<br> 
So, just as with IB0W2L4, it looks like it is a fib-wide problem 
and someone should check the sequence 
and/or the Fib/OFTM hardware.  
<br>
<!-- Author: Eva -->


<!-- Date: Fri Mar 8 19:52:05 2002 -->
<table><tr><td><a href=/elog/notebooks/silicon/Fri_Mar__8_2002_195122.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-397  align='top'></td><td>Power supply plots for SB2W11 morning of 3/8/02. Notice DOIM currents. (HV was ramped up at ~7:51)</a></td></tr></table>
<!-- Author: Eva -->


<!-- Date: Fri Mar 8 19:54:05 2002 -->
<table><tr><td><a href=/elog/notebooks/silicon/Fri_Mar__8_2002_195333.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-399  align='top'></td><td>Power supply plots for SB2W10 morning of 3/8/02, fib partner of SB2W11 which had the init errors.  Notice DOIM currents. (HV was ramped up at ~7:51)</a></td></tr></table>
<!-- Author: Eva -->


<!-- Date: Fri Mar 8 20:23:37 2002 -->
<pre>Gui log now reports when the silicon has been turned on via 
IFIX. During the testing we tried to turn on a tripped ladder 
and could not do so. We did not know that the silicon was in 
run control at the time. 
Since the silicon was booked in the run (cosmics), the FIB would 
not allow the handshake. Steve Nahn's code works. Once the  
silicon was removed from the run, we could reset the ladder. 
</pre>
<!-- Author: Doug B -->


<!-- Date: Sat Mar 9 03:59:35 2002 -->
<pre>LB0W3 5Vdoim undervoltage trip on ch 70 at about 03:25</pre>
<!-- Author: Saverio -->
<!-- Comment: Sat Mar  9 04;31;28 comment by...Saverio -->
<pre>Could not turn it on due to handshake problems although the
silicon was not in a run. As collisions will not start
untill the morning, decide to punt and take out of hdwdb,
instructing the ace to do so (Mario). The handshake problem 
may be due to the PS GUI work recently done. The ace was unable to turn it on even if silicon crates were not being used by RC.

</pre>


<!-- Date: Sat Mar 9 07:18:17 2002 -->
<pre>This message is in response to Saverio's previous entry. 
We did nothing in the PS  gui or server code associated with the 
handshake mechanism in the past 3 weeks. All the server and 
gui code work recently has been related to adding a tag when 
the silicon is turned on via IFIX. Before that we reduced the  
amount of data returned from the CAEN system when trying to get 
the system status. This work speed up the PS gui/server code  
significantly and reduced the number of spontaneous transitions 
greatly. We also stopped broadcasting on &lt;-&gt; standby "spontaneous 
transitions"; note these seem to occur during ramps. 
</pre>
<!-- Author: Doug B -->


<!-- Date: Sat Mar 9 09:17:12 2002 -->
<pre> 
Investigating the handshake problem: 
- L0B0W3 back in hdwdb 
- warm-start faulure, as expected, then I try to turn it on: 
 
FIB side: 
Sent ack to Run Control: 0 READY SUCCESS  
Fib Handshaking... 
Fib Handshaking... 
+++ MessageType 1056 
 
svxinter side 
Sending Request for f830 
Timeout 
ERROR: not all Fibs are initialized 
ERROR: FIB initialization, ps_n = 3setPSStatus,err = -1, detector = 0, ps_n = 3 
 
It looks like a deja vu: 
</pre> 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=91&frame=2&anchor=14:49:41'> 
Petra's entry a while ago</a> This FIB is antisocial: it has a good history of refused handshake 
<br> 
Now trying to reboot both FIB and svxinter crates while we have no beam.<br>
<!-- Author: Saverio -->


<!-- Date: Sat Mar 9 09:26:49 2002 -->
<pre>L00 trip while trying to turn it on 
svxinter says: "L00 B0WPC got a trip" 
 
Story until now: 
Sat Mar 09 08:56:38 CST 2002     ERROR: Power Supply L00 WEST W3 Handshake - No response from the FIB 
Sat Mar 09 09:01:51 CST 2002     ERROR: Power Supply L00 WEST W3 Handshake - No response from the FIB 
Sat Mar 09 09:22:31 CST 2002     ERROR: Power Supply L00 WEST W3 Handshake - FIB initialization failed 
Sat Mar 09 09:22:36 CST 2002     TRIP: L00  WEST  W3 5VDOIM OverVoltage</pre>
<!-- Author: Saverio -->


<!-- Date: Sat Mar 9 09:28:44 2002 -->
<pre>Re-booted the FIB: 
 
b0fib01-&gt; Fib Handshaking... 
Looking for Wedge: 
         f830 
Acknowledge Wedge 0xf830 Slot 16 
 
Fib Handshaking... 
 
SVX processPartition: 0 
 
SUCCESS ! ! 
</pre>
<!-- Author: Saverio -->
<!-- Comment: Sat Mar  9 09;33;08 comment by...Saverio -->
<pre>Well, not completely !
This time svxinter was unhappy of the fib response.

Sat Mar 09 09:27:46 CST 2002     ERROR: Power Supply L00 WEST W3 Handshake - FIB initialization failed
Sat Mar 09 09:27:54 CST 2002     TRIP: L00  WEST  W3 5VDOIM OverVoltage</pre>


<!-- Date: Sat Mar 9 09:36:30 2002 -->
<pre>Communication error ? 
Sat Mar 09 09:34:42 CST 2002     ALARM:   L00  B1  W4 PC is reported OFF, but low/high voltage is above 2/5 V </pre>
<!-- Author: Sav -->
<!-- Comment: Sat Mar  9 09;38;42 comment by...Saverio -->
<pre>I re-set the trips, GUI understood "L00 EAST OFF" and did so as well.</pre>

<!-- Comment: Sat Mar  9 09;41;04 comment by...Saverio -->
<pre>Everything came up nicely for the EAST. Handshake works there.
So, chances are that it is the FIB.
</pre>


<!-- Date: Sat Mar 9 10:04:45 2002 -->
<pre>Using handshake tool now: 
 
 
	 ******* Current Status **************  
 ID	# 	READY					DONE 
f830	1	/_b0fib01_15200752		null 
		--------------------------- 
Server			Count 
/_b0fib01_15200752	1 
	************************************ 
Got 1 Done from /_b0fib01_15200752 
 
 
 
	 ******* Current Status **************  
 ID	# 	READY					DONE 
f830	0	/_b0fib01_15200752		/_b0fib01_15200752 
		--------------------------- 
Server			Count 
/_b0fib01_15200752	0 
	************************************ 
Finished! and  done 
 
 
</pre>
<!-- Author: Saverio -->


<!-- Date: Sat Mar 9 10:17:18 2002 -->
<pre>I paged Steve Nahn and then Andrew Ivanov, 
Steve told me about the manual handshake tool that I used inthe entry above. 
Andrew explained the nature of the problem: 
- GUI reports two problems: Handshake failure and Doim overvoltage (in time order). I was convinced that the  
second was a consequence of the first, actually it is the other way around: 
- Gui gets handshake ack. 
- tries to turn on the PS 
- gets a trip, voltage not present 
- FIB sends back a failure message because it was unable to initialize. 
- GUI reports the fib initialization failure. 
A nice thing to have is a report of init failure at FIB end 
and an instruction page. 
 
The problem is, probably, a hardware one. 
I am going to take this wedge out of HDWdb and make an entry in the Silicon problem DB.</pre>
<!-- Author: Saverio -->
<!-- Comment: Sun Mar 10 19;29;40 comment by...SCN -->
<h3>Nice theory</h3> but it is wrong.
Since the Fib does all 10 channels at once, there is no way
it can do a readback to determine if it worked or not.
In fact, it (probably) never even got the message to init
as this comes only after the PS sees some current draw.
MOst likely the DOIM undervoltage trip means the message was
never sent,  so the statement "Fib sends back a failure message..." is not true.  The GUI says the Fib failed 
even though it never told the Fib to start.
<!-- Comment: Mon Mar 11 13;46;49 comment by...Andrew -->
<pre>It worked as follows:

- Gui gets handshake ack. 
- tries to turn on the PS 
- gets a trip, voltage not present 

     and doesn't send the affirmative message to FIB that
power is ON, because it failed to turn on the power. Handshake
did not work all the way through, that's why it complained about
error with Handshake specification.

There are two messages the GUI reports. The GUI sends a request 
to FIB to turn on power. If there is no response from the FIB,
it says "no response". Otherwise it turns on power and sends 
a message to iFix, after it checks that power is really on, if it
doesnt get message from iFix, it shuts off power.
</pre>
<!-- Date: Sat Mar 9 10:49:02 2002 -->
<pre>LB1W1 reintegrated on 03/08/2001 at 11:12 by Chris Hill</pre>
<!-- Author: Saverio -->
<!-- Comment: Sat Mar  9 12;39;13 comment by...Eva -->
<pre>I checked LB1W1L3 in DEM and found that it has a few-80% errors
since run # 131075.  Bit 1 is stuck high.</pre>
<!-- Date: Sat Mar 9 11:10:32 2002 -->
<pre>handshake tool: 
java si_ss.HvHdskT &lt;&gt; 
from SCN</pre>
<!-- Author: Sav -->


<!-- Date: Sat Mar 9 11:15:56 2002 -->
I made another try loading the new caenvolts.dat from Hdwdb 
no luck. LB0W3 stays off.
<!-- Author: Saverio -->



<!-- Date: Sat Mar 9 11:25:05 2002 -->
<table><tr><td><a href=/elog/notebooks/silicon/Sat_Mar__9_2002_112442.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-401  align='top'></td><td>/cdf/people2/svxii/evah/currents_030602_IB0W2_11am.ps DOIM currents for IB0W2 03/06/02 zoomed in at 11am</a></td></tr></table>
<!-- Author: Eva -->





<!-- Date: Sat Mar 9 13:56:07 2002 -->
DIAGNOSTIC REPORT FOR SB2W10 <br>  
---------------------------- <br> <br> 
 
Many trips on DVDD overcurrent on 3/8/02. No beam.<br> 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=101&frame=2&anchor=13:18:51'> 
See Gino's entry. </a> 
Had this problem in the past many times. <br> 
Here is the elog history: <br> 
 <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=93&frame=2&anchor=03:21:16'> 
Jan 21 </a> <br> 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=93&frame=2&anchor=06:42:55'> 
Jan 21 again </a>  <br> 
<a href= 
'/cgi/elog/elog.pl?nb=2002&action=view&page=61&frame=2&anchor=06:37:14'> 
Jan 21 general elog entry </a> <br> 
<a href= 
'/cgi/elog/elog.pl?nb=2002&action=view&page=92&frame=2&anchor=10:13:57'> 
Jan 31 </a> <br>  
<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=97&frame=2&anchor=16:38:31'> 
Feb 9 </a>  <br> 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=97&frame=2&anchor=22:29:01'> 
Feb 13 </a> <br> 
 <a href= 
'/cgi/elog/elog.pl?nb=2002&action=view&page=132&frame=2&anchor=19:44:18'> 
Feb 13 general elog entry </a> <br> 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=98&frame=2&anchor=15:51:21'> 
Feb 19 </a> <br>  
<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=98&frame=2&anchor=19:39:31'> 
Feb 19 again </a> <br> 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=98&frame=2&anchor=10:14:37'> 
Feb 24 </a> <br>  
<br> 
 
GUI logfile report from yesterday: <br> 
Fri Mar 08 10:45:09 CST 2002 Software protection trip: High DVDD Current  
SVX B2 W10 L2 <br> 
Fri Mar 08 10:45:22 CST 2002 Software protection trip: High DVDD Current  
SVX B2 W10 L2 <br> 
Fri Mar 08 11:13:28 CST 2002 Software protection trip: High DVDD Current  
SVX B2 W10 L2 <br> 
Fri Mar 08 11:13:29 CST 2002 Software protection trip: High DVDD Current  
SVX B2 W10 L3 <br> 
Fri Mar 08 11:13:40 CST 2002 Software protection trip: High DVDD Current  
SVX B2 W10 L2 <br>  
Fri Mar 08 11:13:40 CST 2002 Software protection trip: High DVDD Current  
SVX B2 W10 L3  <br> 
Fri Mar 08 12:19:18 CST 2002 Software protection trip: High DVDD Current  
SVX B2 W10 L2 <br>  
Fri Mar 08 12:19:19 CST 2002 Software protection trip: High DVDD Current  
SVX B2 W10 L3 <br>  
Fri Mar 08 12:19:30 CST 2002 Software protection trip: High DVDD Current  
SVX B2 W10 L2 <br> 
Fri Mar 08 12:19:30 CST 2002 Software protection trip: High DVDD Current  
SVX B2 W10 L3 <br> 
Fri Mar 08 12:31:12 CST 2002 Software protection trip: High DVDD Current  
SVX B2 W10 L2 <br> 
Fri Mar 08 12:31:22 CST 2002 Software protection trip: High DVDD Current  
SVX B2 W10 L2 <br> 
Fri Mar 08 12:45:50 CST 2002 Software protection trip: High DVDD Current  
SVX B2 W10 L2 <br> 
Fri Mar 08 12:52:53 CST 2002 Software protection trip: High DVDD Current  
SVX B2 W10 L2 <br> 
Fri Mar 08 13:00:16 CST 2002 success: Port Card SVX B2 W10 ON <br> 
Fri Mar 08 13:00:53 CST 2002 success: Channel3 SVX B2 W10 to STANDBY <br> 
Fri Mar 08 13:02:04 CST 2002 success: Channel3 SVX B2 W10 OFF <br> 
Fri Mar 08 13:02:14 CST 2002 success: Port Card SVX B2 W10 OFF <br>  
Fri Mar 08 13:02:20 CST 2002 success: Loading CAENvolts.dat <br>  
Fri Mar 08 13:02:52 CST 2002 ALARM: SVX B2 W10 PC is reported  OFF, but  
low/high voltage is above 2/5 V <br>  
Fri Mar 08 13:15:42 CST 2002 success: Port Card SVX B2 W10 ON <br> 
ERROR: PS is STANDBY. Cannot turn Port Card OFF <br> 
ERROR: PS is STANDBY. Cannot turn Port Card OFF <br> 
Fri Mar 08 13:16:56 CST 2002 success: Port Card SVX B2 W10 OFF <br> 
Fri Mar 08 13:17:21 CST 2002 ALARM: SVX B2 W10 PC is reported OFF, but  
low/high voltage is above 2/5 V <br> 
Fri Mar 08 13:17:51 CST 2002 ALARM: SVX B2 W10 PC is reported OFF, but  
low/high voltage is above 2/5 V <br> 
<br> 
Deintegrated by Bolla at 12:51pm 3/8/02. <br> 
Current history plots tell me nothing.  Just see the trips. 
<br> <br> 
DEM shows 0% for all ladders except e1a2. Chip 86->06, wrong capid 10% (in Lester's list). 
<br> 
<br> 
Perhaps we can re-integrate now?  <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=101&frame=2&anchor=13:55:19'> 
See Andrew's entry on GUI code fix. </a>  
<br> 
Passing this on to Detector expert.  
<!-- Author: Eva -->


<!-- Date: Sat Mar 9 19:00:24 2002 -->
<pre>SB2W10 re-integrated in HDWdb. No problem at all to swithc it to SB. </pre>
<!-- Author: Saverio -->


<!-- Date: Sat Mar 9 19:14:37 2002 -->
<pre>IB0W4 failed init. Successful at second attempt</pre>
<!-- Author: Saverio -->