<!-- Title: 22:15:03  Mon Dec 10 2001 -->
<!-- Crewchief: Cigdem -->
<!-- Op1: Jean -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>plan 
---- 
get silicon back into CDF</pre>


<!-- Date: Mon Dec 10 22:31:13 2001 -->
<pre> 
Summary of the IMON monitoring: 
=============================== 
 
Around 18:22 pm some channels went into "pink status", means  
currents are out of the tolarated range. 
After saving the error messages, I unmarked these channels  
and they were fine.  
 
As the shift crew put the Silicon in  
and started to take data,  
I got continuesly alarms, which I couldn't  
unmarked fast enough.  
 
The problematic channels were: 
 
IB0W2,W3,W5 all layers: AVDD currents were low. 
                        the difference between nominal  
	                currents and actual were 100-150mA. 
IB0W2 layer 4         : DVDD current were high. 
                        Latest  Current: ~370.000000 
                        Nominal Current: 285.000000 
 
SB1W7,layer 1 (SOC)    : AVDD current low. 
                        nominal current should be 562 
                        current current was ~525 
 
After taking silicon out again,  
alarms were not anymore present,  
but IB0W2,W3,W5 stayed in "yellow status",  
means caution.  
 
</pre>
<!-- Author: Cigdem Issever -->


<!-- Date: Mon Dec 10 22:38:14 2001 -->
<center><a href='/elog/notebooks/silicon/Mon_Dec_10_2001_223514.ps'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-254' ></a><a href='/elog/notebooks/silicon/Mon_Dec_10_2001_223717.ps'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-256' ></a></center>
<!-- Author: Cigdem Issever -->
<!-- Comment: Mon Dec 10 22;39;44 comment by...Cigdem Issever -->
<pre>these are the imon alarms of ISL and SVX during data taking with
silicon in. </pre>


<!-- Date: Mon Dec 10 23:20:38 2001 -->
<pre>SVX: 
 
SB0W6 layer 0: DVDD currents are too high.  
               nominal ~50mA 
               actual  193mA 
 
alarm stays there. 
 
 </pre>
<!-- Author: Cigdem Issever -->


<!-- Date: Tue Dec 11 00:51:27 2001 -->
<pre>Summary and plan? 
------------------- 
1. Ib0w0 or Ib0w1 caused done timeouts so we eneded up with 
    only fib01 for ISL ( IB0W2,3).  
2. tomorrow, someone needs to stare at the currents plots. 
    there were strange things going on as indicated by Cigdem. 
    It may all be because of wrong reference values in IMON.  
     These need to be reviewed and set.  
2. One of the SOC or g-pig, started changing currents - 
   but recovered when cold start was done - this needs 
   investigation as well. 
3. Need to understand what was wrong with IB0WW0, W1 
   and how it should have been handled. 
4. A lot of stuff happened tonight, but I am too tired to make 
   sense of them.  All currents appear to be in safe state. 
 
  Please day shifters - try to unravel what happened.  call me 
   if things are too unclear.</pre>
<!-- Author: jean -->


<!-- Date: Tue Dec 11 18:23:13 2001 -->
<pre>We tried to reproduce the IB0W0/1 problem seen last night but could not, even with the HDEVB and L3. We'll have to see if it reappears. 
 
I traced the IB0W3L4 problem to bias ratio select being too low. It is fixed, but I would still like to see that SVXMON and silimon detected it. 
 
I turned on IB5W0 and IB5W1. The fib crate (07) hung right after turning on the first portcard. With Petra's guidance I rebooted the crate. IB5W1L1 frequently failed coldstart. It seemed to do this only after taking a run. It would then succeed after a couple tries but fail after the next run. I turned it off pending investigation. Could someone please check the init log to see why it failed? (I checked in the data that there are not any bit problems). 
 
I left the on set at standby. It can (and should) be run with CDF as possible to see what other problems develop. 
 
</pre>
<!-- Author: David -->
<!-- Comment: Wed Dec 12 09;53;52 comment by...SCN -->
IB5W1 L1 init failures
<ul>
<li> 2-11-2001 15:48:53 CST : 1008107333 Init Fib f511 hdi 6 ERRORS: tried 16 chips  - <I>ALL ZEROS- Failed init to download maps</I>
<li>12-11-2001 15:48:55 CST : 1008107335 Init Fib f511 hdi 6 ERRORS: tried 16 chips - <I>Ditto (Probably doing a PED run)</I>
<li>12-11-2001 16:14:51 CST : 1008108891 Init Fib f511 hdi 6 ERRORS: tried 16 chips <I> All Zeros, but whole wedge...</I>
<li>12-11-2001 16:14:52 CST : 1008108892 Init Fib f511 hdi 6 ERRORS: tried 16 chips: <I> Same as last 3 </I>
<li> 12-11-2001 16:16:32 CST : 1008108992 Init Fib f511 hdi 6 ERRORS: tried 16 chips  <I> All fs </I>
<li>12-11-2001 16:22:43 CST : 1008109363 Init Fib f511 hdi 6 ERRORS: tried 16 chips- <I> All fs </I>
<li> 2-11-2001 16:23:41 CST : 1008109421 Init Fib f511 hdi 6 ERRORS: tried 16 chips <I> All fs </I>
<li>  12-11-2001 16:29:00 CST : 1008109740 Init Fib f511 hdi 6 ERRORS: tried 16 chips <I> All fs </I>
<li> 2-11-2001 16:29:17 CST : 1008109757 Init Fib f511 hdi 6 ERRORS: tried 16 chips <I> All fs </I>
<li>12-11-2001 16:29:36 CST : 1008109776 Init Fib f511 hdi 6 ERRORS: tried 16 chips <I> All fs</I>
</ul> that's all for failed inits from IB5W1 L1. I'd chalk up
the All 0s to no power, the all f's I don't really know

<!-- Date: Tue Dec 11 19:31:51 2001 -->
<pre>SVX Error logger summary of  run 134525  
For Silicon DAQ expert; 
   No data : IB0W2L2C6, IB0W2L3C6, IB0W2L4C1, IB0W2L4C5, 
             IB0W3 all layers 
             ( error of IB0W3 started since run 134513 ) 
             SB0W6L0C1 to C2, SB1W3L0C1 and C2 
             ( Error of SB1W3L0 started since run 134449 ) 
   PipeLine out of sync error : SB0W6L0 
             ( Error of SB0W6L0 started since run 134449 )</pre>
<!-- Author: Akiya and Cigdem -->
<!-- Comment: Tue Dec 11 22;28;48 comment by...Akiya, Cigdem  -->
<pre>Igor told us to ignore 

"Stdev Out of Limits", "Occup Out of Limits", "RMS Out of Limits"

errors, until he makes some adjustments. So will ignore them 
from now on. </pre>
<!-- Comment: Tue Dec 11 22;31;09 comment by...Akiya, Cigdem  -->
<pre>(Igor told us to ignore certain errors at the beginning of our     shift.)</pre>
<!-- Date: Tue Dec 11 19:36:13 2001 -->
<pre>The ISL DOIM voltages for B0 Wedges 2,3,5 have started to deviate from the nominal settings.  I unmarked all six of them.  Jean says she will look at the current-time plots.</pre>
<!-- Author: Hung-Chung (happens to be on shift) -->
<!-- Comment: Tue Dec 11 22;25;22 comment by...Hung-Chung -->
<pre>The DOIM voltages had deviated enough to turn the display for ISL B0 W2,3,5 pink again.  This happened about 30 minutes ago.</pre>
<!-- Date: Tue Dec 11 19:38:12 2001 -->
<pre>Run end summary of SVXMon of run 134612 
IB0W1L1  : Duplicate Channel, EOR not found , Strip out of order, 
          wrong chip count  
IB5W0L2   :  EOR not found , Strip out of order, wrong chip count 
IB0W3 all layers : No dat, see previous comments 
SB0W6L0C0, C1, C2, C3 : Pipeline out of sync 
SB0W6L0C1, C2, SB1W3L0C1, C2 : No data 
</pre>
<!-- Author: AKiya and Cigdem -->
<!-- Comment: Tue Dec 11 19;41;58 comment by...Akiya and Cigdem -->
<pre>SVXMon data dump file does not exist.  We guess that because another run is on going.</pre>


<!-- Date: Tue Dec 11 20:07:31 2001 -->
<pre>Checked SVX Error Logger run summary file of run 134613 
 
We do see in this file a new error, which we didn't see in 
the previous error summary files of today: 
 
SB0W6L0C0,C1,C2,C3 do have now in addition to the pipeline 
out of Sync  
 
"Stuck Cell ID |1| Pipeline cell id is stuck at 63"  
 
</pre>
<!-- Author: Akiya and Cigdem  -->
<!-- Comment: Tue Dec 11 22;23;09 comment by...Akiya, Cigdem  -->
<pre>The other errors in the error summary file are the same like in the previous log book entries.</pre>



<!-- Date: Tue Dec 11 22:26:52 2001 -->
<pre>Checked SVX Error run summary file of RUN 134615: 
================================================= 
 
No new error Types or no new ladders with problems. 
Problems seen in this file you find in the previous one, too. 
The stuck bit error messages of  
SB0W6L0 is still there.  
 
</pre>
<!-- Author: Akiya, Cigdem  -->




<!-- Date: Wed Dec 12 00:43:49 2001 -->
<pre>AVDD for SVX b1 wedge7 l1 was pink but returned to "caution" (current now 526mA)when I unMarked it.  Jean says she saw this 
happen a few times before.</pre>
<!-- Author: ath/monitoring ace -->


<!-- Date: Wed Dec 12 01:41:19 2001 -->
<pre>Pink: ISL Barrel 0 (all); all within 100mA of nominal. Cleared on umark. 
      SVX B0 L1 wedge7; 494mA (561 nominal.)  Cleared on unmark.        </pre>
<!-- Author: ath -->
<!-- Comment: Wed Dec 12 02;01;39 comment by...ath -->
<pre>Twice again in the ISL, same state, within 15-20 minutes. </pre>

<!-- Comment: Wed Dec 12 02;29;19 comment by...ath -->
<pre>ISL Again: delta-t 23 minutes.
and at nearly the same time (2:26) SVX in same barrel/layer/wedge as before.</pre>

<!-- Comment: Wed Dec 12 02;55;22 comment by...ath -->
<pre>SVX channel again (2:48); same characteristics.</pre>

<!-- Comment: Wed Dec 12 03;15;57 comment by...ath -->
<pre>Again:
3:03: SVX 
3:10: ISL same channels.</pre>

<!-- Comment: Wed Dec 12 03;39;12 comment by...ath -->
<pre>ISL again, ~3:30.

BTW, "all" above doesn't include wedge0 or (naturally) 4.</pre>

<!-- Comment: Wed Dec 12 03;51;16 comment by...ath -->
SVX: 3:48<br>
ISL: 3:45<p>

same channels, same currents, all clear.


<!-- Date: Wed Dec 12 04:12:18 2001 -->
<pre>ISL: 4:08 am, same channels</pre>
<!-- Author: ath -->


<!-- Date: Wed Dec 12 09:47:04 2001 -->
<pre>ISL West turned off in preparation for crate work during access. 
</pre>
<!-- Author: David -->



<!-- Date: Wed Dec 12 10:11:54 2001 -->
<pre>We measured termination resistances for the FIB command cable 
for B2W4.  Everything is fine.  See the port card e-log for more 
details, and the paper logbook for even more details. 
</pre>
<!-- Author: Gino & Jason -->


<!-- Date: Wed Dec 12 11:34:29 2001 -->
In regards to Done Timeout problems experienced by  
<a href= 
'/cgi/elog/elog.pl?nb=2001&action=view&page=897&frame=2&anchor=04:22:50'> 
Ace</a>, they all come from the Vrb reading out IB0W0/W1.  Last night, there were no accompanying reformatter errors, however, the evening before, there were, which for various reasons were 
not logged outside of the Level 3 nodes.  I asked Ilya to  
send me the log from the Level 3, and looking at it I see 
that HDIs 0-3 all have 0xc4c4c4c4 instead of for IB0W0.   
HDI 4 was not in use, and IB0W1 looks ok.  This would indicate 
that the whole portcard would not give data, not just an individual ladder.  We've seen this <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=61&frame=2&anchor=12:58:04'> 
before</a> where all ladders fail to give data, but noone really 
tracked it down.  The potential problems are 
<li> The FTM channel and the timing cable down to 
the PC transciever 
<li> A power problem affecting all DOIMs or something else in the PS 
<li> The Fib getting hung up receiving data 
</ul> 
 
The strange thing is that this problem appears to be  
trigger dependent, and even then comes in "waves": occurs at a 
DTO every 2 minutes or so,   then disapppears for a while (like maybe 1.2 hour),  then returns.  <P> 
<h4>Proposal for troubleshooting</h4> 
<ul> 
<li> Look for correlation with other variables such as <ol> 
<li> PS current fluctuation 
<li> "instantaneous" trigger rate (if possible) 
<li> Data oddities. 
</ol> 
<li> Download new Vrb firmware to eliminate Done timeouts 
from this Vrb. SB0WB has been using this firmware since<a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=81&frame=2&anchor=13:52:40'> 
Dec. 1</a> with no problems.  The effect should be to keep the  
events we would otherwise timeout on, but those HDIs which did not send data would not be present in the event. 
</ul> 
Here's a list of approximate DONE Timeout times for various runs. The run number is the integer in the file name <pre> 
errorfile134513.log:(MLE) b0svx08:Messenger:8:57:12 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134513.log:(MLE) b0svx08:Messenger:8:58:12 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134513.log:(MLE) b0svx08:Messenger:8:58:54 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134513.log:(MLE) b0svx08:Messenger:9:00:26 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134513.log:(MLE) b0svx08:Messenger:9:02:26 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134513.log:(MLE) b0svx08:Messenger:9:03:47 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134513.log:(MLE) b0svx08:Messenger:9:04:42 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134513.log:(MLE) b0svx08:Messenger:9:05:51 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134513.log:(MLE) b0svx08:Messenger:9:07:19 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134513.log:(MLE) b0svx08:Messenger:9:10:52 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134513.log:(MLE) b0svx08:Messenger:9:11:52 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134513.log:(MLE) b0svx08:Messenger:9:12:29 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134513.log:(MLE) b0svx08:Messenger:9:12:58 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134514.log:(MLE) b0svx08:Messenger:9:20:32 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134612.log:(MLE) b0svx08:Messenger:4:57:19 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134612.log:(MLE) b0svx08:Messenger:4:58:57 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134612.log:(MLE) b0svx08:Messenger:5:11:24 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134612.log:(MLE) b0svx08:Messenger:5:49:16 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134612.log:(MLE) b0svx08:Messenger:5:51:34 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134612.log:(MLE) b0svx08:Messenger:5:54:44 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134612.log:(MLE) b0svx08:Messenger:5:56:54 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134612.log:(MLE) b0svx08:Messenger:6:08:59 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134612.log:(MLE) b0svx08:Messenger:6:12:43 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134612.log:(MLE) b0svx08:Messenger:6:14:20 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134612.log:(MLE) b0svx08:Messenger:6:20:17 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134612.log:(MLE) b0svx08:Messenger:6:27:57 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134612.log:(MLE) b0svx08:Messenger:6:32:54 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134612.log:(MLE) b0svx08:Messenger:6:36:46 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134612.log:(MLE) b0svx08:Messenger:6:43:44 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134612.log:(MLE) b0svx08:Messenger:6:45:40 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134612.log:(MLE) b0svx08:Messenger:6:49:34 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:7:23:38 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:7:28:12 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:7:38:29 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:7:39:55 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:7:45:01 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:7:51:10 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:7:51:38 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:7:54:24 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:7:59:02 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:8:07:09 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:8:12:38 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:8:28:26 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:8:42:15 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:8:45:51 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:8:52:19 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:9:13:51 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:9:17:53 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134613.log:(MLE) b0svx08:Messenger:9:24:00 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:10:41:03 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:10:53:00 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:10:55:19 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:10:55:48 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:10:56:36 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:11:05:47 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:11:06:13 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:11:23:38 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:11:40:28 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:11:40:57 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:11:42:33 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:11:45:16 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:11:49:18 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:11:55:52 PM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:12:10:12 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:12:19:21 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:12:28:59 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:12:31:52 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134617.log:(MLE) b0svx08:Messenger:12:33:28 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134622.log:(MLE) b0svx08:Messenger:2:24:37 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134622.log:(MLE) b0svx08:Messenger:2:26:57 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134622.log:(MLE) b0svx08:Messenger:2:27:58 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134622.log:(MLE) b0svx08:Messenger:2:28:27 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134622.log:(MLE) b0svx08:Messenger:2:34:24 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134622.log:(MLE) b0svx08:Messenger:2:37:06 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134622.log:(MLE) b0svx08:Messenger:2:38:13 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134622.log:(MLE) b0svx08:Messenger:2:43:44 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134622.log:(MLE) b0svx08:Messenger:2:46:05 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134622.log:(MLE) b0svx08:Messenger:2:46:42 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134622.log:(MLE) b0svx08:Messenger:2:48:29 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134622.log:(MLE) b0svx08:Messenger:2:53:58 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134622.log:(MLE) b0svx08:Messenger:2:54:46 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134622.log:(MLE) b0svx08:Messenger:2:55:15 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134622.log:(MLE) b0svx08:Messenger:2:58:35 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134622.log:(MLE) b0svx08:Messenger:3:01:43 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134623.log:(MLE) b0svx08:Messenger:3:11:32 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134623.log:(MLE) b0svx08:Messenger:3:15:02 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134624.log:(MLE) b0svx08:Messenger:3:25:20 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134625.log:(MLE) b0svx08:Messenger:3:45:15 AM->Silicon Timeout:!DONE- Slots:  08 
errorfile134626.log:(MLE) b0svx08:Messenger:4:08:34 AM->Silicon Timeout:!DONE- Slots:  08 
</pre> 
 
Of course, with the correlation, there is a cause and effect question...
<!-- Author: SCN -->
<!-- Comment: Wed Dec 12 12;40;49 comment by...SCN -->
I updated the Vrb firmware for Vrb SN 176, b0svx08 Slot 8
to the 11/06/01 version.


<!-- Date: Wed Dec 12 14:14:13 2001 -->
<pre>Silicon PS work: Moved one ISL power supply to a more  
                 useful location. IB1W0 -&gt; IB0W4 
                 Installed hardware for an PS interlock</pre>
<!-- Author: herndon and napora -->
<!-- Comment: Wed Dec 12 16;27;49 comment by...herndon  -->
<pre>Dterm problem was caused by hooking up 5V to the interlock
box and crate input with interlock disabled.  Problem is
fixed.
</pre>
<!-- Date: Wed Dec 12 14:33:33 2001 -->
<pre>we're getting ready to try and add new 6C ISL ladders.  i 
changed the CC trip-temp to 25 deg-C.  depending on the 
outcome of this test, we'll change it back when we're done. 
 
</pre>
<!-- Author: doug -->


<!-- Date: Wed Dec 12 15:53:31 2001 -->
<pre>finished with tests to power ISL B2 ladders on newly cooled 
ledges. i needed to modify the supply valves for the west side 
and we should leave the CC trip temp at 25 deg-C for now.  once 
SVX is on, we'll revisit powering these.  i'm pretty sure we can 
do so, but we may need to modify some ilock thresholds for those 
RTDs on circuits w/o cooling.  Similarly, we'll wait 
for SVX to be powered prior to turning-on B1 and B4 wedges. 
 
&gt;&gt;&gt; PLEASE LEAVE THE NEW TRIP TEMP AND WEST SUPPLY VALVE SETTING  
&gt;&gt;&gt; AS THEY ARE.  THANKS. 
 
&gt;&gt;&gt; ISL B2 now OFF and disabled in HWDB.</pre>
<!-- Author: Doug -->
<!-- Comment: Wed Dec 12 15;54;19 comment by...doug -->
<pre>I'll email the ilock and cooling people to let them know 'sup.</pre>


<!-- Comment: Wed Dec 12 16;32;16 comment by...Andy H. -->
<pre>Doug's new ISL W supply valve settings, just so I don't lose them:

                        valve/%   flow/lpm   P_cot/psia
                SUP-1:   85%        4.1        10.8
                SUP-2:   75%        2.0        10.6
                SUP-3:   90%        3.6        12.0
                SUP-4:   70%        2.6        11.0
                SUP-5:   70%        2.8        10.4
</pre>


<!-- Comment: Wed Dec 12 18;11;50 comment by...Doug G. email -->
<pre>Here are a few more comments from Dou's email on this subject. 

        "please leave them this way for now.  i will eventually play w/
          them in order to balance P_cot across the first four.  i needed
          to increase the flow to SUP-3 in order to get enough flow thru
          the newly opened line so that we get small delta-T when
          operating ladders.

        &gt; the RTDs on the uncooled circuits are basically measuring the
          ambient temperature (+ any thermal coupling through the CF
          space-frame).  these get quite warm while running.  we should
          consider letting these use a separate (higher) threshold.

  b/c the ambient temperatures will be affected by the SVX heat-load, it
  does not make sense to pursue these studies (or powering B1 and B4 of
  the ISL) until the SVX is powered and we've considered these ilock
  changes.
</pre>
<!-- Date: Wed Dec 12 17:47:36 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030><h3>PLease pay attention to IMON.  If squares fo pink or red, the aces have been instructed to page the silicon pager.  If you're using the system, please leave IMON in an "green" condition. 
Call Eduardo or Jean if you need instructions.<br> 
<br> 
IMON defaults have been adjusted to reflect the ISEL changes in  
some of ISL wedges. 
</td></tr></td></table>
<!-- Author: Jean -->



<!-- Date: Wed Dec 12 18:27:52 2001 -->
<pre>ISL work today: 
 
David returned all ladders to nominal ISEL and Bias Ratio setting 
which will return ISL to nominal current values. 
 
IB0,IB5 integrated, left at standby ready for tonight's shot. 
2 problem ladders intentionally left off. </pre>
<!-- Author: Chris Hill -->


<!-- Date: Wed Dec 12 22:10:52 2001 -->
<pre>DTO's on IB5W4,5 = VRB crate 7, slot 16. 
also on svx07 slots 13,15,16 ( mostly 16) 
Also svx08 slots 8,17 
  
</pre>
<!-- Author: jean -->
<!-- Comment: Wed Dec 12 23;41;14 comment by...Lester -->
<pre>This could easily be a consequence of using the software event 
builder with ignore busy not set.
</pre>
<!-- Date: Wed Dec 12 22:21:43 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>I changed the IMON caution limits on ISL 2,5 volt doims 
from 10ma to 70ma to avoid confusing the shift people. 
 
Amit is looking with Auke's program. 
 
There are clearly many unexplained fluctuations in the  
currents.  </pre></td></tr></td></table>
<!-- Author: Jean -->


<!-- Date: Wed Dec 12 22:42:12 2001 -->
<pre>Amit looked at the DOIM PS plot.  There were definitely glitches upward in many wedges in IB0 and IB5, but not in the SVX ladders. 
Of course there are many fewer of them.</pre>
<!-- Author: Jean, amit -->



<!-- Date: Wed Dec 12 23:41:37 2001 -->
<pre>With Steve's help over the phone, I put version VRB1B06P.hex  
firmware into crate 7, slot 16, serial number 0147, in see 
if that helps the DTO's. Note that this seemed to have 
 fixed the same problem in In IB0W0,1. 
 
  This is the firmware with the VRB timeout.</pre>
<!-- Author: Jean -->
<!-- Comment: Wed Dec 12 23;53;01 comment by...Jean -->
<pre>Unfortunately this didn't appear to help. We are taking out
b0fib07 to see if we can run reasonably this way.
  </pre>


<!-- Date: Thu Dec 13 00:07:45 2001 -->
<pre>IB5W2 DVDD current seems to fluctuae more than anyone else. 
Someone should take a look at the plots.</pre>
<!-- Author: Jean -->


<!-- Date: Thu Dec 13 02:52:06 2001 -->
<pre>While on shift as monitoring ace, I noticed several pink ladders 
in IMON which did not persist.  Each ladder was about 115mA over 
current, but unmarking these immediately showed that the current 
had returned to nominal values. 
 
The pink ladders were: 
 
IB5W2 DVDD 0, 1 
IB0W4 DVDD 1,2,3 
 
These ladders are now all green.</pre>
<!-- Author: Trevor -->


<!-- Date: Thu Dec 13 08:39:16 2001 -->
<pre>We are at the beginning of a new store. 
The aces had many DTOs from svx07 (various slots), I took out the corresponding  
wedges (all ISL east) to find out the trouble maker, but in the  
meantime we lost another VME PS (NET rack = fib04). The fib is not  
responding anymore and we also don't see any LEDs lit on the back of  
the VRBs.  
We continued to get DTOs from the east ISL wedges. After Steve  
suggested that we might not be able to run at all when one of the DAISY chained  
VRBs doesn't get any data out of the FIB, Joel took  
the full silicon out of the run. 
I tried to find a correlation between PS dying and spike in losses, but didn't  
see any. 
The dead of the PS was around 7:50am this morning. 
I paged the general silicon pager to inform Chris about the situation. 
 
Frank Chlebana suggested to investigate the DTOs in a seperate partition with HDWEB  
and a high rate/L2R trigger table.  
Steve said he would come in around 9am, so I leave this decision to him.</pre>
<!-- Author: Petra -->
<!-- Comment: Thu Dec 13 08;39;56 comment by...Petra -->
<pre>I'll put the full detector to SB then.</pre>

<!-- Comment: Thu Dec 13 09;46;25 comment by...Arnd -->
Please see this <a href='/cgi/elog/elog.pl?nb=2001&action=view&page=901&frame=2&anchor=09:35:23'>
elog entry </a> for run control's best estimate of when b0fib04 died.





<!-- Date: Thu Dec 13 11:40:34 2001 -->
<h1> What a fun morning</h1> 
<ul> 
<li> I bypassed all of the SVXII East Vrbs because of the 
dead fib crate.  Should we want to include the 1/2 that is 
still alive, we need to pull VTMs out of the back.  Instead I 
suggest we concentrate on SVXII West 
<li> starting up RC, I found IB5W2 L0 OFF, but RC wanted to 
Init.  I found no reason why it was OFF, nothing in logs, and 
the last command was ISL -> Standby, so I turned it on. 
<li> I found a work-around to the DTO problem I think.  I was 
testing with ISL, and getting DTOs often enough if I include any HDI in the bottom PC of b0svx07 slot 13, or any in b0svx07 slot 16, except that I added them back one by one and all was peachy. 
However, thanks to a push by M. Bishai, I went and checked 
and found that we match the Fib channel mask to the HDIs actually on (were it not for flaky fib firmware, this would be the right thing to do).  So, I changed some code to leave all Fib channels 
as they are in the hdwdb (which is currently on) independent of 
which hdis are on.  DTOs went away. <P> 
We know that this is a bug in the fib firmware, though we haven't 
figured out why.  Leaving all the channels on probably doesn't 
hurt, but we should watch closly 
<li> Other oddity- saw the SRC reporting both BUSY and DONE tos 
to the TS (Both L2 DONE and TS WAIT were asserted).  So while 
a VRB reported a DONE TO, the TS (apparently) picks the BUSY 
TO.  Don't know why, LM can look at this one 
<li> Tried to bias ISL, IB5W5L1 Bias1 OVC (retried, and it 
happened again).  C Hill is looking at it 
<li> Took PED D DPS OFF run 134793  (GPs, ISL B5W5L1 at Standby) 
<li> Took PED D DPS ON run 134795   (ditto) 
Some reformatter errors (two "imposter fib ids", and one  
c4c4 instead of f542 and f543 fib ids.) 
<li> D MODE is really slow... 
</ul> 
News flash, IB5W5L1 was being ramped during these runs...<P> 
And now, we're attemting to change RAMP TRIM across  
everything from 5 to 255<P> 
OK, enough for now...  
<!-- Author: SCN -->
<!-- Comment: Thu Dec 13 12;00;08 comment by...SCN -->
Ah, but we found that the ramptrim change has been <a href=
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=70&frame=2&anchor=08:40:37'>
done</a>.  Now we are taking data in our own partition
while L2 work is going on.  Joel has been told that when the
HRR rate goes below 1 every 2 minutes, he can put us in. 
<P> Run 134799 is a standard Physics Run but with Calibration
Continuous and Auto L2A.
<!-- Date: Thu Dec 13 11:54:30 2001 -->
<pre>Trip on bias0 of IB5W5L1 reported by SCN. 
Found to draw ~1uA up to 90V, ~10uA at 100 V, ~25uA at 110V. 
 
Reduced operating voltage from 110V to 95V.  Will check effect 
in data (read all dps off, run 134798 - 20e8e)</pre>
<!-- Author: Chris Hill -->
<!-- Comment: Thu Dec 13 16;34;00 comment by...David -->
<pre>The landaus still look fine at 95V. Let's leave it that way for a while.
</pre>


<!-- Date: Thu Dec 13 17:48:05 2001 -->
<pre>Unmarking two pink squares on iMon: 
 
IB0W4L3 DVDD Latest/Nom.: 445/315mA 17:42:46 
IB0W4L2 DVDD Latest/Nom.: 425/305mA 17:28:13</pre>
<!-- Author: Hung-Chung (monitoring ace) -->


<!-- Date: Thu Dec 13 18:11:11 2001 -->
<pre>We masked off all of the chips in the following wedges for the 
SiliMon error summary: SB0W6L0, SB0W11L1, SB1W3L0. 
 
This affects the tests zeroCheck, bitCheck, and landauCheck. 
</pre>
<!-- Author: Amit & Jason -->


<!-- Date: Thu Dec 13 18:21:37 2001 -->
<pre>Unmarking pink square on iMon: 
 
SB0W6L0 DVDD Latest/Nom.: 82/37mA 18:19:03</pre>
<!-- Author: Hung-Chung -->
<!-- Comment: Thu Dec 13 18;23;26 comment by...H-C -->
<pre>It stayed pink:

Latest/Nom.: 70/37mA 18:21:41</pre>

<!-- Comment: Thu Dec 13 18;27;49 comment by...H-C -->
<pre>Jean/Amit/Lester have looked at it.</pre>


<!-- Comment: Thu Dec 13 19;03;54 comment by...H-C -->
<pre>Paged Si general pager.</pre>
<!-- Comment: Thu Dec 13 19;04;09 comment by...H-C -->
<pre>&lt;pre&gt;Paged Si general pager.&lt;/pre&gt;</pre>
<!-- Comment: Thu Dec 13 19;33;21 comment by...H-C -->
<pre>The previous entry is a duplicate of the one before it, caused by unknown evil forces over the network.  Please ignore. 

Chris said that I should unmark the wedge and keep an eye on it, although it probably will go pink immediately.  This should not be serious, since it is a guinea pig wedge.  

I have unmasked it.   It's green now.  Yeehaw.</pre>
<!-- Date: Thu Dec 13 18:50:09 2001 -->
<pre>Pink squares on iMon unmarked: 
 
SB0W4L1 DVDD Latest/Nom.: 440/320mA 18:46:34 
SB0W4L2 DVDD Latest/Nom.: 980/305mA 18:46:34 
SB0W4L3 DVDD Latest/Nom.: 900/315mA 18:46:34 
</pre>
<!-- Author: H-C -->
<!-- Comment: Thu Dec 13 20;29;01 comment by...H-C -->
<pre>TYPO above: ISL, not SVX.</pre>
<!-- Date: Thu Dec 13 19:42:00 2001 -->
<pre>Added chips to the "do not check" list in SiliMon from several 
ISL ladders.  See the file (on the b0 cluster)  
/data1/consumer/results/SiliMon/SiliMon134837.sum for a list 
of ladder/chips removed.  
 
I edited this file to get the excluded chip list.  Why doesn't 
every chip in a ladder show up on this list?</pre>
<!-- Author: Amit -->


<!-- Date: Thu Dec 13 20:21:28 2001 -->
<pre>Pink on iMon: 
  IB5W2L0 DVDD 435/310mA(nom) 20:13:23 
  IB5W2L1 DVDD 435/300mA(nom) 20:13:22 
  IB5W2L2 DVDD 480/360mA(nom) 20:17:30 
  IB5W2L4 DVDD 425/275mA(nom) 20:14:04 
All four will be unmarked.</pre>
<!-- Author: H-C -->
<!-- Comment: Thu Dec 13 20;25;21 comment by...H-C -->
<pre>turned pink again: IB5W2L4 DVDD 410/275mA(nom) 20:23:12
I'll unmark this one once more.</pre>


<!-- Comment: Thu Dec 13 20;36;01 comment by...H-C -->
<pre>turned pink once again:
  IB5W2L0 DVDD 445/310mA(nom) 20:28:53 
  IB5W2L1 DVDD 435/300mA(nom) 20:28:53
  IB5W2L3 DVDD 425/300mA(nom) 20:30:48
  IB5W2L4 DVDD 415/275mA(nom) 20:29:34
CO just paged the Si general pager.  Chris called back.
i'll talk to Chris about this after the CO iH-Cs done.</pre>
<!-- Comment: Thu Dec 13 20;57;18 comment by...H-C -->
<pre>Chris instructed me to unmark all the pink squares.  They are gone (for now).</pre>
<!-- Date: Thu Dec 13 20:26:47 2001 -->
<pre>IB5W2L3 DVDD 425/300mA(nom) 20:25:22 
unmarked.</pre>
<!-- Author: H-C -->





<!-- Date: Thu Dec 13 21:05:51 2001 -->
<pre>Run 134844.  SiliMon assaulted the Consumer Operator with a barrage of LandauBad error messages.  Chris was paged.  It turns out that almost each of the powered wedges and layers of the ISL was giving this particular error.  The sole exception was IB5W2L0-4.  Strangely enough, IB5W2L0-4 were the ones that repeatedly turned pink on iMon. 
 
Per Chris's instruction, I looked the bias voltages for the ISL wedges that were giving SiliMon errors.  Nothing out of the ordinary was found.  The run was ended for unrelated reasons.  We will keep an eye or two on the next run.</pre>
<!-- Author: H-C -->


<!-- Date: Thu Dec 13 22:17:18 2001 -->
<pre>SiliMon complaining about "bad landaus" on ISL.  Chris paged, 
checked HV w/ monitoring ace and couldn't find any bias problems. 
 
These errors have gone away as the run progressed, so one suspect 
is low statistics.  A histo with 100 events and mean &lt; 20 is 
flagged as an error, and then again every 50 events. 
 
I have changed this to 500 events for the first check, and then 
every 200 events. 
</pre>
<!-- Author: Amit -->


<!-- Date: Thu Dec 13 22:50:37 2001 -->
<pre>We looked at DVDD currents for IB5W2 from 20:00 to 22:30 on 12/13/01 (today) using Auke's plotting.  The currents for all layers are jumping 
around  by &gt; 100 ma.  The nominal DVDD current is ~300, but 
occasionally it goes up to ~450 ma.  At this higher current 
state, the current is jittery.  Then it falls back to nominal, 
steady voltage. 
 
All layers show this jittery high current behavior.  Bad PS 
module?  </pre>
<!-- Author: Hung-Chung, Amit -->


<!-- Date: Fri Dec 14 06:07:41 2001 -->
Although I didnot put on e-logs about the already-known DVDD overcurrent of IB5W2L0-4, anyway, IMON showed a pink color from this wedge more than 20 times during the store (0:00 ~ 5:00). 
<!-- Author: Soushi -->


<!-- Date: Fri Dec 14 06:09:28 2001 -->
The other wedges except IB5W2 looked fine.(IMON)
<!-- Author: Soushi -->


<!-- Date: Fri Dec 14 09:39:19 2001 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre> I forgot the DAQ pager at home :) 
please try to reach me in my office: 2169 during the day.</pre></td></tr></td></table>
<!-- Author: Petra -->


<!-- Date: Fri Dec 14 11:22:37 2001 -->
<pre>The large current fluctuations on IB5W2 may be understood. This is the wedge which was run latch all before the access to do clustering studies. It is still in latch all mode. Being latch all means it spends more time reading out so that the DVDD current required from the supply is larger so the current fluctuations are bigger and more frequent. 
 
It is now set back to neighbor mode. 
 
SB3W8 will probably need similar checking. 
</pre>
<!-- Author: Chris and David -->