<!-- Title: 15:50:57  Mon Jan 19 2004 -->
<!-- Crewchief: Stan Forrester (ssf) -->
<!-- Op1: Rong-Shyang  Lu -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Silicon</pre>


<!-- Date: Mon Jan 19 16:10:49 2004 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3322&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3322  width='300' align='top'></a></td><td>AVDD current drop on SB0W8L4</td></tr></table>
<!-- Author: ssf -->


<!-- Date: Mon Jan 19 16:52:58 2004 -->
At 0920 run had to be terminated due to failure of SB0W8L4 to init  
(MLE) b0fib02:Messenger:9:23:49 AM->Error Initializing HDI Slot 10 Chan 4: SVX B0W8   
(MLE) b0fib02:Messenger:9:23:49 AM->Statechange failed 
<br><br> 
Where channel = ladder number.  -> SB0W8L4 
<br><br> 
this error message roccurrs throught the shift. 
 
<br> 
DAQ ace reports that throught the shift the crate B0FIB02 needed to be shepparded frequently to get it to pass through config. 
Other times the crate had to be shepparded are 1313, 1319, and 1502 are noted in the log. 
<br><br> 
Abort Gap losses:  Just after 0920 there was a spike in abort gap losses. See <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=view&page=56&frame=2&anchor=15:57:01.606'> 
plot in shift e-log. </a>.  Investigate this for possibel beam incident???? 
<br> 
~1540  Silicon Pager called.  Sheparding no longer configs b0fib02 
<br> 
Hockerized crate - did not fix problem. 
<br> 
David Clark suggested mask init read back. Directions for masking readback using traceSilicon 
<ul> 
	<li>setup fer</li> 
	<li>traceSilicon</li> 
	<ul> 
	  <li>select option 0, enter user name and password</li> 
	  <li>select option 4, enter hex ladder name i.e. e084</li> 
	  <li>add 16 to HDI address, i.e channel 4 becomes 20 this sets bit4 of the HID address "high"</li> 
	  <li>select return a number of times to get through the rest of the process</li> 
	   <li>select 99 to exit</li>
         </ul> 
</ul> 
 
SVXmon shows that for SB0W8L4 that we are only getting ~3 of 14 chips read out.  Combined with a drop in the 
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=181&frame=2&anchor=16:10:49'> 
AVDD </a> looks like an AVDD2. 
<br> 
Update:  1616  current plot for AVDD SB0W8L4 is back at the nominal level following a system HRR.  Not sure why, but in SVXmon "strip Occupancy for Detector W5 - A Phi (SB0W8L4C13)(%) shows occupancy everywhere except in the range ~385 to ~505.  And the Z side - A Z (... C6)(%) has no occupancy before strip 128.
<!-- Author: ssf -->
<!-- Comment: Mon Jan 19 17;22;51 comment by...rainer -->
Happened earliest today 7:30am  in run <a href='http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178120'> 178120 </a>. See annotated main elog entry <a href=' http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=copyentry&page=56&time=13:19:44.061'>  here </a>. The aces should have informed the main pager carrier earlier - my bad since I probably covered it not well enough in Ace Class (no time). 
<!-- Comment: Mon Jan 19 18;03;06 comment by...rainer -->
there is a spike in lostp around 7:30am this time, see  <a href='http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=copyentry&page=55&time=08:05:12.278'> here </a>. but if that means anything ....
<!-- Date: Mon Jan 19 16:54:57 2004 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3323&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3323  width='300' align='top'></a></td><td>updated plot of AVDD for SB0W8L4</td></tr></table>
<!-- Author: ssf -->

<!-- Date: Mon Jan 19 17:00:06 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W8L4 
<li>problem n: 1190 
<li>Status: needs_work
<li>Type: AVDD2
<li>Action: Incident
<li>entered by: sforrest 
<li>description is: Masked after ladder would not init.  See <a href= 'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=181&frame=2&anchor=16:52:58'> silicon e-log entry. </a>   
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B0W8&layer=L4">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Mon Jan 19 17:05:37 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W8L2 
<li>problem n: 1191 
<li>Status: needs_work
<li>Type: RX
<li>Action: Incident
<li>entered by: sforrest 
<li>description is: bit 7 low 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B3W8&layer=L2">report</a></ul>
<!-- Author: Problem DB -->





<!-- Date: Mon Jan 19 18:42:39 2004 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3324&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3324' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3325&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3325' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3326&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3326' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3327&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3327' width='300'></a></center>
<!-- Author: ssf -->
<!-- Comment: Mon Jan 19 18;50;49 comment by...ssf -->
Investigated the relationship between high read out times for svx and high dead times for CDF.  Vertical axis are average read out times with error bars for e2a4.  The Horizontal axis is the CDF deadtime reported on the run summary page.<br><br>
e2a4 was the high read out ladder for 18 runs used.
<br><br>Graphs 1 and 3 are long horizontal scale graphs showing dead times out to ~50%.  Graphs 2 and 4 use a shorter horizontal scale than graphs 1 and 3 to show more detail in the region where most of the data are plotted.



<!-- Date: Tue Jan 20 10:03:56 2004 -->
<pre>Investigated SB0W8L4 via IMON and in svxdaq.   
Currents are stable at the lower plateau observed yesterday. 
(in svxdaq) 5k events written to laser10922.svx on b0dap42.  Within the noise plots it appears that down stream chips appear momentarly and then dissappear in the next event.  This along with the current behavior suggets that the problem no longer intermittant.</pre>
<!-- Author: ssf -->


<!-- Date: Tue Jan 20 10:38:39 2004 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3329&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3330  width='300' align='top'></a></td><td>AVDD for e084, stable at lower level</td></tr></table>
<!-- Author: ssf -->
<!-- Comment: Tue Jan 20 15;33;03 comment by...ssf -->
<pre>
   1.AVDD Current Drop: The above current plot shows a drop of ~480uA on 19 Jan 04. This would be consistent with the loss of 6 chips (80mA per chip).
   2.Initialization History: Ladder failed init 13 times on the 18th and 42 times on the 19th.  Read out masked earlier.
   3.Today's Run: Doesn't Initialize when running with svxdaq. Determined that problem was chip 10.
   4.BE of Chip: ramp pedestal is changed from 6 -> 15 and adc count rose as expected. 
   5.FE of Chip: ISEL bits set to 0, AVDD actually rose when downstream chips recovered (as predicted by Gino).</pre>
<!-- Date: Tue Jan 20 10:43:36 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W8L4 
<li>problem n: 1190 
<li>Status: needs_work
<li>Type: AVDD2
<li>Action: Currents_investigated
<li>entered by: sforrest 
<li>description is: <a href= 'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=181&frame=2&anchor=10:38:39'> Investigated AVDD current. </a> Set nominal AVDD current to 626 with new range of 622-630.  
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B0W8&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Tue Jan 20 11:24:46 2004 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3331&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3331  width='300' align='top'></a></td><td>e084 Avdd for approx last year</td></tr></table>
<!-- Author: ssf -->
<!-- Comment: Tue Jan 20 11;49;08 comment by...ssf -->
e084 AVDD current for the last year.  The recent measurements are just above 600 on the far right.  You can see that AVDD has been making excursions to this same level for some time now, but "recovering."  

<!-- Date: Tue Jan 20 12:30:52 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: IB0W2L1 
<li>problem n: 1178 
<li>Status: needs_observation
<li>Type: Chip_other
<li>Action: IMON_limits
<li>entered by: sforrest 
<li>description is: Changed AVDD limit nominal to 1180, with range of 1190 - 1150.  This follows ISEL bit changes on Saturday. 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=ISL&wedge=B0W2&layer=L1">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Tue Jan 20 12:39:50 2004 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3332&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3332  width='300' align='top'></a></td><td>~one year's worth of current plots for IB0W2L1 Avdd.  Recent current jump due to ISEL bit work on saturday.  However there was a current jump on or about 20 March 03, but no PBD entry or other explaination.</td></tr></table>
<!-- Author: ssf -->
<!-- Comment: Tue Jan 20 18;02;37 comment by...Gino -->
<pre>The jump you point to is very likely the BIAS RATIO SELECT change.
I do not have time to check it out.
</pre>
<!-- Date: Tue Jan 20 12:46:17 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB1W8L4 
<li>problem n: 1192 
<li>Status: needs_observation
<li>Type: AVDD
<li>Action: IMON_limits
<li>entered by: sforrest 
<li>description is: From Rong-Shyang Change the IMon nominal value of SB1W8L4 from 1100 to 1110, and also corresponding range to 1095 - 1115.  
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B1W8&layer=L4">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Tue Jan 20 14:32:17 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W8L2 
<li>problem n: 1191 
<li>Status: needs_access
<li>Type: RX
<li>Action: Diagnosis_update
<li>entered by: sforrest 
<li>description is: Confirmed b7h in svxdaq, needs access to swap fibers. 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B3W8&layer=L2">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Tue Jan 20 14:32:17 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W8L2 
<li>problem n: 1191 
<li>Status: needs_access
<li>Type: RX
<li>Action: Diagnosis_update
<li>entered by: sforrest 
<li>description is: correction b7L 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B3W8&layer=L2">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Tue Jan 20 14:54:47 2004 -->
Added two new run configurations to our DAQtest -> SVX directory in run control: SVXCAL_DPSON and SVXCAL_DPSOFF.  The ACEs should use these run configurations for D-mode calibrations. 
<BR> 
The two separate run configurations are intended to help us avoid cases where both runs are taken with the same DPS mode.  
<!-- Author: Jason N -->


<!-- Date: Tue Jan 20 15:35:51 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W8L4 
<li>problem n: 1190 
<li>Status: needs_observation
<li>Type: AVDD2
<li>Action: Diagnosis_update
<li>entered by: sforrest 
<li>description is: During AVDD2 investigation discovered that setting ISEL bits on offending chip (chip 10) allowed the recovery of the downstream chips as predicted by gino.    <a href= 'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=181&frame=2&anchor=10:38:39'> See e-log entry </a>   
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B0W8&layer=L4">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Tue Jan 20 15:37:47 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W8L4 
<li>problem n: 1190 
<li>Status: needs_observation
<li>Type: AVDD2
<li>Action: Diagnosis_update
<li>entered by: sforrest 
<li>description is: During AVDD2 investigation discovered that setting ISEL bits on offending chip (chip 10) allowed the recovery of the downstream chips as predicted by gino.    <a href= 'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=181&frame=2&anchor=10:38:39'> See e-log entry </a>   
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B0W8&layer=L4">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Tue Jan 20 16:24:40 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W8L2 
<li>problem n: 1191 
<li>Status: needs_access
<li>Type: Optical_other
<li>Action: Diagnosis_update
<li>entered by: sforrest 
<li>description is: Swapped fibers to see if problem would follow fiber.  Swapped  fibers for e382 and e381.  After swap e381 showed c2c2 problem  rather than b7L.  Suspect OBVD corrupted.  Needs light level  measurement during access. 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B3W8&layer=L2">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Tue Jan 20 16:35:02 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W8L2 
<li>problem n: 1191 
<li>Status: needs_access
<li>Type: Optical_other
<li>Action: Diagnosis_update
<li>entered by: sforrest 
<li>description is: Correction C2C2 is due to BECLK cludge rather than OBVD problem.  Suggest swap RX with channel 1 or tune BECLK. 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B3W8&layer=L2">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Tue Jan 20 17:06:38 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W7L2 
<li>problem n: 1046 
<li>Status: needs_observation
<li>Type: AVDD2
<li>Action: IMON_limits
<li>entered by: sforrest 
<li>description is: Placed back into hwdb so we could attempt to repower ladder.   Ladder does not trip, but had to reset IMON limits from 815  to 800 mA.  Shifted range downward by 15mA ( new range 790  to 835) 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B3W7&layer=L2">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Tue Jan 20 17:15:49 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W8L4 
<li>problem n: 1190 
<li>Status: needs_observation
<li>Type: AVDD2
<li>Action: IMON_limits
<li>entered by: sforrest 
<li>description is: changed IMON limits nominal 1010 with range 1006 to 1014.   Increase in current due to recovery of downstream chips. 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B0W8&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Wed Jan 21 02:55:41 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W8L4 
<li>problem n: 1190 
<li>Status: needs_observation
<li>Type: AVDD2
<li>Action: IMON_limits
<li>entered by: sforrest 
<li>description is: changed IMON limit nominal 1025 (just shifted limits),  keeping an eye on it. 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B0W8&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Wed Jan 21 03:39:18 2004 -->
the following links are broken:<pre> 
<b0dap30.fnal.gov> file powerset.dat 
powerset.dat: broken symbolic link to /cdf/code-common/cdfonline/caencontrol/devel/powersetfiles/powerset_01212004_0329AM.dat 
 file CAENvolts.dat 
CAENvolts.dat: broken symbolic link to /cdf/code-common/cdfonline/caencontrol/devel/caenvoltsfiles/CAENvolts_01212004_0327AM.dat 
<b0dap30.fnal.gov>  
</pre> 
rerunning tracesilicon 10 (by stan) to re-create it did not fix the latter. as a consequence, svxinter reports 
<pre>    
                    
-> Conf file is /mnt/autofs/cdf/people2/svxii/daq/PSanywhere/PS.conf  
$CAENCONTROL_DIR                                                      
Mapping file is $CAENCONTROL_DIR/CAEN.map 
Voltage file is $CAENCONTROL_DIR/CAENvolts.dat 
b0dap24:bash: /cdf/code-common/cdfonline/caencontrol/devel/CAENvolts.dat: No such file or directory 
 
PowerSet file is $CAENCONTROL_DIR/powerset.dat 
b0dap24:bash: /cdf/code-common/cdfonline/caencontrol/devel/powerset.dat: No such file or directory 
 ERROR: reading powerset.dat 
LV Currents file is $CAENCONTROL_DIR/CAENLVcurrent.dat 
 
program 
Exception current instruction address: 0x00db2c58 
Machine Status Register: 0x0010b130               
Condition Register: 0x22800088      
Task: 0xea1be8 "t2"         
</pre>  
<!-- Author: rainer -->

<!-- Date: Wed Jan 21 04:24:35 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: LB1W5L0 
<li>problem n: 840 
<li>Status: fixed
<li>Type: RX
<li>Action: Monitoring
<li>entered by: petemc 
<li>description is: Error rate low and appears only intermittently.  Take it off watch list for now. 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=L00&wedge=B1W5&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Jan 21 04:30:59 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB2W3L0 
<li>problem n: 1079 
<li>Status: fixed
<li>Type: RX
<li>Action: Monitoring
<li>entered by: petemc 
<li>description is: Problem has not resurfaced. 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B2W3&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Jan 21 04:33:03 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W9L2 
<li>problem n: 1065 
<li>Status: fixed
<li>Type: RX
<li>Action: Monitoring
<li>entered by: petemc 
<li>description is: Problem has nor recurred. 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B3W9&layer=L2">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Jan 21 04:39:16 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W2L1 
<li>problem n: 784 
<li>Status: needs_observation
<li>Type: Unknown
<li>Action: Diagnosis_update
<li>entered by: petemc 
<li>description is: Large Bit 0 error rates come and go since action in December.  Maybe something else needs to be done? 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B0W2&layer=L1">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Jan 21 04:44:35 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB1W10L0 
<li>problem n: 1167 
<li>Status: needs_observation
<li>Type: Unknown
<li>Action: Monitoring
<li>entered by: petemc 
<li>description is: Low rate (~1%) of garbage data persists.  Monitor to make sure it doesn't get any worse. 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B1W10&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Jan 21 04:48:44 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: LB1W5L2 
<li>problem n: 1100 
<li>Status: fixed
<li>Type: RX
<li>Action: Monitoring
<li>entered by: petemc 
<li>description is: Errors persist at low rate (~1% every 10 runs or so).  Monitor to see if they become persistent enough to warrant action. 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=L00&wedge=B1W5&layer=L2">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Jan 21 04:52:53 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W0L1 
<li>problem n: 1107 
<li>Status: needs_observation
<li>Type: Unknown
<li>Action: Monitoring
<li>entered by: petemc 
<li>description is: Approximately every 10 runs, get 1 run with substantial bit 2 errors.  Monitor to see if it becomes persistent enough to act on. 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B0W0&layer=L1">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Jan 21 04:57:55 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: IB0W0L4 
<li>problem n: 130 
<li>Status: needs_observation
<li>Type: TX
<li>Action: Monitoring
<li>entered by: petemc 
<li>description is: Problem staying at same low rate (~1%) but decreasing in frequency. Watch it a little longer to be sure this trend continues. 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=ISL&wedge=B0W0&layer=L4">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Jan 21 05:05:21 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB4W10L3 
<li>problem n: 1193 
<li>Status: needs_observation
<li>Type: Unknown
<li>Action: Monitoring
<li>entered by: petemc 
<li>description is: Occasional bit 5 high. 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B4W10&layer=L3">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Wed Jan 21 05:18:50 2004 -->
<h3> double whammy night  </h3> 
<li> first, SB3W8L0 does not <a href='http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=copyentry&page=61&time=02:39:15.263'> config </a>. not clear whether it had tripped or not. most likely was still powered. 
<li> I try to mask it from home, but apparently failed, since config failed <a href='http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178252'> again </a>. Luckily, stan spotted it as well and masked the readback. 
<li> on ramping the HV, there was a heartbeat alarm and holy trinity fires. PSGui was frozen. 
<li> (maybe) at the same SB3W8L0 tripped off - unbeknowst to us, since the GUI was in limbo. needs to be checked with the IFIX log files. 
<li> I saw svxinter report suspended robin jobs - assume this is why  GUI died, restart svxinter, but the see the following <a href='http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=181&time=03:39:18'>messages</a> - the powerset.dat and CAENvolts.dat files are missing!  
<li> Stan tries to recreate the files from the hwdb, but he gets error some error messages. the files don't appear. 
<li> page andy hocker and I come in. 
<li> hwdb extract works on b0dap30 but not on b0dap42 - there are a lot of core files in the home directory. I delete 100MB worth of core files, when writing out the files worked. This is odd since <pre> 
 
svxii@b0dap44 ~]$ df -h . 
Filesystem            Size  Used Avail Use% Mounted on 
b0dau30:/cdf/people2   34G   23G   11G  67% /mnt/autofs/cdf/people2 
</pre> 
and 
<pre> 
 quota -v 
Disk quotas for user svxii (uid 1111):  
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace 
b0dap32:/cdf/code-Linux-2.0.34 
                      0       0       0               0       0       0         
b0dap32:/cdf/svx/data1 
                      0       0       0               0       0       0         
b0dap32:/cdf/cdf-cf-data1 
                      0       0       0               0       0       0         
[svxii@b0dap44 ~]$ quota  
Disk quotas for user svxii (uid 1111): none 
[svxii@b0dap44 ~]$ quota -v 
Disk quotas for user svxii (uid 1111):  
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace 
b0dap32:/cdf/code-Linux-2.0.34 
                      0       0       0               0       0       0         
b0dap32:/cdf/svx/data1 
                      0       0       0               0       0       0         
b0dap32:/cdf/cdf-cf-data1 
                      0       0       0               0       0       0         
[svxii@b0dap44 ~]$ </pre> 
 
So these commands don't appear to be very helpful to see the space on the disks .... 
<li> reboot svxinter, works fine, restart GUI OK, find wedge SB3W8L0 tripped. 
<li> there are handshake errors, only getting ladders one by one eventually gets the wedge back to standby. also , ISL B5 is already on - the freeze must have happened during ramping.  
<li> incidently, SB3W8L0 DID config successfully on coldstart when we checked the silicon out with RC. 
  
<!-- Author: Stan, Rainer, Andy H. -->
<!-- Comment: Wed Jan 21 05;46;28 comment by...ssf, rainer -->
netscape communicator seems to produce these core files:
<pre>
file core.24163 
core.24163: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'netscape-commun'
<svxii@b0dap30>file core.1387
core.1387: can't stat `core.1387' (No such file or directory).
<svxii@b0dap30>file core.13873
core.13873: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'netscape-commun'
</pre>
and <pre>
du -h core*
17M	core.13873
14M	core.24163
</pre>


<!-- Comment: Wed Jan 21 05;47;45 comment by...Rainer, ssf -->
<pre>BTW dediced not to reset the links at the time since I knew we had a new ladder in SB3W7L2  so feared hwdb status not in syc with older CAENvolts.dat.</pre>
<!-- Comment: Wed Jan 21 06;05;03 comment by...rainer -->
<pre>incidently,  SB3W8L0 tripped off earlier in the pre-access checkout.</pre>
<!-- Comment: Wed Jan 21 08;21;51 comment by...Disk Janitor -->
I think the quota command only produces useful info on b0dau30 <pre>
<svxii@b0dau30>quota -v
Disk quotas for svxii (uid 1111):
Filesystem    usage   quota   limit    timeleft  files   quota limit    timeleft
/cdf/people21372232 1400000 1500000              19305       0   0            
/cdf/people1   3968       0       0                 10       0   0            
/cdf/people3   5264       0       0                 55       0   0            
/data1       282524  500000 1300000              13925       0   0          </pre>
1400000 - 1372232 = 27768 which is kinda small on /cdf/people2. <P>
So, where are the bad guys?  Here's the scoop:<BR><i> cd ~svxii; du -skh * | grep -e "M"</i><pre>1.3M	11.ps
2.1M	ADC1.ps
2.5M	bin
3.6M	biterror-snapshot3.ps
1.6M	biterror-snapshot4.ps
1.6M	biterror-snapshot5.ps
3.0M	biterror-snapshot.ps
1.3M	bulletin_2.pdf
2.9M	CAENburn-in
2.2M	caentest
<FONT COLOR=RED>17M	cdf5425_Run2b_Silicon.ps
5.8M	CMS_chill_290103.pdf.gz
1.3M	CMS_FINALIII_13_01_031.gds
1.3M	CMS_FINALIII_13_01_03.gds
17M	core.13873
14M	core.24163</FONT>
1.1M	D0-note.ps
<FONT COLOR=RED>839M	daq</FONT>
2.8M	default_rd
9.8M	DOE.ppt
1.7M	evah
14M	gino
7.7M	gino.tar
2.0M	hou_vtx02_rev.ps
7.8M	hv
<FONT COLOR=RED>135M	imon</FONT>
2.3M	L00config
1.6M	la
1.6M	la.PS
1.7M	mbox
2.8M	new_rd
<FONT COLOR=RED>27M	peterun</FONT>
5.1M	proposalnonleaking11dec
<FONT COLOR=RED>58M	public_html</FONT>
2.5M	report-11-27-02.PDF
1.3M	rw
15M	sav
6.3M	sb3w6l2bias.ps
2.8M	sensors.prn
5.1M	silimon
2.1M	sintef2003.pdf
1.2M	snap
1.0M	snapshot2.gif
11M	svt
2.1M	svxmon_160301_b0dap41_27617.errlog
1.5M	svxmon_160894_b0dap41_7580.errlog
1.3M	svxmon_160895_b0dap41_7580.errlog
<FONT COLOR=RED>14M	svxmon_162857.root
15M	svxmon_177799.root
15M	svxmon_178064.root</FONT>
4.9M	tevscript
1.8M	tomsrtbt-2.0.103.tar.gz
</pre>
in the daq directory: <i>du -skh * | grep -e "M"</i>
<pre>
26M	all
18M	anywhere
2.8M	chipFiles
1.6M	diag
33M	fcc
324M	fcc4
65M	isl
1.2M	PSanywhere
368M	svx
</pre>
After <i>source cleanDaq</i> , and <i>rm  */core.*</i>
<pre>
2.1M	all
18M	anywhere
2.8M	chipFiles
1.6M	diag
4.3M	fcc
324M	fcc4
65M	isl
1.1M	PSanywhere
338M	svx
</pre>
After modified cleanDaq (including ~svxii/daq/isl/data, and ~svxii/daq/svx/data)
<pre>
2.1M	all
18M	anywhere
2.8M	chipFiles
1.6M	diag
4.3M	fcc
324M	fcc4
12M	isl
1.1M	PSanywhere
4.0k	setup_crateMon
9.8M	svx
</pre>
Now there is some breathing space <pre>
<svxii@b0dau30>quota -v
Disk quotas for svxii (uid 1111):
Filesystem    usage   quota   limit    timeleft  files   quota   limit    timeleft
/cdf/people2 898132 1400000 1500000              18015       0       0            
/cdf/people1   3968       0       0                 10       0       0            
/cdf/people3   5264       0       0                 55       0       0            
/data1       282524  500000 1300000              13925       0       0          
</pre>
Still, there are a lot of *.hst files, a lot of old .ps and .pdf files etc (what is a .gds file?) and all
the runiib crap in ~svxii/daq/fcc4. <H1>The problem with a group account is that noone ever cleans it 
up</h1> Next time I charge you the going rate for IT professionals.
<!-- Comment: Wed Jan 21 08;59;18 comment by...Rainer -->
that's not quite true - I recently run  cleanDaq, w/o your modifications, but the quota thing clearly sucks. But I agree ...
<h1> PLEASE CLEAN UP YOUR MESS!!! </h1>

<!-- Date: Wed Jan 21 05:34:06 2004 -->
<table><tr><td><a href=/elog/notebooks/silicon/Wed_Jan_21_2004_053356.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3334  align='top'></a></td><td>uncle spy.</td></tr></table>
<!-- Author: Rainer,ssf -->







<!-- Date: Wed Jan 21 14:06:36 2004 -->
<h2> are we dropping events left and right ? </h2> 
 
In run 178165 (one example out of many), SVXMon complains about: 
<pre> 
91002 1074592899 {The number of Silicon VRBs in the data stream has changed from 57 to 24} 
 90972 1074592899 {The number of Silicon VRBs in the data stream has changed from 24 to 57} 
 98369 1074592948 {The number of Silicon VRBs in the data stream has changed from 57 to 24} 
 98421 1074592951 {The number of Silicon VRBs in the data stream has changed from 24 to 57} 
 270857 1074594059 {The number of Silicon VRBs in the data stream has changed from 57 to 24} 
 271562 1074594060 {The number of Silicon VRBs in the data stream has changed from 24 to 57} 
 318297 1074594361 {The number of Silicon VRBs in the data stream has changed from 57 to 24} 
 318350 1074594364 {The number of Silicon VRBs in the data stream has changed from 24 to 57} 
 660339 1074596738 {The number of Silicon VRBs in the data stream has changed from 57 to 24} 
 660465 1074596738 {The number of Silicon VRBs in the data stream has changed from 24 to 57} 
 947096 1074598633 {The number of Silicon VRBs in the data stream has changed from 57 to 24} 
 947155 1074598636 {The number of Silicon VRBs in the data stream has changed from 24 to 57} 
 1204546 1074600355 {The number of Silicon VRBs in the data stream has changed from 57 to 24} 
 1204800 1074600355 {The number of Silicon VRBs in the data stream has changed from 24 to 57} 
 1415210 1074601790 {The number of Silicon VRBs in the data stream has changed from 57 to 24} 
 1415909 1074601790 {The number of Silicon VRBs in the data stream has changed from 24 to 57} 
 1654225 1074603473 {The number of Silicon VRBs in the data stream has changed from 57 to 24} 
 1654330 1074603476 {The number of Silicon VRBs in the data stream has changed from 24 to 57} 
 1699947 1074603793 {The number of Silicon VRBs in the data stream has changed from 57 to 24} 
 1699895 1074603797 {The number of Silicon VRBs in the data stream has changed from 24 to 57} 
 1777919 1074604344 {The number of Silicon VRBs in the data stream has changed from 57 to 24} 
 1778165 1074604344 {The number of Silicon VRBs in the data stream has changed from 24 to 57} 
 1791716 1074604443 {The number of Silicon VRBs in the data stream has changed from 57 to 24} 
 1791846 1074604443 {The number of Silicon VRBs in the data stream has changed from 24 to 57} 
 1837934 1074604767 {The number of Silicon VRBs in the data stream has changed from 57 to 24} 
 1838320 1074604767 {The number of Silicon VRBs in the data stream has changed from 24 to 57} 
 2043749 1074606221 {The number of Silicon VRBs in the data stream has changed from 57 to 24} 
 2043767 1074606225 {The number of Silicon VRBs in the data stream has changed from 24 to 57} 
 2045606 1074606237 {The number of Silicon VRBs in the data stream has changed from 57 to 24} 
 2045851 1074606237 {The number of Silicon VRBs in the data stream has changed from 24 to 57} 
</pre> 
This coincides with a warning of a fluctuation of the # of HDIs from 514 to 200 (314 missing).  The missing HDIs are in evt 1204546: 
<pre> 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W4/L4 1204546 1074600354 {HDI with fib id0 224 fib id1 68 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W5/L0 1204546 1074600354 {HDI with fib id0 224 fib id1 80 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W5/L1 1204546 1074600354 {HDI with fib id0 224 fib id1 81 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W5/L2 1204546 1074600354 {HDI with fib id0 224 fib id1 82 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W5/L3 1204546 1074600354 {HDI with fib id0 224 fib id1 83 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W5/L4 1204546 1074600354 {HDI with fib id0 224 fib id1 84 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W6/L1 1204546 1074600354 {HDI with fib id0 224 fib id1 97 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W6/L2 1204546 1074600354 {HDI with fib id0 224 fib id1 98 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W6/L3 1204546 1074600354 {HDI with fib id0 224 fib id1 99 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W6/L4 1204546 1074600354 {HDI with fib id0 224 fib id1 100 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W7/L0 1204546 1074600354 {HDI with fib id0 224 fib id1 112 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W7/L1 1204546 1074600354 {HDI with fib id0 224 fib id1 113 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W7/L2 1204546 1074600354 {HDI with fib id0 224 fib id1 114 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W7/L3 1204546 1074600354 {HDI with fib id0 224 fib id1 115 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W7/L4 1204546 1074600354 {HDI with fib id0 224 fib id1 116 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W8/L0 1204546 1074600354 {HDI with fib id0 224 fib id1 128 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W8/L1 1204546 1074600354 {HDI with fib id0 224 fib id1 129 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W8/L2 1204546 1074600354 {HDI with fib id0 224 fib id1 130 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W8/L3 1204546 1074600354 {HDI with fib id0 224 fib id1 131 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W8/L4 1204546 1074600354 {HDI with fib id0 224 fib id1 132 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W9/L0 1204546 1074600354 {HDI with fib id0 224 fib id1 144 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W9/L1 1204546 1074600354 {HDI with fib id0 224 fib id1 145 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W9/L2 1204546 1074600354 {HDI with fib id0 224 fib id1 146 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W9/L3 1204546 1074600354 {HDI with fib id0 224 fib id1 147 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W9/L4 1204546 1074600354 {HDI with fib id0 224 fib id1 148 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W10/L0 1204546 1074600354 {HDI with fib id0 224 fib id1 160 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W10/L1 1204546 1074600354 {HDI with fib id0 224 fib id1 161 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W10/L2 1204546 1074600354 {HDI with fib id0 224 fib id1 162 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W10/L3 1204546 1074600354 {HDI with fib id0 224 fib id1 163 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W10/L4 1204546 1074600354 {HDI with fib id0 224 fib id1 164 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W11/L0 1204546 1074600354 {HDI with fib id0 224 fib id1 176 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W11/L2 1204546 1074600354 {HDI with fib id0 224 fib id1 178 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W11/L3 1204546 1074600354 {HDI with fib id0 224 fib id1 179 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B0/W11/L4 1204546 1074600354 {HDI with fib id0 224 fib id1 180 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W0/L0 1204546 1074600354 {HDI with fib id0 225 fib id1 0 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W0/L1 1204546 1074600354 {HDI with fib id0 225 fib id1 1 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W0/L2 1204546 1074600354 {HDI with fib id0 225 fib id1 2 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W0/L3 1204546 1074600354 {HDI with fib id0 225 fib id1 3 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W0/L4 1204546 1074600354 {HDI with fib id0 225 fib id1 4 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W1/L0 1204546 1074600354 {HDI with fib id0 225 fib id1 16 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W1/L1 1204546 1074600354 {HDI with fib id0 225 fib id1 17 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W1/L2 1204546 1074600354 {HDI with fib id0 225 fib id1 18 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W1/L3 1204546 1074600354 {HDI with fib id0 225 fib id1 19 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W1/L4 1204546 1074600354 {HDI with fib id0 225 fib id1 20 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W2/L0 1204546 1074600354 {HDI with fib id0 225 fib id1 32 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W2/L1 1204546 1074600354 {HDI with fib id0 225 fib id1 33 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W2/L2 1204546 1074600354 {HDI with fib id0 225 fib id1 34 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W2/L3 1204546 1074600354 {HDI with fib id0 225 fib id1 35 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W2/L4 1204546 1074600354 {HDI with fib id0 225 fib id1 36 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W3/L0 1204546 1074600354 {HDI with fib id0 225 fib id1 48 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W3/L1 1204546 1074600354 {HDI with fib id0 225 fib id1 49 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W3/L2 1204546 1074600354 {HDI with fib id0 225 fib id1 50 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W3/L3 1204546 1074600354 {HDI with fib id0 225 fib id1 51 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W3/L4 1204546 1074600354 {HDI with fib id0 225 fib id1 52 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W4/L0 1204546 1074600354 {HDI with fib id0 225 fib id1 64 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W4/L1 1204546 1074600354 {HDI with fib id0 225 fib id1 65 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W4/L2 1204546 1074600354 {HDI with fib id0 225 fib id1 66 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W4/L3 1204546 1074600354 {HDI with fib id0 225 fib id1 67 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W4/L4 1204546 1074600354 {HDI with fib id0 225 fib id1 68 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W5/L0 1204546 1074600354 {HDI with fib id0 225 fib id1 80 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W5/L1 1204546 1074600354 {HDI with fib id0 225 fib id1 81 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W5/L2 1204546 1074600354 {HDI with fib id0 225 fib id1 82 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W5/L3 1204546 1074600354 {HDI with fib id0 225 fib id1 83 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W5/L4 1204546 1074600354 {HDI with fib id0 225 fib id1 84 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W6/L0 1204546 1074600354 {HDI with fib id0 225 fib id1 96 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W6/L1 1204546 1074600354 {HDI with fib id0 225 fib id1 97 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W6/L2 1204546 1074600354 {HDI with fib id0 225 fib id1 98 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W6/L3 1204546 1074600354 {HDI with fib id0 225 fib id1 99 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W8/L0 1204546 1074600354 {HDI with fib id0 225 fib id1 128 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W8/L1 1204546 1074600354 {HDI with fib id0 225 fib id1 129 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W8/L2 1204546 1074600354 {HDI with fib id0 225 fib id1 130 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W8/L3 1204546 1074600354 {HDI with fib id0 225 fib id1 131 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W8/L4 1204546 1074600354 {HDI with fib id0 225 fib id1 132 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W9/L0 1204546 1074600354 {HDI with fib id0 225 fib id1 144 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W9/L1 1204546 1074600354 {HDI with fib id0 225 fib id1 145 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W9/L2 1204546 1074600354 {HDI with fib id0 225 fib id1 146 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W9/L3 1204546 1074600354 {HDI with fib id0 225 fib id1 147 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W9/L4 1204546 1074600354 {HDI with fib id0 225 fib id1 148 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W10/L0 1204546 1074600354 {HDI with fib id0 225 fib id1 160 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W10/L1 1204546 1074600354 {HDI with fib id0 225 fib id1 161 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W10/L2 1204546 1074600354 {HDI with fib id0 225 fib id1 162 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W10/L3 1204546 1074600354 {HDI with fib id0 225 fib id1 163 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W10/L4 1204546 1074600354 {HDI with fib id0 225 fib id1 164 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W11/L0 1204546 1074600354 {HDI with fib id0 225 fib id1 176 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W11/L1 1204546 1074600354 {HDI with fib id0 225 fib id1 177 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W11/L2 1204546 1074600354 {HDI with fib id0 225 fib id1 178 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W11/L3 1204546 1074600354 {HDI with fib id0 225 fib id1 179 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B1/W11/L4 1204546 1074600354 {HDI with fib id0 225 fib id1 180 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W0/L0 1204546 1074600354 {HDI with fib id0 226 fib id1 0 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W0/L1 1204546 1074600354 {HDI with fib id0 226 fib id1 1 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W0/L2 1204546 1074600354 {HDI with fib id0 226 fib id1 2 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W0/L3 1204546 1074600354 {HDI with fib id0 226 fib id1 3 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W0/L4 1204546 1074600354 {HDI with fib id0 226 fib id1 4 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W1/L0 1204546 1074600354 {HDI with fib id0 226 fib id1 16 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W1/L1 1204546 1074600354 {HDI with fib id0 226 fib id1 17 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W1/L2 1204546 1074600354 {HDI with fib id0 226 fib id1 18 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W1/L3 1204546 1074600354 {HDI with fib id0 226 fib id1 19 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W1/L4 1204546 1074600354 {HDI with fib id0 226 fib id1 20 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W2/L0 1204546 1074600354 {HDI with fib id0 226 fib id1 32 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W2/L1 1204546 1074600354 {HDI with fib id0 226 fib id1 33 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W2/L2 1204546 1074600355 {HDI with fib id0 226 fib id1 34 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W2/L3 1204546 1074600355 {HDI with fib id0 226 fib id1 35 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W2/L4 1204546 1074600355 {HDI with fib id0 226 fib id1 36 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W3/L0 1204546 1074600355 {HDI with fib id0 226 fib id1 48 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W3/L1 1204546 1074600355 {HDI with fib id0 226 fib id1 49 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W3/L2 1204546 1074600355 {HDI with fib id0 226 fib id1 50 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W3/L3 1204546 1074600355 {HDI with fib id0 226 fib id1 51 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W3/L4 1204546 1074600355 {HDI with fib id0 226 fib id1 52 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W4/L0 1204546 1074600355 {HDI with fib id0 226 fib id1 64 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W4/L1 1204546 1074600355 {HDI with fib id0 226 fib id1 65 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W4/L2 1204546 1074600355 {HDI with fib id0 226 fib id1 66 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W4/L3 1204546 1074600355 {HDI with fib id0 226 fib id1 67 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W4/L4 1204546 1074600355 {HDI with fib id0 226 fib id1 68 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W5/L0 1204546 1074600355 {HDI with fib id0 226 fib id1 80 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W5/L1 1204546 1074600355 {HDI with fib id0 226 fib id1 81 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W5/L2 1204546 1074600355 {HDI with fib id0 226 fib id1 82 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W5/L3 1204546 1074600355 {HDI with fib id0 226 fib id1 83 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W6/L0 1204546 1074600355 {HDI with fib id0 226 fib id1 96 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W6/L1 1204546 1074600355 {HDI with fib id0 226 fib id1 97 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W6/L3 1204546 1074600355 {HDI with fib id0 226 fib id1 99 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W6/L4 1204546 1074600355 {HDI with fib id0 226 fib id1 100 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W7/L0 1204546 1074600355 {HDI with fib id0 226 fib id1 112 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W7/L1 1204546 1074600355 {HDI with fib id0 226 fib id1 113 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W7/L2 1204546 1074600355 {HDI with fib id0 226 fib id1 114 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W7/L3 1204546 1074600355 {HDI with fib id0 226 fib id1 115 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W7/L4 1204546 1074600355 {HDI with fib id0 226 fib id1 116 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W8/L0 1204546 1074600355 {HDI with fib id0 226 fib id1 128 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W8/L1 1204546 1074600355 {HDI with fib id0 226 fib id1 129 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W8/L2 1204546 1074600355 {HDI with fib id0 226 fib id1 130 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W8/L3 1204546 1074600355 {HDI with fib id0 226 fib id1 131 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W8/L4 1204546 1074600355 {HDI with fib id0 226 fib id1 132 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W9/L0 1204546 1074600355 {HDI with fib id0 226 fib id1 144 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W9/L1 1204546 1074600355 {HDI with fib id0 226 fib id1 145 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W9/L2 1204546 1074600355 {HDI with fib id0 226 fib id1 146 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W9/L3 1204546 1074600355 {HDI with fib id0 226 fib id1 147 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W9/L4 1204546 1074600355 {HDI with fib id0 226 fib id1 148 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W10/L0 1204546 1074600355 {HDI with fib id0 226 fib id1 160 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W10/L1 1204546 1074600355 {HDI with fib id0 226 fib id1 161 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W10/L2 1204546 1074600355 {HDI with fib id0 226 fib id1 162 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W10/L3 1204546 1074600355 {HDI with fib id0 226 fib id1 163 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W10/L4 1204546 1074600355 {HDI with fib id0 226 fib id1 164 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W11/L0 1204546 1074600355 {HDI with fib id0 226 fib id1 176 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W11/L1 1204546 1074600355 {HDI with fib id0 226 fib id1 177 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W11/L2 1204546 1074600355 {HDI with fib id0 226 fib id1 178 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W11/L3 1204546 1074600355 {HDI with fib id0 226 fib id1 179 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B2/W11/L4 1204546 1074600355 {HDI with fib id0 226 fib id1 180 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W0/L0 1204546 1074600355 {HDI with fib id0 227 fib id1 0 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W0/L1 1204546 1074600355 {HDI with fib id0 227 fib id1 1 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W0/L2 1204546 1074600355 {HDI with fib id0 227 fib id1 2 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W0/L3 1204546 1074600355 {HDI with fib id0 227 fib id1 3 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W0/L4 1204546 1074600355 {HDI with fib id0 227 fib id1 4 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W1/L0 1204546 1074600355 {HDI with fib id0 227 fib id1 16 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W1/L1 1204546 1074600355 {HDI with fib id0 227 fib id1 17 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W1/L2 1204546 1074600355 {HDI with fib id0 227 fib id1 18 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W1/L3 1204546 1074600355 {HDI with fib id0 227 fib id1 19 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W1/L4 1204546 1074600355 {HDI with fib id0 227 fib id1 20 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W2/L0 1204546 1074600355 {HDI with fib id0 227 fib id1 32 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W2/L1 1204546 1074600355 {HDI with fib id0 227 fib id1 33 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W2/L3 1204546 1074600355 {HDI with fib id0 227 fib id1 35 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W2/L4 1204546 1074600355 {HDI with fib id0 227 fib id1 36 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W3/L0 1204546 1074600355 {HDI with fib id0 227 fib id1 48 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W3/L1 1204546 1074600355 {HDI with fib id0 227 fib id1 49 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W3/L2 1204546 1074600355 {HDI with fib id0 227 fib id1 50 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W3/L3 1204546 1074600355 {HDI with fib id0 227 fib id1 51 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W3/L4 1204546 1074600355 {HDI with fib id0 227 fib id1 52 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W4/L0 1204546 1074600355 {HDI with fib id0 227 fib id1 64 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W4/L1 1204546 1074600355 {HDI with fib id0 227 fib id1 65 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W4/L2 1204546 1074600355 {HDI with fib id0 227 fib id1 66 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W4/L3 1204546 1074600355 {HDI with fib id0 227 fib id1 67 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W4/L4 1204546 1074600355 {HDI with fib id0 227 fib id1 68 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W5/L0 1204546 1074600355 {HDI with fib id0 227 fib id1 80 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W5/L1 1204546 1074600355 {HDI with fib id0 227 fib id1 81 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W5/L2 1204546 1074600355 {HDI with fib id0 227 fib id1 82 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W5/L3 1204546 1074600355 {HDI with fib id0 227 fib id1 83 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W5/L4 1204546 1074600355 {HDI with fib id0 227 fib id1 84 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W6/L0 1204546 1074600355 {HDI with fib id0 227 fib id1 96 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W6/L1 1204546 1074600355 {HDI with fib id0 227 fib id1 97 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W6/L2 1204546 1074600355 {HDI with fib id0 227 fib id1 98 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W6/L3 1204546 1074600355 {HDI with fib id0 227 fib id1 99 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W6/L4 1204546 1074600355 {HDI with fib id0 227 fib id1 100 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W7/L0 1204546 1074600355 {HDI with fib id0 227 fib id1 112 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W7/L1 1204546 1074600355 {HDI with fib id0 227 fib id1 113 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W7/L3 1204546 1074600355 {HDI with fib id0 227 fib id1 115 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W7/L4 1204546 1074600355 {HDI with fib id0 227 fib id1 116 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W8/L0 1204546 1074600355 {HDI with fib id0 227 fib id1 128 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W8/L1 1204546 1074600355 {HDI with fib id0 227 fib id1 129 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W8/L2 1204546 1074600355 {HDI with fib id0 227 fib id1 130 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W8/L3 1204546 1074600355 {HDI with fib id0 227 fib id1 131 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W8/L4 1204546 1074600355 {HDI with fib id0 227 fib id1 132 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W9/L0 1204546 1074600355 {HDI with fib id0 227 fib id1 144 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W9/L1 1204546 1074600355 {HDI with fib id0 227 fib id1 145 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W9/L2 1204546 1074600355 {HDI with fib id0 227 fib id1 146 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W9/L3 1204546 1074600355 {HDI with fib id0 227 fib id1 147 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W9/L4 1204546 1074600355 {HDI with fib id0 227 fib id1 148 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W10/L0 1204546 1074600355 {HDI with fib id0 227 fib id1 160 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W10/L1 1204546 1074600355 {HDI with fib id0 227 fib id1 161 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W10/L2 1204546 1074600355 {HDI with fib id0 227 fib id1 162 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W10/L4 1204546 1074600355 {HDI with fib id0 227 fib id1 164 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W11/L0 1204546 1074600355 {HDI with fib id0 227 fib id1 176 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W11/L1 1204546 1074600355 {HDI with fib id0 227 fib id1 177 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W11/L2 1204546 1074600355 {HDI with fib id0 227 fib id1 178 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W11/L3 1204546 1074600355 {HDI with fib id0 227 fib id1 179 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B3/W11/L4 1204546 1074600355 {HDI with fib id0 227 fib id1 180 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W0/L0 1204546 1074600355 {HDI with fib id0 228 fib id1 0 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W0/L1 1204546 1074600355 {HDI with fib id0 228 fib id1 1 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W0/L2 1204546 1074600355 {HDI with fib id0 228 fib id1 2 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W0/L3 1204546 1074600355 {HDI with fib id0 228 fib id1 3 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W1/L0 1204546 1074600355 {HDI with fib id0 228 fib id1 16 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W1/L1 1204546 1074600355 {HDI with fib id0 228 fib id1 17 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W1/L2 1204546 1074600355 {HDI with fib id0 228 fib id1 18 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W1/L3 1204546 1074600355 {HDI with fib id0 228 fib id1 19 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W1/L4 1204546 1074600355 {HDI with fib id0 228 fib id1 20 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W2/L0 1204546 1074600355 {HDI with fib id0 228 fib id1 32 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W2/L1 1204546 1074600355 {HDI with fib id0 228 fib id1 33 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W2/L2 1204546 1074600355 {HDI with fib id0 228 fib id1 34 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W2/L3 1204546 1074600355 {HDI with fib id0 228 fib id1 35 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W2/L4 1204546 1074600355 {HDI with fib id0 228 fib id1 36 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W3/L0 1204546 1074600355 {HDI with fib id0 228 fib id1 48 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W3/L1 1204546 1074600355 {HDI with fib id0 228 fib id1 49 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W3/L2 1204546 1074600355 {HDI with fib id0 228 fib id1 50 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W3/L3 1204546 1074600355 {HDI with fib id0 228 fib id1 51 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W3/L4 1204546 1074600355 {HDI with fib id0 228 fib id1 52 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W4/L0 1204546 1074600355 {HDI with fib id0 228 fib id1 64 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W4/L1 1204546 1074600355 {HDI with fib id0 228 fib id1 65 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W4/L2 1204546 1074600355 {HDI with fib id0 228 fib id1 66 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W4/L3 1204546 1074600355 {HDI with fib id0 228 fib id1 67 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W4/L4 1204546 1074600355 {HDI with fib id0 228 fib id1 68 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W5/L0 1204546 1074600355 {HDI with fib id0 228 fib id1 80 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W5/L1 1204546 1074600355 {HDI with fib id0 228 fib id1 81 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W5/L2 1204546 1074600355 {HDI with fib id0 228 fib id1 82 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W5/L3 1204546 1074600355 {HDI with fib id0 228 fib id1 83 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W5/L4 1204546 1074600355 {HDI with fib id0 228 fib id1 84 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W6/L0 1204546 1074600355 {HDI with fib id0 228 fib id1 96 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W6/L1 1204546 1074600355 {HDI with fib id0 228 fib id1 97 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W6/L3 1204546 1074600355 {HDI with fib id0 228 fib id1 99 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W6/L4 1204546 1074600355 {HDI with fib id0 228 fib id1 100 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W7/L0 1204546 1074600355 {HDI with fib id0 228 fib id1 112 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W7/L1 1204546 1074600355 {HDI with fib id0 228 fib id1 113 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W7/L2 1204546 1074600355 {HDI with fib id0 228 fib id1 114 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W7/L3 1204546 1074600355 {HDI with fib id0 228 fib id1 115 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W7/L4 1204546 1074600355 {HDI with fib id0 228 fib id1 116 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W8/L0 1204546 1074600355 {HDI with fib id0 228 fib id1 128 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W8/L3 1204546 1074600355 {HDI with fib id0 228 fib id1 131 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W8/L4 1204546 1074600355 {HDI with fib id0 228 fib id1 132 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W9/L0 1204546 1074600355 {HDI with fib id0 228 fib id1 144 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W9/L1 1204546 1074600355 {HDI with fib id0 228 fib id1 145 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W9/L2 1204546 1074600355 {HDI with fib id0 228 fib id1 146 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W9/L3 1204546 1074600355 {HDI with fib id0 228 fib id1 147 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W9/L4 1204546 1074600355 {HDI with fib id0 228 fib id1 148 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W10/L0 1204546 1074600355 {HDI with fib id0 228 fib id1 160 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W10/L1 1204546 1074600355 {HDI with fib id0 228 fib id1 161 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W10/L2 1204546 1074600355 {HDI with fib id0 228 fib id1 162 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W10/L3 1204546 1074600355 {HDI with fib id0 228 fib id1 163 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W10/L4 1204546 1074600355 {HDI with fib id0 228 fib id1 164 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W11/L0 1204546 1074600355 {HDI with fib id0 228 fib id1 176 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W11/L1 1204546 1074600355 {HDI with fib id0 228 fib id1 177 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W11/L2 1204546 1074600355 {HDI with fib id0 228 fib id1 178 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W11/L3 1204546 1074600355 {HDI with fib id0 228 fib id1 179 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B4/W11/L4 1204546 1074600355 {HDI with fib id0 228 fib id1 180 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W0/L0 1204546 1074600355 {HDI with fib id0 229 fib id1 0 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W0/L1 1204546 1074600355 {HDI with fib id0 229 fib id1 1 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W0/L2 1204546 1074600355 {HDI with fib id0 229 fib id1 2 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W0/L3 1204546 1074600355 {HDI with fib id0 229 fib id1 3 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W0/L4 1204546 1074600355 {HDI with fib id0 229 fib id1 4 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W1/L0 1204546 1074600355 {HDI with fib id0 229 fib id1 16 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W1/L1 1204546 1074600355 {HDI with fib id0 229 fib id1 17 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W1/L2 1204546 1074600355 {HDI with fib id0 229 fib id1 18 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W1/L3 1204546 1074600355 {HDI with fib id0 229 fib id1 19 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W1/L4 1204546 1074600355 {HDI with fib id0 229 fib id1 20 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W2/L0 1204546 1074600355 {HDI with fib id0 229 fib id1 32 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W2/L1 1204546 1074600355 {HDI with fib id0 229 fib id1 33 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W2/L2 1204546 1074600355 {HDI with fib id0 229 fib id1 34 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W2/L3 1204546 1074600355 {HDI with fib id0 229 fib id1 35 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W2/L4 1204546 1074600355 {HDI with fib id0 229 fib id1 36 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W3/L0 1204546 1074600355 {HDI with fib id0 229 fib id1 48 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W3/L1 1204546 1074600355 {HDI with fib id0 229 fib id1 49 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W3/L2 1204546 1074600355 {HDI with fib id0 229 fib id1 50 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W3/L3 1204546 1074600355 {HDI with fib id0 229 fib id1 51 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W3/L4 1204546 1074600355 {HDI with fib id0 229 fib id1 52 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W5/L0 1204546 1074600355 {HDI with fib id0 229 fib id1 80 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W5/L1 1204546 1074600355 {HDI with fib id0 229 fib id1 81 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W5/L2 1204546 1074600355 {HDI with fib id0 229 fib id1 82 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W5/L3 1204546 1074600355 {HDI with fib id0 229 fib id1 83 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W5/L4 1204546 1074600355 {HDI with fib id0 229 fib id1 84 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W6/L0 1204546 1074600355 {HDI with fib id0 229 fib id1 96 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W6/L1 1204546 1074600355 {HDI with fib id0 229 fib id1 97 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W6/L2 1204546 1074600355 {HDI with fib id0 229 fib id1 98 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W6/L3 1204546 1074600355 {HDI with fib id0 229 fib id1 99 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W6/L4 1204546 1074600355 {HDI with fib id0 229 fib id1 100 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W7/L0 1204546 1074600355 {HDI with fib id0 229 fib id1 112 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W7/L1 1204546 1074600355 {HDI with fib id0 229 fib id1 113 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W7/L2 1204546 1074600355 {HDI with fib id0 229 fib id1 114 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W7/L3 1204546 1074600355 {HDI with fib id0 229 fib id1 115 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W7/L4 1204546 1074600355 {HDI with fib id0 229 fib id1 116 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W8/L0 1204546 1074600355 {HDI with fib id0 229 fib id1 128 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W8/L1 1204546 1074600355 {HDI with fib id0 229 fib id1 129 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W8/L2 1204546 1074600355 {HDI with fib id0 229 fib id1 130 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W8/L3 1204546 1074600355 {HDI with fib id0 229 fib id1 131 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W8/L4 1204546 1074600355 {HDI with fib id0 229 fib id1 132 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W9/L0 1204546 1074600355 {HDI with fib id0 229 fib id1 144 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W9/L1 1204546 1074600355 {HDI with fib id0 229 fib id1 145 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W9/L2 1204546 1074600355 {HDI with fib id0 229 fib id1 146 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W9/L3 1204546 1074600355 {HDI with fib id0 229 fib id1 147 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W9/L4 1204546 1074600355 {HDI with fib id0 229 fib id1 148 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W10/L0 1204546 1074600355 {HDI with fib id0 229 fib id1 160 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W10/L1 1204546 1074600355 {HDI with fib id0 229 fib id1 161 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W10/L2 1204546 1074600355 {HDI with fib id0 229 fib id1 162 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W10/L3 1204546 1074600355 {HDI with fib id0 229 fib id1 163 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W10/L4 1204546 1074600355 {HDI with fib id0 229 fib id1 164 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W11/L0 1204546 1074600355 {HDI with fib id0 229 fib id1 176 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W11/L1 1204546 1074600355 {HDI with fib id0 229 fib id1 177 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W11/L2 1204546 1074600355 {HDI with fib id0 229 fib id1 178 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W11/L3 1204546 1074600355 {HDI with fib id0 229 fib id1 179 not found in the data stream for trigger 1204546} 
errlog 13 {Missing HDI} SIXDMon Silicon/S/B5/W11/L4 1204546 1074600355 {HDI with fib id0 229 fib id1 180 not found in the data stream for trigger 1204546} 
 
errlog 13 {Wrong HDI Count} SIXDMon Silicon/DAQ 1204546 1074600355 {The number of HDIs in the data stream has changed from 514 to 200} 
</pre> 
Steve Nahn checked that all these HDIs are in fact in the raw data, see <br> /cdf/data05/s1/hardware/svxii/eventDump/r178165_e1204546.raw <br> 
and a similar event <br> 
/cdf/data05/s1/hardware/svxii/eventDump/r178165_e1415210.raw <br> 
 
<h3> So it appears this is an SVXMon artefact </h3>  <p> 
Some comments: <b> 
<li> how closely do pager carriers pay attention to the SVXMon error logger ? 
<li> If this was real, how would it show up for the shift crew (I haven't seen SVXMon go red all over the place) ? Presumably the reformatter would be chocking etc. 
<li> A data structure integrity check is very much needed - and SVXMon should be fixed ASAP to keep online SPL blood pressure at semi-healthy level .... 
  
</b>  
<!-- Author: SCN, Rainer -->


<!-- Date: Wed Jan 21 14:07:38 2004 -->
<table><tr><td><a href=/elog/notebooks/silicon/Wed_Jan_21_2004_140732.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3336  align='top'></a></td><td>Intermittent errors in SVXMon for e463 and e261.</td></tr></table>
<!-- Author: Rainer -->
<!-- Comment: Wed Jan 21 20;10;15 comment by...ssf -->
<pre>Error logger for these two ladders shows ONLY the "missing ?HDI" message noted above.</pre>
<!-- Date: Wed Jan 21 14:21:05 2004 -->
<b> intermittent coherent errors in SB0W9 </b> 
see  
main  
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=view&page=62&frame=2&anchor=09:19:27.151'> 
elog. </a>  Happened in run <ahref='http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178256'> 178256 </a>, not observed since. 
  
<!-- Author: Rainer -->


<!-- Date: Wed Jan 21 18:22:10 2004 -->
Working on the <b> E string </b> 
</p> 
We figured out that the FFO goes into a mode where the READOUT signal is always HIGH. 
</p> 
 
When the FFO is in this mode the poor GB counts all the time and so the E string is pulled every 1.5 usec. 
</p> 
 
Conclusion of the work off today: 
</p> 
<b> this is a failure in the FFO</b>. We do not know if this failure is in all FFOs or only in the one connected to the GB so: 
<ul> 
<li> the FFO giving troubles has been disconnected from the GB and it is now connected to a scope for monitoring only. 
<li> another FFO has been connected to the GB and it is also monitored in another channel of the scope. 
<li> Stefano Rapisarda (the engineer responsible for the FFO) is looking into the schematic to figure out what could be happening inside the board to develop some understanding and how to make it more robust. 
</ul> 
 
So there are two options for the future: 
<ul> 
 <li> <b>option A: only that board has a problem</b>  
  <ul> 
    <li> in this case there will not be any more trouble taking data with CDF. 
    <li> the board will be pulled out of the CDF at our convenience (during an access) and debugged on the bench by Stefano. 
  </ul> 
 <li> <b>option B: every FFO board has the problem </b>  
  <ul> 
    <li> in this case there will still be troubles taking data with CDF. 
    <li> a fix will be developed by Stefano or as an alternative we will work on the logic of the GB FPGA 
  </ul> 
  </ul> 
We will understand which of the two options applies with the next time the problem shows up independently if it is <b>A</b> or <b>B</b> 
 
  
<!-- Author: Steve, Marcel and Gino -->
<!-- Comment: Wed Jan 21 18;26;18 comment by...Gino -->
<pre>On my way out of CDF I went to bodap42-44 to check the status 
of things and I found an svxdaq running.
I am paranoied because svxdaq allows to really interfere with 
data taking so I called the Silicon Pager Carrier that is now 
taking care of it.

Hopefully a comment on it will follow.

</pre>


<!-- Comment: Wed Jan 21 18;35;38 comment by...Gino -->
Also while I was there the monitoring ACE asked me about 
the status of 3 PINKIES in IMON. I did not know what to say 
so I told him to ask people that are more involved with the 
daily silicon operations.
</p>
By looking at the <a href= 
'http://www-cdfonline.fnal.gov/internal/ops/svxii/runii/imon-for-aces.html'> instructions </a> he should page Silicon.

</p> Maybe we should reinforce the instructions or change them if not valid anymore.




<!-- Comment: Wed Jan 21 19;09;08 comment by...ssf -->
<pre>Called in by Gino to investigate why svxdaq was running  apparently with no one around and no work going on.  Looked over the logs, nothing.  From the logs Gino was involved in the E-string work so I don't believe it was that.  Upon trying to shut down svxdaq, I was asked if I wanted to, "Clean up the DAQ, Yeah or Nah?"  Since I did not know the results of either during data taking I called Rainer, who suggested Nah.
My question was who was running svxdaq and why was there no documentation in the e-log?  Was it left on by mistake or did I just ruin an afternoon of someone's efforts?

I'm also curious as to the result of choosing "Yeah" as was my inital inclination.</pre>
<!-- Comment: Wed Jan 21 19;36;58 comment by...rainer -->
<li>ad gino's comments - <h2> on pinkies </h2>  have been a recent plague as we try to find the stable operating points for ladders after the shutdown. isl ladders had isl bits changed, one ladder got AVDD2, but apparently chips came back, some other ladders show +/-5mA current changes etc. in one case, an ISL ladder had an intermittent high occupancy state which was picked up/correlated to current increase/decrease of 20mA. <p>
<h2> all of these changes in current  need to be understood before the imon limits are simply bumped up or down because bumping the limits means knocking out one line of defense! </h2>  on the other hand persistent pinkies will lower the guards of the aces, who are new and get used to anything. I leave this up to the call of the detector working group leader to refine this, but my line of reasoning would that pinkies need to be understood ASAP and limits changed AFTER reason is understood. if understanding cannot be reached on the timescale of a few days, then the pinky limit should be adjusted marking this ladder 'needs work' in PDB and it should not be forgotten. my understanding is that we once have reached the stable operation point, we probably will be able to address the occasional pinky on a timescale shorter than that. 
<li> ad stan's comment - <h2> on b0dap42,43,44 hygiene </h2>  b0dap44 and b0dap43 should either run IMON or PSGUi - not both on  one machine. the machine with PSGUI should furthermore run a minicom window to svxinter permanently - this has proven to be useful. b0dap42 is left free for netscape/elog session and svxdaq sessions - <h2> but these should be closed after use </h2>. In case of stan's comment, it was <b> me </b> who goofed and left the svxdaq open. Apologies. As to the question about Yeah, it would have initialized the fibs etc. with very likely bad conseuqences for the ongoing run.
<!-- Comment: Sat Jan 24 16;51;02 comment by...Gino -->
<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=181&frame=2&anchor=16:46:39'>
more informations have </a> been acquired today for this topic. 

</p>
The conclusion is that 2 FFO out of 2 show the problem but not at the same time.

</p>

So the status is that reality is a combination of option <b>A</b> and <b>B</b>.




<!-- Date: Wed Jan 21 18:31:41 2004 -->
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=view&page=63&frame=2&anchor=18:24:26.188'> 
svxinter glitch</a>   
<!-- Author: Andy H. -->




<!-- Date: Wed Jan 21 19:35:08 2004 -->
<pre>Changed IMON limits on SB0W8L4 again, no other pinkies noted upon arrival.  Note:  PDB now requests tracking data and would not accept entry.  Will attempt to re enter data again in the morning.  (IMON 1010 changed to 1020).</pre>
<!-- Author: ssf -->
<!-- Comment: Wed Jan 21 19;52;32 comment by...ssf -->
<a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B0W8&layer=L4"> See PDB entry </a> for info on work done on this ladder.  ISEL bits were changed so current changes were expected.  <B>  However this ladder is not forgotten!</B> In fact I've been working with this ladder since monday. I've spoken with most of the ACEs multiple times and yes they know what to do.



<!-- Date: Wed Jan 21 21:45:35 2004 -->
<table><tr><td><a href=/elog/notebooks/silicon/Wed_Jan_21_2004_214514.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3338  align='top'></a></td><td><h3>More on the Stuck SRC Syndrome</h3> (which really ought to be called the Stuck FFO syndrome...<BR> 
Just as I was about to start a different test (see Marcel's entry) the Estring fired, and 
HRR did not recover.  It showed the exact symptoms of <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=179&frame=2&anchor=21:08:15'> 
Stuck SRC syndrome</a>.  Looking at the LA (left) I saw 
the GB pulling error every 1.5 [[m]]s because the  
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=181&frame=2&anchor=18:22:10'> 
Fib Fanout was stuck with READOUT high</a>.   It would also be nice if the 
GB looks for an edge to count READOUTs, rather than a level, but 
the FFO getting the signal stuck high is really the cause. <P> 
In addition,<ul>  
<li> we explored many options in attempting to get the FFO unstuck by VME, or sending 
various FFO commands via the GLINK.  We could force the line low, but not make it stay low 
(i.e. when we released the forcing, it came up again).  The only way 
to clear the error was to send a READOUT command to the FFO, but of course this 
cannot be done if the SRC is in ERROR, so we were deadlocked.  This also explains why 
the recovery mechanism works-by disabling the "Fatal" nature of the Estring, you 
can send a READOUT command, which unsticks the FFO. 
<li> We checked another FFO (from b0fib00, not b0fib02) to see if it too was stuck high.  It was not. 
Stefano said even though the "Readout Return" output was not enabled, he would expect it 
to behave the same.  This gives us some indication that it is not the firmware but 
rather just a specific board failure (<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=181&frame=2&anchor=18:22:10'> 
Gino's option A</a>).  We left the Estring with this (new) FFO as its input, hoping 
it will not cause the same stuck syndrome 
</ul> 
Stefano is looking to see if he can make the FFO either not do this, or provide a way 
to clear it by VME, or hopefully both.  In the meantime, if 
it is experienced again, the recover procedure should work.</td></tr></table>
<!-- Author: SCN -->


<!-- Date: Wed Jan 21 21:55:21 2004 -->
<pre>Steve Nahn modified the SRC software, so that it reports  
all Level 1 Accepts seen by the SRC. After successfully testing it, it was installed this evening (see  <a href= "http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=copyentry&page=63&time=20:09:57.110"> here </a>)  
 
With some perl scripts and the java tool si_ss.monChecker 
we are now recording the number of Level 1 Accepts seen by the 
 SRC and by the Scalers. The perl scripts srcparser.pl and 
scalerparser.pl are in <em> /data1/svxData/l1Alog </em>.  
On every halt, the scripts write out the number of Level1 
Accepts seen by SRC and Scalers into two files  
(<em>srcL1A.out</em> and <em>scalerL1A.out</em>) in the same directory. 
<br> 
The processes are running at the moment on <em>b0dap43 </em> 
To start the tool chain after a crash one should proceed as follows: 
<ol> 
<li> setup fer </li> 
<li> setup -j svxii -M ups -m svxii.table -r ~nahn/cvs/svxii </li> 
<li> java si_ss.monChecker "/b0tsi03/status/scalData" | scalerparser.pl & 
</li> 
<li>java si_ss.monChecker "/partition-*/status/svx/vrb/02" | srcparser.pl & 
</li> 
</ol> 
 
When comparing the numbers, keep in mind, that the SRC counter 
 is reset after each halt, so you have to do some math to compare with the scaler numbers 
A typical output from <em>scalerL1A.out</em> looks like 
<pre> 
Date                           RunNumber  Scaler6-19 
Wed Jan 21 21:16:57 CST 2004      178262    31088297 
Wed Jan 21 21:17:37 CST 2004      178262    31391551 
Wed Jan 21 21:17:59 CST 2004      178262    31505543 
</pre> 
and from <em>srcL1A.out</em> 
<pre> 
Date                            L1A Seen 
Wed Jan 21 21:16:57 CST 2004    13427894 
Wed Jan 21 21:17:37 CST 2004      303254 
Wed Jan 21 21:17:59 CST 2004      113992 
</pre> 
We let the things running over weekend to gather some 
 statistics on the Level 1 Accepts 
</pre>
<!-- Author: Steve, Marcel -->
<!-- Comment: Thu Jan 22 23;20;19 comment by...Rainer -->
<pre>looking at the output files in /cdf/data1/svxData/l1Alog, I see <pre>
wallny@ncdf90 l1Alog]$ tail -5 scalerL1A.out srcL1A.out
==> scalerL1A.out <==
Thu Jan 22 08:42:24 CST 2004      178298     4375230
Thu Jan 22 08:42:36 CST 2004      178298     4375230
Thu Jan 22 08:52:41 CST 2004      178298    10219931
Thu Jan 22 08:53:05 CST 2004      178298    10353269
Thu Jan 22 09:05:26 CST 2004      178298    17033343

==> srcL1A.out <==
Thu Jan 22 08:52:41 CST 2004     5844701
Thu Jan 22 08:53:05 CST 2004      133338
Thu Jan 22 09:05:26 CST 2004     6680074
Thu Jan 22 09:38:57 CST 2004           0
Thu Jan 22 09:43:22 CST 2004           0
</pre>
and no java processes on b0dap43

i.e. the jobs appeared to have stopped. did not appreciate RT server disasters ? current file moved to
srcL1A.out_012204-0905
scalerL1A.out_012204-0905
and jobs restarted.first test shows all L1s seen so far.




</pre>
<!-- Comment: Mon Jan 26 10;31;48 comment by...Marcel -->
<pre>Thanks for restarting
I guess I have to check for RT server disasters somehow

Marcel</pre>
<!-- Date: Wed Jan 21 23:35:59 2004 -->
<p><font size=+1> D-mode Calibration Job Started </font>
<p><pre>
Job id            : 309
Calibration type  : standard
Run period tag    : Jan_2004
DPS-off run       : 178276
DPS-on run        : 178273
Requested by      : Jeff & Anadi
Main process id   : 30905531
Comment           : solenoid ON, HV ON, with beam
</pre>
<!-- Author: CDF silicon D-mode calibration job server v1.4 -->



<!-- Date: Wed Jan 21 23:42:25 2004 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3339&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3339' width='300'></a></center>
<!-- Author: Rainer -->
<!-- Comment: Wed Jan 21 23;45;04 comment by...Rainer -->
Wacky trip of several wedges on software protection trip while config for a dmod calibration - b0svx00 turned red as well but unable to get error messages reported. What happened
<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=view&page=63&frame=2&anchor=23:14:10.172'>
here ? </a>

<!-- Comment: Wed Jan 21 23;46;52 comment by...Rainer -->
<pre>Wed Jan 21 22:48:59 CST 2004     Software protection trip: High DVDD Current  SV
X B0 W0 L1 current 154.0 softLimit 150.0
Wed Jan 21 22:49:01 CST 2004     Software protection trip: High DVDD Current  SV
X B0 W8 L1 current 161.0 softLimit 150.0
Wed Jan 21 22:49:02 CST 2004     Software protection trip: High DVDD Current  SV
X B5 W0 L1 current 156.0 softLimit 150.0
Wed Jan 21 22:49:13 CST 2004     Software protection trip: High DVDD Current  SV
X B2 W4 L1 current 155.0 softLimit 150.0
Wed Jan 21 22:49:15 CST 2004     Software protection trip: High DVDD Current  SV
X B5 W0 L1 current 156.0 softLimit 150.0
Wed Jan 21 22:49:15 CST 2004     TRIP: SVX  B0  W0 DVDD0 OverCurrent
Wed Jan 21 22:49:15 CST 2004     TRIP: SVX  B0  W0 DVDD1 OverCurrent
Wed Jan 21 22:49:16 CST 2004     TRIP: SVX  B0  W0 DVDD3 OverCurrent
Wed Jan 21 22:49:16 CST 2004     TRIP: SVX  B0  W8 DVDD0 OverCurrent
Wed Jan 21 22:49:16 CST 2004     TRIP: SVX  B0  W8 DVDD1 OverCurrent
Wed Jan 21 22:49:16 CST 2004     TRIP: SVX  B0  W8 DVDD2 OverCurrent
Wed Jan 21 22:49:16 CST 2004     TRIP: SVX  B0  W8 DVDD3 OverCurrent
Wed Jan 21 22:49:16 CST 2004     TRIP: SVX  B5  W0 DVDD0 OverCurrent
Wed Jan 21 22:49:16 CST 2004     TRIP: SVX  B5  W0 DVDD1 OverCurrent
Wed Jan 21 22:49:16 CST 2004     TRIP: SVX  B5  W0 DVDD2 OverCurrent
Wed Jan 21 22:49:26 CST 2004     Software protection trip: High DVDD Current  SV
X B2 W4 L1 current 222.0 softLimit 150.0
Wed Jan 21 22:49:27 CST 2004     TRIP: SVX  B2  W4 DVDD0 OverCurrent
Wed Jan 21 22:49:27 CST 2004     TRIP: SVX  B2  W4 DVDD1 OverCurrent
Wed Jan 21 22:49:27 CST 2004     TRIP: SVX  B2  W4 DVDD3 OverCurrent
Wed Jan 21 22:51:51 CST 2004     success: Resetting trips
</pre>

<!-- Date: Thu Jan 22 03:42:05 2004 -->
<p><font size=+1> D-mode Calibration Job Finished </font>
<p><pre>
This is a computer-generated silicon D-mode calibration status report.
Silicon D-mode calibration job 309 FAILED.

Job info
--------
Calibration type : standard
DPS-off run      : 178276
DPS-on run       : 178273
Requested by     : Jeff & Anadi
Submitted        : 2004-01-21 21:35:38
Completed        : 2004-01-22 01:41:37
Status           : FAILURE
Comment          : solenoid ON, HV ON, with beam

Last processing stage reported : calculating DPS-on pedestals
Failure diagnostics :
child process exited abnormally
</pre>
<!-- Author: D-mode script  -->



<!-- Date: Thu Jan 22 08:53:20 2004 -->
<pre>Checked the scope in the counting room (e-string FFO investigation) and no trigger found.</pre>
<!-- Author: Gino -->
<!-- Comment: Thu Jan 22 15;07;47 comment by...Gino -->
<pre>again</pre>
<!-- Comment: Fri Jan 23 08;51;09 comment by...Gino -->
<pre>again</pre>
<!-- Date: Thu Jan 22 09:34:10 2004 -->
<pre>Incident Abort gap losses shot up to over 250k Hz, (sudden jump not a drift) (will past plot later) tevmom alarmed.  The shift crew paused the run and took silicon to stby.  Losses are coming down but still in the 9k Hz range and I have requested that silicon remain out until MCR calls back and says they are through tuning or have other wise addressed the incident. 
 
Somebody fry a rat?</pre>
<!-- Author: Rong, ssf -->
<!-- Comment: Thu Jan 22 09;51;51 comment by...ssf, Rong -->

<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=view&page=65&frame=2&anchor=09:46:27.207'>
RF Glitch plot posted by R.J. Tesarek </a>  and 
<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=view&page=65&frame=2&anchor=09:47:36.167'>
Abort gap losses for incident </a>.  Tevmon is green, Ops manager want to wait until confirmation from MCR that this was an issolated glitch before including silicon.  Sounds good to me. 



<!-- Date: Thu Jan 22 10:10:30 2004 -->
<pre>We were asked by Rainer to check the Bunch X error in run 177384. 
Most of them are happened at SB1W0 and SB1W1 but also some other ladders occasionally. The reason seems to be a bit error during the communication between SRC to FIB? (not sure yet, but should be among DAQ modules) They are either have bit6(64) or bit7(128) low (expected to be high). 
 
Errors are something like :  
errlog 7 {Bunch X Out of Sync} SIXDMon Silicon/I/B1/W0/L0 405 1073357100 {Bunch crossing is 7, expected 71} 
errlog 7 {Bunch X Out of Sync} SIXDMon Silicon/I/B1/W0/L1 405 1073357100 {Bunch crossing is 7, expected 71} 
errlog 7 {Bunch X Out of Sync} SIXDMon Silicon/I/B1/W0/L2 405 1073357100 {Bunch crossing is 7, expected 71} 
errlog 7 {Bunch X Out of Sync} SIXDMon Silicon/I/B1/W0/L3 405 1073357100 {Bunch crossing is 7, expected 71} 
errlog 7 {Bunch X Out of Sync} SIXDMon Silicon/I/B1/W0/L4 405 1073357100 {Bunch crossing is 7, expected 71} 
errlog 7 {Bunch X Out of Sync} SIXDMon Silicon/I/B1/W1/L0 405 1073357100 {Bunch crossing is 7, expected 71} 
errlog 7 {Bunch X Out of Sync} SIXDMon Silicon/I/B1/W1/L2 405 1073357100 {Bunch crossing is 7, expected 71} 
errlog 7 {Bunch X Out of Sync} SIXDMon Silicon/I/B1/W1/L3 405 1073357100 {Bunch crossing is 7, expected 71} 
errlog 7 {Bunch X Out of Sync} SIXDMon Silicon/I/B1/W0/L0 25625 1073357115 {Bunch crossing is 5, expected 133} 
errlog 7 {Bunch X Out of Sync} SIXDMon Silicon/I/B1/W0/L1 25625 1073357115 {Bunch crossing is 5, expected 133} 
errlog 7 {Bunch X Out of Sync} SIXDMon Silicon/I/B1/W0/L2 25625 1073357115 {Bunch crossing is 5, expected 133} 
errlog 7 {Bunch X Out of Sync} SIXDMon Silicon/I/B1/W0/L3 25625 1073357115 {Bunch crossing is 5, expected 133} 
errlog 7 {Bunch X Out of Sync} SIXDMon Silicon/I/B1/W0/L4 25625 1073357115 {Bunch crossing is 5, expected 133} 
errlog 7 {Bunch X Out of Sync} SIXDMon Silicon/I/B1/W1/L0 25625 1073357115 {Bunch crossing is 5, expected 133} 
errlog 7 {Bunch X Out of Sync} SIXDMon Silicon/I/B1/W1/L2 25625 1073357115 {Bunch crossing is 5, expected 133} 
errlog 7 {Bunch X Out of Sync} SIXDMon Silicon/I/B1/W1/L3 25625 1073357115 {Bunch crossing is 5, expected 133} 
</pre>
<!-- Author: Rong and Stan -->


<!-- Date: Thu Jan 22 10:40:31 2004 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3340&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3340  width='300' align='top'></a></td><td>SB0W8L4</td></tr></table>
<!-- Author: ssf -->
<!-- Comment: Thu Jan 22 10;59;35 comment by...ssf -->
<pre>Suspected AVDD2 for this ladder on the 19th, recovered remining chips on 20th by setting ISEL bits on offending chip to zero, hench the higher currents starting on that date.  Even when stable, the current varies within a band about some nominal value +/-5 mA.  The allowed range on this ladder is +/- 4mA.  </pre>




<!-- Date: Thu Jan 22 11:10:59 2004 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3342&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3342  width='300' align='top'></a></td><td>SB0W8L4 AVDD</td></tr></table>
<!-- Author: ssf -->
<!-- Comment: Thu Jan 22 11;14;18 comment by...ssf -->
<pre>Close up of current variation whether nominal is at 620, 1010 or 1020 is greater than the +/- 4 uA range we currently are using.</pre>


<!-- Date: Thu Jan 22 11:27:07 2004 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3344&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3344  width='300' align='top'></a></td><td>latest run</td></tr></table>
<!-- Author: ssf -->

<!-- Date: Thu Jan 22 11:43:45 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: IB1W0/1All 
<li>problem n: 1039 
<li>Status: needs_work
<li>Type: Other
<li>Action: Diagnosis_update
<li>entered by: sforrest 
<li>description is: (Rong's investigation) SB1W0/1 giving bunch crossing errors,  either have bit6(64) or bit7(128) low (expected to be high).  Rong believes this could be a  communications error between the  SRC and the FIB.  
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=ISL&wedge=B1W0/1&layer=All">report</a></ul>
<!-- Author: Problem DB -->




<!-- Date: Thu Jan 22 19:19:00 2004 -->
<pre>SB2W2L4 tripped on DVDD software protection trip.  Hand shake problem with wedge. Rainer powered each ladder individually.  Layer 4 continued to trip, removed from hwdb and resummed the run.</pre>
<!-- Author: ssf -->
<!-- Comment: Thu Jan 22 19;33;17 comment by...ssf -->
<pre>DVDD history plot from IMON not avaliable due to restart of IMON by ACES to clear  red trip indication.  Apparently "Unmark" will not do this and IMON has to be restarted</pre>
<!-- Comment: Thu Jan 22 21;21;42 comment by...rainer -->
it does not unmark tripped ladders in the local window - this is a bug. it will always unmark with the global unmark button. this the aces did not do.
so we need a plot from davids current plotter to reconstruct what happened with this guy. please have a look.
<!-- Date: Thu Jan 22 19:23:17 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB2W2L4 
<li>problem n: 672 
<li>Status: needs_quiet_time
<li>Type: Chip_other
<li>Action: Incident
<li>entered by: sforrest 
<li>description is: DVDD software protection trip.  Could not power ladder.   Removed from Hwdb.  Investigate during next quiet time. 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B2W2&layer=L4">report</a></ul>
<!-- Author: Problem DB -->






<!-- Date: Thu Jan 22 23:30:26 2004 -->
<center><a href='/elog/notebooks/silicon/Thu_Jan_22_2004_233013.ps'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3348' ></a><a href='/elog/notebooks/silicon/Thu_Jan_22_2004_233003.ps'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3346' ></a></center>
<!-- Author: Rainer -->
<!-- Comment: Thu Jan 22 23;45;39 comment by...rainer -->

so DVDD of SB2W2L4 simply shot through the roof. it had  ~2% c2c2
issues ala
e22412fc 8d0800f3 8c088b08 8908c2c2  
and intermittent truncated events. no most likely a gonner. 


<!-- Comment: Fri Jan 23 09;23;23 comment by...Gino -->
I do not have time to document the 20 minutes of digging that I did but the solution to have the ladder back is to <b>change the powere supply</b>.
</p>
Maybe somebody wanna take the challenge and dig?


<!-- Comment: Fri Jan 23 15;28;02 comment by...Rainer -->
<pre>I have dug already on this guy quite a bit last night and would pass the challenge on to the detector working group and the power supply people. It is encouraging that you think we can get this guy back in data taking.</pre>
<!-- Comment: Sat Jan 24 01;32;11 comment by...Gino -->
Finally I found some time (not able to sleep):
<ul>
<li> The message in the GUI said that the ladder tripped on software protection trip and it also reported the value of the current (that was above 1000 mA). This informations are posted online, are very valuable, and imply that:
<ul>
<li> Being a SPT the GUI received that high value for at least 4-5 (I do not remember) readings in a row.
<li> This value is above the Hardware tripping limit of the PS (I think this is about 400 mA) and so something fishy is going on because the PS did not trip but the GUI turned the wedge OFF because of SPT.
<li> Clearly the PS misbehaved
</ul>
<li> now by digging on the PS log files you figure out that the SB2W2L4 DVDD has been reading the values that should be associated with SB2W2L4 AVDD so:
<ul>
<li> this can only be due to:
<ul>
<li> the digital part of the power supply (<b> very likely</b>)  (the analog part is doing fine being the ladder did not trip) or
<li> the comunication between the PS and the GUI (including the HDW ie CAENET SVXINTER and so on) (<b> very unlikely</b> being the comunication and the associated HDW crate by crate)
<li> the code for the GUI or the code that pipes the data on the GUI (<b>very unlikely</b> but yes also software can have its own glitches) 
</ul>
<li>as far as this specific trip is concern <b>the ladder is doing fine</b> (as long as we do not power cycle it 100 times trying to get it back when the fault is in the PS and power cycle all the ladders in that crate for hockerization). 
</ul>
</ul>
</p>
The Power supply should be swapped as soon as possible. And from there see the stability and if the problem reappears than start questioning the crate and the comunication between the crate and the GUI.
</p>
I suggest to make the plot above with the correct scales including current and voltages for both AVDD and DVDD for the incriminated ladder. This might help to understand the description given here.
</p>
Once again (funny enough) I have never seen this failure mode before. 

</p> BTW it took me 40 minutes to make this entry and 20 to do the digging. I really think I have to improve my typing skills.
<!-- Comment: Sat Jan 24 01;33;40 comment by...Gino -->
Also I like the Italian Accent on <b>powere supply</b>
<!-- Comment: Mon Jan 26 09;07;28 comment by...Gino -->
<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=181&frame=2&anchor=09:00:45'>
Here </a> is the plot I was asking for. 

<!-- Date: Fri Jan 23 12:00:29 2004 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3349&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3349  width='300' align='top'></a></td><td>pinkies starting about 10:10</td></tr></table>
<!-- Author: ssf -->


<!-- Date: Fri Jan 23 12:07:54 2004 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3350&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3350  width='300' align='top'></a></td><td>IMON plot after taking system through coldstart</td></tr></table>
<!-- Author: ssf -->


<!-- Date: Fri Jan 23 16:06:48 2004 -->
<p><font size=+1> D-mode Calibration Job Finished </font>
<p><pre>
This is a computer-generated silicon D-mode calibration status report.
Silicon D-mode calibration job 309 FAILED.

Job info
--------
Calibration type : standard
DPS-off run      : 178276
DPS-on run       : 178273
Requested by     : Jeff & Anadi
Submitted        : 2004-01-21 21:35:38
Completed        : 2004-01-23 14:06:02
Status           : FAILURE
Comment          : solenoid ON, HV ON, with beam

Last processing stage reported : comparing to the reference calibration
Failure diagnostics :
The calibration failed the following reference comparison criteria:

           Cut Name    Sensor Set   Cut Value   Data Value
-------------------    ----------   ---------   ----------
dps_off_ped abs max    SVX_PHI      0.5         0.580768  
 dps_on_ped abs max    SVX_PHI      0.2         0.244694  

Reference calibration runs are 174789 (DPS off) and 174788 (DPS on)
</pre>
<!-- Author: D-mode script run_calib_analysis_noph.tcl -->
<!-- Comment: Fri Jan 23 16;50;46 comment by...Jason -->
This is good news, believe it or not.  I was able to recover the calibration which got stuck when pcdaq.lbl.gov went down.
<BR>
This calibration is OK for making used sets, and it has the latest ladder configuration.
<BR>
<!-- Date: Fri Jan 23 16:29:32 2004 -->
<pre>SVT SPYmon plots not avaliable for this run (error in .ps file).  </pre>
<!-- Author: ssf -->
<!-- Comment: Fri Jan 23 16;59;32 comment by...ssf -->
<pre>see next entry</pre>
<!-- Date: Fri Jan 23 16:48:36 2004 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3352&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3352  width='300' align='top'></a></td><td>SVT SPYmon plot</td></tr></table>
<!-- Author: ssf -->
<!-- Comment: Fri Jan 23 16;59;03 comment by...ssf -->
<pre>I had trouble opening this plot on b0dap44 as well. No trouble on another machine. Perhaps the disc is filling up again?</pre>



<!-- Date: Fri Jan 23 23:12:00 2004 -->
<pre>Used quiet time to try to reintegrate and power SB2W2L4.  
  Used traceSilicon to place the ladder online in hwdb  
(option 4), and then wrote CAENvolts_hwdb.dat (option 10).  
  In PSGui clicked on load CAENvolts.dat and then tried  
to take the ladder from "Off" to "Standby."  Gui buttons  
indicate that the transition takes place, an appropriate  
message appears in the window, and then transitions back  
to "Off" without tripping or further message.   
 
At this point quiet time lost and ladder was taken offline,  
hwdb was written again and loaded in the PSGui. <p> 
The next step will be to hockerize the crate and then try  
again.  Before doing so however I will need to turn off the  
power to all of the ladders in that powered by that CAEN  
crate. 
 
In directory  
	/mnt/autofs/cdf/people2/svxii/cratelocation 
 
the command $ grep SB2W2 b0fib00  returned 
 
 SVX   SB2W2-024 NW Top    Bot       2       Pit     35  Settings       2. 
</pre><p> 
meaning that the CAEN crate is "NW Top Botttom." <a href='http://www.pha.jhu.edu/~reid/si_workshop.pdf'> Reid's Lake Geneva talk </a> has a look up table on page 2 which shows that this is CAEN crate 2 and on the reset pannel is labled "NW Top 2." <p> 
<pre> 
The command   $grep "NW Top    Bot" b0fib00     returns  
 
 SVX   SB0W0-006 NW Top    Bot ... 
 SVX   SB0W2-025 NW Top    Bot ...   
 SVX   SB0W3-034 NW Top    Bot ...      
 SVX   SB1W0-007 NW Top    Bot ...   
 SVX   SB1W2-027 NW Top    Bot ...   
 SVX   SB1W3-038 NW Top    Bot ...  
 SVX   SB2W2-024 NW Top    Bot ...      
 SVX   SB2W3-031 NW Top    Bot ...        
 ISL   IB0W1-031 NW Top    Bot ...    
 
these are the wedges which need to be turned off.</pre>  
<!-- Author: ssf -->
<!-- Comment: Fri Jan 23 23;44;38 comment by...ssf -->
<b>It Lives!</b>Hockerized as detailed above.  SB2W2L4 placed back on line, hwdb file written, PSGui loaded CAENvolts.dat in PSGui and all wedges repowered. e224 tripped several times but eventualy stayed in standby. 

<!-- Date: Sat Jan 24 00:21:45 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB2W2L4 
<li>problem n: 672 
<li>Status: needs_quiet_time
<li>Type: Chip_other
<li>Action: Removed_hdwdb
<li>entered by: sforrest 
<li>description is: Quiet time studies, ladder was powered after hockerization.   Ladder removed from hwdb over concerns for ladder reliability. Needs more quiet time to prove ladder reliability. 
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B2W2&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Sat Jan 24 00:22:54 2004 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3353&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3353' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3354&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3354' width='300'></a></center>
<!-- Author: ssf -->
<!-- Comment: Sat Jan 24 00;23;47 comment by...ssf -->
<pre>Proof that this ladder lives!</pre>



<!-- Comment: Sat Jan 24 01;51;35 comment by...rainer -->
We'll study the ladder a bit more during next quiet time - mike was so anxious beating rob in efficiency while  he stands in for rob in florida.


<!-- Date: Sat Jan 24 02:14:46 2004 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3355&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3355  width='300' align='top'></a></td><td>SVT SPY Mon</td></tr></table>
<!-- Author: ssf -->


<!-- Date: Sat Jan 24 08:12:45 2004 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3356&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3356' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3357&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3357' width='300'></a></center>
<!-- Author: Rong -->
<!-- Comment: Sat Jan 24 08;21;43 comment by...Rong -->
<pre>The AVDD and DVDD plots of SB1W1L4 which started to show pinky since 7:40. DVDD seems to be stable.</pre>

<!-- Comment: Sat Jan 24 08;28;03 comment by...ssf -->
<pre>problem looks to have begun earlier ~0645</pre>

<!-- Comment: Sat Jan 24 09;10;06 comment by...Rong -->
<pre>Stan and I check SVXMon for occupancy and mean charge plots for this ladder and nothing obviously wrong. Stan suggest to issue an HRR but the current level didn't change too much. Now the state of IMON for e114 is in either green and yellow.</pre>
<!-- Date: Sat Jan 24 08:32:21 2004 -->
Entered in Silicon database: <ul>
<li>Ladder: SB1W1L4 
<li>problem n: 1102 
<li>Status: needs_work
<li>Type: AVDD
<li>Action: Currents_investigated
<li>entered by: rong 
<li>description is: AVDD current drop by 45mA. see report <a href= 'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=181&frame=2&anchor=08:12:45'> user link text </a>   
<li>See <a href="http://www-cdfonline.fnal.gov/internal/usr/lmiller/cgi-bin/newproblemdb/DBreport.pl?detsys=SVX&wedge=B1W1&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Sat Jan 24 08:56:04 2004 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3359&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3360' ></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3361&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3362' ></a></center>
<!-- Author: Rong -->
<!-- Comment: Sat Jan 24 08;57;50 comment by...Rong -->
<pre>AVDD history plots since day900. AVDD currrent level was adjusted  by 40mA on 12/02/2003. It seems to be decreasing gradually.</pre>


<!-- Date: Sat Jan 24 09:01:09 2004 -->
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=view&page=70&frame=2&anchor=06:54:26.614'> 
Simultaneous Silicon Trip and loss of PSGui </a>.   
<pre> 
Trigger inhibits on both SVX and ISL, and missing low voltage  
bar in HV summary for both detectors.-> suspected multiple trips  
or at least IFix thinking there were. ACE reported that IMON  
showed no Trips (RED) and so I had the ACE kill and restart IMON  
to see if it would pick up some indication upon restart. Because  
we had lost indication of exactly what ladders tripped I walked  
the ACE through trip procedure using HV Summary rather than  
turning on individual ladders.</pre> 
<pre> 
1. Reset trips in recovered PSGui. 
2. Took all of Silicon to Stby via HV Summary controls to place  
Silicon in a known state. 
3. After PSGui indicated that Detector was in stby, had ACE turn  
HV back on.   
4. When HV up, trigger inhibits went away and the shift crew  
resumed the run. 
5. Requested that SVXMon plots be placed in the shift e-log  
after 10 minutes running hoping that would catch any chip or  
ladder problems. 
 
0856 now and SVXmon shows no problems. I'm not even certain we  
had a trip.  Log files show no trips, but that could be due to  
the loss of the GUi and it was never logged. 
</pre> 
 
 
 

<!-- Author: ssf -->
<!-- Comment: Sat Jan 24 16;51;02 comment by...Andy H. -->
<pre>If you could grab the svxinter output from this incident, that
would be helpful (maybe too late to figure out this incident's
output, since svxinter can't time-stamp...)

For further diagnostics (more as a note to self, but anyone
else should feel free to follow up):

PS log file:
/cdf/svx/data1/datalog/ps/20040124.si_

At time stamp 06:13:15, things start to look crazy.
GUI gives three more updates, then dies.
GUI restored at 06:17:19.

logfile legend:
column 1: Channel number (lookup_table.txt in same directory 
          maps to physical device --- is it accurate?  Haven't
          checked in a while)
column 2: Channel status: 0=ON, 1=OFF, 2=STBY, -1=error
column 3: CAEN status word: bit 16 = ON, bit 1 = present,
          others... read SY527 manual
column 4: V
column 5: I
</pre>
<!-- Comment: Sat Jan 24 17;07;42 comment by...Rainer -->
<h1> Analysis of Inhibits in Run <a href='http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178390'> 178390 </a> </h1> or <h1> hey you scooped me </h1>
<p><h3> from PSGui log file </h3>
<pre>
Sat Jan 24 01:34:35 CST 2004     IFix -  success: Silicon ON  
Sat Jan 24 06:17:00 CST 2004     IFix -  success: Silicon ON  
Sat Jan 24 06:20:38 CST 2004     IFix -  success: Silicon ON  
JAVA GUI Interface for Silicon Power Supply Control
July,2000
Andrew Ivanov, David Westbrook, Mark Kruse
andrew@fnal.gov, mkruse@fnal.gov

--== GUI Startup Complete ==--
Sat Jan 24 06:22:47 CST 2004     Established communication with iFix
(...)
Sat Jan 24 06:39:54 CST 2004     IFix -  success: Silicon to STANDBY  
(...)
Sat Jan 24 06:42:50 CST 2004     IFix -  success: Silicon ON  

</pre>
timestamps the restart of the PSGui, also  see below.
   
<p>
<h3> svxinter  output </h3> <pre>
detector = 0, status = 0
detector = 2, status = 0
Caenet_write: Error number -1 ffffffff received
group_info:cr_n = 2, response = -1
Caenet_write: Error number -1 ffffffff received
group_info:cr_n = 2, response = -1
Caenet_buffer: Error number 7 7 received
group_info:cr_n = 2, response = 7
SVX B0W0L5 spontaneously turned off
turning it back on
Sending Request for e005
Timeout
ERROR: not all Fibs are initialized
ERROR: FIB initialization, ps_n = 0SetChannelStatus, err = -1
checkSt[1][0][5] = 1
SVX B0W0L4 spontaneously turned off
turning it back on
Sending Request for e004
Timeout
[...]
ISL B0W1L5 spontaneously turned off	  Sending Request for f015
SVX B0W0L5 spontaneously turned off	  Sending Request for e005
SVX B0W2L5 spontaneously turned off	  Sending Request for e025
SVX B0W3L5 spontaneously turned off	  Sending Request for e035
SVX B1W0L5 spontaneously turned off	  Sending Request for e105
SVX B1W2L5 spontaneously turned off	  Sending Request for e125
SVX B1W3L5 spontaneously turned off	  Sending Request for e135
SVX B2W2L5 spontaneously turned off	  Sending Request for e225
SVX B2W3L5 spontaneously turned off	  Sending Request for e235
</pre>
which all correspond to crate#2 NW Top Bottom. <p>


<h3> SVXIICON/ALMON output </h3> <pre>

01/24/2004 00:05:31.3 [SVXIICON] SVX_PS_LOGGING                 CFN            NOT OK      CAEN GUI and PSMon running? (HV heartbt 
[...]
01/24/2004 00:14:34.3 [SVXIICON] SVX_PS_LOGGING                 OK                 OK      CAEN GUI and PSMon running? (HV heartbt 
[... NB: we were there ....]
01/24/2004 06:08:00.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:08:01.5 [CDFS1   ]  G_OP_NAME_1                      I/O - General Error. Command rejected                            
01/24/2004 06:08:31.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:08:36.0 [CDF_S1  ] R_SOUND_ALM                    COMM          ???????      Sound Rack Alarm - CDF3                 
01/24/2004 06:08:36.0 [CDF_S1  ] CRY_WE_START_DATE                                                                                  
01/24/2004 06:08:36.0 [CDF_S1  ] CRY_NDE_START_DATE                                                                                 
01/24/2004 06:08:36.0 [CDF_S1  ] CRY_EDE_START_DATE                                                                                 
01/24/2004 06:08:43.0 [CDF_S1  ] G_HTR_CT2_2                    OK               0.00      COT Cold Trap 2 Regeneration Heater     
01/24/2004 06:08:43.0 [CDF_S1  ] G_GSAEDR                       OK               0.00      PNEUMATICS EAST DROP RATE               
01/24/2004 06:08:43.0 [CDF_S1  ] G_GSAWDR                       OK               4.91      PNEUMATICS EAST DROP RATE               
01/24/2004 06:08:43.0 [CDF_S1  ] G_EV_LN2_CT2                   OK               OPEN      PV-5853; COT Cold Trap 2 LN2 Supply     
[...]
01/24/2004 06:16:16.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:16:39.3 [SVXIICON] SVX_PS_LOGGING                 CFN            NOT OK      CAEN GUI and PSMon running? (HV heartbt 
[...] 
</pre>
This was most likely the tweety alarm reported<a href='http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=view&page=70&frame=2&anchor=06:50:09.271'> here </a> 
<pre>
01/24/2004 06:22:32.3 [SVXIICON] SVX_PS_LOGGING                 OK                 OK      CAEN GUI and PSMon running? (HV heartbt 
01/24/2004 06:22:59.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:23:30.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:24:01.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:24:32.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:25:03.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:25:27.3 [SVXIICON] SVX_PT_COTBHE1                 OK              12.98 PSIA SVX COT BH1 EAST PSIA                   
01/24/2004 06:25:34.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:25:48.3 [SVXIICON] SVX_PT_COTBHE1                 HI              13.17 PSIA SVX COT BH1 EAST PSIA                   
01/24/2004 06:26:05.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
[repeated 23 times]
01/24/2004 06:38:29.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:38:51.3 [SVXIICON] SVX_PT_COTBHE1                 OK              12.98 PSIA SVX COT BH1 EAST PSIA                   
01/24/2004 06:39:00.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:39:12.3 [SVXIICON] SVX_PT_COTBHE1                 HI              13.07 PSIA SVX COT BH1 EAST PSIA                   
01/24/2004 06:39:31.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:40:02.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:40:33.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:41:04.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:41:35.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:42:06.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:42:37.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:43:08.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:43:28.3 [SVXIICON] SVX_PT_COTBHE1                 OK              12.97 PSIA SVX COT BH1 EAST PSIA                   
01/24/2004 06:43:40.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:43:49.3 [SVXIICON] SVX_PT_COTBHE1                 HI              13.14 PSIA SVX COT BH1 EAST PSIA                   
01/24/2004 06:44:11.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:44:42.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
01/24/2004 06:44:54.3 [SVXIICON] SVX_PT_COTBHE1                 OK              12.99 PSIA SVX COT BH1 EAST PSIA                   
01/24/2004 06:45:13.1 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                    
</pre>
<h1> Conclusion </h1>

there seems to have been a communication problem with crate#2. not clear to me whether the the CAEN Gui crash at 6:16am was before that
trip (a cause ?) or unrelated - can't get a time stamp on the trigger inhibit. The PSGui freeze was witnessed earlier around midnight -
likely connected to a network glitch ( See main <a
href='http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=copyentry&page=70&time=00:13:53.656'>
elog </a>). For sure, the auto-recovery does not help - all the ladders were tried to turn back twice, of course to no avail, since the fib is busy. <p>
Can we get a timestamp on the trigger inhibit from PSMOn ??? Also, if we could log svxinter in a more graceful way( hate the CR conversions) - I capture the minicom output for now into minicom.cap, but that's not so hot.

<!-- Comment: Sat Jan 24 17;10;28 comment by...Rainer -->
Also, it appears the wedges of crate #2 where turned off (there was a trigger inhibit) but that did not bother IFIX, according to the ALMon log file above. Also, there was a second incident I will describe shortly, see below.


<!-- Comment: Sat Jan 24 18;26;31 comment by...Rainer -->
I also don't understand this  - <pre>
Sat Jan 24 06:17:00 CST 2004     IFix -  success: Silicon ON  
Sat Jan 24 06:20:38 CST 2004     IFix -  success: Silicon ON  
</pre>
did the ace try to ramp up the silicon as a result of the holy trinity alarm (IFIX Gui communication ?)

<!-- Date: Sat Jan 24 13:16:11 2004 -->
<b> L1 accounting </b> <p> 
from run <a href='http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178390'> 178390 </a>  
<pre>Sat Jan 24 04:43:04 CST 2004    34646968        Sat Jan 24 04:43:05 CST 2004      178390    34646968 
Sat Jan 24 04:58:30 CST 2004     8733420        Sat Jan 24 04:58:31 CST 2004      178390    43380388 
Sat Jan 24 05:17:51 CST 2004    11562711        Sat Jan 24 05:17:52 CST 2004      178390    54943099 
Sat Jan 24 05:22:34 CST 2004     2588743        Sat Jan 24 05:22:34 CST 2004      178390    57531842 
Sat Jan 24 06:10:08 CST 2004    30833286        Sat Jan 24 06:10:05 CST 2004      178390    88365128 
Sat Jan 24 06:23:16 CST 2004     9959950        Sat Jan 24 06:23:17 CST 2004      178390    98325078 
Sat Jan 24 07:11:02 CST 2004    21449903        Sat Jan 24 07:11:03 CST 2004      178390   119774981 
Sat Jan 24 07:23:57 CST 2004    10389177        Sat Jan 24 07:23:58 CST 2004      178390   130164158 
Sat Jan 24 08:31:35 CST 2004    52836173        Sat Jan 24 08:31:36 CST 2004      178390   183000331 
Sat Jan 24 08:39:38 CST 2004     5974353        Sat Jan 24 08:39:39 CST 2004      178390   188974684 
Sat Jan 24 09:11:44 CST 2004    23649689        Sat Jan 24 09:11:45 CST 2004      178390   212624373 
Sat Jan 24 09:23:02 CST 2004     8070409        Sat Jan 24 09:23:04 CST 2004      178390   220694782 
Sat Jan 24 09:32:36 CST 2004     5802999        Sat Jan 24 09:32:37 CST 2004      178390   226497781 
Sat Jan 24 10:11:48 CST 2004    27398258        Sat Jan 24 10:11:50 CST 2004      178390   253896039 
Sat Jan 24 10:59:42 CST 2004    31903601        Sat Jan 24 10:59:43 CST 2004      178390   285799640 
Sat Jan 24 11:44:10 CST 2004    34267084        Sat Jan 24 11:44:12 CST 2004      178390   320066724 
Sat Jan 24 11:44:36 CST 2004         881        Sat Jan 24 11:44:38 CST 2004      178390   320067605 
Sat Jan 24 11:45:00 CST 2004         545        Sat Jan 24 11:45:02 CST 2004      178390   320068150 
 </pre> 
The SRC scalers sum up to 320068150 so no L1A MIA.  
<!-- Author: Rainer -->
<!-- Comment: Sat Jan 24 13;25;52 comment by...Rainer -->
there were 3 L2 decision timeouts in this run.


<!-- Date: Sat Jan 24 16:46:39 2004 -->
Today the E-string fired again and CDF got stuck in the well known mode where we have to play dirty tricks to go back to data taking. 
</p> 
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=181&frame=2&anchor=18:22:10'> 
Previous work </a> concluded that the FFO readout signal that is used for the GB to detect resonances gets stuck HIGH in the FFO. As a consequence monitoring tools were installed in order to monitor 2 FFOs at the time. 
 
</p> 
What we figured out today is that <b>  only one of the two FFO being monitored got stuck in that mode </b>. Rainer is gonna pipe this information to the Engineer responsible for the FFO board.  
<!-- Author: Gino -->
<!-- Comment: Sat Jan 24 17;50;53 comment by...Rainer -->
Gino suggested to tighten the Estring parameters at the end of the store to see whether we get a resonance condition fired out of normal data taking situation, out of which we could be able to recover, or not. In this case, again, there was a DAQ error preceeding (or caused ??) by the resonance fatal error - <pre>
(MLE) b0xft00:Messenger:3:05:26 PM->Event 2115452: Bunchcounters in slot 13 (BC=109) and slot 10 (BC=108) disagree (L2B=0) 
(MLE) b0xft00:Messenger:3:05:26 PM->Runtime Error 6, Event 2115452: Bunch counter mismatch, mismatch count = 1
(MLE) b0dap73.fnal.gov:Thread-13847:3:18:36 PM->FrontEnd Error: VRB_SVX_02 
(MLE) b0dap73.fnal.gov:Thread-13847:3:18:36 PM->Requested Halt-Recover-Run issued [errmon]
CT: 2004.01.24 15:18:37
18'37" 1 crate/s: b0svx02(112),  in error.[RXPT]b0cot12:Messenger:3:18:36 PM->Runtime Error 1, Event 2269407: Timeout on global TDC Done -->  
 Additional Information: 

 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!! 

 - Timeout on global TDC Done      
 - More > 1 crate in Done TO.      
 - Auto-SHEPHERD WILL NOT BE ISSUED.  
[MLE]
...
</pre>
The odd thing is - where does the automatic HRR come from ? I thought we had taken that out. Requested by the TDCs ? anyway, the run auto -recovered, and got stuck.<p>
Incidentially, to recover 

the FFO we need to send one L1 so the 'Activiate' part of these
<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=180&frame=2&anchor=16:10:46'>
instructions </a> is really important - something we did not know at the time when the instruction was drafted. 
<!-- Comment: Sun Jan 25 05;16;57 comment by...Rainer -->
<h3> Some Resonance Estring parameter tweaking done </h3> <p>
.. and we could <b> HRR out of every single FATAL error!!! </b>. See main 
<a href=
'/cgi/elog/elog.pl?nb=2004&action=view&page=73&frame=2&anchor=05:03:12.738'>
elog </a>. Note that flipping the hardware database values were not picked up by the SRC even with refresh hwdb - very odd.

<!-- Date: Sat Jan 24 17:27:48 2004 -->
<h1> Another PS trip </h1> or hey, don't we like to fumble around with that PS system <p><b> PSGui log </b> <pre> 
 
Sat Jan 24 13:49:59 CST 2004     Illegal condition: Portcard OFF, Ladder ON; turning OFF   SVX B5 W7 
Sat Jan 24 13:50:55 CST 2004     success: Resetting trips 
Sat Jan 24 13:52:10 CST 2004     success: Power Supply SVX B5 W7 ON 
Sat Jan 24 13:52:19 CST 2004     TRIP: SVX  B5  W7 AVDD0 OverCurrent 
Sat Jan 24 13:52:49 CST 2004     success: Channel0 SVX B5 W7 ON 
Sat Jan 24 13:52:56 CST 2004     TRIP: SVX  B5  W7 AVDD0 OverCurrent 
Sat Jan 24 13:53:21 CST 2004     success: Resetting trips 
Sat Jan 24 13:53:42 CST 2004     success: Resetting trips 
Sat Jan 24 13:53:50 CST 2004     success: Channel0 SVX B5 W7 to STANDBY 
Sat Jan 24 13:54:19 CST 2004     success: Resetting trips 
Sat Jan 24 13:54:24 CST 2004     success: Channel0 SVX B5 W7 ON 
Sat Jan 24 13:54:49 CST 2004     success: Resetting trips 
</pre> 
<b>IFIX Log </b> 
<pre> 
01/24/2004 13:52:13.0 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                     
01/24/2004 13:52:41.3 [SVXIICON] SVX_PS_TRIP                    CFN              TRIP      B5W7 layer 0, maybe others               
01/24/2004 13:52:44.0 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                     
01/24/2004 13:53:15.0 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                     
01/24/2004 13:53:46.0 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                     
01/24/2004 13:54:00.3 [SVXIICON] SVX_PS_TRIP                    OK                 OK      This field filled at alarm time          
01/24/2004 13:54:17.0 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                     
01/24/2004 13:54:48.0 [HTRDATA ]  HTCWATCHDOGTIMER                BLOCK IS RESET                                                     
</pre> 
<b> svxinter </b> log file was lost due to resizing the window 
<p> 
i.e.there was no IFIX alarm when the PSGui detected the illegal condition - only when the ace tried to power the wedge back on, did one ladder trip on AVDD, which caused a trip. <p> 
seems like in cases like these, we only rely on the software inhibit - now since the LV/HV is most likely ON, this is not a problem, but if the wedges really would transition to OFF, then this would only be caught 30-60s later. <p> 
 
Now this is the <b>  SIXTH </b> time since beginning of the year there is a PS oddity ( See  
<a href=' http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=view&page=25&frame=2&anchor=00:28:45.779'> here </a>,  
<a href=' http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=180&time=16:55:07'> here </a>, 
<a href=' http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=180&time=20:44:19'>here</a>, 
<a href=' http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=180&time=02:18:19'>here</a>, and 
<a href='http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=180&time=08:37:56'>here</a>), two of the portcard and two of the spontaneous transition type, apparently not coming from the same crate. So .... 
<center> <h1> WHAT's GOING ON ?? </h1></center>  
<!-- Author: Rainer -->




<!-- Date: Sat Jan 24 19:28:12 2004 -->
<b> pager carriers </b> - please see note about silicon trigger inhibit status  
 
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=view&page=72&frame=2&anchor=18:33:36.578'> 
here </a>   
<!-- Author: Rainer -->


<!-- Date: Sat Jan 24 23:04:04 2004 -->
<b> reminder </b> I have found system.cfg of ~/daq/svx 
with  b0svx0x crates commented out. this is in general not a good idea. you should run SVXDAQ always with the b0svx0x crates included.  
<!-- Author: Rainer -->



<!-- Date: Sun Jan 25 15:50:07 2004 -->
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=view&page=74&frame=2&anchor=15:27:19.249'> 
TeVMon overridden </a> at beginning of store.  TeVMon green at a few miniutes into the store, 1547.
<!-- Author: ssf -->


<!-- Date: Sun Jan 25 18:35:14 2004 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3363&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3363  width='300' align='top'></a></td><td>SVT Spy mon</td></tr></table>
<!-- Author: ssf -->


<!-- Date: Sun Jan 25 19:57:31 2004 -->
<h1> another un-recoverable Resonance error </h1> <p> 
reported <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=view&page=75&frame=2&anchor=19:04:17.795'> 
here.</a> Auto-HRR still effective, so no choice but go for  
dkclark's <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=180&frame=2&anchor=16:10:46'>recovery</a> procedure. I would have liked to try fumbling with the resonance parameters to see whether we could HRR out of it - the idea being the with a high count rate, one could on recover send the L1A needed to reset the FF0 before the SRC fires again. Another HALT could then be used to reset the limits to the default.  <p> 
again only one FFO fired - scope trace to follow. LA and OS reset.  
<!-- Author: Rainer -->
<!-- Comment: Sun Jan 25 20;17;25 comment by...Rainer -->
Donatella T says that in this run <a href='http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178435'> 178435 </a> there was RXPT errors first which are clear via auto-HRR - the recover appear to have been pending (?) when the resonance detector error came through.

<pre>
 (MLE) b0l3pcom1.fnal.gov:main:6:53:06 PM->Host b0eb24.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 8 for event 718802.
The VRB had no event.
(MLE) b0l3pcom1.fnal.gov:main:6:53:06 PM->Host b0eb25.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 8 for event 718802.
The VRB had no event.
(MLE) b0l3pcom1.fnal.gov:main:6:53:06 PM->Host b0eb23.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 8 for event 718802.
The VRB had no event.
CT: 2004.01.25 18:53:10
53'10" 1 crate/s: b0svx02(112),  in error.[RXPT]
(MLE) b0dap73.fnal.gov:Thread-26061:6:53:10 PM->FrontEnd Crate Error Condition from:
VRB_SVX_02

(MLE) b0dap73.fnal.gov:Thread-26061:6:53:10 PM->FrontEnd Error: VRB_SVX_02
(MLE) b0dap73.fnal.gov:Thread-26061:6:53:10 PM->Requested Halt-Recover-Run issued [errmon]
(MLE) b0l3pcom1.fnal.gov:main:6:53:07 PM->Host b0eb24.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 10 for event 718802.
The VRB had no event.
(MLE) b0l3pcom1.fnal.gov:main:6:53:07 PM->Host b0eb25.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 10 for event 718802.
The VRB had no event.
(MLE) b0l3pcom1.fnal.gov:main:6:53:07 PM->Host b0eb23.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 10 for event 718802.
The VRB had no event.
(MLE) b0l3pcom1.fnal.gov:main:6:53:08 PM->Host b0eb24.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 12 for event 718802.
The VRB had no event.
(MLE) b0l3pcom1.fnal.gov:main:6:53:08 PM->Host b0eb25.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 11 for event 718802.
The VRB had no event.
(MLE) b0l3pcom1.fnal.gov:main:6:53:08 PM->Host b0eb23.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 12 for event 718802.
The VRB had no event.
CT: 2004.01.25 18:53:12
53'12" 1 crate/s: b0svx02(112),  in error.[RXPT]b0svx02:Messenger:6:53:08 PM->SRC Fatal Error:Sl 5 Resonance Detected
 -->
 Additional Information:

 Attention !!!. FERML_SRC_FATALITY ERROR !!!

[MLE]
(MLE) b0l3pcom1.fnal.gov:main:6:53:09 PM->Host b0eb24.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 13 for event 718802.
The VRB had no event.
(MLE) b0l3pcom1.fnal.gov:main:6:53:09 PM->Host b0eb23.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 16 for event 718802.
The VRB had no event.
(MLE) b0l3pcom1.fnal.gov:main:6:53:09 PM->Host b0eb25.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 12 for event 718802.
The VRB had no event.
(RC)   18:53:13 Halt -> HALTED
(MLE) b0l3pcom1.fnal.gov:main:6:53:10 PM->Host b0eb24.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 15 for event 718802.
The VRB had no event.
(MLE) b0l3pcom1.fnal.gov:main:6:53:10 PM->Host b0eb23.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 18 for event 718802.
The VRB had no event.
(MLE) b0l3pcom1.fnal.gov:main:6:53:10 PM->Host b0eb25.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 16 for event 718802.
The VRB had no event.
(MLE) b0l3pcom1.fnal.gov:main:6:53:11 PM->Host b0eb24.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 16 for event 718802.
The VRB had no event.
(MLE) b0l3pcom1.fnal.gov:main:6:53:11 PM->Host b0eb23.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 20 for event 718802.
The VRB had no event.
(MLE) b0l3pcom1.fnal.gov:main:6:53:11 PM->Host b0eb25.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 17 for event 718802.
The VRB had no event.
(MLE) b0l3pcom1.fnal.gov:main:6:53:12 PM->Host b0eb24.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 18 for event 718802.
The VRB had no event.
(MLE) b0l3pcom1.fnal.gov:main:6:53:12 PM->Host b0eb25.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 18 for event 718802.
The VRB had no event.
(MLE) b0l3pcom1.fnal.gov:main:6:53:13 PM->Host b0eb24.fnal.gov, task tRec_0
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 20 for event 718802.
The VRB had no event.
CT: 2004.01.25 18:53:18
53'18" 1 crate/s: b0svx02(112),  in error.[RXPT]
(RC)   18:53:24 Recover -> RECOVERED
(MLE) b0tsi00:Messenger:6:53:23 PM->TS failed when trying to Run (State 2)
</pre>
why did these crates b0eb23,24,25 aka b0svx06,07,08 go into error in the first place ? What is cause, what is effect ?

<!-- Date: Sun Jan 25 19:58:33 2004 -->
<table><tr><td><a href=/elog/notebooks/silicon/Sun_Jan_25_2004_195829.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3365  align='top'></a></td><td>scope trace from recent un-recoverable SRC resonance error.</td></tr></table>
<!-- Author: rainer -->
<!-- Comment: Sun Jan 25 20;18;07 comment by...Rainer -->
documented 
<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=181&frame=2&anchor=19:57:31'>
here </a>


<!-- Date: Mon Jan 26 02:29:43 2004 -->
<h3> And Another One </h3> See  
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=view&page=75&frame=2&anchor=00:18:52.413'> 
here </a>  
 
<h1> Synopsis of recent stuck -resonance Occurences: </h1> 
<h3> run 
<a href='http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178391'> 178391 </a> errorfile 
<a href='http://www-cdfonline.fnal.gov/daq/runSummary/errorlog/errorfile178391.log'> 178391 </a> reformatter  
<a href='http://www-cdfonline.fnal.gov/daq/runSummary/errorlog/errorfile178391_reformatter.log'> 178391 </a></h3> 
<p> 
 
<pre><font color="red"><b> 
(MLE) b0dap73.fnal.gov:Thread-13847:3:18:36 PM->FrontEnd Error: VRB_SVX_02  
</b><font color="black"> 
(MLE) b0dap73.fnal.gov:Thread-13847:3:18:36 PM->Requested Halt-Recover-Run issued [errmon] 
CT: 2004.01.24 15:18:37 
<b> 18'37" 1 crate/s: b0svx02(112),  in error.[RXPT]b0cot12:Messenger:3:18:36 PM->Runtime Error 1, Event 2269407: Timeout on global TDC Done -->   </b> 
 Additional Information:  
 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!!  
 
 - Timeout on global TDC Done       
 - More > 1 crate in Done TO.       
 - Auto-SHEPHERD WILL NOT BE ISSUED.   
[MLE]<b> 
18'37" 1 crate/s: b0svx02(112),  in error.[RXPT]b0cot16:Messenger:3:18:36 PM->Runtime Error 1, Event 2269407: Timeout on global TDC Done -->  </b> 
 Additional Information:  
 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!!  
 
 - Timeout on global TDC Done       
 - More > 1 crate in Done TO.       
 - Auto-SHEPHERD WILL NOT BE ISSUED.   
[MLE] 
<b>18'37" 1 crate/s: b0svx02(112),  in error.[RXPT]b0cot02:Messenger:3:18:36 PM->Runtime Error 1, Event 2269407: Timeout on global TDC Done -->  </b> 
 Additional Information:  
 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!!  
 
 - Timeout on global TDC Done       
 - More > 1 crate in Done TO.       
 - Auto-SHEPHERD WILL NOT BE ISSUED.   
[MLE] 
CT: 2004.01.24 15:18:38 
<b> 18'38" 1 crate/s: b0svx02(112),  in error.[RXPT]b0cot17:Messenger:3:18:36 PM->Runtime Error 1, Event 2269407: Timeout on global TDC Done -->  </b> 
 Additional Information:  
 
 Attention!!!. FERML_TDC_NOT_READY_GLOBAL !!!  
(...) 
[<i>several  other cot crates, see below </i>] 
(...) 
<font color="black"> 
<b> 18'38" 1 crate/s: b0svx02(112),  in error.[RXPT]b0svx02:Messenger:3:18:36 PM->SRC Fatal Error:Sl 5 Resonance Detected  
 --></b>  <font color="black"> 
<font color="green"><b> 
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error for b0svx02: Sl 5 Resonance Detected  
  
 HRR is STRONGLY RECOMMENDED.  
</b><font color="black"> 
[MLE]<font color="blue"><b> 
(RC)   15:18:39 Halt -> HALTED </b> <font color="black"> 
CT: 2004.01.24 15:18:45 
<b> 18'45" 21 crate/s: b0cot00(48), b0cot01(49), b0cot02(50), b0cot03(51), b0cot04(52), b0cot05(53), b0cot06(54), b0cot07(55), b0cot08(56), b0cot09(57), b0cot10(58), b0cot11(59), b0cot12(60), b0cot13(61), b0cot14(62), b0cot15(63), b0cot16(64), b0cot17(65), b0cot18(66), b0cot19(67), b0svx02(112),  Done TO.1 crate/s: b0svx02(112),  in error.[RXPT]</b> 
(MLE) b0dap73.fnal.gov:Thread-13847:3:18:45 PM->Done Timeout: COT_00 COT_01 COT_02 COT_03 COT_04 COT_05 COT_06 COT_07 COT_08 COT_09 COT_10 COT_11 COT_12 COT_13 COT_14 COT_15 COT_16 COT_17 COT_18 COT_19 VRB_SVX_02  
<font color="blue"> <b> 
(RC)   15:18:46 Recover -> RECOVERED </b> <font color="black"> 
(MLE) b0tsi00:Messenger:3:18:47 PM->TS failed when trying to Run (State 2) 
<font color="black"> 
</pre> 
 
<h3> Run 
<a href='http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178435'> 178435 </a> errorfile 
<a href='http://www-cdfonline.fnal.gov/daq/runSummary/errorlog/errorfile178435.log'> 178345 </a> reformatter  
<a href='http://www-cdfonline.fnal.gov/daq/runSummary/errorlog/errorfile178435_reformatter.log'> 178345 </a>  
</h3> 
<p> 
<pre> 
MLE) b0l3pcom1.fnal.gov:main:6:53:06 PM->Host b0eb24.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 8 for event 718802. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:53:06 PM->Host b0eb25.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 8 for event 718802. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:53:06 PM->Host b0eb23.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 8 for event 718802. 
The VRB had no event. 
CT: 2004.01.25 18:53:10 
<b> 53'10" 1 crate/s: b0svx02(112),  in error.[RXPT] </b> 
(MLE) b0dap73.fnal.gov:Thread-26061:6:53:10 PM->FrontEnd Crate Error Condition from: 
VRB_SVX_02 
<font color="red"> <b> 
(MLE) b0dap73.fnal.gov:Thread-26061:6:53:10 PM->FrontEnd Error: VRB_SVX_02  
</b> <font color="black"> 
(MLE) b0dap73.fnal.gov:Thread-26061:6:53:10 PM->Requested Halt-Recover-Run issued [errmon] 
(MLE) b0l3pcom1.fnal.gov:main:6:53:07 PM->Host b0eb24.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 10 for event 718802. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:53:07 PM->Host b0eb25.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 10 for event 718802. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:53:07 PM->Host b0eb23.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 10 for event 718802. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:53:08 PM->Host b0eb24.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 12 for event 718802. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:53:08 PM->Host b0eb25.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 11 for event 718802. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:53:08 PM->Host b0eb23.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 12 for event 718802. 
The VRB had no event. 
CT: 2004.01.25 18:53:12 
<b> 53'12" 1 crate/s: b0svx02(112),  in error.[RXPT]b0svx02:Messenger:6:53:08 PM->SRC Fatal Error:Sl 5 Resonance Detected  
 -->  </b> 
<font color="green"><b> 
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error for b0svx02: Sl 5 Resonance Detected  
 
 HRR is STRONGLY RECOMMENDED.  
</b> <font color="black"> 
[MLE] 
(MLE) b0l3pcom1.fnal.gov:main:6:53:09 PM->Host b0eb24.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 13 for event 718802. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:53:09 PM->Host b0eb23.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 16 for event 718802. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:53:09 PM->Host b0eb25.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 12 for event 718802. 
The VRB had no event. 
<font color="blue"> <b> 
(RC)   18:53:13 Halt -> HALTED </b> <font color="black"> 
(MLE) b0l3pcom1.fnal.gov:main:6:53:10 PM->Host b0eb24.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 15 for event 718802. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:53:10 PM->Host b0eb23.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 18 for event 718802. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:53:10 PM->Host b0eb25.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 16 for event 718802. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:53:11 PM->Host b0eb24.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 16 for event 718802. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:53:11 PM->Host b0eb23.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 20 for event 718802. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:53:11 PM->Host b0eb25.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 17 for event 718802. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:53:12 PM->Host b0eb24.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 18 for event 718802. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:53:12 PM->Host b0eb25.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 18 for event 718802. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:53:13 PM->Host b0eb24.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 20 for event 718802. 
The VRB had no event. 
CT: 2004.01.25 18:53:18 
<b>53'18" 1 crate/s: b0svx02(112),  in error.[RXPT] </b> 
<font color="blue"> <b>  
(RC)   18:53:24 Recover -> RECOVERED 
</b> <font color="black"> 
(MLE) b0tsi00:Messenger:6:53:23 PM->TS failed when trying to Run (State 2) 
</pre> 
<p> 
 
<font color="black"> 
 
</pre> 
<font color="black"> 
 
<p><h3> run 
<a href='http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178438'> 178438 </a> errorfile 
<a href='http://www-cdfonline.fnal.gov/daq/runSummary/errorlog/errorfile178438.log'> 178438 </a> reformatter  
<a href='http://www-cdfonline.fnal.gov/daq/runSummary/errorlog/errorfile178438_reformatter.log'> 178438 </a> 
<p></h3> 
<pre> <font color="red"> <b> 
(MLE) b0dap73.fnal.gov:Thread-26061:11:56:33 PM->FrontEnd Error: COT_18  
</b><font color="black"> 
(MLE) b0dap73.fnal.gov:Thread-26061:11:56:33 PM->Requested Halt-Recover-Run issued [errmon] 
CT: 2004.01.25 23:56:33 
<b> 56'33" 1 crate/s: b0cot18(66),  in error.[RXPT] </b> 
(MLE) b0l3pcom1.fnal.gov:main:11:56:27 PM->Host b0eb24.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 8 for event 2133227. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:11:56:27 PM->Host b0eb23.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 8 for event 2133227. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:11:56:27 PM->Host b0eb25.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 8 for event 2133227. 
The VRB had no event. 
(...) 
[<i> and more errors from slot 10,12,13,16,all three CPUs repeat </i>] 
(...) 
(MLE) b0l3pcom1.fnal.gov:main:11:56:30 PM->Host b0eb24.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 13 for event 2133227. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:11:56:30 PM->Host b0eb25.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 12 for event 2133227. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:11:56:30 PM->Host b0eb23.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 16 for event 2133227. 
The VRB had no event. 
<font color="blue"> <b> 
(RC)   23:56:37 Halt -> HALTED 
</b> <font color="black"> 
(...) 
[<i> and more slots, all three CPUs repeat </i>] 
(...) 
(MLE) b0l3pcom1.fnal.gov:main:11:56:33 PM->Host b0eb25.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 18 for event 2133227. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:11:56:34 PM->Host b0eb24.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 20 for event 2133227. 
The VRB had no event. 
 
<font color="blue"> <b>  
(RC)   23:56:48 Recover -> RECOVERED 
</b> <font color="black">  
(RC)   23:56:50 Run -> ACTIVE 
(MLE) b0dap73.fnal.gov:Thread-26061:12:00:52 AM->FrontEnd Crate Error Condition from: 
VRB_SVX_02 
 <font color="red"> <b> 
(MLE) b0dap73.fnal.gov:Thread-26061:12:00:52 AM->FrontEnd Error: VRB_SVX_02 </b><font color="black"> 
(MLE) b0dap73.fnal.gov:Thread-26061:12:00:52 AM->Requested Halt-Recover-Run issued [errmon] 
CT: 2004.01.26 00:00:54 
<b>0'54" 1 crate/s: b0svx02(112),  in error.[RXPT]b0svx02:Messenger:12:00:47 AM->SRC Fatal Error:Sl 5 Resonance Detected  
 --></b>   
<font color="green"><b> 
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error for b0svx02: Sl 5 Resonance Detected  
  
 HRR is STRONGLY RECOMMENDED.  
[MLE] 
</b> <font color="black"> 
<font color="blue"> <b> 
(RC)   0:0:56 Halt -> HALTED 
</b> <font color="black"> 
CT: 2004.01.26 00:01:00 
<b>1'0" 1 crate/s: b0svx02(112),  in error.[RXPT] </b> 
<font color="blue"> <b> 
(RC)   0:1:2 Recover -> RECOVERED 
</b> <font color="black"> 
(MLE) b0tsi00:Messenger:12:00:58 AM->TS failed when trying to Run (State 2) 
</pre> 
<p> 
<font color="black"> 
 
<h3> 
 
 
A good one ('recovered HRR') for contrast: 
 
<a href='http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=178400'> 178400 </a> errorfile 
<a href='http://www-cdfonline.fnal.gov/daq/runSummary/errorlog/errorfile178400.log'> 178400 </a> reformatter  
<a href='http://www-cdfonline.fnal.gov/daq/runSummary/errorlog/errorfile178400_reformatter.log'> 178400 </a> 
</h3> 
 
<pre><font color="red"><b> 
(MLE) b0dap73.fnal.gov:Thread-13847:1:18:06 AM->FrontEnd Error: VRB_SVX_02  
</b> <font color="black"> 
(MLE) b0dap73.fnal.gov:Thread-13847:1:18:06 AM->Requested Halt-Recover-Run issued [errmon] 
CT: 2004.01.25 01:18:07 
<b>18'7" 1 crate/s: b0svx02(112),  in error.[RXPT] </b> 
<b> 18'7" 1 crate/s: b0svx02(112),  in error.[RXPT]b0svx02:Messenger:1:18:00 AM->SRC Fatal Error:Sl 5 Resonance Detected  
 --></b>   
<font color="green"><b> 
 Additional Information:  
 
 Attention !!!. FERML_SRC_FATALITY ERROR !!!  
  
 SRC Fatal Error for b0svx02: Sl 5 Resonance Detected  
  
 HRR is STRONGLY RECOMMENDED.  </b> 
<font color="black"> 
[MLE] 
<font color="blue"> <b> 
(RC)   1:18:9 Halt -> HALTED </b> 
<font color="black"> 
CT: 2004.01.25 0:18:14 
<b>18'14" 1 crate/s: b0svx02(112),  in error.[RXPT]</b> 
<font color="blue"><b> 
(RC)   1:18:15 Recover -> RECOVERED 
 
(RC)   1:18:18 Run -> ACTIVE 
</b><font color="black"> 
CT: 2004.01.25 01:18:28 
18'28" AUTOMATIC HRRs disabled from EH Options Menu.[EH Info] 
</pre> <font color="black">
<!-- Author: Rainer -->
<!-- Comment: Mon Jan 26 02;36;12 comment by...Rainer -->
I conclude that I cannot conclude. I don't see a significant difference.
<!-- Date: Mon Jan 26 02:32:23 2004 -->
<table><tr><td><a href=/elog/notebooks/silicon/Mon_Jan_26_2004_023219.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3367  align='top'></a></td><td>Stuck resonance scope trace - same sucker.</td></tr></table>
<!-- Author: rainer -->
<!-- Comment: Mon Jan 26 02;33;42 comment by...Rainer -->
belongs to this guy
<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2004&action=view&page=75&frame=2&anchor=00:18:52.413'>
here </a> 






<!-- Date: Mon Jan 26 09:00:45 2004 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3368&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-3369  width='300' align='top'></a></td><td>SB2W2L4 current and voltages (see comment)</td></tr></table>
<!-- Author: Gino -->
<!-- Comment: Mon Jan 26 09;06;17 comment by...Gino -->
<ul>
<li>BLACK: DVDD Voltage in V multiplied 100 times
<li>RED:   DVDD Current in mA
<li>BLUE:  AVDD Voltage in V multiplied 100 times
<li>GREEN: AVDD Current in mA
</ul>
</p>
This plot shows how before the SPT the DVDD values became equal to the AVDD values.
</p>
This should help to understand the explanation given <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=181&frame=2&anchor=23:30:26'>
here </a>