<!-- Title: 10:14:55  Mon Jan  8 2007 -->
<!-- Crewchief: Ricardo Eusebi (MPC) -->
<!-- Op1: Ulrich Husemann (SPL) -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Keep it running smoothly.</pre>


<!-- Date: Mon Jan 8 10:56:28 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10576&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10577  width='300' align='top'></a></td><td>Store 5166 is in with 240 lum. Losses are < 600Hz. SVTmon and SVXmon look Ok.</td></tr></table>
<!-- Author: Ricardo -->


<!-- Date: Mon Jan 8 13:34:05 2007 -->
<pre>Beam just aborted. Ace is doing the post-quench checkup.</pre>
<!-- Author: Ricardo  -->
<!-- Comment: Mon Jan  8 13;49;02 comment by...Ricardo -->
<pre>The beam was (clenly) aborted because of a trip in a PS.
The Silicon post-quench chekup passed.
</pre>


<!-- Date: Mon Jan 8 14:04:28 2007 -->
<h3>Downtime work</h3> 
We are going to take a shot at: 
<ul> 
<li>AutoHocker installation 
<li>b0svx04 power supply swap 
<li>SVXINTER mezzanine card installation 
</ul> 
As a first step, Ricardo switched of the Silicon, and I have changed the SVX chiller set point to -6&deg;C. Stay tuned.
<!-- Author: Ulrich -->
<!-- Comment: Mon Jan  8 21;20;38 comment by...Ricardo & Ulrich -->
Here are more details about the work we did today, making use of the quiet time and Tevatron studies after the abort today. A picture gallery will follow as well.

<ul>

<li> The silicon was turned off completely, and the SVX chiller temperature set-point was increase from -10&deg;C to -6&deg;C.</li>

<li> The power supply for crate b0svx04 was swapped by Roberto and Lou in under 1/2 hour. The supply had shown a fan failure late last year. The PSM alarm for the supply has been un-silenced after the work was done.</li>

<li> On the way to implementing the AutoHocker the following actions were taken:
  <ul>
     <li> The digital output card was installed in SVXIICON by Mark Knapp. Mark also upgraded the anti-virus software.</li>
     <li> The old Hockerization panel was replaced the brand new panel prepared for AutoHocker by George and Wayne. The cable from the PC to the card is only 1 m long, therefore the new panel resides the same rack as SVXIICON, 1RR34F (one rack to the left of the old panel in 1RR34G). To make things finally fit, we had to lift SVXIICON by 5 cm. Unfortunately, now the monitor does not fit on top of it anymore. Maybe we can figure out a more elegant solution later.</li>
     <li> PSMon.exe was set up to run with all hardware functions to access the digital output card. With this step, the AutoHocker was in principle functional.</li>
     <li> We changed the iFix page for manual hockerization to show that it is now the real thing. In addition, the page got linked to the Silicon Cooling page.</li>
     <li> We have also updated the Ace instructions and informed the Aces about the new procedure.</li>
  </ul>
</li>


<li> The following tests were done to check the functionality of AutoHocker:
  <ul>
     <li> With the switch set to "Manual" the connectivity of all the buttons were tested by hockerizing all the 16 crates.</li> 
     <li> A communication loss was simulated by pulling the CAENet cable of every crate in SVXINTER. The software detected all of them correctly.</li>
     <li> The iFix page that allows to manually hockerize the crates was tested crate by crate. We found that at some point in the chain the order of crates was reversed, i.e. we hockerized crate 16 when we clicked on crate 1. So we swapped the connectors accordingly and changed the labels on the new panel.</li>
     <li> We also discovered that the time during which a crate loses communication after a manual hockerization is long enough to trigger another hockerization by the automatic mechanism. To prevent this, we added the same 1-minute timeout after a manual hockerization that was already introduced for the automatic version last week.</li>
  </ul></li>

<li> We have also made some progress towards the new TevMon: The detector was off anyway, so Bill Badgett installed the mezzanine card on the SVXINTER crate CPU which will pick up the HV permit signal for the Silicon in the future. He also installed a new kernel which supports the new card. The SVXINTER code does not seem to have problems with the new kernel.</li>

<li> The version of IMON in <tt>~svxii/imon_dev_Jan08_2007</tt> was tested. We kept it running for the night but didn't make it official yet. The new features include:
  <ul>
      <li> The channels marked as tripped will now recover to 'off', 'bad' or 'caution' automatically once the trip is cleared in the CAEN supplies ("Reset Trips" button on PS GUI): no unmarking will be necessary. </li>
      <li> Since the AutoHocker is in place, the alarm pop-up window reporting the need for hockerization was taken out. </li>
      <li> The opportunity was taken to test IMON for the diagnosis of spontaneous switch-off of ladders and power supplies. This should result in fewer and more correct pop-up windows. </li>
  </ul> </li>


<li>We returned to -10&deg;C and powered the Silicon back on, which was mostly smooth, except for the following incidents:
  <ul>
    <li>We started the cool-down a little late and almost triggered a cooling trip for BH temperatures above 2&deg;C. We changed the PLC trip limit in the last minute...</li>
    <li>A couple of times a software protection trip took out f02x. The DVDD current of f020 went above the soft limit of 600 mA. After some attempts, the current would finally stay below 600 mA. The envelope of the DVDD current curve over time showed an exponential decay behavior - some strange turn-on effect.</li>
    <li>b0fib02 refused to go through CONFIG a couple of times, claiming that the HDI for f052 failed to initialize properly. We tried the usual magic: FIB crate reboot, power-cycling the wedge, switching things on and off ladder by ladder, etc. What finally fixed it was a <tt>reset_crate</tt> for b0fib02. The VME reset brought things back in line.</li>
  </ul></li>
</ul>



<!-- Date: Mon Jan 8 21:21:24 2007 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10580&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10580' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10578&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10578' width='300'></a></center>
<!-- Author: Ulrich -->
<!-- Comment: Mon Jan  8 21;22;05 comment by...Ulrich -->
Picture gallery 1: new Silicon Cooling and Hockerization iFix pages.


<!-- Date: Mon Jan 8 21:24:45 2007 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10581&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10582' ></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10585&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10586' ></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10587&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10588' ></a></center>
<!-- Author: Ulrich -->
<!-- Comment: Mon Jan  8 21;25;38 comment by...Ulrich -->
Picture Gallery 2: George with his opus, from the front, from the back and finally installed.


<!-- Date: Mon Jan 8 21:27:50 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10589&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10589  width='300' align='top'></a></td><td><h3>New AutoHocker Instructions</h3> 
These new instruction have been posted in the control room and sent to all Aces.</td></tr></table>
<!-- Author: Ulrich -->
<!-- Comment: Mon Jan  8 21;28;37 comment by...Ulrich -->
I've had enough for today. I'll change the web pages tomorrow.


<!-- Date: Tue Jan 9 06:10:54 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10590&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10591  width='300' align='top'></a></td><td>Store 5171 is in with initial lum of 180. SVT spymonn looks good. Losses are low as always.</td></tr></table>
<!-- Author: Ricardo -->


<!-- Date: Tue Jan 9 07:49:11 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10592&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10592  width='300' align='top'></a></td><td><h3>Cooling Trip</h3> 
Around 6:50 this morning, we had another SVX/L00 cooling trip. The chiller compressor stuttered a couple of times before it kicked in again. The picture shows that the effect is real, because at the same time the chiller evaporator temperature goes up, indicating a lack of cooling. All of SVX and L00 went off. The Ace brought everything to STANDBY, which caused also all of SVX and L00 to go to STANDBY (except for a bunch of trips, which we are recovering now).</td></tr></table>
<!-- Author: Ricardo & Ulrich -->
<!-- Comment: Tue Jan  9 22;42;00 comment by...Jennifer & Ulrich -->
<h3>Some More Thoughts On Today's Trip</h3>
Over the course of the day, we discussed this morning's cooling trip. A couple of thoughts are collected here:
<ul>
  <li><b>Add timer on compressor signal?</b> 1-2 minutes without the compressor should not impose danger on the silicon. If the Silicon is in danger, there are other mechanisms to trip the Silicon. We can ride out such a glitch by adding a timer in the PLC code. The timer would only trip the Silicon after the compressor is off continuously for, say, 2 minutes. The timer would only require only a relatively simple change to the PLC code, so it could be done regardless of the further debugging of the chiller.</li>
<p>
  <li><b>Relay, freeze protection or flow switch?</b> We checked the schematics of the chiller. There are several ways of tripping the compressor. When it trips on low or high head-pressure, the trip is latched and needs to be reset. Such a failure mode is not compatible with what we saw, i.e. the compressor going off and on again. There are three other failure modes which could explain what we see: low flow on the chiller (measured by a flow switch), a faulty relay to switch the compressor, and the freeze protection mechanism.
<p>
  The hottest candidate among these is the freeze protection. It would warm up the compressor, compatible with what we see. The set point for the freeze protection is around -12&deg;C, 2&deg;C lower than the chiller set point. It seems that the chiller is running about 1&deg;C colder than in the summer. It could be that the limit is hit for a minute, causing the trip. We will try and reproduce this failure mode once the detector is off by increasing the freeze protection set point. If this is really the problem, we could either increase the set point by 0.5&deg; or lower the freeze protection (which bears the danger of freezing the coolant).</li>
<p>
<li><b>Swapping chillers or controllers?</b> Other options are to switch to the spare chiller or to change the controller. Given the small statistics, it is unclear if we would find anything this way.</li>
</li>
</ul>
Thanks to Bill Noe, Dean Beckner, Jim Loskot, and Rich Schmitt for their input.
<!-- Date: Tue Jan 9 08:57:22 2007 -->
<pre>AVDD of e474 is a bit larger than it's limits. This is perfectly logical in the history of this channel. Since we are not taking data I'll set it to ignore it for now and see how it behaves for the day. 
</pre>
<!-- Author: Ricardo -->


<!-- Date: Tue Jan 9 10:45:57 2007 -->
<pre>It looks like the XFT problem was fixed. A run was taken without Silicon and worked OK. I ramped up the silicon and we are back to taking data. </pre>
<!-- Author: Ricardo -->
<!-- Comment: Tue Jan  9 10;46;57 comment by...Ricardo -->
<pre>Forgot to say that the L1 and L2 rates were OK. I'm now waiting for the prescales to kick in.
</pre>


<!-- Date: Tue Jan 9 11:33:33 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10593&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10593  width='300' align='top'></a></td><td>Bias of e424 is very large. This developed over the weekend from 100 to about 164. Otherwise this was a very good ladder without entries in the problem database. 
I checked the strip distribution and rms charge in SVXmon and there were no spikes.   
Besides blaming micron, and hockerize the PS crate, does anyone has an alternative explanation ? 
 
</td></tr></table>
<!-- Author: Ricardo -->
<!-- Comment: Tue Jan  9 11;42;05 comment by...Ricardo -->
<pre>For the moment I keep it Ignored.
</pre>


<!-- Date: Tue Jan 9 11:43:10 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10594&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10594  width='300' align='top'></a></td><td>AVdd current for e474. This current is just above the limits. 
I'll increase the limits.</td></tr></table>
<!-- Author: Ricardo -->

<!-- Date: Tue Jan 9 11:45:16 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB4W7L4 
<li>problem n: 1062 
<li>Status: fixed
<li>Type: Other
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: Change the Avdd limits.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>1051</td><td>1051</td></tr>
<tr><td><b>Lower</b></td><td>1005</td><td>1005</td></tr>
<tr><td><b>Upper</b></td><td>1065</td><td>1078</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B4W7&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Tue Jan 9 14:21:05 2007 -->
<h3>MCR Computer Room Power Outage: No Clock</h3> 
In today's 9:00 meeting, it was announced that some maintenance work is needed in the MCR computer room. One of the consequences is that the entire Tevatron will be without a clock during that time. We will have to switch off the Silicon. There is no date and time available yet, but it will be directly after termination of one of the next stores .
<!-- Author: Ulrich -->


<!-- Date: Tue Jan 9 21:00:26 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10595&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10595  width='300' align='top'></a></td><td>I got paged because large bias current of e294. As the plot shows large current behaviour was developed relatively recently. I asked the SciCo to set to ignore for now, but will take a better look tomorrow.</td></tr></table>
<!-- Author: Ricardo -->
<!-- Comment: Tue Jan  9 21;04;41 comment by...Ulrich -->
Didn't we have this guy in Ignore anyway?

<!-- Comment: Tue Jan  9 21;09;48 comment by...Ricardo -->
<pre>No. If I remember correctly that was DVDD 494 and Bias 424.
</pre>

<!-- Comment: Tue Jan  9 21;18;51 comment by...Ulrich -->
Yes, see <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=334&frame=2&anchor=13:00:43'>
last Saturday</a>.

<!-- Comment: Tue Jan  9 21;29;46 comment by...Ricardo -->
I know it was marked in the past but it was green (and unmarked) today during data-taking, so I left it so.


<!-- Date: Tue Jan 9 21:44:14 2007 -->
<h3>Och ai the noo</h3> 
 
<p>There be wee biccies from the land of the brave in the sili office</p>
<!-- Author: Scotty McDoughal Haggis  -->



<!-- Date: Tue Jan 9 23:17:27 2007 -->
<h3>Electron Lens Trip</h3> 
We had an electron lens trip. TevMon went only into warning (10 minute delay) and AGIGI2 went only up to 2e9, but the Ace played safe and brought the Silicon to STANDBY. See the plots in the <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2007&action=view&page=27&frame=2&anchor=22:46:57.128'>main e-log</a>. Here is what MCR has to say: 
<pre> 
22:52:39-  T:L1VAC (TEL vacuum interlock) tripped the TEL off, which prompted CDF to put its Si into stand-by. Ops responded and found all ion pumps running, along with the solenoid, and correctors. A reset was issued to T:L1VAC and it cleared, and the modulator's output returned to normal. Channels three and four were tripped locally - the digital status for T:L1VAC showed L1VAC3 Low Trip, L1VAC1 1st Trip, and L1VAC1 Low Trip all tripped. The TEL looks stable now and notified CDF of this. - marty 
-- Tue Jan 9 23:07:48 comment by...marty --  The TEL was off for about five minutes and AGIGI2 never exceded 2.1 E9. 
</pre>
<!-- Author: Ulrich -->
<!-- Comment: Wed Jan 10 08;42;34 comment by...Ricardo -->
<pre>Statistics: The electron lens tripped at 22:28 and the Silicon was brought to stand by at 22:34 which means that the ace discovered the trip about 6 minutes after it actually happened. Perhaps TevMon should be improved ? </pre>
<!-- Comment: Wed Jan 10 10;27;49 comment by...Ricardo & Ulrich -->
Looking into things more carefully this morning, things happened in the following order:
<ol>
<li>The electron lens tripped off at 22:28. TevMon did not go to WARNING because T:L1COLI is still idle due to a problem with reading T:VSCRAP. So the Ace did not realize that there was a problem.</li>
<li>The electron lens came back on at 22:34 and cleaned the abort gap. T:AGIGI2 was still below 2 at that point. The abort gap cleaning caused a spike in C:B0PAGC, and the Ace took the Silicon to STANDBY.</li>
</ol>
We have notified Ray Culbertson about the TevMon problem already yesterday and sent a reminder today. This should be fixed a.s.a.p.
<!-- Date: Wed Jan 10 07:50:09 2007 -->
<pre>Got paged because a large jump of DVDD current in e414. If failed to go back after two coldstart, but data-taking brought it back to normal.</pre>
<!-- Author: Ricardo -->


<!-- Date: Wed Jan 10 07:55:22 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10596&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10596  width='300' align='top'></a></td><td>Store 5172 is in with initial lum of about 230E30. SVT and SVX Mon look ok.</td></tr></table>
<!-- Author: Ricardo -->
<!-- Comment: Wed Jan 10 08;10;10 comment by...r -->
<pre>was 213E30, not 230E30.
</pre>



<!-- Date: Wed Jan 10 08:52:06 2007 -->
<pre>Electron lens tripped off for less than an minute and was recovered before any action was needed.</pre>
<!-- Author: Ricardo -->


<!-- Date: Wed Jan 10 09:00:27 2007 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10597&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10597' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10598&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10598' width='300'></a></center>
<!-- Author: Ricardo -->
<!-- Comment: Wed Jan 10 09;01;36 comment by...Ricardo -->
Long (left) and short (right) Bias0 history of f020. I will increase the limits.

<!-- Date: Wed Jan 10 09:03:44 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB0W2L0 
<li>problem n: 1786 
<li>Status: needs_observation
<li>Type: DVDD
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: Bias0 imon limits need update.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>62</td><td>62</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100</td><td>120</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B0W2&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Jan 10 09:05:33 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB0W2L0 
<li>problem n: 1786 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: The previous entry should be Bias0 NOT AVDD<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>62</td><td>62</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100</td><td>120</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B0W2&layer=L0">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Wed Jan 10 09:32:47 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10599&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10599  width='300' align='top'></a></td><td>A shorter history plot of Bias e424. We will likely increase the IMON limits (so we can unignore it in IMON) and keep this channel in observation. 
</td></tr></table>
<!-- Author: Ricardo -->

<!-- Date: Wed Jan 10 09:45:41 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB4W2L4 
<li>problem n: 1873 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: Since this ladder decided to have large bias we are forced to changed the imon limits after about a week of observation. <table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>37</td><td>37</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100</td><td>180</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B4W2&layer=L4">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Jan 10 09:57:58 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ricardo
<li>Description of Problem: Fixed by the ace on 4 minutes with experts looking over their shoulders.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=85">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Wed Jan 10 10:09:19 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10600&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10601  width='300' align='top'></a></td><td>Here is a graph of the DVDD history of e414. </td></tr></table>
<!-- Author: Ulrich -->

<!-- Date: Wed Jan 10 10:11:17 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB4W1L4 
<li>problem n: 1875 
<li>Status: needs_observation
<li>Type: DVDD
<li>Action: Incident
<li>entered by: husemann 
<li>description is: The DVDD current of this ladder climbed up while the Silicon was in STANDBY, see <a href= 'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=10:09:19'>here</a>. 2-3 COLDSTARTs were needed to bring it back again. No problems during data-taking. This is the second time this happens in the last week. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B4W1&layer=L4">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Jan 10 10:11:17 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB4W1L4 
<li>problem n: 1875 
<li>Status: needs_observation
<li>Type: DVDD
<li>Action: Incident
<li>entered by: husemann 
<li>description is: The DVDD current of this ladder climbed up while the Silicon was in STANDBY, see <a href= 'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=10:09:19'>here</a>. 2-3 COLDSTARTs were needed to bring it back again. No problems during data-taking. This is the second time this happens in the last week. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B4W1&layer=L4">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Jan 10 10:14:59 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB2W9L4 
<li>problem n: 1670 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: This ladder has roughly erratic bias current (See <a href= '/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=21:00:26'> here </a>). <table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>103</td><td>103</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>260</td><td>280</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B2W9&layer=L4">report</a></ul>
<!-- Author: Problem DB -->




<!-- Date: Wed Jan 10 10:54:42 2007 -->
Rich Krull installed security patches on NCDF205 this morning. The logic analyzer software has been restarted and re-armed.
<!-- Author: Ulrich -->


<!-- Date: Wed Jan 10 11:00:15 2007 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10602&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10602' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10603&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10603' width='300'></a></center>
<!-- Author: Ricardo -->
<!-- Comment: Wed Jan 10 11;04;42 comment by...Ricardo -->
Dvdd current of e480 after the PS was replaced. Left during data-taking. Right without taking data. The PS has a large offset and there are no current plans to swap it. The channel has been marked ignored for a long time and I will change the IMON limits accordingly.

<!-- Date: Wed Jan 10 11:10:56 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB4W8L0 
<li>problem n: 1777 
<li>Status: fixed
<li>Type: DVDD
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: Since there are no current plans to fix the supply IMON limits are changed to match the PS. The variance is kept at 8.4<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>80</td><td>80</td></tr>
<tr><td><b>Lower</b></td><td>70</td><td>1</td></tr>
<tr><td><b>Upper</b></td><td>200</td><td>20</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B4W8&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Jan 10 11:14:25 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ricardo
<li>Description of Problem: I spent this time investigating and fixing IMON limits for several channels during the past three days.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=86">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Wed Jan 10 11:45:57 2007 -->
We had our first Auto-hockerization! 
This was the first time that Auto-hockerization fires since its installation two days ago. The sequence of events happended while I was staring at the screen :  
<ul>  
<li> Caen crate 16 lost communication. IMON marked all its ladders to BLUE and popped up a window indicating to wait until the ladders change to BLACK before powering them back on.</li> 
<li> The Ifix Auto-hocker panel displayed Crate 16 RED and proceed to reset it.</li> 
<li> Trigger Inhibit kicked in and RC was set to halt. </li> 
<li> The ace quickly turned all the ladders back on and continue data taking.</li> </ul> 
 The total downtime was only about 3 minutes. 
</ul> 
 
Going through the svxiicon elog Ulrich also noted that during what we thought was a spontaneous switch-off earlier in the morning, Auto-hocker also fired. The question is now whether Auto-hocker mistakenly fired after the spontaneous switch-off, or the Auto-hocker fired first and the ace missed it and thought it was a spontaneous switch-off. Keep tuned. 
 
<!-- Author: Ricardo and Ulrich -->


<!-- Date: Wed Jan 10 13:21:57 2007 -->
<h3>Tevatron End of Store Studies</h3> 
From Brian Drendel's e-log: 
<pre> 
Experts would like to move the crystal collimator during the last two hours of this store.  The collision halls would likely want to turn off during this time.  
</pre> 
The store will be kept until late owl shift.
<!-- Author: Ulrich -->

<!-- Date: Wed Jan 10 17:20:26 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB1W1L0 
<li>problem n: 1860 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: Bias1 limit changed<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>50</td><td>50</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100</td><td>125</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B1W1&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Jan 10 17:22:00 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB1W0L4 
<li>problem n: 1872 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: Increasing Bias1 limit<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>56</td><td>56</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100</td><td>125</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B1W0&layer=L4">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Jan 10 17:24:35 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB2W0L4 
<li>problem n: 809 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: Bias1 limit increased.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>140</td><td>140</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>160</td><td>185</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B2W0&layer=L4">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Jan 10 17:26:39 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB5W2L0 
<li>problem n: 149 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: Bias1 limit increased.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>125</td><td>125</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>150</td><td>180</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B5W2&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Jan 10 17:27:33 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ricardo
<li>Description of Problem: Changed a bunch of bias imon limits.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=87">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Thu Jan 11 00:05:23 2007 -->
<h3>ISL Chiller Problems</h3> 
<ul> 
  <li> Around 19:30 the shift crew contacted us because there was a large number of bias pinkies. We realized that the ISL temperatures on the Real Time Plotter (reminding us of the ISL cooling problem) had jumped up by 1-2&deg;C. So the higher ISL temperatures resulted in higher bias currents which caused the pinkie alarm.</li> 
  <li>The source of the higher temperatures was tracked down to the ISL chiller itself. We had a look at the chiller together with the cryo tech (Steve Gordon). Usually it cycles between "Cooling" and "Hot Gas". This cycle was somehow not working, and we found no way of getting it back on. The freon sight glass showed bubbles, which would point to low freon charge on the chiller. 
  <li>Steve also realized that the "Run Chiller" request was not working in the interlock, so he hit "Start Chiller", but that didn't fix the problem either.</li> 
  <li>The next thing we tried is power-cycling the chiller. For that we planned to switch off the Silicon gracefully, but we didn't come to this point, because ISL tripped on low flow during our attempts to throttle the flow with the manual bypass valve. The trip was not silenced in the cryo area since Steve was down in the alcove, so the auto-dialer paged a bunch of people. Sorry, Jennifer, Ankush, and Rong-Shyang.</li> 
  <li>We didn't have to switch off SVX and L00 either, because the Ace hit the Crash Button. He heard the sono-alarm from the ISL cooling trip and got confused by the messages "Interlock not enabled" in the PS GUI log file. We have sent mail to Dean and Mark to have them removed.</li>  
  <li>We got the ISL chiller running again, after hitting the Britney Spears flow switch with a blunt device. Without heat load, it would cool down to 6&deg;C ok. After a couple of minutes it dropped out for a while again but came back by itself. The cycle of "Cooling" and "Hot Gas" looked not like before, with the "Cooling" cycle being longer than usual. This points to reduced ability to cool, which in turn points again to low freon charge. So we decided to swap chillers. 
  <li>None of us had ever done a chiller swap, but Jim Loskot had created instructions in iFix. With the instructions we could do the job, but it took much longer than expected because of two mistakes in the instructions (which I apparently overlooked when I was proofreading them) 
  <ol> 
    <li>The spare chiller doesn't have a MV-2001W valve like the ISL chiller. The MV-2301W valve has to be moved instead.</li> 
    <li>There is another valve on the chillers which is not mentioned in the instructions, MV-2200W. This valve had to be closed on the ISL chiller and opened on the spare chiller. 
  </ol> 
</li> 
<li>Like in the ISL chiller, we had to correct for the miscalibration of the temperature probe. We changed the Input 1 Bias to +1.5&deg;C to reach the target chiller return temperature of 6&deg;C.</li> 
<li>Steve recovered the racks, and we rebooted all FIB crates. For some reason, the reboot was quite slow.</li> 
<li>At this point we tried to implement what is now called the 
"Lazy pager carrier switch-on". This procedure brings the whole of the silicon from off to 
STANDBY in a couple of simple steps: 
<ul> 
  <li> Press the iFix button to STANDBY </li> 
  <li> While the PS come up we keep doing Coldstarts in a standalone version of RC.</li> 
  </ul></li> 
  <li>Unfortunately while doing that we did not watch the temperature of the compromised ISL cooling line CCE1.1. By the time we realized that the temperature is at 20&deg;C and shooting up, we were not able to switch off fast enough.  The temperature passed 24&deg;C and tripped ISL B4 and finally exceeded  25&deg;C resulting in an "ISL temperature &uuml;bertrip", which took out the entire Silicon again!</li> 
  <li>Then we re-powered all of of Silicon the "old-fashioned SPL way", and kept out the four ladders served by ISL CCE1.1, f401, f403, f451, and f453. We had to hockerize crate #6 because of DOIM pinkies in e09x (very conveniently via iFix). Otherwise everything came back fine.</li> 
  <li>While carefully watching the temperature of the cooling line, we tried to power up these four remaining ladders one by one. Even with only two of the four switched on, the cooling couldn't beat the load, and the temperature was growing too fast. So we took these four ladders out of the database and resumed data-taking with silicon.</li> 
  <li>We cleaned up by rebooting SVXINTER and restarting the PS GUI and IMON.</li> 
  <li>SVXMon had to be restarted to pick up the changes in the hardware HB.</li> 
</ul> 
  
<!-- Author: Ricardo & Ulrich -->
<!-- Comment: Thu Jan 11 09;44;27 comment by...Ulrich -->
A RAM technician came in this morning to have a look the ISL chiller. He is looking into the freon charge and clogged filters in the freon circuit.
<!-- Comment: Thu Jan 11 11;10;12 comment by...Ulrich -->
Update: the RAM technician found a leak in the seal of the safety relief valve of the chiller. So as we guessed yesterday, the chiller was leaking freon. The tech is now working on fixing the leak, exchanging filters, and re-charging the chiller with freon.
<!-- Comment: Thu Jan 11 14;50;25 comment by...Ulrich -->
Update: The RAM technician fixed the leak and re-charged the ISL chiller. It is now running on a dummy load and can be swapped in again at our convenience.
<!-- Comment: Fri Jan 12 17;45;26 comment by...Ulrich -->
End-of-week update: we will keep the system running using the spare chiller over the weekend and swap back next week. The ISL chiller has been running fine on a dummy load so far. Before I leave, I switched off the ISL chiller, closed the main valve on it with the manual control box, and flipped the switch on the heater which created the dummy load (without flow it had already stopped heating).
<!-- Date: Thu Jan 11 00:30:43 2007 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10604&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10604' width='300'></a></center>
<!-- Author: Ulrich -->
<!-- Comment: Thu Jan 11 00;31;01 comment by...Ulrich -->
Temperature history of the compromised cooling line. By now the temperature has stabilized around 18&deg;C - without any heat load!


<!-- Date: Thu Jan 11 00:34:35 2007 -->
Two SRC error during the last hour worth mentioning: L2A w/o L1A (logic analyzer was not armed at the time), and resonance. Now the LA is re-armed.
<!-- Author: Ulrich -->


<!-- Date: Thu Jan 11 02:12:18 2007 -->
<pre>While restarting a run (due to wcal problems) b0fib02 failed config. Ace follow the flowchart with no avail. Powercyle SB0W5 and that solved the problem.</pre>
<!-- Author: Ricardo -->


<!-- Date: Thu Jan 11 08:56:21 2007 -->
<pre>Fib02 decided not to pass config complaining about SB0W5L2. Powercycling the ladder did not work, rebooting the crate didn't work, and using reset_crate didn't work either.  
While doing this e3a4 tripped and keep tripping on DVDD4 OverCurrent. I could only recover momentarely by going to ON and then to stand-by (don't ask me why), but it tripped again. 
The PDB shows says that this ladder always trips when the North Muon wall is moved. We will leave e3a4 off until the access is finished. 
</pre>
<!-- Author: Ricardo -->
<!-- Comment: Thu Jan 11 11;38;11 comment by...Jose -->
This ladder tripped 2 or 3 times last time they moved the wall. It seems that this strange behavior is back.
<!-- Date: Thu Jan 11 09:46:16 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ricardo
<li>Description of Problem: Using SVXdaq Ulrich and I confirmed that the readback of e502 did not work properly all the time, so we masked it out.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=88">Incident Report</a></ul>
<!-- Author: Incident DB -->
<!-- Comment: Tue Jan 16 10;18;55 comment by...Ricardo -->
<pre>It's e052 not e502.
</pre>
<!-- Date: Thu Jan 11 09:52:31 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ricardo
<li>Description of Problem: ISL chiller problem, chiller was swap, ladder of compromised cooling line taken out. See details <a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=00:05:23'>here </a>.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=89">Incident Report</a></ul>
<!-- Author: Incident DB -->


<!-- Date: Thu Jan 11 09:55:06 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ricardo
<li>Description of Problem: Paged at night because b0fib02 didn't pass config. See entry <a href='http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=02:12:18'>here</a>.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=90">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Thu Jan 11 09:56:03 2007 -->
<pre>We are in access. Sasha is doing quiet time studies. 
</pre>
<!-- Author: Ricardo -->

<!-- Date: Thu Jan 11 10:02:48 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB3W0L2 
<li>problem n: 1536 
<li>Status: needs_observation
<li>Type: Chip_capid
<li>Action: Diagnosis_update
<li>entered by: sasha 
<li>description is: CAP_ID errors on chip a0 magically disappear on 01/06/07. All our previous attempts to fix this problem with changing Rdriver and ISEL bits failed. This is one of those ladders where we can only declare the current situation. So, we should keep an eye on this ladder for ~1 week before declaring it temporarely fixed again. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B3W0&layer=L2">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Thu Jan 11 10:07:17 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W2L1 
<li>problem n: 784 
<li>Status: fixed
<li>Type: Chip_bit_error
<li>Action: Diagnosis_update
<li>entered by: sasha 
<li>description is: Bit0 high problem is gone. Fixed, at least for now. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B0W2&layer=L1">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Thu Jan 11 10:08:07 2007 -->
On my request, KC Cahill has swapped in a new vacuum pump VP-1A for ISL this morning. No problems. Note: this is completely unrelated to the ISL chiller problems we had yesterday.
<!-- Author: Ulrich -->

<!-- Date: Thu Jan 11 10:12:37 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB1W8L4 
<li>problem n: 1801 
<li>Status: needs_work
<li>Type: Chip_capid
<li>Action: Diagnosis_update
<li>entered by: sasha 
<li>description is: Ulrich was able to reproduce CAP_ID problems in SVXDAQ last week. He tried to change ISEL bits to fix the problem, but with no success. CAP_ID problem might be related to the AVDD2 problem reported earlier. It's unlikely we can do anything, but we can try few more times. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B1W8&layer=L4">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Thu Jan 11 10:16:36 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: husemann
<li>Description of Problem: We had another SVX/L00 cooling trip. The chiller compressor stuttered a couple of times before it kicked in again. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=91">Incident Report</a></ul>
<!-- Author: Incident DB -->
<!-- Comment: Thu Jan 11 10;17;07 comment by...Ulrich -->
This is the Incident DB entry for the trip on Tuesday.




<!-- Date: Thu Jan 11 13:40:18 2007 -->
f853 tripped and was recovered.
<!-- Author: ricardo -->


<!-- Date: Thu Jan 11 14:12:50 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10605&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10606  width='300' align='top'></a></td><td><h3>Store 5174</h3> 
Store  5174 came in with an initial luminosity of 237e30. The losses were slightly higher than usual (1.7 kHz for C:LOSTP). SVTSpyMon and SVXMon look fine.</td></tr></table>
<!-- Author: Ricardo & Ulrich -->


<!-- Date: Thu Jan 11 14:20:11 2007 -->
<h3>Summary of quiet time studies</h3> 
 
<p> 
<li>e184 -- wrong CAP_ID for chip 88. It can be reproduced in SVXDAQ. Any attempts to fix a problem by changing settings failed. The problem is likely to be related to AVDD2. I suggest to check data from this chip and sparse it out if there is nothing meanigful.</li> 
<li>e4b4 -- wrong CAP_ID for chip a3 and garbage data. This is a known old problem. Chip was previously sparsed out. All attempts to fix this chip failed. I suggest to sparse out chip a3.</li> 
<li>e224 -- drop readout problem with a long history (ladder was affected by quench). It can be reproduced in SVXDAQ. The problem can be fixed only if we cut a chip chain starting from chip 89 (chip 8a to be the last chip). Current error rate is ~30%. I guess we should proceed with this change only if the rate is close to 90%.</li> 
<li>Many SVX L0 ladders tripped when Tev started to inject beam. All were successfully recovered.</li> 
<li>e3a4 was tripping on DVDD overcurrent when North Wall was out. I powered this ladder up after the end of access. The only weird thing is that Run Control would not notice that the ladder tripped. It would pass Config without problems even if the ladder was off. I checked a few times that the ladder was included in HWDB. I also restarted RC.</li> 
</p>
<!-- Author: sasha/Takayuki -->

<!-- Date: Thu Jan 11 14:21:44 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB2W2L4 
<li>problem n: 1423 
<li>Status: needs_work
<li>Type: Chip_bit_error
<li>Action: Diagnosis_update
<li>entered by: sasha 
<li>description is: The problem can be fixed only if we cut a chip chain starting from chip 89 (chip 8a to be the last chip). Current error rate is ~30%. I guess we should proceed with this change only if the rate is close to 90%. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B2W2&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Thu Jan 11 14:58:29 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: husemann
<li>Description of Problem: Conor detected reformatter errors from the Silicon. While ending the run, b0fib07 got stuck in the transition. The Ace rebooted the crate, and it came back fine. I analyzed a bad event with our <a href="http://b0sili01.fnal.gov/FormatterError.html">Reformatter Error Tool</a>, and it turned out that all FIB headers in b0fib07 were missing.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=92">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Thu Jan 11 17:05:42 2007 -->
We just had Ace meeting. Guess how many Aces were there! Hint: it is a number which does not exist in Roman numerals...
<!-- Author: Ulrich -->
<!-- Comment: Fri Jan 12 12;15;17 comment by...sasha -->
I suppose it's ZERO :-) 
<!-- Date: Fri Jan 12 07:17:18 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB5W7L0 
<li>problem n: 1876 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: Need to bump the bias limits.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100</td><td>120</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B5W7&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Fri Jan 12 07:22:30 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W1L1 
<li>problem n: 1762 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: Bumping the bias limits, ladder seems to be aging gracefully albeit fast.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>120</td><td>140</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B3W1&layer=L1">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Fri Jan 12 07:24:40 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB0W3L1 
<li>problem n: 1868 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: Bumping bias0 current limit.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>120</td><td>120</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B0W3&layer=L1">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Fri Jan 12 07:30:53 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB1W2L3 
<li>problem n: 1627 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: Bias0 growing as expected.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>150</td><td>170</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B1W2&layer=L3">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Fri Jan 12 07:33:53 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB1W5L3 
<li>problem n: 1603 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: Bias0 increasing as expected.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100</td><td>120</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B1W5&layer=L3">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Fri Jan 12 07:39:12 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB5W2L0 
<li>problem n: 149 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: Bias0 aging as expected.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>110</td><td>125</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B5W2&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Fri Jan 12 07:55:51 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ricardo
<li>Description of Problem: b0svx00 and svx04 went red in RC while starting a new run. Event builder of the b0svx04 was the culprit and the ace resetted it sucessfuly.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=93">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Fri Jan 12 08:26:27 2007 -->
<pre>f053 report zero current for  bias0. We need to hockerize PS crate 7 when we stop taking data.</pre>
<!-- Author: Ricardo -->
<!-- Comment: Fri Jan 12 08;28;44 comment by...r -->
Forgot to say that ladder is giving data.


<!-- Comment: Fri Jan 12 13;26;27 comment by...Ricardo -->
<pre>Crate 7 hockerized. Current readback went back to normal when in turned the ladder bias on. I turned off again and I'm waiting for shot setup.
 </pre>
<!-- Date: Fri Jan 12 08:53:34 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10607&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10607  width='300' align='top'></a></td><td>Bias current of e123. I'll bump the limit to 820, but we need to keep this ladder in observation.</td></tr></table>
<!-- Author: Ricardo -->
<!-- Comment: Fri Jan 12 08;59;42 comment by...Ricardo -->
<pre>Strip 150 (plus/minus 10) in the SVXMon plot for  C0-C4 shows a 100% ocuppancy. It does not looks like a pinhole as the neighbors have very small occupancy.

</pre>
<!-- Comment: Fri Jan 12 12;19;22 comment by...Ricardo -->

The actual strip numbers are 164 to 168, corresponding to chip 88 channels 24 to 28. These five strips have 100 percent occupancy. I took a look at the raw data; for these I found that channel 25 returns more than 9 adcs, but 26 returns always f3. <br>
If only channel 25 were hot, the nearest neighbor logic would read channel 24,25 & 26, and every now and then channel 27 and 28 too. What is strange is that we seem to be reading the five channels in every event, which could be explained if channels 25 and 27 were both hot.<br><br>
  
As a sidenote, according to the production folders in the Silicon office, this ladder had shorts in readout channels 26 and 27 and were half-disconnected. In addition, a comment written by our very own Juan Pablo mentions that there were bad capacitors on channels 24, 25 and 26. <br>
I couldn't make a consistent scenario that explain this behaviour AND the large bias current. I've speculated at some point that a pinhole developing in channel 26 (which always reports f3 anyways) could leak charge to channels 25 and 27, but then the charge of 25 and 27 must show correlations and it does not. The more plausible theory for now is that 25 and 27 are hot channels, and that the large bias has nothing to do with them.

   


<!-- Date: Fri Jan 12 08:55:10 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB1W2L3 
<li>problem n: 683 
<li>Status: needs_longterm_monitoring
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: Bumping limits again.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>770</td><td>820</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B1W2&layer=L3">report</a></ul>
<!-- Author: Problem DB -->




<!-- Date: Fri Jan 12 10:16:55 2007 -->
<h3>Emergency Recovery Procedures Updated</h3> 
I have updated the Silicon emergency procedures on the web and in our folder. They can be found under: 
<a href="https://www-cdfonline.fnal.gov/internal/ops/svxii/procedures/silicon-emergency-recovery.html">https://www-cdfonline.fnal.gov/internal/ops/svxii/procedures/silicon-emergency-recovery.html</a>. I have also moved over the Silicon cooling procedures to <a href="https://www-cdfonline.fnal.gov/internal/ops/svxii/procedures/silicon-emergency-cooling.html">https://www-cdfonline.fnal.gov/internal/ops/svxii/procedures/silicon-emergency-cooling.html</a>.
<!-- Author: Ulrich -->





<!-- Date: Fri Jan 12 13:58:02 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10610&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10611  width='300' align='top'></a></td><td>Dvdd current in hight state for e414. Introducing the brute-force coldstart; if one coldstart brings the current down (13:38), many coldstart work even better.  Looks like the current wend too low now, I wonder if it is going to stay there.</td></tr></table>
<!-- Author: Ricardo -->


<!-- Date: Fri Jan 12 15:34:27 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10612&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10612  width='300' align='top'></a></td><td>Store 5175 is in with a initial lum. of 250E30. SVTspymon and  SVXmon are ok. </td></tr></table>
<!-- Author: Ricardo -->

<!-- Date: Fri Jan 12 15:57:42 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB1W4L0 
<li>problem n: 1567 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: The Bias1 current tends to be slightly above limit when the Silicon is biased. Bumping up the limit to avoid unnecessary alarms.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>1086.61</td><td>1086.61</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>1210.0</td><td>1250.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B1W4&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Fri Jan 12 16:08:23 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: husemann
<li>Description of Problem: The second time this week, all of SVX tripped (except for the PortCards). Again, this points to a glitch in the SRC. The logic analyzer didn't trigger on SRC error, there were no clock errors, and there were so we don't know why this happen. We will work on a more sophisticated trigger.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=94">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Fri Jan 12 17:26:25 2007 -->
<h3>Second thoughs about last nights DAQ problem</h3> 
Today Conor looked into the incident last night when b0svx00 and b0svx05 failed partition and we could not connect to them, not even using vxcom. The more we discuss it, the more confused we get. From the logging we got the following impression: 
<ul> 
<li>The run was ended around 5:10 because many crates were in a bad state and the RTserver was hosed.</li> 
<li>The Ace had to manually override the END transition because of stuck crates.</li> 
<li>At about the same time, the SCPU catches a VME bus error and is paralyzed</li> 
<li>The crate CPUs failed PARTITION, and when we tried to log into the crate CPUs, we couldn't do so via vxcom</li> 
<li>The Ace followed the EVB instructions to reset the SCPUs in b0svx00 and b0svx05, which fixed all the problems. 
</ul> 
This leaves a bunch of questions: 
<ul> 
<li>Suppose the Ace reset the SCPU: the SCPU reset does not touch the VME bus, so why would this fix the crate CPU as well.</li> 
<li>Suppose the Ace reset the crate CPU by mistake: what would happend to the SCPU?</li> 
</ul> 
So we thought of a procedure for these kinds of incidents: 
Whenever a Silicon expert is called to debug such a problem, s/he should also check if the SCPU is up and running. After a reset of the crate CPU, a reset of the SCPU may also be required.  
<!-- Author: Ulrich -->
<!-- Comment: Sat Jan 13 04;37;50 comment by...Marcel -->
Well, that is not entirely correct, my dear Watson !<br>
If the SCPU dies while accessing the VME bus, the whole VME bus is blocked. If the SVX CPU now tries to access the VME bus, it detects  the VME_BUS BUSY (Still hold high by the SCPU) and has to wait. At som point, there are timeouts (long) and the whole CPU appears to be stuck. When you then reset the SCPUS, the VME_BUS BUSY is pulled low, the svx CPUS can happiliy access the bus and you are good to go. <br>
THE SCPU has a PCI-to-VME bridge, which exposed some features before, a VME BUS error (however it occured, could as well be a bus collision), so it gets stuck and blocks the BUS.  By resetting the sCPUS, you reset the entire board including the PCI-to-VME bridge and you are good to go. <br>
If you reset the crate CPU, nothing happens with the SCPU, but the crate CPU may still get stuck when it tries to access the BUS again. <br>
In Summer 2005 the Klute, Ron Rechenmacher, Sebastian  and I (with SCN advising as well) encountered a series of these incidents (which we all documented in the Si-elog). Basically all problems we saw with CPU's stuck where with concurrent bus access attemps and VME_BUS errors. ir the really interested, Vince has an excellent book about the VME bus, which a least Sebastian and I found very useful.<br>

Best wishes from the rainy island !<br>
<!-- Date: Fri Jan 12 18:15:47 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ricardo
<li>Description of Problem: S4W8 tripped and did not recover because of Software protection trip on DVDD. The CAEN (#13) was given corrupted readback and needed hockerization.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=95">Incident Report</a></ul>
<!-- Author: Incident DB -->


<!-- Date: Fri Jan 12 19:58:01 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: husemann
<li>Description of Problem: Automatic hockerization of CAEN crate #2. Recovered by the Ace in 3 minutes.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=96">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Fri Jan 12 21:43:29 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10613&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10613  width='300' align='top'></a></td><td>This the ladder that's giving problems</td></tr></table>
<!-- Author: Ricardo -->

<!-- Date: Fri Jan 12 22:19:02 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ricardo
<li>Description of Problem: e0b4 tripped on Avdd current, at the same time that the plug trip. Losses were not spike during at that time. It looks like the Avdd hardware limits (1400) for e0b4 were too close to the very high Avdd current (1375) and fluctuating. We powercycled several times the wedge, hockerize crate 1, and even reboot b0fib00 with no sucess. We decided to take this  ladder out of the HDB until the next store. By the time we finish with this RC was running again (they had problems with pcal crates, etc). SVXMon was restarted and the CO warned that SVTspymon may have a hole at B0W11.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=97">Incident Report</a></ul>
<!-- Author: Incident DB -->


<!-- Date: Fri Jan 12 22:53:16 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W11L4 
<li>problem n: 1877 
<li>Status: needs_work
<li>Type: AVDD
<li>Action: Incident
<li>entered by: husemann 
<li>description is: Ladder taken out of the database because it started <a href= 'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=21:43:29'>tripping on AVDD overcurrent</a>. The current was stable below 1000 mA, and now it jumped up just below the hardware trip limit of 1400 mA. Unfortunately, taking the ladder out results in an SVT hole. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B0W11&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Sat Jan 13 01:29:22 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10614&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10614  width='300' align='top'></a></td><td>pinky several times. </td></tr></table>
<!-- Author: Soo-Bong -->


<!-- Date: Sat Jan 13 01:31:50 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10615&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10615  width='300' align='top'></a></td><td>pinky several times</td></tr></table>
<!-- Author: Soo-Bong -->


<!-- Date: Sat Jan 13 01:44:59 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10616&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10616  width='300' align='top'></a></td><td>Pinky -> HRR -> 'unmark' button disappear  </td></tr></table>
<!-- Author: Soo-Bong -->
<!-- Comment: Sat Jan 13 02;10;19 comment by...Ricardo -->
<pre> Please post these plots in the MAIN CDF e-log.
</pre>

<!-- Date: Sat Jan 13 04:12:01 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ricardo
<li>Description of Problem: Pinkies, Pinkies and Pinkies throughout the night.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=98">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Sat Jan 13 11:32:25 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ricardo
<li>Description of Problem: Run 232026 found with consecutive yellow ladders in ISL B3 (see entry <a href='/cgi/elog/elog.pl?nb=2007&action=view&page=37&frame=2&anchor=03:54:39.826'>here </a>). Run was restarted and things went back to normal. Downtime was short, but time to fix was large as they couldn't get SVXMon running properly.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=99">Incident Report</a></ul>
<!-- Author: Incident DB -->
<!-- Comment: Sat Jan 13 11;40;15 comment by...Ulrich -->
Did you stay awake that long? The other option would have been to quickly check the raw data from those ladders. I also wonder why it took them over 1 hour to detect the missing FIB (unless it developed later in the run).


<!-- Date: Sat Jan 13 12:04:38 2007 -->
<h3>Logic Analyzer: New Trigger</h3> 
After the second complete SVX trip on which the logic analyzer did not trigger, we extented the trigger logic a little. The trigger before was on the rising edge of SRCERR SVX or SRCERR ISL. The most likely cause for the trips is corrupted chip commands. So the new trigger checks if either of the FIB SVX signals is missing for more than 120 ns. We are not 100% sure if we got all of them in our list. For instance, this morning I had to add 0x55. The list includes: 0x40, 0x41, 0x45, 0x46, 0x4c, 0x52, 0x53, 0x55, 0x62, 0x64, 0x7e, 0xc0, 0xc1, 0xc5, 0xc6, 0xcc. Let's see if we learn something...
<!-- Author: Ricardo, Mark, Ulrich -->


<!-- Date: Sat Jan 13 12:07:28 2007 -->
<pre>The AVDD current of e182 is about 1150 when the hardware limit is 1200. Let's make we check this ladder whenever we put e0b4 back in. 
</pre>
<!-- Author: Ricardo -->

<!-- Date: Sat Jan 13 18:31:24 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W11L4 
<li>problem n: 1877 
<li>Status: needs_work
<li>Type: AVDD2
<li>Action: Incident
<li>entered by: husemann 
<li>description is: This ladder shows an AVDD2 like behavior, but it is a layer-4 ladder. AVDD jumped up to around 1400 mA, and DVDD jumped down from 105 mA to 90 mA. Only the first two chips (8d and 8c) return good data, so we chopped the rest of the chip chain and set all ISEL bits to 0 for them. We did not change the bias ratio and Rdriver values, as we encountered problems in SVXDAQ when we tried. After that the AVDD current dropped to below 1200 mA. Just to be sure, we have also increased the trip limit to 1600 mA. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B0W11&layer=L4">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Jan 13 18:43:13 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ricardo
<li>Description of Problem: Ladder e0b4 was put back into the HDB. It was originally taken out because it failed to switch on due to AVDD overcurrent. After closer inspection it looks like this is a "jumper" failure, in which the AVDD current increases while the DVDD decreases. Also ran the ladder in SVXdaq and saw that chip #2 was given corrupted data. For the moment we decided to chop the chain to only the first two chips.   In addition we increased the AVDD hardware limit from 1400 to 1600. With the reduction on the number of chips the AVDD is now stable at about 1275, while the DVDD is stable at about 105. This needs further observation.<br>As we noticed before (see <a href='http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=12:07:28'>here</a>) e182 also had large AVDD current close to the hardware limit. In this case we also increased the trip limit  from 1200 to 1400. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=100">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Sat Jan 13 18:48:57 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10617&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10618  width='300' align='top'></a></td><td>Store 5176 is in with an initial lum of about 217E30. SVXMon and SVT both look OK. </td></tr></table>
<!-- Author: Ricardo -->

<!-- Date: Sat Jan 13 18:49:49 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB1W8L2 
<li>problem n: 4 
<li>Status: needs_observation
<li>Type: AVDD2
<li>Action: Currents_investigated
<li>entered by: husemann 
<li>description is: The AVDD current of this ladder has gone up again more than 1100 mA. We increased the hardware trip limit to 1400 mA, just to move it away from the current value. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B1W8&layer=L2">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Jan 13 19:06:28 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W11L4 
<li>problem n: 1877 
<li>Status: needs_work
<li>Type: AVDD2
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: Due to the new values of hardware trip limits we need to bump up the AVDD limits in IMON.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>991</td><td>??</td></tr>
<tr><td><b>Lower</b></td><td>980</td><td>980</td></tr>
<tr><td><b>Upper</b></td><td>1075</td><td>1290</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B0W11&layer=L4">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Jan 13 19:12:27 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB1W8L2 
<li>problem n: 4 
<li>Status: needs_observation
<li>Type: AVDD2
<li>Action: IMON_limits
<li>entered by: ricardo 
<li>description is: Increasing the IMON limits after increasing the hardware trip limits.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>607</td><td>607</td></tr>
<tr><td><b>Lower</b></td><td>580</td><td>950</td></tr>
<tr><td><b>Upper</b></td><td>825</td><td>1200</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B1W8&layer=L2">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Sat Jan 13 19:13:01 2007 -->
<h3>Changing CAEN LV Currents</h3> 
Another lesson in traps and pitfalls while changing CAEN LV current limits: We changed the limits in the HWDB via the cardEditor, and also via the cardEditor, we wrote the new file. For this file you just pull 64 kB worth of data out of the HWDB. This is done by an SQL script executed via "sqlplus". It turned out that this procedure took 15 minutes to complete and paralyzed the entire DB, claiming that the SQL script locked CHIPCHAINS. As a consequence, Run Control was not able to partition a non-silicon calibration run, which should not use CHIPCHAINS at all. Why does it take so long and why does it interfere with everybody else? We should have a look at the script.
<!-- Author: Ricardo & Ulrich -->


<!-- Date: Sat Jan 13 19:36:45 2007 -->
<h3>DMode Calibration Settings Changed</h3> 
I have changed the settings for the two DMode calibration configurations. Both DaqTest/SVX/SVXCAL_DPSON and DaqTest/SVX/SVXCAL_DPSOFF now have the settings we tested successfully last time (Interval0 = 25000, all L3 nodes included). The next DMode calibration is scheduled for tomorrow.
<!-- Author: Ulrich -->

<!-- Date: Sun Jan 14 16:01:54 2007 -->
<p><font size=+1> D-mode Calibration Job Started </font>
<p><pre>
Job id            : 457
Calibration type  : standard
Run period tag    : Jan_2007
DPS-off run       : 232064
DPS-on run        : 232063
Requested by      : Tatiana/Ulysses
Main process id   : 9774
Comment           : Ulysses actually did the calibs
</pre>
<!-- Author: CDF silicon D-mode calibration job server v1.4 -->


<!-- Date: Sun Jan 14 17:32:48 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ricardo
<li>Description of Problem: e2a0 refused to turn to stand-by. At store termination the crew turned the silicon to stand-by, but the ifix screen reported that SVX was not completely down, and the PSGUI reported that SVX was still on on the SVX ratio button. However, the window to the left reported the usual "success - Silicon brought to stand-by". The crew contacted MCR to delay the end of store. <br>The ace confirmed that all of PSGUI was yellow, and that the bias readings in imon were according to stand-by. <br>I gave instruction to the crew to proceed to end the store thinking that it was probably a ifix/PSGUI/svxinter problem.I systematically scanned the wedges of SVX and found out that e2a0 was still on and green on the PSgui (the ace missed it since layer 0 is barely a pixel on the gui). This was confirmed by imon which reported a voltage of 114. The CAEN refused to turned it off or stand-by so I hockerized the crate.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=101">Incident Report</a></ul>
<!-- Author: Incident DB -->
<!-- Comment: Sun Jan 14 17;46;19 comment by...Ricardo -->
I tried to hockerize from the IFIX page, and saw the "Hockerizing" label on the button, but nothing turned off in what I thought was an eternity. So I ran downstairs and did it manually. I'm not sure whether the PS in that funny state needed a second reset to work, or I lost the patient way too fast.



<!-- Date: Sun Jan 14 19:03:42 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10619&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10620  width='300' align='top'></a></td><td>Store  5177 in with an IL=250E30 and lostP <1100Hz. SVTmon and SVXmon look OK.</td></tr></table>
<!-- Author: Ricardo -->


<!-- Date: Sun Jan 14 19:11:02 2007 -->
<pre>It looks PS#4 is returning zero Bias1 current for f214. Will need hockerization when the store is dropped.  </pre>
<!-- Author: Ricardo -->

<!-- Date: Sun Jan 14 22:09:59 2007 -->
<p><font size=+1> D-mode Calibration Job Finished </font>
<p><pre>
This is a computer-generated silicon D-mode calibration status report.
Silicon D-mode calibration job 457 FAILED.

Job info
--------
Calibration type : standard
DPS-off run      : 232064
DPS-on run       : 232063
Requested by     : Tatiana/Ulysses
Submitted        : 2007-01-14 16:01:49
Completed        : 2007-01-14 22:09:58
Status           : FAILURE
Comment          : Ulysses actually did the calibs

Last processing stage reported : comparing to the reference calibration
Failure diagnostics :
The calibration failed the following reference comparison criteria:

             Cut Name    Sensor Set   Cut Value   Data Value
---------------------    ----------   ---------   ----------
dps_off_noise rel max    SVX_Z        0.025       0.029532  
     bad_chan rel max    SVX_PHI      0.3         0.550984  

Reference calibration runs are 220872 (DPS off) and 220869 (DPS on)
</pre>
<!-- Author: D-mode script run_calib_analysis.tcl -->
<!-- Comment: Mon Jan 15 08;19;37 comment by...Ulrich -->
Timo says that the calibration failed with respect to the reference run but looks very similar to the last calibration we took. He will discuss with Jason if this indicates that we need a new reference.
<!-- Date: Mon Jan 15 08:15:25 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ricardo
<li>Description of Problem: Ladder SB0W10 tripped (and kept tripping) on DVDD software protection. Instructed the ace to hockerize PS crate 1.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=102">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Mon Jan 15 08:18:05 2007 -->
All day yesterday and over night, we had intermittent ISL warnings in iFix. The reason was (again) the East ISL Portcard COT face pressure. The average value is at 12.8 psi or so, but once in a while, the pressure fluctuates up to the warning limit of 13.0 psi. As for a couple of other cooling lines before, I have bumped up the warning limit to 13.3 psi. This is still far enough away from the alarm limit of 13.5 psi and the PLC trip limit of 13.8 psi.
<!-- Author: Ulrich -->



<!-- Date: Mon Jan 15 08:30:17 2007 -->
<h3>Summary of the Week</h3>  
 
<b>Monday, January 8, 2007 </b> 
<ul>  
  <li> Store 5166 is in with an initial lum. of 240E30. Beam aborted 2.5 hrs later.</li> 
  <li> Downtime work; 
     <ul> 
        <li> AutoHocker installed.</li> 
        <li> b0svx04 power supply swaped.</li> 
        <li> SVXINTER mezzanine card installed.</li>  
        <li> Documentation updated.</li>  
     </ul> (see details <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=14:04:28'>here</a>) </li> 
</ul> 
 
<b>Tuesday, January 9, 2007</b> 
<ul> 
  <li>Store 5171 is in with initial lum. of 180E30.</li> 
  <li> SVX/L00 cooling trip, details are <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=07:49:11'>here</a>. </li> 
  <li> Bias of e424 very large. Ladder had no previous history.</li> 
  <li> Bias current of e294 still large and erratic. See <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=21:00:26'> here</a>. </li> 
  <li> Electron lens trip. Ace discovered it after 6 minutes, see <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=23:17:27'>here</a>.</li> 
</ul> 
 
 
<b>Wednesday, January 10, 2007</b> 
<ul> 
  <li> Store 5172 is in with an initial lum. of 230E30.</li> 
  <li> General cleanup of potential pinky bias in IMON.</li> 
  <li> First Auto-hockerization!, details <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=11:45:57'>here</a>. </li> 
</ul> 
 
 
<b>Thursday, January 11, 2007</b> 
<ul> 
  <li> ISL Chiller Problems. Decided to change to the spare chiller. Compromised ladder had to be taken out of the HDB. See the full <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=00:05:23'> story</a>. </li> 
  <li> SRC errors: L2A w/o L1A, and resonance.</li> 
  <li> b0fib02 failed config. SB0W5 had to be powercyled.</li> 
  <li> Access for 3 hours.</li> 
  <li> New vacuum pump VP-1A swapped for ISL this morning.</li> 
  <li> e3a4 <a href='http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=08:56:21'>tripped</a> on DVDD OverCurrent, a known feature when the North Muon wall is moved.</li> 
  <li> Masked out the readback of e502. </li> 
  <li> Quiet time studies. See the <a href='http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=14:20:11'>summary</a>. </li> 
  <li> Store 5174 in with an initial luminosity of 237E30.</li>  
  <li> Reformatter errors from the Silicon: FIB headers in b0fib07 were missing.</li>  
  <li> NO ACES SHOWED AT THE ACE MEETING</li> 
 
</ul> 
 
 
<b>Friday, January 12, 2007</b> 
<ul> 
  <li> More checks for potential pinkies in IMON.</li>  
  <li> b0svx00 and svx04 went red in RC while starting a new run. Evb of b0svx04 was resetted.</li> 
  <li> The Emergency Recovery Procedures were updated (see <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=10:16:55'>here</a> ) </li> 
  <li> e414 shows high Dvdd current. </li> 
  <li> Store 5175 is in with an initial lum. of 250E30</li> 
  <li> All of SVX tripped (second time this week), see <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=16:08:23'> here</a>. </li> 
  <li> S4W8 tripped on DVDD "Software protection trip". Needed hockerization. </li>  
  <li> e0b4 tripped on AVDD current. AVDD was large and hardware limits  too close to it. Taken out of the HDB. </li> 
  <li> Pinkies throughout the night. </li> 
  <li> Run 232026 with consecutive yellow ladders. Run was restarted. Details <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2007&action=view&page=37&frame=2&anchor=03:54:39.826'>here</a>.</li> 
</ul> 
 
 
<b>Saturday, January 13, 2007</b> 
<ul> 
  <li> Logic analyzer: new trigger implemented to understand SVX trips </li> 
  <li> e0b4 shows AVDD2 behavior. Chop the readout chain and increase the hardware limits.</li> 
  <li> e182 shows also large AVDD current. Increased the hardware limits.</li> 
  <li> Store 5176 with initial lum of 217E30.</li> 
  <li> DMode calibration settings changed, see <a href='http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=335&frame=2&anchor=19:36:45'>here</a>. </li> 
</ul> 
 
 
<b>Sunday, January 14, 2007</b> 
<ul> 
  <li> DMode calibration taken. </li> 
  <li> Store 5177 in with initial lum of 250E30. </li> 
</ul> 
 
 
<b>Monday, January 15, 2007</b> 
<ul> 
  <li> SB1W10 kept tripping on DVDD current via software protection trip. Hockerized crate 1.</li> 
</ul>  
<!-- Author: Ricardo -->