<!-- Title: 11:06:03  Mon Nov 20 2006 -->
<!-- Crewchief: Satyajit -->
<!-- Op1: Ulrich (SPL) -->
<!-- Op2: Jose (SPL) -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
My first MPC week! So can't be optimistic even if I am supposed to. 
<font color=green>Experts on high alert!</font>


<!-- Date: Mon Nov 20 11:42:47 2006 -->
<h3>Checks on PSMon.exe</h3> 
I have had a look at PSMon.exe to find out why we didn't get an iFix alarm yesterday when the bias voltage for e182 dropped to zero. As it turns out, that is not what the program is actually checking. The logic is rather like this: <em>For any bias line check the status bit of the channel.  If the channel does not indicate ON although the nominal bias voltage for the channel is greater than 0, then the PS_CAEN_OFF alarm shows up in iFix. </em> 
<p> 
It is straightforward to add another check, e.g. if the voltage readback is smaller than, say, 1V, and pull the iFix alarm also in this case.  
<!-- Author: Ulrich -->


<!-- Date: Mon Nov 20 11:47:09 2006 -->
The other problem is also easy to fix: When a communication failure is detected in more than one crate, the alarm string can be easily composed of a list of crates.
<!-- Author: Ulrich -->


<!-- Date: Mon Nov 20 13:54:07 2006 -->
<h3> Suggestions from the today's Meeting </h3> 
 
I have changed the functionality of the smile face in b0dasvx01 (bottom screen). Now executes the part of Ignacio's script that produces screenshots. You can ask ACEs to click on that and you will see the result on: 
 
<blockquote><a href=https://www-cdfonline.fnal.gov/cgi/internal/ops/svxii/screenshots.pl> Last Screenshot </a></blockquote> 
Links in silicon pages will be also updated to point to this page. As suggested in the meeting, I have modified IMON to look for ladders with "VBIAS < 0.5". It will display a window and a sono alarm asking to page silicon if any ladder with this problem is found. Hopefully, this will help us to catch faster problems with non biased ladders. IMON modifications are under test. 
 
  
<!-- Author: Jose -->
<!-- Comment: Mon Nov 20 14;54;42 comment by...Satyajit -->
<pre>Thanks Jose, this is very convenient!</pre>

<!-- Comment: Mon Nov 20 16;23;17 comment by...Ulrich -->
I have updated the link on the Silicon web page.
<!-- Date: Mon Nov 20 15:05:05 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W6L0 
<li>problem n: 1800 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: satyajit 
<li>description is: Increasing the bias upper limit.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>58.74</td><td>95.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>110.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B3W6&layer=L0">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Mon Nov 20 15:59:48 2006 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10332&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10332' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10333&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10333' width='300'></a></center>
<!-- Author: Ulrich -->
<!-- Comment: Mon Nov 20 16;01;27 comment by...Ulrich -->
<h3>SRC Error: Weird Glitch</h3>
Around 14:45, we had a weird glitch in almost all SRC lines, which cause an SRC error. River dancing? Here is the error log:
<pre>
48'59" b0svx06:Messenger:Nov 20, 2006 02:49:00 PM->SRC Fatal Error:Sl 6 L2A w/o L1A 
 -->  
 Additional Information: 

 Attention !!!. FERML_SRC_FATALITY ERROR !!! 
 
 SRC Fatal Error from b0svx06: Sl 6 L2A w/o L1A 
 
 - HALT the run                   
 - record any relevant information in e-log     
 - RECOVER - RUN  
</pre>
And indeed, L1 is about the only line which is not high at the point of the glitch.



<!-- Date: Mon Nov 20 19:00:22 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10338&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10338  width='300' align='top'></a></td><td> 
 Ladder e184 started on Friday to give Errors <a href= 
http://b0sili01.fnal.gov/cgi-bin/bitreporter/makeladder.pl?beginrun=225908&endrun=228405&ladder=S184>bit reporter</a>. Seems that one of the chips is missing since Friday/Saturday. 
 
<pre> 
Event Number: 256 
<font color="yellow"><b>10</b></font><font color="yellow"><b>50</b></font><font color="blue">0d</font><font color="cyan"><b>0b</b></font> <font color="yellow"><b>04</b></font><font color="yellow"><b>10</b></font><font color="yellow"><b>41</b></font><font color="yellow"><b>00</b></font> <font color="yellow"><b>00</b></font><font color="yellow"><b>00</b></font><font color="yellow"><b>00</b></font><font color="yellow"><b>00</b></font> <font color="yellow"><b>00</b></font><font color="yellow"><b>00</b></font><font color="yellow"><b>00</b></font><font color="yellow"><b>00</b></font> <font color="yellow"><b>00</b></font><font color="yellow"><b>00</b></font><font color="yellow"><b>00</b></font><font color="yellow"><b>00</b></font> <font color="yellow"><b>00</b></font><font color="yellow"><b>00</b></font><font color="yellow"><b>00</b></font><font color="yellow"><b>00</b></font> <font color="yellow"><b>08</b></font><font color="yellow"><b>16</b></font><font color="yellow"><b>08</b></font><font color="yellow"><b>16</b></font> <font color="yellow"><b>00</b></font><font color="yellow"><b>00</b></font><font color="yellow"><b>00</b></font><font color="yellow"><b>00</b></font>  
 
<font color="black">e1</font><font color="black">84</font><font color="green">3e</font><font color="cyan">fd</font> <font color="green"><b>8d</b></font><font color="black">29</font><font color="blue"><b>00</b></font><font color="black">f3</font> <font color="green"><b>8c</b></font><font color="black">29</font><font color="blue"><b>0c</b></font><font color="black">f3</font> <font color="blue"><b>0d</b></font><font color="black">09</font><font color="blue"><b>0e</b></font><font color="black">03</font> <font color="green"><b>8b</b></font><font color="black">29</font><font color="green"><b>8a</b></font><font color="black">29</font> <font color="blue"><b>28</b></font><font color="black">02</font><font color="blue"><b>29</b></font><font color="black">0c</font> <font color="blue"><b>2a</b></font><font color="black">03</font><font color="green"><b>89</b></font><font color="black">29</font> <font color="green"><b>88</b></font><font color="red"><b>00</b></font><font color="green"><b>87</b></font><font color="black">29</font> <font color="red">500ns</font> 
 
<font color="green"><b>a6</b></font><font color="black">29</font><font color="green"><b>a5</b></font><font color="black">29</font> <font color="green"><b>a4</b></font><font color="black">29</font><font color="green"><b>a3</b></font><font color="black">29</font> <font color="green"><b>a2</b></font><font color="black">29</font><font color="green"><b>a1</b></font><font color="black">29</font> <font color="green"><b>a0</b></font><font color="black">29</font><font color="blue"><b>53</b></font><font color="black">f3</font> <font color="blue"><b>54</b></font><font color="black">0a</font><font color="blue"><b>55</b></font><font color="black">1b</font> <font color="blue"><b>56</b></font><font color="black">01</font><font color="blue"><b>7f</b></font><font color="black">08</font> <font color="yellow"><b>c1</b></font><font color="yellow"><b>c1</b></font><font color="yellow">00</font><font color="yellow">00</font>          <font color="green">560ns</font> 
</pre></td></tr></table>
<!-- Author: Jose -->


<!-- Date: Mon Nov 20 22:39:06 2006 -->
Was paged by the shift crew around 8:50pm when TevMon 
<a href= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=878&frame=2&anchor=20:52:15.248'> 
reported</a> that the electron lens is experiencing problem. The crew stopped the run and took silicon to standby. They consulted 
with MCR and got them to work on it. I told them to page me back 
when electron lens is active and the losses are under control. 
The shift crew called me back around 10:05pm that electron lens 
is back in operation and that the losses are under control. 
After checking the loss plots I asked them to 
<a href= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=878&frame=2&anchor=22:18:02.161'> 
include silicon</a>.  
<!-- Author: Satyajit -->


<!-- Date: Mon Nov 20 23:02:26 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10341&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10342  width='300' align='top'></a></td><td><h4>Strange happenings!</h4> 
I accessed the IMON/PSGui snapshot page and see a pinky on 
the cell S_BIAS_B3W6L0 as shown here. The timestamp is 
Mon Nov 20 22:50:08 CST 2006, which is what the time is now. 
However, the monitoring ACE doesn't see it! How is this 
possible? 
</td></tr></table>
<!-- Author: Satyajit -->
<!-- Comment: Tue Nov 21 06;56;59 comment by...Jose -->
<pre>The snapshot page displays the last image saved. The time stamp shows the "present time" that may not coincide with snapshot. To update images you have to ask the ace to click in the smile face or run Ignacio's script.</pre>
<!-- Comment: Tue Nov 21 10;38;57 comment by...Ignaico -->
Note that the only smily face button looking at the web page is the one on the lower screen (b0dasvx01).
<!-- Date: Tue Nov 21 01:07:40 2006 -->
Was paged a couple of times about a  
<a href= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=879&frame=2&anchor=00:18:30.871'> 
transient PSM alarm</a> 
which clears up quickly w/o intervention. There is no  
corresponding entry in the PSGui logs. I assume this is not  
something to worry about unless it happens frequently and/or 
persists. Have instructed the ACE to keep logging the occurances 
and page me if they happen several times.
<!-- Author: Satyajit -->



<!-- Date: Tue Nov 21 07:59:01 2006 -->
<h3>Fault Relay Power Supply Replacement</h3> 
This morning Roberto and Dervin swapped in a new power supply for the fault relay of the cooling controls. The old supply broke during the power glitch last Thursday, and we were running on a bench supply since then. Here is what was done: 
<ul> 
<li>After the detector was brought into "end of store", we switched it off completely. In the course of that, a couple of the inner layer SVX ladders tripped on DVDD overcurrent.</li> 
<li>We silenced the sono-alarms and shut down flow to the detector.</li> 
<li>We changed the SVX temperature set point to -6&deg;C to keep the COT warm.</li>
<li>Roberto and Dervin swapped the power supply, and we confirmed that the cooling crash button worked properly and tripped the fault relay. Note that the <em>FIB racks did not trip</em> during this exercise. However, they tripped last Thursday. I think this was caused by the aces hitting the crash button (although I told them there is no need to do so).</li>
<li>We re-established flow to the detector. At this point, an old problem re-occured: the <em>hit-me-baby-one-more-time</em> flow switch. This is a flow switch on the ISL chiller which had gone stuck in the past. This caused the chiller to think it is on low flow, and the compressor would not kick in. We replaced the old switch with a new one during the shutdown and did not encounter problems so far. Now I had to hit the switch again to start the compressor.</li>
<li>We requested a resin exchange for SVX and ISL, as the SVX coolant conductivity is above its alarm limit after the power glitch. We'll wait for this work to be finished, before we start to bring the detector to STANDBY.</li> 
</ul> 
Thanks a lot, Roberto and Dervin!
<!-- Author: Jose & Ulrich -->



<!-- Date: Tue Nov 21 10:48:18 2006 -->
<h3>Resin exchange</h3> 
KC exchanged the resin for both SVX and ISL. The detector was still off at that time. The ISL power supply enables were dropped due to low flow during the procedure.
<!-- Author: Ulrich -->


<!-- Date: Tue Nov 21 11:10:32 2006 -->
<table><tr><td><a href=/elog/notebooks/silicon/Tue_Nov_21_2006_111025.ps align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10346  align='top'></a></td><td>This ladder started giving high rate of CAP_ID errors on 11/18/06. Jose also noted <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=328&frame=2&anchor=19:00:22'>here</a> that chip88 didn't return any data. We studied this ladder in SVXDAQ. After 5K events, no errors were observed in both RA and NN modes. </td></tr></table>
<!-- Author: sasha/jeannine -->

<!-- Date: Tue Nov 21 11:17:36 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB1W8L4 
<li>problem n: 1801 
<li>Status: needs_observation
<li>Type: Chip_capid
<li>Action: Incident
<li>entered by: sasha 
<li>description is: This ladder started giving high rate (~100%) of CAP_ID errors on 11/18/06. At the same time, chip88 stopped returning any data. Jeannine and I studied this ladder during quiet time, and we could not reproduce the error in SVXDAQ. The ladder was OK after 5K events in NN and RA modes. Also, all chips seemed to be OK <a href= 'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=328&frame=2&anchor=11:10:32'> see here</a>. We need to keep an eye on this ladder for some time.    
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B1W8&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Tue Nov 21 14:17:41 2006 -->
<h3>Test Version of PSMon.exe</h3> 
Our power supply monitoring program <tt>PSMon.exe</tt> runs on SVXIICON. The purpose of the program is the following: 
<ul> 
<li>Read the voltages and currents from SVXINTER via SmartSockets and publish them to the iFix database.</li> 
<li>Process requests from iFix High Voltage Summary page (Global ON-STANDBY-OFF).</li> 
</ul> 
In the new version I added some functionality: 
<ul> 
<li>In case of communication failures from multiple crates: append all crate numbers to the iFix string published on the Global Alarms page.</li> 
<li>For a ladder with a nominal voltage greater than 5V, make sure the voltage readback is greater than 0.1V. Otherwise, the same PS_CAEN_OFF iFix alarm as for spontaneous switch-offs is issued.</li> 
<li>The program has hooks for the new AutoHocker functions and is linked against the National Instruments libraries. The functions are called, but all communication with the digital output card (which is not yet installed in SVXIICON) is disabled internally. It will still be useful for debugging, as a log file is written for every hockerization decision.</li>   
</ul> 
Originally, I also wanted to send mail to the alphanumeric pager, but I ran into problems with "managed" vs. "unmanaged" code in M$ Visual Studio. The mechanism to allow Hockerization from iFix is still missing. 
<p> 
The plan is to run the new version during working hours and fall back to the older version over night.
<!-- Author: Ulrich -->
<!-- Comment: Tue Nov 21 19;53;41 comment by...Ulrich -->
The test version ran fine so far, but there weren't any communication failures. I rolled back to the default version for the night. More testing tomorrow.
<!-- Date: Tue Nov 21 14:48:16 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10347&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10348  width='300' align='top'></a></td><td><h3>Store 5086 Init. lumi 226e^30</h3> 
Silicon was included from the beginning after losses became 
stable. 
</td></tr></table>
<!-- Author: Satyajit -->
<!-- Comment: Tue Nov 21 15;09;29 comment by...Satyajit -->
Soon after the scraping was over the LOSTP started climbing
steadily from ~ 1 to 2 kHZ. After a (seemingly) tune attempt it
settled and started ramping down slowly. This is the time we 
decided to have silicon integrated.

<!-- Date: Tue Nov 21 15:05:08 2006 -->
There were about 5 pinkies at the beginning of this store which 
cleared quickly, except for the bias voltage on f140 which started 
off at a pretty high value of 1274 and decaying slowly. This should 
get back to its max. value of 1210 within half an hour.
<!-- Author: Satyajit -->



<!-- Date: Tue Nov 21 16:00:33 2006 -->
<h3>Crate 11 Spontaneous Switch-Off</h3> 
CAEN Crate 11 just switched itself off spontaneous. The shift crew pulled me out of the Silicon office, and I recovered the corresponding wedges. No problems.
<!-- Author: Ulrich -->


<!-- Date: Tue Nov 21 18:57:54 2006 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10349&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10349' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10350&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10350' width='300'></a></center>
<!-- Author: Jose -->
<!-- Comment: Tue Nov 21 19;00;28 comment by...Jose -->
Event Sizes before and after the upgrade to 4 VRB crates for the ISL+L00. You can find more information in Conor's <a href=http://mit.fnal.gov/~conor/cdf/evb/evb_fragment_sizes.html>page</a>. 


<!-- Date: Tue Nov 21 19:08:49 2006 -->
<pre>TeVMon had a problem of refresh. The Ace put silicon to standby and then came to ask. The ACNET plots were updating and everything looked fine. Silicon was put in back for data taking. "Problem" was solved restarting TeVMon. </pre>
<!-- Author: Jose -->



<!-- Date: Wed Nov 22 02:36:07 2006 -->
<h3>Another spontaneous PS crate turnoff</h3> 
Was paged around 1:22 am about a PS trip issue. On checking the 
IMON/PSGui snapshot I saw several ISL and L00 wedges are RED which 
are indicative of a PS crate turn off. With the ACE I went through 
a normal trip recovery procedure. After failing to recover L00 west 
W0 in one attempt I got Ulrich online. He had to use a few tricks 
to get the wedges back up.
<!-- Author: Satyajit, Ulrich -->
<!-- Comment: Wed Nov 22 02;42;54 comment by...Ulrich -->
Boy, that was a tough one. Luckily, the ace (Nate) was up to speed, so it was easy to walk him through the procedure of turning on things one by one and issuing CONFIG in between a.s.a.p. We tried to switch off the Port Card and turn on the wegde as a whole. This ended in a trip on DVDD3 overcurrent. Next we tried to switch the wedge on ladder by ladder. For L0 and L3 we had to try twice before they would stay on.

<!-- Comment: Wed Nov 22 02;45;03 comment by...Ulrich -->
The crate in question was #7.


<!-- Comment: Wed Nov 22 09;42;28 comment by...sasha -->
<pre>This crate misbehaved last week too. I remember at least two cases.
Is there anything wrong with it?</pre>
<!-- Comment: Wed Nov 22 10;29;41 comment by...Ulrich -->
Hard to say. The crate is in NW Bot Top. Maybe the radiation at that spot is higher than elsewhere (remember that the corresponding FIB crate power supply has been moved to the beam on the side). Jeannine should have more statistics on this. Any comment from the old guys? 
<!-- Comment: Wed Nov 22 10;32;41 comment by...Ankush (unqualified old guy) -->
<p>That&#39s just plain old stats for you....it&#39s happened a few times where it appears one crate has more problems for a couple of weeks and then it's quiet again.</p>
<!-- Date: Wed Nov 22 02:48:23 2006 -->
<h3>SVX B1W10 Corrupted Readback</h3> 
At around the same time as the spontaneous switch-off, we also had a problem with one of the SVX wedges. All current readbacks were zero. Also here I could rely on the monitoring ace to recover things. First we tried to power-cycle the wedge - no success. Then I asked him to hockerize crate #1. After he recovered all wedges in that crate, the current readings were back to normal.  So from the Silicon side we are good to go, but data-taking is held off by L2 problems.
<!-- Author: Satyajit & Ulrich -->


<!-- Date: Wed Nov 22 09:33:38 2006 -->
<h3>b0svx01 got stuck</h3> 
The shift crew had to start a new run because b0svx01 got stuck and would not sync up with the rest of the partition after a reboot. After that everything was fine again.
<!-- Author: Ulrich -->





<!-- Date: Wed Nov 22 13:15:27 2006 -->
Two SRC errors within the last 90 minutes. The first one was a real resonance conditions, for the second, the LA was not yet re-armed. The error log does not show anything, so it could be just a short clock glitch due to river-dancing.
<!-- Author: Ulrich -->


<!-- Date: Wed Nov 22 15:13:55 2006 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10351&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10351' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10352&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10352' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10354&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10354' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10355&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10355' width='300'></a></center>
<!-- Author: Ulrich, Jose, Ignacio & Mark -->
<!-- Comment: Wed Nov 22 15;23;12 comment by...Ulrich, Jose, Ignacio & Mark -->
It was noticed by the shift crew that SVXMon was not seeing any data for SVX in run 228581.  When we dumped a raw event, however, we did see data from SVX in the event.  Upon further investigation we found that the code that reads out the SIXD bank was failing with the following error:

<pre>
%ERLOG-e SIXD_StorableBank read aborted:
         /home/cdfsoft/dist/packages/TrackingObjects/V00-02-05/src/TRYRun2SiStripSet_nndata.cc
        : 142  HDI header invalid. header =  257 GlobalLibraryLogger
         TRYRun2SiStripSet::readSIXDBank_nndata () 22-Nov-2006 13:53:09 CST
         run = 228581 event = 1021688
</pre>

The problem was fixed by starting a new run.  We are now investigating the data from this run to see why exactly this piece of code was failing.


<!-- Comment: Wed Nov 22 18;01;29 comment by...Mark & Jose -->
We had a look to the raw data (a lot of fun!!) for events in the run with problems. There is a piece of data that it is clearly bad:

<h3> Good Event </h3>
<pre>
      300: 00002e38 00002e54 00002e62 00002e74
      304: 00002ea0 00002eb8 00002f0c 00002f16
      308: 00002f22 00002f3e 00002f9a 00002fb4
      312: 00003002 0000300e 0000300e 0000301e
      316: 0000304a 00003056 0000305e 00003088
      320: 0000309e 00003104 00003104 00003104
      324: 00003104 00003104 00003104 0000310e
      328: 0000311a 0000314a 00003162 00003178
      332: 0000318e 0000319a 000031aa 000031c0
      336: 000031dc 000031e6 000031f6 0000320c
      340: 0000321c 00003230 00003242 00003262
      344: 00003278 00003292 000032ac 000032c2
      348: 000032e0 000032fc 00003314 0000332a
      352: 0000337c 0000338a 0000339a 000033ac
      356: 000033c8 000033d4 0000341c 00003446
      360: 0000345e 00003474 00080000 47fee000
</pre>

<h3> Bad Event </h3>

<pre>
      300: 000042be 00004307 0000432c 0000434d
      304: 00004385 0000439c 000043fd 00004449
      308: 0000446b 000044b5 000044d4 000044ff
      312: 0000453d 00004587 000045e9 00004628
      316: 0000467d 000046a5 000046b7 000046db
      320: 00004716 0000474e 00004779 0000479f
      324: 000047cf 000047d6 000047e1 000047f0
      328: 00004804 0000482e 01010000 03030202
      332: 05050404 07070606 09090808 0b0b0a0a
      336: 0d0d0c0c 0f0f0e0e 11111010 13131212
      340: 15151414 17171616 19191818 1b1b1a1a
      344: 1d1d1c1c 1f1f1e1e 21212020 23232222
      348: 25252424 27272626 29292828 2b2b2a2a
      352: 2d2d2c2c 2f2f2e2e 31313030 33333232
      356: 35353434 37373636 39393838 3b3b3a3a
      360: 3d3d3c3c 3f3f3e3e 41414040 43434242
      364: 45454444 47474646 49494848 4b4b4a4a
      368: 4d4d4c4c 4f4f4e4e 51515050 53535252
      372: 55555454 57575656 59595858 5b5b5a5a
      376: 5d5d5c5c 5f5f5e5e 61616060 63636262
      380: 65656464 67676666 69696868 6b6b6a6a
      384: 6d6d6c6c 6f6f6e6e 71717070 73737272
      388: 75757474 77777676 79797878 7b7b7a7a
      392: 7d7d7c7c 7f7f7e7e e0007f7f 832888fd
</pre>

 The problem is at the begining of the SVX banks. It could be a coincidence (data from the first VRB is corrupted) or more likely to a problem with the EVB (?).

<!-- Date: Wed Nov 22 15:28:32 2006 -->
When we started a new run e052 was not returning any data.  An HRR fixed the problem.
<!-- Author: Ulrich, Jose, Ignacio,Mark -->


<!-- Date: Wed Nov 22 17:19:02 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10356&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10356  width='300' align='top'></a></td><td>Here are the event sizes I see for run 228429, which is the run for that <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=328&frame=2&anchor=18:57:54'> 
Conor looked at.</a>  Seems to be in good agreement.</td></tr></table>
<!-- Author: Mark -->
<!-- Comment: Wed Nov 22 19;06;34 comment by...Ignacio -->
 It looks the same. I maybe wrong, but it seems  our ISL/L00 are still the largest...
<!-- Comment: Thu Nov 23 06;00;45 comment by...Marcel (Sitting in the rain in England) -->
Well, that is the case by design. A standard SVX VRB controls ten ladders with 88 chips and there are always 6 VRBs per crate ( a perfect symmetry to  quote SCN) <br>
An ISL VRB has also ten ladders but with 160 chips. So even only reading chip ids, the ISL one will be larger. The basic idea was that you keep the event size reasonable low, if you take the lower occupancy of the ISL into account. <br>
On top you have the layer 00, which is in readall anyway. <br>
The only way, the SVX can ever beat the ISL/L00 combo concerning event size is if the occupancy in the SVX becomes extremly high. Which Marks plot seem to indicate 350E30
<!-- Comment: Thu Nov 23 09;35;58 comment by...Mark -->
It is true that ISL/L00 crates are still probably the largest in CDF below ~300e30, but they are only on the order of 10% larger than the other CDF crates.  It may be that there is some room for improvement with threshold optimization.  It will be interesting to see what the trigger experts have to say about store 5086 which came in at 225e30.  It looks like the L2A rate briefly peaked at ~900 Hz but was below 800 Hz for most of the time at the beginning of the store.  They may need to change the trigger table to really push the L2A rate.
<!-- Comment: Thu Nov 23 10;01;32 comment by...Ignacio -->
In my comment above I meant the largest in CDF. We were speculating that we may be have become second after the b0svx09 introduction.
<!-- Comment: Thu Nov 23 12;10;54 comment by...Marcel -->
we'll probably always be the largest in CDF, given that we own 3/4 of the readout channels :-)

<!-- Comment: Thu Nov 23 12;11;50 comment by...Marcel -->
800 Hz seems low anyway, shouldn't be that low ...
<!-- Comment: Sat Nov 25 13;18;15 comment by...Ankush (sitting in sunny NJ) -->
<p>It's quite likely a new trigger table will be needed to exploit the bandwidth properly.  The 800 Hz could have been limited to keep the deadtime low.</p>
<!-- Date: Wed Nov 22 18:59:20 2006 -->
I have updated the web  
<a href= 
'https://www-cdfonline.fnal.gov/internal/ops/svxii/dump_raw/dump_raw.html'> 
web page </a> that dumps the raw data being taken at the moment and 
allows you to parse it. Now you can fill in the ladder range you what to display while the program it's running, which is a bit  more convenient.
<!-- Author: Ignacio -->


<!-- Date: Wed Nov 22 20:31:03 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB2W0L0 
<li>problem n: 1802 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: jgarcian 
<li>description is: Increasing limits for this ladder.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>56.2357</td><td>95</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100</td><td>110</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B2W0&layer=L0">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Wed Nov 22 22:06:42 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10357&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10357  width='300' align='top'></a></td><td>Another resonance in SVX.</td></tr></table>
<!-- Author: Ulrich -->


<!-- Date: Thu Nov 23 02:41:30 2006 -->
Came in towards the end of shotsetup when pbars had finished 
loading. To my dismay there was a TeV quench at flattop. Although 
we were at standby I asked the monitoring ACE to complete the 
post-quench checkout anyway. The 
<a href= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=885&frame=2&anchor=02:19:57.514'> 
BLM loss plot</a> 
shows that this was a clean abort. The whole detector passed 
through a COLDSTART just fine and there were no pinkies. The 
current prediction is to have another shotsetup <B><I>in a few 
hours</I></B>. I would prefer watching it from home.
<!-- Author: Satyajit -->
<!-- Comment: Thu Nov 23 17;53;54 comment by...Satyajit -->
I guessed this right ;-) MCR now expects to begin shotsetup around
9pm. This makes <b><i>in a few hours</i></b> = 19 hrs!


<!-- Date: Fri Nov 24 05:13:32 2006 -->
Got paged aound 4 am that final protons have been loaded. Came in 
when scraping was just finished. The losses were all stable, well 
below 5 kHz. While bringing silicon to ON, ISL B0W3, B1W3-5 and 
B2W0 tripped off. They were recovered quickly wedge-by-wedge and 
the data taking was resumed.
<!-- Author: Satyajit -->
<!-- Comment: Fri Nov 24 07;28;48 comment by...Ulrich -->
That was a spontaneous switch-off of CAEN crate #3.
<!-- Date: Fri Nov 24 05:30:59 2006 -->
At the beginning of the store about 8 bias pinkies were created. 
All except SB5W0L2 cleared off quickly. The latter is quite  
close to but not within its limits. Would watch it for some time 
before taking any action.
<!-- Author: Satyajit -->
<!-- Comment: Fri Nov 24 05;42;35 comment by...Satyajit -->
<pre>SB5W0L2 bias pinky cleared.</pre>
<!-- Date: Fri Nov 24 05:40:50 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10358&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10359  width='300' align='top'></a></td><td><h3>Store 5089 Init. Lumi. 190.62e^30</h3> 
Silicon was integrated from the beginning. Run resumed after 
5 ISL wedges were recovered from trips. 
</td></tr></table>
<!-- Author: Satyajit -->



<!-- Date: Fri Nov 24 06:20:28 2006 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10362&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10363' ></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10364&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10364' width='300'></a></center>
<!-- Author: Satyajit -->
<!-- Comment: Fri Nov 24 06;22;42 comment by...Satyajit -->
The DVDD current of SB3W9L3 does show a downward excursion over
time. Despite the high variance (26.8) the lower bound is out of
range. It would be better to lower the limit from 90.0 to 80.0
till some corrective action is taken.



<!-- Date: Fri Nov 24 12:08:15 2006 -->
I am not sure if they did the correct thing... hopefully yes. 
 
<pre> 
Fri Nov 24 07:18:08 
During the HRR, b0Fib03 went red:  
following the flow chart I did a ping then a Reset/Shepherd, which recovered the fib crate.  
</pre>  
<!-- Author: Jose -->


<!-- Date: Sat Nov 25 07:26:25 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10365&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10366  width='300' align='top'></a></td><td>SVTSpyMon for store #5092, initial luminosity 217E30</td></tr></table>
<!-- Author: Tuula (on ACE shift) -->
<!-- Comment: Sat Nov 25 07;32;17 comment by...Tuula -->
<pre>No yellow cells in SVXMon.</pre>

<!-- Comment: Sat Nov 25 10;38;37 comment by...Satyajit -->
<pre>Thanks Tuula!</pre>


<!-- Date: Sat Nov 25 14:11:54 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB2W5L1 
<li>problem n: 1803 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: Bumping up IMON limit for bias current - just the regular increase.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>56.6033</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>130.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B2W5&layer=L1">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 14:15:41 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB5W0L2 
<li>problem n: 1756 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: Whenever we did not run for a couple of hours, this Micron ladder shows the typical turnon curve. Otherwise the ladder is behaving normal. Bumping up limit to avoid pinkies during this time.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>100</td><td>105</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>110.0</td><td>150.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B5W0&layer=L2">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 14:18:07 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: IB0W0L3 
<li>problem n: 1739 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: Now adjusting BIAS0 limit as well.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>66.4446</td><td>101.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>120.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B0W0&layer=L3">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 14:20:50 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: IB0W2L3 
<li>problem n: 1774 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: Now also BIAS1 needs to be bumped up.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>61.1405</td><td>101.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>120.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B0W2&layer=L3">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 14:25:00 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: IB2W1L3 
<li>problem n: 1757 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: Also the BIAS1 current limit needs to be bumped up (already yellow).<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>69.6362</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>150.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B2W1&layer=L3">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 14:27:05 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: IB3W1L2 
<li>problem n: 1746 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: Also adjusting BIAS1 current limit.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>55.9675</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>150.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B3W1&layer=L2">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 14:29:09 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: IB3W0L0 
<li>problem n: 1717 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: BIAS1 current runs into limit again. Bumping it up<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>90.2771</td><td>124.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>125.0</td><td>150.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B3W0&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 14:39:30 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: IB3W0L1 
<li>problem n: 1537 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: Another pre-emptive strike on the current limit for BIAS1.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>107</td><td>140.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>140.0</td><td>180.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B3W0&layer=L1">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 14:47:52 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: IB3W0L2 
<li>problem n: 1531 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: Pre-emptive strike on the BIAS0 current for this ladder. We should keep an eye on it however.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>132.0</td><td>140.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>140.0</td><td>180.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B3W0&layer=L2">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 14:54:05 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: IB5W3L3 
<li>problem n: 1804 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is:  The BIAS0 current for this ladder just slipped over the limit. Bumping up the limit.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>60.0355</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>120.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B5W3&layer=L3">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 14:56:09 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: IB5W2L3 
<li>problem n: 1805 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: Bumping up IMON limit on BIAS0 line (just slipped over the limit).<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>49.2424</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>120.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B5W2&layer=L3">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Sat Nov 25 15:22:28 2006 -->
<h3>Pre-emptive Strike on Bias Current Limits</h3> 
During the last few days, a couple of bias pinkies showed up in SVX and ISL. In a pre-emptive strike, I bumped up a bunch of limits, specifically for  
<ul> 
<li>e251 
<li> e502 
<li> f003 BIAS0 
<li> f023 BIAS1 
<li> f213 BIAS1 
<li> f300 BIAS1 
<li> f301 BIAS1 
<li> f302 BIAS0 
<li> f312 BIAS1 
<li> f523 BIAS0 
<li> f533 BIAS0 
</ul> 
While I "blindly" bumped up these limits, I think it is worthwhile revisiting Ignacio's and Jeannine's study of bias currents in the light of the ISL cooling problem.
<!-- Author: Ulrich -->


<!-- Date: Sat Nov 25 15:22:42 2006 -->
<h3>Strange Jump in SVX Bias and DVDD Currents</h3> 
While I was working on this, we had another of these strange collective jumps in the bias and DVDD currents. This happened just for SVX, but not for L00 and ISL. We checked ladders from different CAEN supplies, and the effect is all over the place. Strangely, ISL ladders in the same CAEN supply do not show the effect. Currently, the following SVX bias lines are pink: e011, e152, e250, e310, e461. I will bump up the current limits for those as well. Any idea what could cause the jump?
<!-- Author: Ulrich -->
<!-- Comment: Sat Nov 25 15;23;50 comment by...Ulrich -->
I will also bump up the yellows: e260, e310, and e462.

<!-- Comment: Sat Nov 25 15;47;35 comment by...Ulrich -->
It may be worth mentioning that almost all of channels for which I bumped up the limit had still their initial current limits of 100 &mu;A. The exceptions were e260 and e4a2 (which were only yellow anyway).
<!-- Date: Sat Nov 25 15:25:55 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W2L1 
<li>problem n: 1806 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: The bias current went above the limit during a collective jump of all SVX bias and DVDD currents. Bumping up the limit.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>60.7698</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>125.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B0W2&layer=L1">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 15:27:48 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB1W5L2 
<li>problem n: 1807 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: The bias current went above the limit during a collective jump of all SVX bias and DVDD currents. Bumping up the limit.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>73.2187</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>125.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B1W5&layer=L2">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 15:29:45 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB2W5L0 
<li>problem n: 1808 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: The bias current went above the limit during a collective jump of all SVX bias and DVDD currents. Bumping up the limit.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>57.0615</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>125.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B2W5&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 15:31:50 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB2W6L0 
<li>problem n: 1723 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: The bias current went above the limit during a collective jump of all SVX bias and DVDD currents. Bumping up the limit.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>85.2832</td><td>115.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>115.0</td><td>150.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B2W6&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 15:33:54 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W1L0 
<li>problem n: 1809 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: The bias current went above the limit during a collective jump of all SVX bias and DVDD currents. Bumping up the limit.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>57.7307</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>125.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B3W1&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 15:38:17 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W5L0 
<li>problem n: 1810 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: The bias current went above the limit during a collective jump of all SVX bias and DVDD currents. Bumping up the limit.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>59.6254</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>125.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B3W5&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 15:40:08 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB4W0L0 
<li>problem n: 1811 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: The bias current went above the limit during a collective jump of all SVX bias and DVDD currents. Bumping up the limit.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>59.6497</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>125.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B4W0&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 15:44:16 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB4W6L1 
<li>problem n: 1812 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: The bias current went above the limit during a collective jump of all SVX bias and DVDD currents. Bumping up the limit.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>65.7739</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>125.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B4W6&layer=L1">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Sat Nov 25 15:45:52 2006 -->
Entered in Silicon database: <ul>
<li>Ladder: SB4W10L2 
<li>problem n: 1703 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: husemann 
<li>description is: The bias current went above the limit during a collective jump of all SVX bias and DVDD currents. Bumping up the limit.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>134.0</td><td>145.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>144.0</td><td>180.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B4W10&layer=L2">report</a></ul>
<!-- Author: Problem DB -->




<!-- Date: Sat Nov 25 20:05:21 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10368&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10368  width='300' align='top'></a></td><td>Was paged by the monitoring ACE that e114's AVDD current 
<a href= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=893&frame=2&anchor=19:20:30.168'> 
remains pink</a> 
even after unmarking twice, followed by an HRR. I see that this 
ladder has a history of losing ~40 mA AVDD current from time to 
time. From the (current plotter) history plot I see that it took 
a similar plunge about a month ago and recovered in a few days. So we would keep watching it for now. 
</td></tr></table>
<!-- Author: Satyajit -->
<!-- Comment: Sat Nov 25 20;14;43 comment by...Ulrich -->
That's right. Two other checks before we start worrying about the ladder: 
<ol>
<li>40 mA is less than you expect for an entire chip to drop out (that's more like 80 mA)</li>
<li>I have checked the raw data: all chips look ok.</li>
</ol>

<!-- Comment: Sat Nov 25 20;41;52 comment by...Satyajit -->
<pre>Thanks, Ulrich! Is there a tool to plot ADC or noise of a ladder vs channels for a run?

</pre>

<!-- Comment: Sat Nov 25 21;03;38 comment by...Ulrich -->
Yes, you can use Badgett's RootFileBrowser, available from the <a href="http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=228688">Run Summary page of a given run</a>. This tools allows you to browse through all histograms which SVXMon produces, including ADCmon and the SVXMon Strip Plots (mean charge and occupancy).


<!-- Date: Sun Nov 26 00:51:23 2006 -->
There were several yellow cells in SVXMON, <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2006&action=view&page=894&frame=2&anchor=00:36:08.759'> 
see</a>. First time I saw everything was yellow in Cell Id SVX Chip Status Map.  
 
I thought it was SVXMON problem, so tryed HRR and restarting SVXMON. It didn't help. Then I asked to restart the run (we were also asked to take a beam-on calibration some time during the shift). It helped.
<!-- Author: Tuula -->
<!-- Comment: Sun Nov 26 01;53;58 comment by...Satyajit -->
Was paged around 1:15am by the CO whether to mark Run 228691 bad
for silicon. This Run lasted for little less than an hour. I
checked the strip level plots for SVX in SVXMon and found them
empty. I suggested the CO to mark the Run bad for silicon.


<!-- Comment: Sun Nov 26 03;02;36 comment by...Tuula -->
<pre>I noticed Cell Id SVX Chip Status Map was all yellow again. We restarted the run, and now it is OK.</pre>

<!-- Comment: Sun Nov 26 08;10;38 comment by...Ulrich -->
This has happened before on <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=328&frame=2&anchor=15:13:55'>
Wednesday</a>. Mark and Jose found that the SIXD bank was corrupted before data from the first ladder arrives. Also then, restarting the run helped. Now we have to find out what the reason for this corruption is.

<!-- Comment: Sun Nov 26 09;51;44 comment by...Ulrich -->
I have wrote mail to the old guys to ask for some advice on the matter. Also I posted a yellow shift crew reminder to check SVXMon whenever a new run is started into the main e-log.


<!-- Date: Sun Nov 26 10:08:16 2006 -->
ISL cooling update: after the power glitch the temperature in the compromised cooling line CCE1.1 was 4-5&deg;C higher than before. The situation was slightly better after we tripped to cooling to exchange the fault relay power supply. This morning between 6:15 and 6:20, the temperature dropped by 3&deg;C, and is now at 15.68&deg;C. Bubbles...
<!-- Author: Ulrich -->


<!-- Date: Sun Nov 26 15:13:48 2006 -->
<h3>More changes to PSMon.exe</h3> 
I made use of the quiet time after the Tevatron lost the store this morning to add more stuff to PSMon.exe. The last round of changes took place <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=328&frame=2&anchor=14:17:41'> 
last  
Tuesday</a>. Here is a list of the changes: 
<ul> 
<li>Added code to detect spontaneous switch-offs: the code simply counts the number of wedges in a given CAEN crate which are completely off (AVDD, DVVV, BIAS, PortCard). If at least one wedge which is supposed to be on is off, the PS-CAEN-OFF iFix alarm is triggered, and the alarm string tells the shift crew that crate <i>xy</i> has switched off spontaneously. In addtion, the switch-off is appended to a log file on SVXIICON. I hope this helps them detecting this failure mode quicker.</li> 
<li>Added code to process Hockerization requests from iFix: A while ago I had created two new iFix tags per CAEN crate. One indicates that the crate lost communication and can be used to trigger iFix animations. The other one is a flag that indicates a Hockerization request for a given crate. PSMon.exe picks up this request, calls the Hockerization function and resets the flag.</li> 
</ul> 
After the changes had been successfully tested with dummy code, I went ahead and switched off wedges in the real detector to see if this is correctly piced up by the new code. After some debugging, this was actually the case. I have also switched off all wedges in CAEN crate 8 and pulled the CAENet cable on the first floor to simulate a communication failure. Also this is picked up correctly, and a (dummy) Hockerization is performed. 
<p> 
While working on this, I have also found a copy-and-paste bug in the L00 iFix page. The page wasn't showing the L00 PS-CAEN-OFF alarm string but rather the PS-ALARM string. I have also worked ont he iFix page to manually hockerize crates. I have implemented the full functionality for the crate 8. Now it is just a matter of transfering everything to 15 other crates. 
<p> 
Currently I have the new version of the code running, but I will roll back to the old version before I leave the lab.
<!-- Author: Ulrich -->
<!-- Comment: Sun Nov 26 16;17;59 comment by...Ulrich -->
Enough work in the freezer for today. I rebooted the iFix PC and rolled back to the old version of PSMon.exe. All alarms to be taken seriously from now on.

<!-- Comment: Sun Nov 26 16;48;18 comment by...Satyajit -->
<pre>Thanks! I have turned ON the alphanumeric pager.
</pre>


<!-- Date: Sun Nov 26 20:07:57 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10372&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10372  width='300' align='top'></a></td><td>I was checking some of the changes in IMON for next week. I started it and I got the message: "Channels with Zero Bias. Possible INHIBIT problem. ISL Bias B5W2 L1". I have checked the voltage in PSGUI for that ladder and in Bias0 is 0.4 V (should be 2 V). I had the limit for the message in "if (V<0.5)".  
Looking to the history of this ladder, it seems that it has been for a long time 0.4 V for this ladder. I will need to decrease the IMON warning limit.</td></tr></table>
<!-- Author: Jose -->


<!-- Date: Sun Nov 26 21:32:27 2006 -->
Got paged around 8:50pm after pbars have been loaded. Since there 
wasn't enough time to come to B0 control room, I decided to watch 
the progress from home. Around 9:11pm the scraping was complete. 
After TevMon turned green the monitoring ACE pasted the  
<a href= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=896&frame=2&anchor=21:18:29.790'> 
losses plot</a> 
on the shift e-log. Everything looked normal. I gave a green 
signal to include silicon. Silicon HV turned ON without any problem.  
<!-- Author: Satyajit -->
<!-- Comment: Sun Nov 26 21;40;54 comment by...Satyajit -->
Also reminded the CO to check the SVXMon plots for SVX before
anything else. Things look normal as of now.



<!-- Date: Sun Nov 26 21:51:19 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10373&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10374  width='300' align='top'></a></td><td><h3>Store 5095  Init. Lumi. 110x10^30</h3> 
Silicon was included from the beginning.</td></tr></table>
<!-- Author: Satyajit -->
<!-- Comment: Sun Nov 26 21;59;20 comment by...Satyajit -->
As mentioned above, there is no yellow cell in SVXMon status maps.


<!-- Date: Sun Nov 26 22:34:24 2006 -->
Got an iFix alarm around 10:20pm saying that something in SVX is 
OFF. Checked to see that PS crate 14 has spontaneously turned OFF. 
Checked with the ACE about the status. They are struggling to  
bring RC to a HALT state. After they get to it, the MonACE would 
recover the trips. Nate is quite experienced in this.
<!-- Author: Satyajit -->
<!-- Comment: Sun Nov 26 23;17;31 comment by...Ulrich -->
It is hard to reconstruct what happened from the logs, but it seems that first b0fib06 turned red (as you may expect when wedges are off). It is unclear in which state RC was at this point. In any case, the aces followed the flow chart to recover the crate and to go to HALT. Then b0svx02 turned red and would not follow into HALT. At this point Satyajit brought me into the game. So I logged into b0svx02, which looked fine and advised them to end the run, overriding the failing HALT transition, and recover the wedges which were off using COLDSTART. After that, a new run with silicon was started without problems.


<!-- Date: Mon Nov 27 09:32:38 2006 -->
<h3>Week summary</h3><p> 
<h4>Monday</h4> 
<lu> 
 <li>PSMon.exe development list: (Ulrich)</li> 
 <lu> 
  <li>emit an alarm if voltage readback &lt; 1V</li> 
  <li>comm. failure to multiple crates; list of crates in alarm message</li> 
  <li>option for auto-Hocker. 
 </lu> 
 <li>If VBIAS &lt; 0.5V in IMon emit a popup window and sono alarm. (Jose)</li> 
 <li>Smiley face on the bottom-right corner takes snaphots. (Jose)</li> 
 <li>Weird glitch in ISL/L00 SRC. 6 L2A w/o L1A. River dancing suspected.</li> 
 <li>Ladder e184 produces errors in bitreporter. A chip missing.</li> 
 <li>Electron lens turns off. Silicon at standby till its recovery.</li> 
 <li></li> 
</lu> 
 
<h4>Tuesday</h4> 
<lu> 
 <li>Cooling interlock 24V PS replaced.</li> 
 <li>SVX/ISL resin changed</li> 
 <li>Ladder e184 investigated on SVXDaq. No error found! All chips intact.</li> 
 <li>test version of PSMon.exe tried out</li> 
 <li>Store 5086</li> 
 <li>Crate 11 spontaneous switch off</li> 
 <li>After upgrade to 4 VRB crates for the ISL+L00, event sizes reduce. Still the biggest CDF user?</li> 
 <li>Crate 7 spontaneous switch off</li> 
</lu> 
 
<h4>Wednesday</h4> 
<lu> 
 <li>SVX B1W10 Corrupted Readback</li> 
 <li>b0svx01 got stuck</li> 
 <li>SVXMon was not seeing any data for SVX in run 228581. The problem was fixed by starting a new run</li> 
 <li>Resonance in SVX at 22:06.</li> 
</lu> 
 
<h4>Thursday</h4> 
<lu> 
 <li>TeV quench at flattop</li> 
</lu> 
 
<h4>Friday</h4> 
<lu> 
 <li>Spontaneous switch-off of CAEN crate #3</li> 
 <li>Store 5089</li> 
</lu> 
 
<h4>Saturday</h4> 
<lu> 
 <li>Strange Jump in SVX Bias and DVDD Currents. Just for SVX, but not for L00 and ISL</li> 
</lu> 
 
<h4>Sunday</h4> 
<lu> 
 <li>SVXMon yellows again</li> 
 <li>More changes to PSMon.exe</li> 
 <li>Store 5095</li> 
 <li>PS crate 14 spontaneous turn OFF. b0svx02 in stuck state.</li> 
</lu> 
 
<h4>Monday</h4> 
<lu> 
</lu> 
 
<p>  
<!-- Author: Satyajit -->