<!-- Title: 10:46:24  Mon Sep 29 2008 -->
<!-- Crewchief: Oscar (MPC) -->
<!-- Op1: Saty (SPL) -->
<!-- Op2: Superman (the rest) -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
We begin with a shot setup and hoping for a quiet time and tons of data.


<!-- Date: Mon Sep 29 11:42:05 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12919&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12919  width='300' align='top'></a></td><td>Store 6460 came in with an initial luminosity of 250E30. Looses are fine and silicon there from the beginning. SVXMon is also fine.</td></tr></table>
<!-- Author: Oscar -->


<!-- Date: Mon Sep 29 20:40:04 2008 -->
Previous store quenched in the afternoon... new shot setup currently happening. Everythings seems to be fine.
<!-- Author: Oscar -->


<!-- Date: Mon Sep 29 21:27:42 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12920&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12921  width='300' align='top'></a></td><td>New store 6462 in with an initial luminosity of 250E30. Everything looks fine and silicon was in from the begining (however, we lost some data due to Level 3).<br> 
SVXMon and SpyMon plots look fine. Let us hope this store last!</td></tr></table>
<!-- Author: Oscar -->


<!-- Date: Tue Sep 30 08:27:02 2008 -->
At the EOS PS#10 had a spontaneous switch off. After the detector was at stand-by, I re-enabled SAR and it recovered fine the wedges that were off. However, right after that, all ladders in IB5W0 turned pink. I did a COLDSTART and now everything looks fine.<br> 
Dervin plans to do some work on the clock after they drop the store. I will stay around in case there are ploblems. He claims the previous times they did something like this the detector was not affected.
<!-- Author: Oscar -->


<!-- Date: Tue Sep 30 09:44:40 2008 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>Massive trip in SVX and some wedges in L00</pre></td></tr></td></table>
<!-- Author: Oscar -->
<!-- Comment: Tue Sep 30 09;46;53 comment by...Oscar -->
It seems that Steve Hahn (that is very sorry :) wanted to rewset the errors in the clock and he reset the clock. All SVX went down but SAR is recovering it with no issues. I came to the CR to check everything is fine. I will do coldstart after SAR finish.<br>
As an aside, the clock glitch also turned off the PortCards (so I was wrong in the discussion we had on Monday).

<!-- Comment: Tue Sep 30 09;53;44 comment by...Oscar -->
Coldsart went fine. SAR recovered the whole in 5/10 minutes... which is not bad. When I did it manually a couple of years ago it took me more than 1 hour.<br>
I had a few issues with SB4W4... for some reason the PortCard reported a strange value and SAR was trying to recover. One channel was marked as tripped... I am not sure what happened. When I disabled SAR to take a look, everything was fine.<br>
SAR is enabled again.

<!-- Comment: Tue Sep 30 09;56;06 comment by...Oscar -->
I also put back the BeamPermit that I was have killed from svxinter to avoid it interfere with anything in the recovery... and make it much faster. It is already back and working fine.

<!-- Date: Tue Sep 30 11:26:30 2008 -->
Entered in Silicon database: <ul>
<li>Ladder: IB3W0L2 
<li>problem n: 1536 
<li>Status: needs_work
<li>Type: Chip_capid
<li>Action: Diagnosis_update
<li>entered by: sergo 
<li>description is: Run svxdaq to diagnose the problem further. The ladder has bit5 high, causing the cap_id and unknown channel errors at high rate.  Needs fiber swap to diagnose further. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B3W0&layer=L2">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Tue Sep 30 11:37:58 2008 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W7L2 
<li>problem n: 2660 
<li>Status: needs_work
<li>Type: Chip_capid
<li>Action: Diagnosis_update
<li>entered by: sergo 
<li>description is: Tested e372 which is not returning data (according to SVXMon). The channel did good data while running svxdaq, however it has a cap_id error for chip 89. According to the PDB this channels has chip 89 sparsed out, because it is the first chip the Error Monitor takes it as reference marking the rest of the chips as bad. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B3W7&layer=L2">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Tue Sep 30 11:43:33 2008 -->
Entered in Silicon database: <ul>
<li>Ladder: SB5W6L4 
<li>problem n: 345 
<li>Status: unfixable
<li>Type: Optical_other
<li>Action: Diagnosis_update
<li>entered by: sergo 
<li>description is: The ladder is dead - not returning any data now. It had bits 6 & 7 low in the past. Since we have many good RXs we can try to revive it by doing RX swap during next acccess. However since past RX swaps were unsuccessful, doubtful that this will fix problem. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B5W6&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Tue Sep 30 11:44:00 2008 -->
There has been a radiation abort from the silicon during the Tevatron Beam Studies. MCR claims it has been due to a sudden and unexpected increase in losses. Came to th Control Room to check.
<!-- Author: Oscar -->
<!-- Comment: Tue Sep 30 11;50;24 comment by...Oscar -->
When trying to do a ColdStart, svx08 got stuck in transition, so I rebooted (connect to it and type reboot). Everything is again fine. The Coldstart was also successful.
<!-- Date: Tue Sep 30 11:47:12 2008 -->
Entered in Silicon database: <ul>
<li>Ladder: SB4W9L3 
<li>problem n: 1511 
<li>Status: needs_observation
<li>Type: Chip_bit_error
<li>Action: Diagnosis_update
<li>entered by: sergo 
<li>description is: Bit 1 high error has re-appeared at high rate. Will try to do fiber swap during next access.  
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B4W9&layer=L3">report</a></ul>
<!-- Author: Problem DB -->




<!-- Date: Tue Sep 30 21:41:03 2008 -->
New store 6464 in after some issues on scraping (that did not affect CDF, just some delay). Initial luminosity was 231E30. I still do not have the SpyMon plot... it seems that something happens to TrigMon. 
SvxMon plots look fine.
<!-- Author: Oscar -->


<!-- Date: Tue Sep 30 21:43:46 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12922&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12923  width='300' align='top'></a></td><td>Finally I got the SpyMon plot working... everything is fine and I can go home.</td></tr></table>
<!-- Author: Oscar -->


<!-- Date: Wed Oct 1 06:09:56 2008 -->
I got paged at the end of the store to get the permission to reboot the silicon PC. I gave them the ok. We should perhaps tell them to wait until 8:00 meeting if the message appears in the middle of the night (I was already awake, anyway). 
<!-- Author: Oscar -->
<!-- Comment: Wed Oct  1 11;10;39 comment by...Mark -->
The cryo guys have always mentioned it during the 8am meeting (at least, for me)... or was it the SciCo who was worried?
<!-- Date: Wed Oct 1 07:32:20 2008 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12924&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12925' ></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12928&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12929' ></a></center>
<!-- Author: Oscar -->
<!-- Comment: Wed Oct  1 07;38;11 comment by...Oscar -->
These are the 2 DOIM currents we have ignored in IMON (the two lines in SB0W7). There has been some increase in the current, getting closer to the nominal, but not close enough yet.


<!-- Comment: Wed Oct  1 20;42;24 comment by...Oscar -->
There has been a simultaneous decrease in the two currents at the moment of biasing the detector to start datataking. In case it helps... it seems that the variations we are observing (at least today) are correlated with the biasing of the detector or the start  of the data processing.
<!-- Date: Wed Oct 1 07:40:04 2008 -->
The Ace reports that ProcMon sometimes complains about IMon not responding, and for crazy amounts of time (10000 seconds, things like that). He did not make an entry in the e-log since after rechecking IMon was ahppy again in ProcMon. I decided to restart IMon in case it is a memory issue or so.<br> 
First I will deal with a pinkie in BIAS for L00.
<!-- Author: Oscar -->

<!-- Date: Wed Oct 1 07:43:43 2008 -->
Entered in Silicon database: <ul>
<li>Ladder: LB1W1L1 
<li>problem n: 2662 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: oscar 
<li>description is: Chaginmg limits since BIAS current reached the upper limit and we got a pinkie in IMON. However, the values do not correzxpond to the previous entry. I am not sure about the reason.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>18.7</td><td>18.7</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100</td><td>130</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=L00&wedge=B1W1&layer=L1">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Wed Oct 1 07:48:01 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12930&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12931  width='300' align='top'></a></td><td>Store 6465 came in with an initial instantaneous luminosity of 220E30. Everything was fine regarding the losses and the silicon<br> 
Ace told me he had to sherpherd crate fib00 because it failed config and fib06 because of Heap corrupt.</td></tr></table>
<!-- Author: Oscar -->



<!-- Date: Wed Oct 1 14:43:06 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12932&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12933  width='300' align='top'></a></td><td>Bias line in Imon for SB0W5L2, set to ignore by the Scico (Andreas). when I came back to check, the current seems to come back to its correct value. I unmark it.</td></tr></table>
<!-- Author: Oscar -->


<!-- Date: Wed Oct 1 20:39:39 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12934&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12935  width='300' align='top'></a></td><td>New store 6467 in with an initial instantaneous luminosity of 221E30. Everything is fine and silicon was in from the beginning (which was very smooth this time).</td></tr></table>
<!-- Author: Oscar -->



<!-- Date: Wed Oct 1 20:48:26 2008 -->
We have an autohockerization of CAEN crate #7. SAR recovered the wedges quick and nicely. We are taking data again.
<!-- Author: Oscar (still here) -->


<!-- Date: Wed Oct 1 22:11:13 2008 -->
I was paged because SB0W2L0 DVDD cell is pink... Plot, posted in the main <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2008&action=view&page=825&frame=2&anchor=22:01:34.468'>e-log</a> shows 0 current. I think it is corrupted read-back so hockerize crate #2 should fixed it. I instructed shiftcrew to page me if a chance for that happens.
<!-- Author: Oscar -->


<!-- Date: Thu Oct 2 07:46:06 2008 -->
The ace (Stefano) mentions me that ProcMon complains from time to time that IMon is not responding for a crazy amount of second (~20000). I think it is either a communication glitch or a ProcMon problem, since I do not seem that IMon has problems (although there have been several pinkies during the owl shift).
<!-- Author: Oscar -->

<!-- Date: Thu Oct 2 08:05:46 2008 -->
Entered in Silicon database: <ul>
<li>Ladder: IB0W2L3 
<li>problem n: 2189 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: oscar 
<li>description is: Chainging limit to avoid pink cell since current is normal, and just increasing due to aging.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>125</td><td>125</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>280</td><td>310</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B0W2&layer=L3">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Thu Oct 2 08:08:40 2008 -->
Entered in Silicon database: <ul>
<li>Ladder: IB4W3L0 
<li>problem n: 2385 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: oscar 
<li>description is: Changing upper limit in IMON for BIAS0 to avoid a pink cell due to the steady increase caused by againg.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>200</td><td>200</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>160</td><td>200</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B4W3&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Thu Oct 2 08:10:45 2008 -->
Entered in Silicon database: <ul>
<li>Ladder: IB4W2L0 
<li>problem n: 2369 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: oscar 
<li>description is: Changing upper limit in IMON for BIAS1 line, increase due to aging.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>110</td><td>110</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>225</td><td>255</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B4W2&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Thu Oct 2 08:12:50 2008 -->
Entered in Silicon database: <ul>
<li>Ladder: IB5W2L1 
<li>problem n: 2427 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: oscar 
<li>description is: Changing the upper limit in IMON for BIAS1 line, to avoid a pinky due to the aging.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>150</td><td>150</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>265</td><td>300</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B5W2&layer=L1">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Thu Oct 2 08:14:44 2008 -->
Entered in Silicon database: <ul>
<li>Ladder: IB5W1L3 
<li>problem n: 2413 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: oscar 
<li>description is: Changing the upper IMON limit for BIAS1. Increase due to aging.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>150</td><td>150</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>290</td><td>320</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B5W1&layer=L3">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Thu Oct 2 09:04:49 2008 -->
I hockerized CAEN crate #2 after the EOS in order to recover the DVDD current that was 0 (SB0W2L0). After hockerization, current is back to normal. Unmarked.
<!-- Author: Oscar -->


<!-- Date: Thu Oct 2 09:05:55 2008 -->
I made a request for a NOISE BIAS SCAN with medium priority. I am currently testing the RC configuration to avoid issues (although in NOISE scans we just use the current COSMIC table).
<!-- Author: Oscar -->
<!-- Comment: Thu Oct  2 09;07;02 comment by...Oscar -->
Test was successful.


<!-- Date: Thu Oct 2 09:59:00 2008 -->
I asked Bill Noe to have the ISL vacuum pump VP1A (currently running, with a pressure of 1.9 PSI, on the upper edge of running okay), swapped out and rebuilt.  Also, I asked him to have the resin bed for ISL replaced.  We have enough time between stores to get this done, and if everything goes well, it should be transparent.
<!-- Author: Tom Junk -->


<!-- Date: Thu Oct 2 11:14:22 2008 -->
<pre>There must have been a momentary dip in the ISL flows, as there was a low-flow trip.  The flows recovered right away but we had low-flow interlocks.  Reset interlocks and turned on silicon without incident. </pre>
<!-- Author: Tom Junk -->


<!-- Date: Thu Oct 2 11:16:48 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12936&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12936  width='300' align='top'></a></td><td>Some relevant histories with the resin bed and vacuum pump swap.  ISL VP2A is not very strong, either -- we should swap it out and rebuilt it too -- possibly during the shutdown.</td></tr></table>
<!-- Author: Tom Junk -->


<!-- Date: Thu Oct 2 13:37:38 2008 -->
When trying to ramp the HV up, svxinter got frozen and the PSGui unresponsive. The detector was set to ON, but iFix did not get the signal. I tried restarting the PSGui and it did not help.<br> 
Everything worked fine again when I start a new conexion to svxinter using a new terminal.
<!-- Author: Oscar -->


<!-- Date: Thu Oct 2 14:28:03 2008 -->
We had a lot of action at the beginning of the new store (6469) because of the HV controllers. It is not clear what caused the problem, but we had hard time to recover wedges in Crate #13. It seems that SAR crashed because it could not connect properly to the Hardware Database, but also we needed (?) to reboot svxinter and the PSGui.<br> 
The bottom line is that it is unclear what was preventing fib06 to config the chips, so SAR worked but all ladders tripped in DVDD overcurrent. However, sherpherding the fib did not work (and other chips in the same fib were fine).<br> 
Also, although we tried to recover the wedges manually, it did  not pay. It is clear that SAR was more efficient. However, we do not know if it has problems because fibs were in a partition (that we open to recover).   
<!-- Author: Sasha, Saty and Oscar -->


<!-- Date: Thu Oct 2 14:43:50 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12937&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12938  width='300' align='top'></a></td><td>Store 6469 came in with an instantaneous luminosity of 186E30. Everything from the beam size was fine, but we had a lot of troubles to set the silicon ready for data-taking. After fixing them, silicon was included and we are collecting good data, when the COT hold the voltage.</td></tr></table>
<!-- Author: Oscar -->


<!-- Date: Thu Oct 2 21:33:40 2008 -->
They had an autohockerization of crate #13 and some issues with fib06 (see <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2008&action=view&page=828&frame=2&anchor=21:30:15.045'>main e-log</a>. Everything seems to be fine now, but it is remarkable that are the same players which gave us troubles at the beginning of the store.
<!-- Author: Oscar -->

<!-- Date: Fri Oct 3 06:55:50 2008 -->
<p><font size=+1> D-mode Calibration Job Started </font>
<p><pre>
Job id            : 518
Calibration type  : standard
Run period tag    : Oct_2008
DPS-off run       : 267619
DPS-on run        : 267618
Requested by      : Miguel
Main process id   : 23958
</pre>
<!-- Author: CDF silicon D-mode calibration job server v1.4 -->



<!-- Date: Fri Oct 3 07:34:52 2008 -->
SB0W5L2 had during the night several excursions in the AVDD current ... currently is also pink. Since this is due to the test of the noise-scan table (my guess... we always have issues with this guy when going to a read-all mode), I decided to powercycle the wedge. This helped in some of the noise-scans.
<!-- Author: Oscar -->
<!-- Comment: Fri Oct  3 07;44;37 comment by...Oscar -->
Done, everything seems to be fine. I will keep an eye on it to confirm it. It would be interesting to see if data is fine for the runs taken in this store. Specifically 267614 and 267616.<br>
Byt the way, after I turned the wedge off, SAR recovered it so fast that the time it was off does not appear in the IMON history plot.

<!-- Date: Fri Oct 3 10:40:50 2008 -->
<p><font size=+1> D-mode Calibration Job Finished </font>
<p><pre>
This is a computer-generated silicon D-mode calibration status report.
Silicon D-mode calibration job 518 finished successfully.

Job info
--------
Calibration type : standard
DPS-off run      : 267619
DPS-on run       : 267618
Requested by     : Miguel
Submitted        : 2008-10-03 06:55:46
Completed        : 2008-10-03 10:40:34
Status           : SUCCESS
</pre>
<!-- Author: D-mode script run_calib_analysis.tcl -->



<!-- Date: Fri Oct 3 16:06:34 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12939&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12939  width='300' align='top'></a></td><td>SVX cooling line blockage.  At approximately 2:00 PM today the flow in SVX BHW3 dropped very rapidly to zero, with no prior bad behavior.  We tried opening and closing the flow control valve for it but had no success in recovering the flow in this line.  The corresponding PCOT, DPT and temperatures indicate also that there is no supply pressure in this manifold and no cooling.</td></tr></table>
<!-- Author: Tom Junk, Miguel Mondragon, Sasha Sukhanov, Satyajit Behari -->


<!-- Date: Fri Oct 3 16:20:41 2008 -->
<h2>SVX Cooling Line Blockage</h2> 
We investigated the zero-flow cooling line SVX BHW3.  Our current best guess is that the supply electrovalve has stopped working.  Either its solenoid has burnt out or shorted, or otherwise is not responding to its control voltage/current.  We identified the terminal block in the first floor that corresponds to this valve -- it's block 823, and corresponds to analog output WY4107 in the PLC.  We were able to change the voltage across the terminals by changing the valve settings in iFIX.  80% open (the default setting) corresponds to 8.4V, and 0% corresponds to 2.0V.  We also saw 8.4V across the neighboring pair of terminals for BHW2, which is also set at 80% open. 
<p> 
We unscrewed the red wire going to the valve and measured the resistance between the red and black wires -- it is 500 Ohms.  We will investigate with the valve manual to see what we should expect.  The best case is that the valve solenoid has stopped working and we can just replace it.  
<p> 
 A worse scenario is that something is clogging the line.  The fact that this circuit's flow has been stable as long as we looked (90 days back it's all stable) makes us less likely to believe a blockage due to something getting stuck in there.  The granularity of the problem is a supply circuit, and not a single cooling line after the manifold, and the pressure has dropped on the PCOT too.  The pipes are quite large between the chiller and the PCOT transducer, also making us not suspect a spontaneous blockage, but rather a valve closing.   
<p> 
There is also a mechanical valve in parallel with the supply electrovalve -- we will check that too but do not believe it could spontaneously close.
<!-- Author: Tom Junk, Miguel Mondragon, Sasha Sukhanov, Satyajit Behari -->
<!-- Comment: Fri Oct  3 23;33;16 comment by...Tom -->
<pre>Oops -- the mechanical valve is in series with the electrovalve on the supply pipe.</pre>
<!-- Date: Fri Oct 3 16:25:02 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12940&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12940  width='300' align='top'></a></td><td>Interestingly, the pressures exponentially settled in to an intermediate value after the incident.</td></tr></table>
<!-- Author: Tom,Miguel,Sasha,Satyajit -->


<!-- Date: Fri Oct 3 16:30:07 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12941&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12941  width='300' align='top'></a></td><td>We are running a bit warmer than usual, though the corresponding bulkhead is off.</td></tr></table>
<!-- Author: Tom, Sasha, Miguel, Satyajit -->


<!-- Date: Fri Oct 3 16:55:41 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12942&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12942  width='300' align='top'></a></td><td><h3>SVX bulkhead-0 masked off!</h3> 
Until the electro-valve is fixed (unlikely before coming 
Sunday) we will run silicon w/o bulkhead 0. Tom and Miguel 
are working on activating the SVX sono-alarm. In the meanwhile 
we will be on <b>high alert</b> about the iFix alarms! 
</td></tr></table>
<!-- Author: Satyajit,Sasha -->


<!-- Date: Fri Oct 3 17:00:09 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12943&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12944  width='300' align='top'></a></td><td>Store 6470 came in with an instantaneous luminosity of 88E30. We had some issues because of a cooling line with no flow. We are currently running with silicon but without bulkhead 0 of the SVX due to that problem.</td></tr></table>
<!-- Author: Oscar -->


<!-- Date: Fri Oct 3 17:45:52 2008 -->
<pre>On request of the shift crew, Miguel and I forced the interlock status to be OK for the CCW3 interlocks that had tripped.  T We did this for the low-flow interlock, the delta-P interlock and the temperature interlock. Steve Gordon says there is a condition Quadlog still doesn't like.  We will back out of forcing variables.</pre>
<!-- Author: Tom, Miguel -->
<!-- Comment: Fri Oct  3 18;09;22 comment by...Tom, MIguel -->
We unforced the interlocks, leaving them tripped.  The SVX sono-alarm is now disabled as before.  Turns out QuadLog is smarter than we are.  It detected the condition that the cooling had failed (probably the zero flow in BHW3) but that the interlock had been bypassed, and shut down all the racks.
<p>
So.. unless we can construct a sono-alarm out of everything except BHW3's interlocks (and keep the interlocks tripped and whatever QuadLog looks at - we'd be sunk if it's the sono-alarm signal itself), we're out of luck trying to re-enable the sono-alarm with our flow zeroed out as it is.
<p>
Furthermore, on thinking about it, disabling the interlocks is a bad idea since it allows the power supplies to be turned on with zero flow.
<p>
The scenario JJ was worried about is that another cooling problem could occur and we'd be without a sono-alarm.  Normally interlocks protect us, but the crash button is something the shift crew needs instructions for.  Now we only have the PLC interlocks and iFIX's audible "Silicon Alarm" to protect us (and QuadLog).  JJ wants explicit instructions for the shift crew in this state (which will last at least until we go in on an access for this).



<!-- Comment: Fri Oct  3 22;24;36 comment by...Oscar, Satyajit, Sasha, Tom and Miguel -->
While recovering from the "All Racks Shutdown" we faced two major
issues. When we restarted PSGui, there were error messages on the svxinter console complaining about failed RtServer communications.
We had the ACE restart RtServer. Later SAR failed repeatedly to
recover ladders citing they were aborted by Interlocks and that
the ladders are neither in standby or ON state! PSGui logs
suggested these were due to stuck CAEN crates. We hockerized most 
of the crates to recover the ladders (manually).

<!-- Comment: Fri Oct  3 22;42;49 comment by...Andy H. -->
<pre>Go to the SVX interlock page.  For BHW3, click the text that says "ACTIVE" to toggle it to "INACTIVE."  This will effectively mask BHW3 from the sono-alarm.  This kind of situation is exactly what we designed that "ACTIVE/INACTIVE" scheme for. </pre>
<!-- Comment: Fri Oct  3 22;47;54 comment by...Andy H. -->
<pre>Also, keep an eye on BHW3 temperatures.  Even though the BHW3 wedges are interlocked off, nearby powered wedges are likely to heat up the BHW3 temps if there is no flow to BHW3.</pre>
<!-- Comment: Fri Oct  3 23;37;12 comment by...Tom -->
<pre>Thanks Andy!  I didn't know about that!</pre>
<!-- Comment: Sat Oct  4 13;05;20 comment by...Miguel M -->
Perhaps the reason why SAR was complaining about the Interlocks was that BHW3's interlocks were in active state. It would be good to observe after an eventual hockerization if the complain has gone away. 
<!-- Date: Fri Oct 3 19:52:36 2008 -->
<h3>SVX Sono-Alarm is currently disabled!</h3> 
We have pasted <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2008&action=view&page=831&frame=2&anchor=19:42:52.900'> 
instructions for the shift crew</a> to take all the Silicon Alarms 
seriously until flow in the bulkhead-0 is restored.  
<!-- Author: Tom, Miguel, Oscar, Sasha, Satyajit -->
<!-- Comment: Sat Oct  4 09;37;17 comment by...Miguel M / Tom J -->
We have set the interlocks as "INACTIVE" for SVX BHW3. Then we have re-enabled the sono-alarm and passed the standard instructions to the shift crew.
<!-- Date: Sat Oct 4 02:02:10 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12945&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12946  width='300' align='top'></a></td><td>New store 6471 came in with an initial instantaneous luminosity of 84E30. Everything looks fine (I check from home... shiftcrew has my phone number).<br> 
Spymon plot looks fine, considering SVX-B0 is off.</td></tr></table>
<!-- Author: Oscar -->


<!-- Date: Sat Oct 4 09:17:25 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12947&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12947  width='300' align='top'></a></td><td>BHW3 temps are stable with most temps < 5C and one at around 7.5C.  Dean says Warren will be in on Monday and knows where all the spare e-valves are and can help replace it.  The e-log indicates that these valves vibrate at 300 Hz and have failed in the past.  Asco tech support says they're really only good for 3 years and "wear and tear" gets to them at about that time.</td></tr></table>
<!-- Author: Tom -->




<!-- Date: Sat Oct 4 16:03:49 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12948&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12948  width='300' align='top'></a></td><td>Taking advantage of the quiet time, I have hockerized crate #13 that it is the associated one to the AVDD channel that was pink because it was a bit high. Now it is back to normal and as green as it can be.</td></tr></table>
<!-- Author: Oscar -->
<!-- Comment: Sat Oct  4 16;43;54 comment by...Oscar -->
As an aside SAR was not able to recover the wedges, just part of it, giving the same error as yesterday. I do not think it is going to work if we have a trip during data taking.


<!-- Comment: Sun Oct  5 14;06;50 comment by...Satyajit -->
This is serious! Weren't there any trips till the store was dropped
for the shutdown?
<!-- Date: Sat Oct 4 20:58:14 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12949&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12950  width='300' align='top'></a></td><td>Store 6475 came in with a luminosity of 108E30... let us hope that this is the last low-luminosity store.<br> 
Everything was fine and the ace put the silicon at the beginning... we are still missing Bulkhead 0.</td></tr></table>
<!-- Author: Oscar -->


<!-- Date: Sun Oct 5 06:52:27 2008 -->
<h3>Shutdown: Work on Power Supplies</h3>
<!-- Author: Mark/Oscar/SashaS -->
<!-- Comment: Sun Oct  5 06;53;50 comment by...SashaS -->
Set Auto-Hocker to Manual mode in preparation for the access to retrieve 2 power supplies.
<!-- Comment: Sun Oct  5 07;27;50 comment by...SashaS -->
Took out of HWDB ladders e37* and f45*

<!-- Comment: Sun Oct  5 07;48;59 comment by...SashaS -->
e3a4 tripped ASA NW has been moved. Taken out of HWDB

<!-- Comment: Sun Oct  5 07;50;18 comment by...Mark / Oscar / SashaS -->
Supplies were removed, everything else checks out fine. I'll bring them to FCC for repair tomorrow

<!-- Comment: Sun Oct  5 08;27;08 comment by...SashaS -->
Put Auto-Hocker back to Auto mode


<!-- Comment: Sun Oct  5 14;07;26 comment by...Satyajit -->
<pre>Great! Thanks!!</pre>

<!-- Comment: Sun Oct  5 16;31;03 comment by...Oscar -->
<table border=0><td><tr><td bgcolor=#e0e030>We need to put an updated reference paper... they paged me because two wedges were blue.</td></tr></td></table>


<!-- Comment: Mon Oct  6 01;03;25 comment by...Satyajit -->
Done!
<!-- Date: Sun Oct 5 22:33:24 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12953&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12954  width='300' align='top'></a></td><td>This is the new peak seen in ADC distribution for e290.</td></tr></table>
<!-- Author: Tino -->


<!-- Date: Sun Oct 5 22:37:04 2008 -->
<h3>Monitoring Summary (Run 267563-267718)</h3> 
 
<br> 
 
I have checked ADCMon plots during this week from run 267563 (0x4152b) 
 to run 267718 (0x415c6).  
<br>Apart known troublesome ladders( e250, f533,...), the new things are: 
 
<br><br> 
 
<li> e495 show a bad ADC distribution for runs 267617 (0x41561) and 267686 (0x415a6)</li> 
 
<li>e372 no data for runs: 267563(0x4152b), 267582(0x4153e) and 267718(0x415c6). <br>The rest of runs, this ladder give data but presents high bit errors.</li> 
 
<li>f302 show problems again in all the runs I have checked.</li> 
 
<li>Noted that e290 has a big new peak in ADC distribution since september.</li>  
<!-- Author: Tino -->



<!-- Date: Mon Oct 6 08:42:39 2008 -->
<h2>Summary of the week</h2> 
 
Quiet week except for the Cooling incident that leads us to lose Bulkhead-0 
of the SVX and learn something else about the interlocks. Hopefully we recover 
fine from the started shutdown. 
 
<h3>Monday</h3> 
<ul> 
   <li>  Store 6460 with initial luminosity 250E30. Everything fine until it 
         quenched in the afternoon. 
   </li> 
   <li>  Store 6462 with initial luminosity 250E30. Everything fine. 
   </li> 
</ul> 
 
<h3>Tuesday</h3> 
<ul> 
   <li>  Spontaneous switch-off at EOS. SAR worked although some ladders needed 
         some coldstart to get the proper config. 
   </li> 
   <li>  Massive trip in SVX due to the resetting of the clock. SAR Recovered it 
         fine. 
   </li> 
   <li>  Sergo did some quiet-time studies. 
   </li> 
   <li>  Radiation abort that did not cause troubles, except the complication to 
         find our RadCo expert. 
   </li> 
   <li>  Store 6464 with 231E30. Everything fine but it was a long scraping. 
   </li> 
</ul> 
 
<h3>Wednesday</h3> 
<ul> 
   <li>  PC rebooted (I was paged at 6am). 
   </li> 
   <li>  Two DOIM currents are ignored in IMON (SB0W7) because their values 
         are a bit lower. Changes in currents observed in the past. 
         See entry <a href='http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=425&frame=2&anchor=07:32:20'>here</a>. 
   </li> 
   <li>  Changing IMon limits for aging rise in the Bias current. 
   </li> 
   <li>  Store 6465 with luminosity 220E30. Everything was fine. Ace needed to 
         sherpherd crate b0fib00 while getting ready for the data taking because 
         it failed transition. 
   </li> 
   <li>   Store 6467 with an initial luminosity of 221E30. Everything fine. 
   </li> 
   <li>   Autohockerization of crate 7. 
   </li> 
</ul> 
 
<h3>Thursday</h3> 
<ul> 
   <li>  Hockerized crate 2 because of corrupted read back in DVDD current 
         (value was 0) for SB0W2L0. 
   </li> 
   <li>  Quiet time cooling work <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=425&frame=2&anchor=09:59:00'>here</a>. 
   </li> 
   <li>  Store 6469 with initial luminosity of 186E30. Lots of troubles because 
         our HV controllers got crazy (and I got confussed). Unclear the reason, 
         but fib06 and PS crate 13 seems to be in the middle. 
   </li> 
   <li>  Autohockerization of crate 13, while recovering, fib06 gave troubles to 
         the shiftcrew. 
   </li> 
   <li>  D-mode calibration taken. 
   </li> 
   <li>  SB0W5L2 shows some issues with AVDD current because of a test of the 
         noise-to-bias scan table. Powercycling the wedge helped, as we usually do 
         after noise scans. 
   </li> 
</ul> 
 
<h3>Friday</h3> 
<ul> 
   <li> SVX Cooling line blockage (see details 
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=425&frame=2&anchor=16:06:34'>here</a>). We lost Bulkhead 0 of SVX. 
   </li> 
   <li>  Luckily there were issues at the Recycler and luminosity was low before 
         during the weekend and before the shutdown. 
   </li> 
   <li>  When trying to get a stable configuration with the alarms, the interlocks 
         and SVX-B0 off, the Quadlog switched off everything (crash-button effect). 
   </li> 
   <li>  Recovery was a headache since we needed to restart everything (svxinter, 
         PSGui, RTServer,...) and SAR did not cooperate. 
   </li> 
   <li>  Reference is <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=425&frame=2&anchor=17:45:52'>here</a>. 
   </li> 
</ul> 
 
<h3>Saturday</h3> 
<ul> 
   <li>  Store 6471 with luminosity 84E30 (we missed those good old times!). 
         Everything fine, except SVX-B0 was off. 
   </li> 
   <li>  Hockerize crate 13 to get AVDD current to the correct value. SAR did not 
         help me this time either. 
   </li> 
   <li>  Store 6475 with initial luminosity of 108E30. Everything fine. 
   </li> 
</ul> 
 
<h3>Sunday</h3> 
<ul> 
   <li>  We started the shutdown at 6:50am... take two PS out for repairing. 
   </li> 
   <li>  Report is <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=425&frame=2&anchor=06:52:27'>here</a>. 
</ul> 
 
<h3>Monday</h3> 
<ul> 
   <li>  Scheduled power outage at 7am. We are in recovering mode now. 
   </li> 
   <li> Monitoring summary. 
   </li> 
</ul> 
 
<b>Question of the week:</b> Will be this my last MPC shift?
<!-- Author: Oscar -->