<!-- Title: 10:48:12  Mon Feb  5 2007 -->
<!-- Crewchief: Mark (MPC) -->
<!-- Op1: Roberto (Shadow) -->
<!-- Op2: Ulrich (SPL) -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
Hopefully the MI comes back soon.


<!-- Date: Mon Feb 5 11:22:36 2007 -->
<pre>ADCmon watching. 
Week from Monday 29 January to 
          Monday 5 February 
 
Runs observed: from 233098 to 233258 
Comments: none, correct data acquisition. 
 
 
</pre>
<!-- Author: Roberto -->


<!-- Date: Mon Feb 5 11:47:27 2007 -->
<h3>Quota/Disk Space Checking Script</h3> 
After last weeks problems with the quota for /cdf/onln/home/svxii, we took a look at /cdf/onln/home/svxii/hv/smartso/makespace.pl which runs as a cron job from svxii@b0gateway and does some level of quota and disk space checking.  We found that this script only checks for free disk space and quota space for /cdf/onln/data/svxii, and not the home area.  Also, it was sending email to Ankush, Andy Hocker and David Stuart.  We added checking for the home area and updated the emails to Ignacio, Jose and Ulrich.  It sends an email if the free disk or quota space drops below 5% for the home area or 10% for the data area.
<!-- Author: Ignacio & Mark -->


<!-- Date: Mon Feb 5 14:33:15 2007 -->
<h2>Silicon Collision Hall Rack Trips</h2> 
While Ulrich was working on the ISL flows the Silicon collision hall racks tripped off.  After about 15 minutes, we turned the racks back on and they tripped again about 5 minutes later.  We found that the rack trips were happening at the same time as flow trips from the ISL work.  The theory is that one of the pressures was spiking (see Ulrich's entry for more details on this theory).  About 10 minutes before anti-protons finished loading, the ISL flows were reestablished and we were able to turn the collision hall crates back on.  We then hurridly rebooted the FIB crates and started to turn on the detector in preperation for the store.  After we had the SVX on the Tevatron quenched at flat top, so we are now turning the silicon on at a more leisurely rate.
<!-- Author: Ignacio, Jose, Ulrich, Mark -->
<!-- Comment: Mon Feb  5 15;04;29 comment by...Mark & Ignacio -->
The silicon is back on and we have run 5000 events in ColdStart without errors. 
<!-- Comment: Mon Feb  5 16;10;30 comment by...Mark -->
We biased the silicon and the detector checks out ok with the bias checker.  The CCE1 temperature is steady at 20.8.  
<!-- Date: Mon Feb 5 14:42:55 2007 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10720&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10720' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10721&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10721' width='300'></a></center>
<!-- Author: Ignacio -->
<!-- Comment: Mon Feb  5 14;44;28 comment by...Mark, Jose, Ignacio -->
Here are the screenshots for IMON/PSGUI/svxinter from the collision hall rack trips.  Notice that that the PS crates have lost communication because they have no power.


<!-- Date: Mon Feb 5 14:50:29 2007 -->
<h3>Tevatron Abort During Shot Setup</h3> 
During shot setup the beam was lost due to human error.  From MCR elog:<br> 
"14:36:24-   Jonathan Walters reports that Klystron 3's cooling fan on the control panel's door stopped funtioning. Ken Hartman is aiding him in assessing the problem. - R" 
The ACE <a href= 
'/cgi/elog/elog.pl?nb=2007&action=view&page=107&frame=2&anchor=14:22:36.737'> 
checked the BLMs </a> and we did not see any significant radiation.  
<!-- Author: Mark, Ignacio -->
<!-- Comment: Mon Feb  5 14;56;53 comment by...Mark -->
I copied the wrong MCR elog entry:<br>
"14:38:00-   In the process of high potting the Main Injector before the access, ops accidentally pressed the Tevatron "test/reset" button on the Power Supply Loop controller. This opened Tevatron PS VCBs. There was no quench, but alas, the store was lost. - marty"





<!-- Date: Mon Feb 5 16:13:46 2007 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10722&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10723' ></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10724&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10724' width='300'></a></center>
<!-- Author: Mark & Ignacio -->
<!-- Comment: Mon Feb  5 16;17;07 comment by...Ignacio & Mark -->
After turning the silicon back on, the DVDD current for f400 is quite erratic and below the nominal value, even while we are not taking data.  A look at the "Latest Values" in the "Channel Info" window reveals that the current is jumping around with a frequency much higher than the history logging rate.  A look at the long term history (right) shows this type of behavior happening on occasion.

<!-- Comment: Mon Feb  5 16;45;40 comment by...Mark -->
We ran this ladder in svxdaq, and it reads out just fine.


<!-- Comment: Mon Feb  5 17;24;56 comment by...Mark -->
I have ignored this in IMON for now.
<!-- Date: Mon Feb 5 17:24:34 2007 -->
The default fer release was updated with changes to the ReadoutList class.  I rebooted b0fib01 and b0svx09 to pick up the changes and was able to take them through partition and config with no problems.  I then rebooted the remainder of the crates and we ran 5000 events in Coldstart without any problems.
<!-- Author: Mark -->



<!-- Date: Mon Feb 5 18:38:04 2007 -->
<h3>Problems with E-Valve for CCE1</h3> 
Last night we had another drop on the flow measured in ISL cooling circuit CCE1. The symptoms were compatible with the e-valve closing a little, thus reducing the flow, the pressure, and the differential pressure at the same time (as opposed to an obstruction in the cooling line, for which you would expect an increase in the pressures and a drop in flow). 
 
<h4>Trouble-Shooting</h4> 
We tried to trouble-shoot the valve as far as we could without access to the collision hall. Warren Bowman helped us with this task a great deal, thanks! First of all, we had to remind ourselves how the system actually works: 
<ul> 
  <li>All e-valves are controlled by analog output cards (<a href="http://www.sea.siemens.com/automat/product/plc/505/au505man.html">Siemens 505-6208B</a>). With the PLC, you set digital values between 6400 and 32000 that are matched to voltages of 2-10 VDC or equivalently currents of 4-20 mA.</li> 
  <li>The e-valves are of the type <a href="http://www.ascovalve.com/Common/PDFFiles/Product/Posiflow_7.51R3.pdf">Asco Proportional Solenoid Valve SD8202G84V</a>. These valves are controlled by applying input voltages of 0-24 VDC. We are using the valve with an electronic control unit (8908A001) that translates either 0-10 VDC or 0-20 mA, or 4-20 mA into the proper input voltage. The control unit has an adjustable offset, so there is some hope that we can calibrate the problem away, once we get to the valve.</li. 
  <li>For some reason, there are no spares of this type of valve available at B0. We have ordered two valves and two control units each, the estimated delivery time is 8 weeks.</li> 
</ul> 
 
Next we did a couple of checks with the system: 
<ul> 
<li>We checked the voltage output for a couple of e-valves, including the problematic one. The voltages are consistent directly at the digital output module and at the wiring cabinet. The voltages are linear within 1%, with an offset of 2 V. The problematic valve puts out the correct voltage.</li> 
<li>We also tried to power-cycle the control unit by disconnecting the 24 VDC line in the wiring cabinet (brown wire in the bottom rail of the terminal block) and reconnecting it. During the recovery we learned that this exercise did not fix the problem (and luckily did not make it worse either).</li> 
</ul> 
 
<h4>System Recovery: Rack Trip</h4> 
 
We had great difficulties bringing up the East ISL flow again. The system was extremely touchy and tripped the flow many times, even when we were opening the valves very carefully. The most likely cause of the trips was over-pressure in CCE1. We have also managed to trigger the QuadLog interlock twice during the procedure, triggering a complete trip of all our collision hall racks. Despite its name, the logging capabilities of QuadLog are rather poor, so we cannot tell for sure, but our best guess is that the upward fluctuation in pressure went above 18 psi, which would trigger the QuadLog in addition to the PLC. 
<p> 
In the end we managed to get the flows to an acceptable state. We had to open the e-valve for CCE1 in steps of 1-2% in order not to trip anything, and ended up with highly tweaked valve settings: 
<ul> 
  <li>CCE1: 89% (default: 75%)</li> 
  <li>CCE2: 60% (default: 75%)</li> 
  <li>CCE3: 70% (default: 75%)</li> 
  <li>CCE4: 70% (default: 85%)</li> 
  <li>PCW: 75% (default: 80%)</li> 
</ul> 
 
<h4>Next Step: Access</h4> 
 
The situation is stable now, but we must expect more e-valve glitches. We still have 10% valve opening as headroom, but it is getting closer. We will request a 2-hour access to the collision hall during day-time, such that Warren can trouble-shoot the e-valve directly at the manifold. For the work at the manifold, the East IMU steel has to be opened by at least 50 cm.  
<!-- Author: Jose, Ignacio, Mark, Warren, Ulrich -->


<!-- Date: Mon Feb 5 23:56:26 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10725&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10726  width='300' align='top'></a></td><td>Store 5215 is in with an astounding initial instantaneous luminosity of 86e30.  Trigger rates look ok up to and including the first prescale change.  SVTSPYMon and SVXMon are fine.  CCE1 flow is holding steady at 1.8 lpm.</td></tr></table>
<!-- Author: Mark -->
<!-- Comment: Mon Feb  5 23;57;48 comment by...Mark -->
f54/5 is not giving the high rate of pipeline out of sync that we saw in the previous store before we put the old FIB back.

<!-- Comment: Tue Feb  6 00;10;00 comment by...Mark -->
Bias check ok as well.
<!-- Date: Tue Feb 6 00:09:00 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W3L4 
<li>problem n: 1920 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: mhartz 
<li>description is: Bump up bias imon limits<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>117.8</td><td>143</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>145</td><td>165</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B3W3&layer=L4">report</a></ul>
<!-- Author: Problem DB -->




<!-- Date: Tue Feb 6 02:10:41 2007 -->
I was paged for a <a href= 
'/cgi/elog/elog.pl?nb=2007&action=view&page=109&frame=2&anchor=01:51:58.772'> 
pinkie on e554 AVDD</a> that does not go away with HRRs.  The current stepped down by about 70 mA.  I took a look at the occupancy plots, and all chips are reading out, so it does not look like an AVDD2 problem.  This ladder has some history of AVDD moving around in the past so I told the shift crew to "ignore" it, but keep and eye on it.  I will investigate it in more detail in the morning.
<!-- Author: Mark -->
<!-- Comment: Tue Feb  6 07;58;02 comment by...Mark -->
After the step down in current, the current slowly decreased over the night at a rate of about 1 mA per hour.  Will investigate further.


<!-- Date: Tue Feb 6 08:16:13 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10727&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10728  width='300' align='top'></a></td><td>The DVDD current for f400 is back to normal now.</td></tr></table>
<!-- Author: Mark -->


<!-- Date: Tue Feb 6 08:34:39 2007 -->
We had an auto-hocker of PS crate 5.  There was a slight hickup in recovering as the ACE had gone to Recover instead of Halt.  In all it took about 6 minutes to recover.
<!-- Author: Mark -->
<!-- Comment: Tue Feb  6 08;35;33 comment by...Mark -->
We recovered with power by crate, and it worked just fine.


<!-- Date: Tue Feb 6 09:19:03 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10729&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10729  width='300' align='top'></a></td><td>Here is the IAvdd history for e554.  We have similar downward excursions of the current in the past that last a couple of days and then recover.  The ladder continues to give good data, so this current is ignored for now.</td></tr></table>
<!-- Author: Mark -->

<!-- Date: Tue Feb 6 09:23:13 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB5W5L4 
<li>problem n: 1623 
<li>Status: needs_observation
<li>Type: AVDD
<li>Action: Diagnosis_update
<li>entered by: mhartz 
<li>description is: This AVDD current has changed again (dropped by 70 mA).  The ladder is running fine so I will hold off on changing the limits and see if it recovers as it has done in the past. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B5W5&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Tue Feb 6 10:13:45 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10730&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10731  width='300' align='top'></a></td><td>IAvdd for e554 came back.</td></tr></table>
<!-- Author: Mark -->


<!-- Date: Tue Feb 6 12:28:34 2007 -->
All of the wedges that talk to b0fib05 tripped during Tevatron studies.  This is probably due to a SEU in the FFO.  We tried to use the new "power by FIB" option in PSGUI, but it did not successfully turn on all of the wedges, so they had to be done wedge by wedge.  We took the opportunity to go over trip recovery procedures with the ACE.
<!-- Author: Jose, Sasha, Mark -->
<!-- Comment: Tue Feb  6 12;37;03 comment by...Mark -->
I don't see any corresponding spike in losses.


<!-- Date: Tue Feb 6 16:37:00 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10732&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10733  width='300' align='top'></a></td><td>Store 5216 is in with luminosity of 208e30.  We saw again the problem where SVXMon does not see any data for SVX, so we restarted the run.  After that, SVXMon is ok.  Trigger rates look fine.  CCE1 flow is near the warning limit of 1.70.</td></tr></table>
<!-- Author: Mark -->


<!-- Date: Tue Feb 6 18:40:49 2007 -->
<h3>Another Drop in ISL CCE1 Flow</h3> 
Today around 15:00, another glitch in the ISL CCE1 flow occured. This time, the flow went down from 1.84 lpm to 1.71 lpm. The glitch happened during pbar loading, so we were hesitant to crank up the flow before the current store. We have left a  
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2007&action=view&page=111&frame=2&anchor=18:23:07.032'> 
policy</a> in the main e-log how to handle the situation. We will get paged when the average flow is approaching 1.55 lpm. Then we will come in and adjust valve settings. Apart from that, we are considering the following: 
 
<ul> 
<li>The hardware trip limit is at 1.5 lpm. We are thinking of lowering this value in the PLC, but that should only be done between stores, ideally when the detector is off. We also have to make sure that we can still cool the detector properly with lower flow.</li> 
<li>We have put out an access request to trouble-shoot the e-valve in the collision hall (East ISL manifold). We hope that we can either find a problem with the wiring or adjust the control unit on the valve.</li> 
<li>We have ordered spare valves and control units yesterday. Today Del Allspach has managed to shorten the delivery time significantly: the control units will be shipped tomorrow, and the valves take 4 weeks.</li> 
</ul>
<!-- Author: Ulrich -->
<!-- Comment: Wed Feb  7 13;12;24 comment by...Jennifer :-) -->
Does anyone have a friend or a Dad that works with this valve company? ;-)
<!-- Date: Tue Feb 6 21:42:39 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: mhartz
<li>Description of Problem: We saw the problem with extra data at the beginning of the SIXD bank at the beginning of store 5216.  The symptom is all of SVX yellow in SVXMon.  A quick restart of the run fixed it.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=130">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Tue Feb 6 22:37:53 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10734&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10734  width='300' align='top'></a></td><td>IAVDD for e114 <a href= 
'/cgi/elog/elog.pl?nb=2007&action=view&page=111&frame=2&anchor=22:16:01.001'> 
dropped</a> by about 50 mA.  From the occupancy plot it looks like chip 8a is stopped returning data. </td></tr></table>
<!-- Author: Mark -->
<!-- Comment: Tue Feb  6 22;47;19 comment by...Mark -->
From the SVXMon log I only see no data massage for chip 8a.  No other error messages for this ladder.

<!-- Comment: Wed Feb  7 08;45;42 comment by...Mark -->
The current came back and chip 8a is now returning data.  For some reason the front end for this chip was dead for a bit and then came back to life.
<!-- Date: Wed Feb 7 07:49:21 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB0W1L1 
<li>problem n: 1600 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: mhartz 
<li>description is: Bump up bias limit.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>125</td><td>180</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>145</td><td>200</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B0W1&layer=L1">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Feb 7 07:52:14 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB5W0L4 
<li>problem n: 1784 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: mhartz 
<li>description is: Bump up bias1 limit.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>51.2</td><td>110</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100</td><td>130</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B5W0&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Wed Feb 7 08:23:25 2007 -->
We had another global bias current jump that can possibly be correlated with a prescale change.
<!-- Author: Mark -->

<!-- Date: Wed Feb 7 08:38:35 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB4W8L4 
<li>problem n: 1590 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: mhartz 
<li>description is: Bumped up bias current limit.  Was near limit when we had one of the global bias current jumps.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>64.7</td><td>100</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100</td><td>140</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B4W8&layer=L4">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Feb 7 08:41:38 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB5W5L0 
<li>problem n: 1849 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: mhartz 
<li>description is: Increase bias current limit.  Was near limit and went pink when we had a global bias jump.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>100</td><td>115</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>110</td><td>130</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B5W5&layer=L0">report</a></ul>
<!-- Author: Problem DB -->




<!-- Date: Wed Feb 7 08:52:28 2007 -->
It would be good if you silicon people have TeVMon fixed... it becomes useless if it issues a warning that has t obe ignored every 10minutes...
<!-- Author: Sebastian -->
<!-- Comment: Wed Feb  7 09;54;49 comment by...Ulrich -->
The reasoning behing this was the following:
<ul>
<li>The new TevMon will be ready soon</li>
<li>If we disable the check of C:B0PLOS (the warning is issued if this get below 20 Hz), we lose the ability to check if the loss counters are working properly. From discussions with Koji Terashi we learned that an alternative check would require a little more thought (e.g. measuring the in-time rate instead of the out-of-time one).
</ul>
The check of the C:B0PLOS is ignored in the latest version of <i>fer</i>, whenever that will be released or we ask Ray to patch it.
<!-- Date: Wed Feb 7 08:52:41 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB1W1L4 
<li>problem n: 1921 
<li>Status: needs_observation
<li>Type: AVDD
<li>Action: Diagnosis_update
<li>entered by: mhartz 
<li>description is: Chip 8a has a very low rate of the front end going dead and not returning any data.  When it happens it usually goes away on its own after a few hours or so.  Keep an eye on it to make sure the rate of 'no data' from this chip is not increasing. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B1W1&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Wed Feb 7 11:18:28 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB3W1L4 
<li>problem n: 1598 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: mhartz 
<li>description is: Adjust BIAS0 limits.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>77.3</td><td>160</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>150</td><td>190</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B3W1&layer=L4">report</a></ul>
<!-- Author: Problem DB -->




<!-- Date: Wed Feb 7 14:35:38 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10735&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10735  width='300' align='top'></a></td><td>The electron lense tripped off, but the losses did not cause a Tevmon alarm until the lense came back on and the proton abort gap losses jumped up.  The ACE correctly turned the Silicon to standby at this point, and we then turned it back on as soon as it had finished the transition.</td></tr></table>
<!-- Author: Mark -->
<!-- Comment: Wed Feb  7 14;46;09 comment by...ronmoore -->
An electron lens trip is only a warning until the TEL is off for more than 10 minutes.  There was a warning generated when the TEL initially tripped off.  The new TevMon worked correctly as currently implemented.

<!-- Comment: Wed Feb  7 14;49;07 comment by...Ulrich -->
Right, but it went into alarm when the TEL kicked back in and the proton abort gap losses shot up. Is that an alarm situation in which we want to be in STANDBY or not? The danger (if any) is gone by the time everything is in STANDBY.
<!-- Comment: Wed Feb  7 15;00;52 comment by...ronmoore -->
I fully expect the abort gap *losses* to shoot up when the TEL is switched back on after a trip.  If you want to have an alarm generated sooner, let me know...that's not my call (I'm wearing my programmer hat here).  For this condition, I copied what is in the old TevMon.  I thought the 10 minute delay was implemented to prevent nuisance trips since it takes a while for beam to fill up the abort gap to unsafe *intensities* anyway.
<!-- Comment: Wed Feb  7 15;04;20 comment by...ronmoore -->
<pre>As a side note, the CDFMON OAC is not running the latest version anyway due to a rollback of a problematic release of the whole controls gov tree.  I'll restart the OAC after the current store is terminated.  (The warning should have been generated immediately after the TEL-1 trip, not 30 secs or so later.)</pre>
<!-- Date: Wed Feb 7 14:47:06 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: mhartz
<li>Description of Problem: The electron lense tripped.  Losses were fine until the lense came back at which point the the proton abort gap losses spiked and Tevmon went to danger.  The silicon was placed in standby and turned back on almost immediately.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=131">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Wed Feb 7 14:53:16 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB1W2L1 
<li>problem n: 1507 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: mhartz 
<li>description is: Bumped up Vbias1 limits.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>76.4</td><td>160</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>150</td><td>190</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B1W2&layer=L1">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Feb 7 15:00:18 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB2W1L1 
<li>problem n: 1629 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: mhartz 
<li>description is: Bumped up Vbias1.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>115</td><td>150</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>135</td><td>170</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B2W1&layer=L1">report</a></ul>
<!-- Author: Problem DB -->





<!-- Date: Wed Feb 7 20:34:21 2007 -->
<h3>The ISL E-Valve Saga</h3> 
 
After the Drendel had made up his mind that the store was to be dropped exactly at 16:35, we worked on two issues regarding the problematic e-valve in the East ISL system, ISL-EV-CCE1SUP. We have changed the low-flow trip limit from 1.5 lpm to 1.3 lpm by changing the PLC code. After that we went on controlled access to trouble-shoot the valve and its control unit. The bottom line is: we are back to sufficient flow into CCE1, at least for now. 
 
<h4>Preparation</h4> 
In preparation for the work, we switched off the entire detector as soon as CDF was in end-of-store mode. At the same time, we increased the SVX chiller temperature set point from -10°C to -6°C to keep the COT warm. As a precaution, we changed the SVX bulkhead and ISL silicon and hybrid trip limits from +21°C to +4°C. In order to avoid the plethora of alarms during the switch-off, we stopped IMON temporarily, and in anticipation of a rack trip, we set the AutoHocker to manual mode. 
 
<h4>Interlock Code Change</h4> 
We have changed the default value for the ISL bulkhead low flow trip limit variable (V151) from 12263 (1.5 lpm) to 11481 (1.3 lpm). Then we changed the current value accordingly. Note that the latter change only takes effect after the PLC is back from programming mode to run mode. Next we re-initialized the ISL analog input module in PLC Crate 2 Slot 1 with the new value. This was achieved by resetting the timer which runs the initialization sequence (TCC24=0). 
<p> 
To validate the change it was not sufficient to force one flow reading below the new trip limit and validate that indeed the low flow trip is issued. Instead we opened one of the valves to positions slightly below and slightly above the trip limit and verified that the corresponding out-of-range bit flipped correctly. 
 
<h4>Collision Hall Access</h4> 
During the interlock code change, Del Allspach and Mary Convery already started to open the East IMU steel, so that we could access the East ISL manifold. Warren Bowman and I (Ulrich) followed. Del and Warren opened up the controller unit of the e-valve and measured the supplied voltages. They turned out to be the same as upstairs, both for the 24 VDC device power and the voltage to adjust the valve. This finding rules out problems with the electrical connection between the 1st floor and the collision hall. 
<p> 
Next we inspected the controller unit. The dip switches are set such that it reacts on 4-20 mA currents, compatible with the specs of the PLC analog output module. The solenoid resistance also did not reveal anything. We also compared settings, voltages, and solenoid resistance with the e-valve that serves the PCE. In summary, there is no obvious failure of any device downstairs. 
<p> 
The controller unit has a potentiometer to change the offset in valve settings. We cranked it up by 1.5 turns to give us more headroom. We were checking and adjusting valve settings from the collision hall with my laptop - very useful. At some point during this procedure we had another rack trip. We kind of expected that after our previous experience with pressure inequalities in the system. 
After the potentiometer adjustment, we could get to 1.6 lpm flow at a valve setting of 50%. The only weird thing is that the minum flow for the line was at 0.4 lpm, but the other parameters (pressures, temperature) were compatible with zero flow. This may be a miscalibration issue(?). Bringing back the flow (at least preliminary) worked fine, so we wrapped up in the collision hall, and Del and Mary closed the IMU steel. 
 
<h4>Checkout</h4> 
After the work was done, we re-established flow to East ISL pretty quickly. We tried to go back to the default values for all valves but the one we touched, but the pressure was just a little too high (probably due to the degrading vacuum, now at 1.6 psi). After letting the system swing in for an hour or so, the settings are: 
<ul> 
<li>CCE1: 54%</li> 
<li>CCE2: 70%</li> 
<li>CCE3: 75%</li> 
<li>CCE4: 80%</li> 
<li>PCE: 80%</li> 
</ul> 
 
Next we went back to the nominal SVX chiller temperature set point and trip limits, while switching on the detector without problems. The whole exercise left the compromised cooling line CCE1.1 almost unchanged, the output temperature is at 20.9°C. We did the usual checkout, including running 5000 events through coldstart and biasing the detector. Everything looks fine. 
 
<h4>Next Steps</h4> 
It is very likely that the valve controller unit is the culprit. We are expecting delivery of a spare controller unit within the next days, so that we can easily swap it out during another access. We will see if the flow continues to drop occasionally, but we should have enough headroom to ride it out now. 
 
<p> 
Thanks a lot to Warren, Del, and Mary for their help tonight!
<!-- Author: Jose, Ignacio, Mark, Ulrich -->


<!-- Date: Wed Feb 7 22:54:00 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10736&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10737  width='300' align='top'></a></td><td>Store 5218 in with initial luminosity of 178e30.  Trigger rates, SVXMon and bias checker are all ok.</td></tr></table>
<!-- Author: Mark -->
<!-- Comment: Wed Feb  7 23;00;46 comment by...Mark -->
CCE1 flow is holding steady around 1.90 lpm.  Temperature is at steady at 20.99.
<!-- Date: Wed Feb 7 22:58:52 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB1W4L0 
<li>problem n: 1567 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: mhartz 
<li>description is: Bumping up bias1 upper limit to avoid alarms at beginning of stores.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>1087</td><td>1230</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>1250</td><td>1400</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B1W4&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Wed Feb 7 22:59:36 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: mhartz
<li>Description of Problem: Came in for start of store 5218.  Everything went smoothly.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=132">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Thu Feb 8 08:20:50 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB3W0L4 
<li>problem n: 1506 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: mhartz 
<li>description is: Bumped up bias0 current limit.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>161</td><td>210</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>200</td><td>250</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B3W0&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Thu Feb 8 08:25:07 2007 -->
Store 5218 was <a href= 
'/cgi/elog/elog.pl?nb=2007&action=view&page=115&frame=2&anchor=03:30:35.100'> 
cleanly aborted</a> at 3:30 due to bad HFU at C3.   
<!-- Author: Mark -->


<!-- Date: Thu Feb 8 08:26:59 2007 -->
The beam was aborted during shot setup after protons were loaded due to D0 BLMs.  It was <a href= 
'/cgi/elog/elog.pl?nb=2007&action=view&page=115&frame=2&anchor=07:54:56.735'> 
clean</a> for us.
<!-- Author: Mark -->


<!-- Date: Thu Feb 8 09:47:00 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10738&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10739  width='300' align='top'></a></td><td>Store 5219 in with initial lumi of 155e30.  The f00/1 FIB was red in SVXMon initially so we did an HRR after which SVXMon looks fine.  Trigger rates look ok.</td></tr></table>
<!-- Author: Mark -->


<!-- Date: Thu Feb 8 16:21:42 2007 -->
<h3>New Bitreporter Feature</h3> 
I have added efficiency information to the ladder summaries in bitreporter for SVX and ISL (L00 is still to come).  If you want to check out the new version, it is at:<br> 
<a href="http://b0sili01.fnal.gov/cgi-bin/bitreporter-devel/bitreporter.pl">http://b0sili01.fnal.gov/cgi-bin/bitreporter-devel/bitreporter.pl</a> 
<br> 
Let me know if you find any bugs.  As soon as I have added L00 and tested it, I will move it to the standard release.
<!-- Author: Mark -->


<!-- Date: Thu Feb 8 20:06:18 2007 -->
Studies schedule from the Run Coordinator e-log: 
 
<pre> 
# Crystal Collimator Studies:   From 0600 until 0800 hours Dean Still will do Crystal Collimator studies.  These studies have the potential of being more destructive, so the experiments may chose to turn off.  Make sure to notify both experiments before this study starts. 
</pre>  
<!-- Author: Jose -->
<!-- Comment: Thu Feb  8 21;22;59 comment by...Ulrich -->
I guess that means that at least the silicon will be in STANDBY?

<!-- Comment: Fri Feb  9 07;54;10 comment by...Mark -->
Yes.


<!-- Date: Fri Feb 9 08:17:10 2007 -->
The flow for CCE1 had dropped to 1.7 lpm, so Ulrich adjusted the supply valve from 54% to 57%.  This raised the flow to 1.95 lpm, and the interlocks did not even trip once.
<!-- Author: Ulrich & Mark -->


<!-- Date: Fri Feb 9 08:20:30 2007 -->
There was an SRC resonance error that was caught by the analyzer last night.  The trace looked like the standard resonance trace.
<!-- Author: Mark -->


<!-- Date: Fri Feb 9 08:29:08 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10740&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10741  width='300' align='top'></a></td><td>ISL flows since the access on Wednesday.  It looks like the stored value for CCE1 is encoded with less bits when it is above a certain value around 1.8 lpm.  We saw a drop in the flow rate of about 0.25 lpm over the course of about 36 hours.</td></tr></table>
<!-- Author: Mark -->


<!-- Date: Fri Feb 9 10:12:20 2007 -->
<h3>Two new SVX supplies arrived from CAEN!</h3> 
Yesterday we received a box from CAEN with our two brand new SVX supplies (we're still waiting on the two ISL supplies we ordered).  Before surrendering them to the Fermilab processing system, I ran them both through a quick loadbox test.  I checked every channel held voltage and current, tried setting low voltage values and bias trip limits, checked the overvoltage and overcurrent and enable trips... both supplies checked out good!  Once they get checked-in, I'll run them through the full burn-in tests, and (hopefully) we'll have two new spare SVX!<br>
<!-- Author: Jen -->


<!-- Date: Fri Feb 9 11:00:58 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10742&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10743  width='300' align='top'></a></td><td>Store 5220 is in with initial inst. luminosity of 245e30.  b0fib04 went red at the beginning of the store, so we had to reboot it.  SVXMon looks fine, as do the trigger rates and the bias checker.  It looks to me that the occupancy for e06 in the SVTSPYMon plot is a bit lower than usual.  Investigating..</td></tr></table>
<!-- Author: Mark -->
<!-- Comment: Fri Feb  9 11;16;45 comment by...Mark -->
After talking to Stefano, and looking at this wedge in the errorlogger, I think e06 is fine.

<!-- Date: Fri Feb 9 14:13:07 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB1W1L0 
<li>problem n: 1922 
<li>Status: needs_longterm_monitoring
<li>Type: Chip_capid
<li>Action: Monitoring
<li>entered by: boveia 
<li>description is: DSEM shows CAP id for chip 82 is bad for 24% of parsed events for run 233764. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B1W1&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Fri Feb 9 14:33:42 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: LB0W5L3 
<li>problem n: 1923 
<li>Status: needs_longterm_monitoring
<li>Type: Unknown
<li>Action: Monitoring
<li>entered by: boveia 
<li>description is: For ~10% of events in run 233764, chip 80 (also see thread 1241) reads out a cap id of 3f followed by a sequence of junk data.  The first channel data is 'ee'.  Subsequent channels read out 'ff'.  Watch out for this to become a more frequent, reproducible problem.  Otherwise, ignore it.   
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=L00&wedge=B0W5&layer=L3">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Fri Feb 9 14:40:41 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB5W4/5All 
<li>problem n: 1689 
<li>Status: needs_observation
<li>Type: Other
<li>Action: Diagnosis_update
<li>entered by: mhartz 
<li>description is: It turns out that Fib 122 and 119 have a few bad sockets in the J3 connector. This could have caused by slot 19 of the crate in B0 teststand or by the collision hall b0fib07 slot 11 when we swap 122 by 107. We should keep an eye on the mechanical safaty of the boards we take from the collision hall, in particular 107. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B5W4/5&layer=All">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Fri Feb 9 14:40:41 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB5W4/5All 
<li>problem n: 1689 
<li>Status: needs_observation
<li>Type: Other
<li>Action: Diagnosis_update
<li>entered by: mhartz 
<li>description is: It turns out that Fib 122 and 119 have a few bad sockets in the J3 connector. This could have caused by slot 19 of the crate in B0 teststand or by the collision hall b0fib07 slot 11 when we swap 122 by 107. We should keep an eye on the mechanical safaty of the boards we take from the collision hall, in particular 107. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B5W4/5&layer=All">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Fri Feb 9 17:46:25 2007 -->
<h3>FIB/J3 Problems</h3> 
Last week we swapped FIB 122 into b0fib07 slot 11 to address occasional FIB wide pipeline out of sync problems for f54/5.  FIB 107 came out and was run in the teststand without problems.  In the meantime, the rate of pipeline out of sync errors greatly increased  with FIB 122 in slot 11 of b0fib07.  We made an access on 2/4 to put FIB 107 back into b0fib07 and took FIB 122 out.  After taking FIB 122 out we tried running it in the teststand and saw problems getting events with the DEM (similar problems may have been seen before FIB 122 was swapped into b0fib07).  At this time we did a visual inspection of the J3 slot were the FIBs are placed in the teststand.  We found that at least one pin that appeared to be bent.  A visual inspection of the the J3 connectors did not show any obvious problems on the FIBs, so we tried running FIB 119 and FIB 122 again in another slot in the teststand.  Once again there were problems taking events with FIB 122.  A visual inspection of the second slot we ran in once again showed damage.  At this point we did not want to put either of the FIBs into a teststand, so we took them over to FCC and had Vince and Ken Treptow look at them.  When we looked at FIB 119 and 122 under the microscope we found that there was in fact damage to the sockets which are used for the following FIB->FF0 status lines: FIB_COMMAND_OVERFLOW, FIB_FLOW_CTL, FIB_TRUNCATE, FIB_DEAD_CHANNEL and RESERVED.  The J3 connector can probably be replaced, but it will take some time.  If these sockets are not used by the system (we think they are) then they could just be drilled out, which would be much easier.  We will will determine what the pins are used for and then ask prep to go ahead with the necessary repairs.<br> 
<b>To summarize the damage:</b> We have two slots in the teststand that have damaged J3 pins and two FIBs with damaged J3 sockets, although it is not clear what orginally caused the damage.  There are still free slots in the teststand so that is not an immediate concern.  The two FIBs can be fixed, but it may take some time.  There is also a FIB/Slot combo in the collision hall (FIB 107 in b0fib07 slot 11), that may have been the root cause of all of this damage.  The ladders read out by FIB 107 show a very low rate of FIB wide pipeline out of sync errors, so for now we should just leave that FIB alone.  If it has to come out in the future we should make an inspection of the J3 backplane and connectors before putting anything in that slot.
<!-- Author: Ignacio & Mark -->


<!-- Date: Sat Feb 10 06:27:15 2007 -->
I was paged for a couple of pinkies in IMON.  First AVDD for e554 dropped like it did earlier this week.  The ladder looks like it is returning normal data so we are just monitoring it for now.  Second, the bias current for e424 jumped up.  The bais for this ladder has been behaving strangely since 1/06/17, but the data from the ladder is good.  Once again I told the shift crew to continue monitoring it.  The current is at about 200 uA which is still well below the trip limit of 1000.
<!-- Author: Mark -->

<!-- Date: Sat Feb 10 11:34:00 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: mhartz
<li>Description of Problem: Came in for trigger table test.  The purpose of this test was to look at a number of L2 triggers that are now DPS instead of lumi enabled.  First ran without silicon and all rates looked normal.  Next we did a run with silicon.  After the DPS kicked in the rates were about the same as they were with the default trigger table.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=135">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Sat Feb 10 11:41:38 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10746&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10746  width='300' align='top'></a></td><td>x</td></tr></table>
<!-- Author: x -->


<!-- Date: Sat Feb 10 11:41:59 2007 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10744&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10745' ></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10746&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10746' width='300'></a></center>
<!-- Author: Mark -->
<!-- Comment: Sat Feb 10 11;46;24 comment by...Mark -->
Immediate and long term hitory for e424 bias.  The bad behavior starts around dsday 2210.


<!-- Date: Sat Feb 10 12:11:59 2007 -->
We ran the dmode calibrations but the job info submission failed with the message:</br> 
<i> 
Error: can't open connection to the calibration job server on cdffarm0.fnal.gov. 
<br><br> 
This problem has been reported to nielsenj@fnal.gov together with the job parameters. Please do not send another e-mail.</i><br> 
I will assume that this message is correct and not take any other action.
<!-- Author: Mark -->


<!-- Date: Sat Feb 10 12:14:38 2007 -->
To update, the flow for CCE1 is currently at about 1.78 lpm.  We should have at least 4 hours before the next store comes in if we decide we want to adjust the valve opening.
<!-- Author: Mark -->
<!-- Comment: Sat Feb 10 15;30;01 comment by...Jennifer :-) -->
I came in this afternoon to have a look at the CCE1 loop. It was a bit low (less than 1.75 lpm) to begin the store with, so I adjusted the following:
<p>
<li>CCE1 supply valve: 57% &rarr; 59%, in 2 steps of 1% decrease. This now allows around 1.95 - 2.0 lpm flow to our clogged line. Pressures on COT face remain in range: 12.7 psi - 12.9 psi. </li>
<li>CCE2 supply valve: 70% &rarr; 69%. I have been receiving warnings for the CCE2 COT face pressure via e-mail (meaning his pressure was ~ 13.0 psi). He is now comforatble at 12.5 - 12.8 psi.</li>
<p>
I have also made a main e-log entry, reminding the shift crew to page me (Jennifer) if flows decrease below 1.5 lpm. Thanks to Ulrich for moving the PLC limit down to 1.3 lpm, giving us a bit more room to avoid tripping.



<!-- Date: Sat Feb 10 18:07:56 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10747&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10748  width='300' align='top'></a></td><td>Store 5221 is in with initial inst. luminosity of 149e30.  SVXMon looks ok, trigger rates are fine and bias checker is ok.  The bias current for e424 is misbehaving again, but the SVT efficency for that wedge looks normal.</td></tr></table>
<!-- Author: Mark -->
<!-- Comment: Sat Feb 10 18;11;15 comment by...Mark -->
Took at the look at the ADCMon distributions for e424.  I see a normal looking physics tail.


<!-- Date: Sat Feb 10 23:02:35 2007 -->
<h3>Lost RF Station</h3> 
RF station 1 tripped and the shift crew took silicon to standby as the <a href= 
'/cgi/elog/elog.pl?nb=2007&action=view&page=123&frame=2&anchor=22:54:49.212'> 
losses jumped up</a>.  After MCR called to say that the beam should be stable, they checked that it was stable in ACNET, and waited for Tevmon to go green.  They paged me just to make sure that it was ok to put the Silicon back in, and I gave them the go ahead.
<!-- Author: Mark -->

<!-- Date: Sun Feb 11 10:25:37 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W6L4 
<li>problem n: 1924 
<li>Status: needs_work
<li>Type: DVDD
<li>Action: Removed_hdwdb
<li>entered by: husemann 
<li>description is: e364 kept tripping on DVDD overvoltage, taken out of the HWDB. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B3W6&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Sun Feb 11 10:36:51 2007 -->
<h3>DVDD Overvoltage Trip in SVX B3W6L4 and PS GUI Problems</h3> 
e364 kept tripping on DVDD overvoltage. After a first recovery attempt over the phone we got stuck because RC was still in RECOVER. After that was solved, we had problems with the PS GUI which got frozen (update counter running, but no response to any clicks) and had to be restarted. There was some confusion about the restart, because the instructions close the the Silicon terminal were still pointing to the old version. We have fixed that on the sheet, but we need to change the web page as well! 
<p> 
We first suspected a corrupted readback from the power supply and asked the Ace to Hockerize crate #14. This did not help, so we decided to take the ladder out of the hardware database and investigate the problem between stores. We have also asked the CO to restart SVXMon and the Ace to refresh from the hardware database. While ramping the Silicon again, the PS GUI got stuck again, in the middle of biasing ISL. After a second restart of the PS GUI and a reboot of svxinter, everything came back up fine.
<!-- Author: Mark & Ulrich -->
<!-- Comment: Sun Feb 11 10;52;01 comment by...Mark -->
Just a clarification on the first recovery attempts.  The shift crew tried to recover from HALT first, and this failed.  The second time they tried to recover they were either in RECOVERED or ACTIVATE so there was the expected FIB communication error.  At that point they paged me and we tried another recovery from HALT.  So we had a couple of "good" recovery attempts before the PSGUI problems.  It is not clear to me if there is any relation between the PSGUI getting stuck and the attempt to recover while not in HALT.

<!-- Comment: Sun Feb 11 12;32;05 comment by...Jose -->
I have seen once a PS-GUI not responding to any click but refreshing. The problem at that time was that the "ALARM ACKNOWLEDGE" window (hidden somewhere behind the GUI and not visible for the ACE) had the focus. After clicking/closing that ALARM window the PS-GUI was reponding again.


<!-- Comment: Sun Feb 11 13;11;36 comment by...Mark -->
Ah, that sounds like reasonable explanation.
<!-- Comment: Sun Feb 11 13;18;29 comment by...Mark -->
After the store I took another look at e364 to see if there is any chance of recovering it.  I tried hockerizing again and turning on individually and with the whole wedge, but to no avail.  I took a look in the ps log file and noticed that there are no entries for e365 DVDD after 09:12, which is the time of the first recovery attempt.  For the initial trip and first attempt at recovery, it was tripping on DVDD overcurrent.  From the second attempt at recovering onward the ladders has tripped on overvoltage with no entry in the ps logs.  The ladder is back out of the hdwdb.
<!-- Comment: Sun Feb 11 13;21;55 comment by...Jose -->
For the moment, I have modified the ACEs web page to point to the new directory "/cdf/onln/data/svxii/caencontrol/PSfilesNew" (I should have done this before...) Probably, we should change the new directory name to be as the old one and make the new PSGUI the official version.

<!-- Date: Sun Feb 11 12:50:50 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: mhartz
<li>Description of Problem: e364 tripped on DVDD overcurrent and the shift crew could not recover it as it subsequently tripped on DVDD overvoltage.  I was paged and we tried to recover it one more time at which point the PSGUI became unresponsive.  We tried to restart it, but ran into a problem as the instructions for restarting the new version were not posted.  After getting the new version running, we hockerized the ps crate, at which point Ulrich showed up in the control room and took over the recovery.  After the hockerization, the ladder was still tripping, so it was taking out of the hdwdb.  The PSGUI crashed one more time while the silicon was being biased and had to be restarted again.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=137">Incident Report</a></ul>
<!-- Author: Incident DB -->






<!-- Date: Sun Feb 11 16:08:24 2007 -->
Forgot to mention that I opened up the supply valve for CCE1 once more, from 59% to 62% to compensate for the drop in flow over the last day. It seems that we should do this between stores as part of our daily routine until the spare controller is swapped in.
<!-- Author: Ulrich -->

<!-- Date: Mon Feb 12 08:38:00 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W6L4 
<li>problem n: 1924 
<li>Status: needs_work
<li>Type: DVDD
<li>Action: Diagnosis_update
<li>entered by: mhartz 
<li>description is: Took another look at this.  When trying to recover, the transisitions do not even show up in the ps logs.  Still unable to bring it back. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B3W6&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Mon Feb 12 08:41:50 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10749&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10750  width='300' align='top'></a></td><td>Store 5224 came in at 250e30.  SVXMon, trigger rates and bias checker are ok.  You can see the lower SVT occupancy for e36 due to taking out e364.</td></tr></table>
<!-- Author: Mark -->


<!-- Date: Mon Feb 12 09:26:45 2007 -->
<h3>Weekly Summary</h3> 
<ul> 
<li>Monday 
  <ul> 
  <li>Update /cdf/onln/home/svxii/hv/smartso/makespace.pl script to check quotas and disk space in svxii home and data areas.  Send mail to SPLs. 
  <li>Erratic dvdd current for f400.  Still the case - ignored in IMON. 
  <li><a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=339&frame=2&anchor=18:38:04'> 
Investigation of ISL CCE1 cooling line</a>.   Causes a few Quadlog interlock trips of collision hall racks, but back up barely in time for store... 
  <li>Tevatron abort during shot setup pushes store 5215 back to late evening, measly 86e30. 
  <li>f54/5 checks out ok (FIB was reswapped on Sunday). 
  </ul> 
<li>Tuesday 
  <ul>  
  <li>Auto-hocker of PS crate 5.  Recovered with power by crate, after confusion about RC not being in HALT. 
  <li>Avdd current for e554 drops by 70 mA.  Came back up for about a day and then back down where it currently is.  Ignored in IMON. 
  <li>All wedges in b0fib05 trip during Tev end of store studies.  SEU of FFO?  Tried recovering with power by FIB, but some wedges did not turn on. 
  <li>ISL CCE1 flow <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=339&frame=2&anchor=21:42:39'> 
continues to fall</a>.  Down to 1.71 during shot setup, so did not adjust valve opening. 
  <li>Store 5216 in with 208e30.  No SVX data in SVXMon due to SIXD bank problem; restarted run immediately to fix this. 
  <li>e114 loses chip 8a with what looks like a dead front end.  Came back after about 10 hours. 
  </ul> 
<li>Wednesday 
  <ul> 
  <li>Global bias jump correlated with prescale change. 
  <li>Tevmon fixed to not complain about C:B0PLOS below 20 Hz 
  <li><a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=339&frame=2&anchor=14:35:38'> 
Electron lense tripped off</a>.  Silicon stayed on while lense was off, but when lense came back on the proton abort gap losses spiked (as expected) and Tevmon temporarily went into danger.  Shift crew correctly took silicon to standby, and was able to bring it back on almost immediately. 
  <li><a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=339&frame=2&anchor=20:34:21'> 
Access</a> to look at controls for CCE1 evalve.  Adjustment of potentiometer gives more headroom.  Looks like a control problem. 
  <li>Store  5218 in at 178e30.  CCE1 line at 1.90 lpm. 
  </ul> 
<li>Thursday 
  <ul> 
  <li>Store 5218 aborted due to bad HFU at C3. 
  <li>Abort during shot setup due to D0 BLMs. 
  <li>Store 5219 in at 155e30 
  <li><a href="http://b0sili01.fnal.gov/cgi-bin/bitreporter-devel/bitreporter.pl">Bit reproter with new feature</a>.  Inefficiencies shown on ladder reports.  Still need to clean this up. 
  </ul> 
<li>Friday 
  <ul> 
  <li>CCE1 valve opend from 54% to 57%, flow from 1.7 lpm to 1.95 lpm.  Estimate that flow is dropping at rate of 0.25 lpm per 36 hours. 
  <li>Two new SVX Caen supplies are here. 
  <li>Store 5220 in at 245e30.  b0fib04 red at beginning, had to be rebooted.  New SVT patters, occupancy looks normal. 
  <li>Find <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=339&frame=2&anchor=17:46:25'> 
problems with FIB 122, 119 J3 sockets and two slots in the testand</a>.  
  <li>Bias current for e424 jumps up.  Was erratic for most of weekend.  Now is fairly steady at 565 uA (former upper limit was 280 uA).  Ignored in IMON. 
  </ul> 
<li>Saturday 
  <ul> 
  <li>Trigger table test for L2 triggers where lumi enable is replaced by dps.  OK for silicon. 
  <li>Ran dmode calibrations. Submission failed, should check with J Neilsen and co. if we do here back about this calibration. 
  <li>Jennifer <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=339&frame=2&anchor=12:14:38'> 
adjusted CCE1 valve.</a> 
  <li>Store 5221 in at 149e30. 
  <li><a href= 
'/cgi/elog/elog.pl?nb=2007&action=view&page=123&frame=2&anchor=23:10:28.640'> 
Lost RF station</a>.  Silicon to standby and then back on when beam was stable and loses back down. 
  </ul> 
<li>Sunday 
  <ul> 
  <li>e364 tripped on dvdd overcurrent.  Could not recover and subseqently tripped on dvdd overvoltage.  Had to take it out and have requested access to investigate.  <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=339&frame=2&anchor=10:36:51'> 
The gory details.</a> 
  <li>No beam in tevatron for rest of the day after store dropped at noon because of electron cooling problems related to pellatron discharges and Vesda alarm in D0 collision hall 
  </ul> 
<li>Monday 
  <ul> 
  <li>Store 5224 in at 250e30 
  </ul> 
</ul>  
<!-- Author: Mark -->