<!-- Title: 10:59:48  Mon Oct  6 2008 -->
<!-- Crewchief: Satyajit (MPC) -->
<!-- Op1: Sasha (SPL) -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
Starting with the 4 days accelerator shutdown. Hope to be up and  
running by the weekend!  


<!-- Date: Mon Oct 6 14:16:17 2008 -->
A busy morning!  The power outage started a couple of minutes before 7:00 AM and was over by 7:20.  We were still recovering flows with the backup pumps when the main power came back, and we proceeded to check the chillers and recover flows with the main pumps.  We moved the temperature setpoints as per the JHA to warm up for the COT work.  The COT East baggie was cut at 10:35 this morning, and we were fully warm by then. 
<p> 
Dew points have been far away from our minimum temperatures, but we did notice a rise in the sili baggie dew points; the west side in the sili baggie is dry but the east side shows some signs of humidity.  Possibly there is a leak between the volumes bounded by the baggies, and that the volume inside the sili baggie is under pressure relative to the COT baggie volume. 
<p> 
The online database was unavaliable this morning as firmware for a disk controller was upgraded. It is available again now, but the power supplies are LOTO'ed.  Nonetheless, checkout of FIBs and SRCs succeeded.  We will do checkouts when COT work is finished this afternoon.
<!-- Author: Tom Junk, Miguel Mondragon, Sasha Sukhanov, Satyajit Behari -->


<!-- Date: Mon Oct 6 15:03:53 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12955&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12955  width='300' align='top'></a></td><td><h3>Evolution of Dewpoints</h3> 
After a period of stability about noon, the dewpoints in the COT baggie (COTE_DEWPT & COTW_DEWPT) started rising about 1:30 pm, in the east side it reached a value of 4 C; this is to compare with the ISL minimum temperature which is 12 C. Humidity was also rising to about 34%. So we called the COT guys in Collision Hall (Bob Wagner et al); they decided to spray foam around the slit on the baggie; that would help to reduce the inflow of air.<br><br> 
The dewpoints in the Silicon baggie (SIBAG*_DEWPT, SVX_DEWPT, ISL_DEWPT) are around -8 C and look stable; this is to compare with the SVX minimum of -1 C. <br> 
</td></tr></table>
<!-- Author: Miguel M / Tom J -->


<!-- Date: Mon Oct 6 16:21:18 2008 -->
Dew point interlock -- sili baggie East dew points got to -4C which is the min. temp minus 3C which is the interlock limit.  We had been in contact with the COT people -- they had sprayed some DAP foam around the slit in the baggie but it wasn't enough.  The JHA tells us to ask them to close up the baggie and wait for the dew points to fall, and we did.  COT repair crew needed some time to think about what they wanted to do anyway.
<!-- Author: Miguel Mondragon, Tom Junk -->
<!-- Comment: Mon Oct  6 16;36;29 comment by...MM -->
We sucessfully recovered the flows. The dewpoint temperatures are still going down, -15 C in the silicon baggie, -8 in COT baggie.
<!-- Date: Mon Oct 6 16:29:56 2008 -->
<pre>COT people raised the N2 flow rate in order to drive dewpoints down, but they cannot work with the higher N2 flow because of the ODH.  The dewpoints drop much faster than they rise, and thus they can work in batches.  Also, they had more than one point they were working on, and were considering working on only one thing at a time.</pre>
<!-- Author: Tom, Miguel -->



<!-- Date: Mon Oct 6 17:05:18 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12956&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12956  width='300' align='top'></a></td><td><h3>Power cuts are here to help!</h3> 
Today after the power outage the flow of BHW3 of SVX has come back to life (red line). The e-valve was unresponsive last Friday.</td></tr></table>
<!-- Author: Miguel M / Tom J -->
<!-- Comment: Mon Oct  6 17;12;35 comment by...MM / TJ -->
It is worthy to point out that we did not try to open the e-valve since the incident of last Friday. 


<!-- Date: Mon Oct 6 17:25:35 2008 -->
<pre>I re-enabled the sono-alarm component for BHW3 by clicking on "inactive" to make it "active".</pre>
<!-- Author: Tom -->
<!-- Comment: Mon Oct  6 23;07;19 comment by...Satyajit -->
Nice! After the e-valve inspection/replacement is finished we will
put the wedges back in HWDB.
<!-- Date: Mon Oct 6 18:12:48 2008 -->
<h4>COT access continuees</h4> 
They slited the baggie again at about 17:20.  We have been watching the dewpoints; they look safe so far.<br>
<!-- Author: Miguel M / Tom J -->


<!-- Date: Mon Oct 6 18:13:27 2008 -->
<pre>The COT baggies are repaired for the evening, and the N2 flows have been returned to their normal values.  The dew points are dropping.  The COT guys would like to slit the baggie again tomorrow morning at 8:30.  We decide to stay warm tonight instead of cycling the temperature back to cold for the 12 or so hours we have until we'd have to warm back up again.</pre>
<!-- Author: Tom, Miguel, Satyajit -->



<!-- Date: Tue Oct 7 10:15:47 2008 -->
<h3>BHW3 e-valve electronics replacement</h3> 
 
Warren and I replaced the electronic controller on the solenoid valve for SVX BHW3.  The new one performs like a champ -- we get the same flow as before with a 50% valve setting (previously it had been at 80%).  Warren will test the module with a voltage source and scope when time allows, and let us know if he finds anything.  Swapping the electronics is very easy.
<!-- Author: Tom Junk, Warren Bowman, Miguel Mondragon -->
<!-- Comment: Tue Oct  7 13;04;13 comment by...Tom Junk, Warren Bowman -->
Warren checked out the controller we removed with a power supply and oscilloscope.  He says it works just fine.  He says the new one has a higher gain on its control by choice, so it's not an indication that the older one is somehow weaker that it required a higher setting in order to work. I still have better peace of mind now that it is swapped out with a new one.
<!-- Comment: Thu Oct  9 14;29;46 comment by...Tom, Warren, Miguel -->
<pre>Warren tested it today again and found that it failed at 80% and higher.  Apparently they have two sub-ranges and the higher one (above 80%) requires some kind of power boosting, and it failed there.  We took this as justification to replace as many of the SVX valve electronics as possible (we had four spares and swapped out the ones that are normally the most open).
 </pre>
<!-- Date: Tue Oct 7 11:59:29 2008 -->
The COT guys are out of the collision hall. It seems that they sealed the baggie; the dew points are going down.<br>
<!-- Author: Miguel M -->


<!-- Date: Tue Oct 7 12:42:43 2008 -->
<pre>Bill Badgett showed us yesterday how to run the screen snapshot service.  You can view screen snapshots at <a href="http://www-cdfonline.fnal.gov/daq/daqExpert/daqmon.html">this link</a> and there is also a link to start the service on a computer.  Bill added VNODE02 to the list of computers authorized to send snapshots, but we found yeseterday VNODE02 did not have Java installed.   Mark Knapp installed the Java runtime environment on VNODE02, and put the snapshot service in the startup script.  I has been asking for "do you trust content from this server" when starting up, and we've been clicking "yes".  We'll see if it asks that the next time someone logs into VNODE02.  Until then, happy snapshotting!</pre>
<!-- Author: Tom Junk, Mark Knapp -->
<!-- Comment: Tue Oct  7 13;00;42 comment by...Tom -->
Here is a <a href="http://www-cdfonline.fnal.gov/daq/daqExpert/daqmon.html">clickable link</a> corresponding to the URL above.
<!-- Date: Tue Oct 7 12:55:55 2008 -->
<h3>DPT Meters</h3> 
Warren and I went in on access to look at the DPT meters which are not functioning.  Before we started, the DPT meter for SVX_PCE was reading erratic values, and SVX_PCW, ISL_PCE, and ISL_CCW2's meters were reading zero or small negative numbers. 
<p> 
The DPT meters for SVX_PCW and ISL PCE were valved off, as noted before when Ricardo and Tom investigated.  Warren first checked that they were receiving 24V supply voltage, and then we opened the valves connecting them to the system.  Both of these meters started reading stable, if smaller than expected, values.  We will observe their outputs over time (at least on Friday we have another power cut and likely a flow trip so we can see if they respond to that). 
<p> 
ISL CCW2 was reading also zero, and had 24V on it.  We attempted to dismount it but were unsuccessful.  The bolts are very tight, and two of them are underneath elbows for the polyflow lines input to them.  The mechanical mounts are also flimsy, and could not hold the torque needed to dislodge a mounting bolt.  We checked a spare meter to see if the bolts were threaded in the standard direction and they are.  We were afraid of breaking the hoses and left it as is.  When we switched it back on, it read a nonzero value for about a minute before reverting back to its zero reading. 
<p> 
SVX_PCE was inaccessible due to plug source work -- the area was roped off. 
<p> 
We will monitor the exhaust flow rates out of both SVX and ISL, since we changed the valve status of these DPT meters.
<!-- Author: Tom Junk, Warren Bowman -->




<!-- Date: Tue Oct 7 13:15:58 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12958&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12958  width='300' align='top'></a></td><td><h3>Vacuump Pump Incident of July 15</h3> 
Remember the incident where the poly-flow of Vacuum Pump (VP1) of ISL was broken? This plot traces the evolution of the incident. The pressure of the air separator (dark blue) goes up; at 3.6 PSI the automatic control system switches the second pump VP2 on (red line). When pressure stabilizes around an average of 3.6 PSI, then VP2 is switched off and on. The confused cryo techs noticed the problem and tried to fix it manually by turning off/on the pumps. Apparently they did the wrong choices (the best was to switch VP1 off only). The pressure of the return flow (light blue) is highly correlated to the pressure of the air separator.</td></tr></table>
<!-- Author: Miguel M / Tom J -->


<!-- Date: Tue Oct 7 13:59:26 2008 -->
<h3> Vacuum pump maintenance work </h3> 
KC came and said he was planning on replacing ISL VP2A, as we had requested.  It pulls a vacuum of 2.0 to 2.1 PSIA, which is on the high side for normal running.  It was our backup pump, but was weak.  The one KC pulled out last week (was VP1A when it was pulled out) has a white, crystalline residue on the inside of the pumping chamber.  He'll clean it off and rebuild it. 
<p> 
The brand new pump (it pulls 1.1 PSIA on the bench with no load) is pumping now and KC is replacing VP2A.
<!-- Author: Tom, KC -->


<!-- Date: Tue Oct 7 14:05:40 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12961&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12961  width='300' align='top'></a></td><td>The dewpoint (red) and humidity (dark blue) have grown up in COT East. We have been informed that the COT guys are working in Collision Hall.</td></tr></table>
<!-- Author: Miguel M / Tom J -->
<!-- Comment: Tue Oct  7 14;33;34 comment by...Miguel M / Tom J -->
The silicon baggie dewpoint rose too. At this moment we are about 4  C below the trip level.

<!-- Comment: Tue Oct  7 14;52;09 comment by...MM / TJ -->
Tom called Bob Wagner (in Collision Hall) when the dewpoint was 2 C below the trip level; they closed the baggie and now the dewpoint are about 9 C below the trip level and dropping.


<!-- Date: Tue Oct 7 15:39:44 2008 -->
We complained when the dewpoint got as warm as -6C in the sili baggie on the East side.(our trip limit is -4C = -1C (lowest temp) - 3C (safety margin).  Bob Wagner said he only had a small slit in the West side open.  He raised the N2 flow and closed up, and the dewpoints fell back down to -25. 
<p> 
He started work again on the west side, and sili baggie dewpoints rose again, but only to -20, much better.  Bob is closing up again, will rase the flow for a bit, and then slit the baggie again.
<!-- Author: Tom Junk -->
<!-- Comment: Tue Oct  7 19;35;37 comment by...Miguel M -->
The COT work is done for today. They will resume tomorrow at 7:30. The dewpoints are going down to usual values.


<!-- Date: Tue Oct 7 20:12:34 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12962&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12962  width='300' align='top'></a></td><td><h3>Evolution of Dewpoints Monday-Tuesday</h3> 
The plots show dewpoints for SVX. The constant line is the trip level. This afternoon they were dangerously high, so the baggie had to be closed up. Yesterday there was a trip (the guys were out of the collision hall).</td></tr></table>
<!-- Author: Miguel M. -->


<!-- Date: Wed Oct 8 07:32:54 2008 -->
<pre>Another rainy day in Batavia.  COT guys are entering the collision hall to start work.  Hall humidity s 42%.</pre>
<!-- Author: Tom Junk -->


<!-- Date: Wed Oct 8 07:38:09 2008 -->
<pre>Missing some numbers on the Alarms page in iFIX:  SVX_TE_CHSUP, SVX_TE_CHLR_1 (evaporator temp. 1), and the quadlog status are all "@@@@".  Interestingly, the warning and alarm values (where applicable) are also @@@@, so it's almost certainly a communication failure with S3.</pre>
<!-- Author: Tom Junk -->


<!-- Date: Wed Oct 8 08:05:55 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12963&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12963  width='300' align='top'></a></td><td>KC replaced the vacuum pump ISL_VP2A yesterday.  The vacuum improved from 2.0 to 1.8 PSIA (KC got 1.1 on the bench out of the pump with no load).  I am mystified by the flow rate going out and the O2 reading.</td></tr></table>
<!-- Author: Tom Junk, MIguel Mondragon -->


<!-- Date: Wed Oct 8 11:00:57 2008 -->
Put back online in HWDB all ladders in SVX B0 which had been excluded last week due to e-valve failure. The following ladders are keep offline: e014, e023, e034, e060, e084 and e0b1
<!-- Author: SashaS -->


<!-- Date: Wed Oct 8 12:16:10 2008 -->
<h3>COT access done</h3> 
They are done with the COT work. The baggie has been sealed. The humidity is going down, now it is at 10% in COT. We will shortly start cooling down.<br>
<!-- Author: Miguel M / Sasha S / Satya -->
<!-- Comment: Wed Oct  8 14;15;34 comment by...Miguel M -->
The humidity is now below 3% and dewpoints below -23 C; the sealing  and N2 flow seem efficient.
<!-- Date: Wed Oct 8 14:11:48 2008 -->
<h3>Cooling-down</h3> 
The chiller set-points are now at their regular values; -10 C for SVX and 6 C for ISL. We cooled down in three steps. The warning/alarm regular levels have been restored in IFIX; as well as  the trip levels in PLC. The temperatures seem stable; the detector is in standby. All cooling related alarms have been unsilenced in IFIX. Sasha S and Sergo are doing the check-out.<br>
<!-- Author: Miguel M / Tom J -->



<!-- Date: Wed Oct 8 15:25:02 2008 -->
<h2> Detector checkout! </h2> 
We checked out both sides, east and west, everything looks ok. e393  was returning 0's for noise and dnoise but last pulse and pedestal looked ok, we are not sure if this is a new thing - will test further and will keep an eye on the quality of data coming from this ladder after shutdown.
<!-- Author: SashaS/Sergo -->


<!-- Date: Wed Oct 8 18:32:22 2008 -->
<h3>Preparations for the New Vacuum Pump Control Code</h3> 
We made a back-up of the PLC code and edited it in the test-stand (after a suggestion from Andy H). We modified the control of the vacuum pumps to catch-up eventual problems as that of last July 15. The code passed our tests. We plan to implement the changes in the running PLC tomorrow.<br> 
We also edited the Ifix DB with two new variables we will need.<br>
<!-- Author: Miguel M / Tom J -->
<!-- Comment: Wed Oct  8 18;39;27 comment by...Miguel M / Tom J -->
<pre>The request change form found in:

CDF Online --- &gt; Operations Home ---&gt; Operations Procedures

has been filled out and approved by Phil S (Detector Operations Leader) and Sasha S (SPL).</pre>


<!-- Date: Thu Oct 9 13:46:22 2008 -->
<h3>Air-Cooled Chiller Work</h3> 
This morning, starting at 8:00 AM, Jim Loskot, Jim Humbert, and Bruce Vollmer installed tees on the SVX chiller's CHW input.  They were finished by 11:00, and we restored flows.  We were chilling ISL the whole time but had turned off flows to SVX.  In hindsight, the backup pump was running and we could have flowed the still-cold coolant, but we had e-valve replacement work to do anyhow which required stopping flows.  
<!-- Author: Tom -->


<!-- Date: Thu Oct 9 13:53:20 2008 -->
<h3>E-Valve Electronics Replacement</h3> 
At about 11:00 AM Warren Bowman, Miguel Mondragon, and Tom Junk replaced the electronics for the e-valves for SVX_BHE2, SVX_BHW1, SVX_BHW2, and SVX_PCW, with new controllers.  SVX_BHW3 was replaced on Tuesday morning.  Warren was able to reproduce the failure mode of SVX_BHW3's e-valve electronics on his test bench; earlier it was reported that it was working fine, but that was only in the lower end of the range.  When asked to open the valve past 80%, the controller failed.  Warren has thrown that controller away. 
<p> 
The four we replaced today were chosen because they were the most open valves, at 75% to 85% open by default. Warren has chosen a new calibration setting for the replacements, so we hunted around for proper operating points for them, usually around 50%.  The new settings are: 
<pre> 
 
BHE1   70%   (unchanged, old valve still in there) 
BHE2   70%   (unchanged, old valve still in there) 
BHE3   50%   new controller 
L00E   37%   (unchanged, old valve still in there) 
PCE    65%   (unchanged, old valve still in there) 
 
BHW1   50%   new controller 
BHW2   50%   new controller 
BHW3   50%   new controller 
L00W   70%   (unchanged, old valve still in there) 
PCW    60%    new controller 
 
</pre> 
We will keep the four working ones we pulled out as preventative maintenance as emergency spares, and order new ones.  
<!-- Author: Tom Junk, Miguel Mondragon, Warren Bowman -->


<!-- Date: Thu Oct 9 13:55:40 2008 -->
<h3>Filter Replacement</h3> 
KC Replaced the microparticle filters on ISL and SVX this morning.  No adverse effects, but we did trip flows off to ISL (probably CCE1 PCOT) when that happened.  We recovered quickly.
<!-- Author: Tom Junk, KC Cahill -->


<!-- Date: Thu Oct 9 13:57:47 2008 -->
<h3>Warmup</h3> 
In anticipation of having detector power off this afternoon due to DAQ and database outages, as well as a life safety test, we changed the chiller setpoints to -5C for SVX and 10C for ISL.  This will prevent freezing of the portcards and hopefully will help keep COT happy.
<!-- Author: Tom Junk, Miguel Mondragon -->
<!-- Comment: Thu Oct  9 13;58;59 comment by...Tom -->
<pre>We will probably need to stay warm through tomorrow, since we will also not have power first for the power cut at 7:00 AM, and second for the generator swap work which starts after the power is restored.</pre>



<!-- Date: Thu Oct 9 19:31:27 2008 -->
<h3>PLC Code change implemented; vacuum pumps</h3> 
It is meant to improve the control of the vacuum pumps. In case that the pressure at the air separator goes too high, the system will switch to the second pump (failover) only once. This would prevent an incident as that of July 15 (see <a href= '/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=414&frame=2&anchor=16:54:05'> entry</a> and <a href= '/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=426&frame=2&anchor=13:15:58'> history plot above</a>). Andy Hocker helped us a lot. <br>
<!-- Author: Miguel M / Tom J  -->
<!-- Comment: Thu Oct  9 19;56;58 comment by...MM / TJ -->
The new code requires a new Ifix variable, a sort of "Failover enabled" which we have already implemented in the Ifix DataBase, but needs to be controlled from the Ifix GUI.  We will ask Jim Loskot to put it there as well as an Ifix alarm.


<!-- Date: Fri Oct 10 18:19:47 2008 -->
Went into access to replace the 2 power supplies we took out earlier in the week, as well as fix the Bias trip limits on LB1W5, which was replaced in Aug with incorrect limits. SB3W7 checked out OK (eventually), but IB4W5 gave some problems. It first only turned on channel 4 with the new loadbox, then all but channel 0 with the old loadbox (Bias00 was Ovv). When hooked to the new loadbox again, it showed the same Bias00 Ovv. We hope evertyhing checks out fine when things are turned on again
<!-- Author: Mark, Saty, SashaS, Roberto -->
<!-- Comment: Sun Oct 12 01;06;15 comment by...Mark -->
f450 was still tripping Ovv, so it was taken otu the HWDB. If we get an opportunity, we would like to take it to the test stand and reproduce the Ovv trip.
<!-- Date: Fri Oct 10 20:04:24 2008 -->
<h2>Generator Swap</h2> 
The generator swap followed the JHA procedure closely, except for the following items, more or less in the order in which they happened.  We went into the generator swap procedure shortly after recovering from a power outage, and the detector power supplies had been off since the previous day, for e-valve work, a life safety test and DAQ being down. 
<ul> 
<li> We blew a 5A fuse on one of the 24V power supplies in the first floor racks when moving it from generator power to alternate power.  We had a PLC error until we found another fuse, turned on the 24V supply, and re-initialized the modules.  Suggested mitigation: turn off the 24V supplies first before unplugging them.  We did this when re-plugging them. 
<li> The SVX vacuum pump we plugged into alternate power with an extension cord did not turn on.  We tried again with another extension cord, and also pushed the "reset" button on the receptacle a few times just to make sure.  It worked the second time. 
<li>  At 10:00, after we had established operating with the new power configuration, the PLC stopped being aware the vacuum pumps on alternate power were being powered.  There is a signal sent back from the alcove verifying that the pumps are on, and we lost it.  The consequences are that the solenoid valves for the vacuum pumps closed and the pressures in the air separators rose.  We forced on those inputs in the PLC indicating which vacuum pumps were on alternate power and got the valves to open. 
<li>  Related to this, while the pressure rose in the ISL air separator, the flows tripped off, and the level in the air separator rose to 29 inches.  We figure we drained the detector and plumbing as air leaked in, and the water went to the air separator.  There is a "high-level" trip closing the e-valves to one of the vacuum pumps in ISL (but not the other, just the one we were using).  The high pressure in the air separator kept the chiller pump from switching on, giving us a catch-22 situation. After consulting with Rob Roser, we overrode the high-level trip and pumped it down (the alternative was to drain some of the water and put it back when we were done).  Rob's suggestion succeeded and we were able to re-establish vacuum and flows. 
<li>  Most seriously, the power to SVX_PV_9001W was lost.  This is a solenoid valve with compressed air assist which shuts off the CHW supply to both the SVX and ISL chillers.  It is hooked up to a generator power circuit in hard conduit, and we did not notice this when making the list of things on generator power.  Shortly after generator power was cut, the SVX and ISL chillers' compressors tripped on high head pressure.  We were unable to restart the compressors until the CHW supply returned.  We operated in a power-cut mode, circulating cooling fluids which were not being cooled, until the power was restored.  The highest temperature the coolants got for SVX was about 13 degrees. 
<b>We should address this before the next generator swap</b>.  There may be a manual override button on the side of the solenoid control.  The aseembly hall pit was an ODH Class 2 hazard area, and the valve is about 10 feet off the floor, behing a lot of piping, making this awkward. 
<li>  We lost power to the PLC and its crates about a half hour before generator power was restored.  The UPS was drained.  The battery backup of the PLC program worked, and we did not have to reload the ladder logic, but we did lose settings of PID loops.  We also re-initialized the crates. 
</ul> 
In all it was an exciting day.  We are now recovered, and have learned quite a bit.  Thanks to Andy and Rob for advice.  Thanks to the process system crew: Bill Noe, Jim Loskot, KC Cahill, and all who helped us out.
<!-- Author: Tom Junk, Miguel Mondragon, Sasha Sukhanov, Satyajit Behari -->

<!-- Date: Fri Oct 10 20:08:08 2008 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W7L3 
<li>problem n: 2666 
<li>Status: needs_work
<li>Type: Unknown
<li>Action: Removed_hdwdb
<li>entered by: sukhanov 
<li>description is: This ladder fails CONFIG in RC. In svxdaq it runs just fine and does not show any problem 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B3W7&layer=L3">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Fri Oct 10 21:00:24 2008 -->
<h3>L00 flow warnings and alarms</h3> 
Cryo's alarms rang several times today on low flow for L00, both East and West.  Their flows are between 2.3 and 2.5 LPM, and the 
cryo low limits are 2.5 LPM.  Historically the flows had been 2.3 or so (looking back 10 days).  We first tried raising the flow rate to get away from the cryo low-flow alarm, but ended up hitting the PCOT alarms on the high side.  It turns out the warning and alarm limits were reset today as a consequence of the generator swap, and these must have been set higher than they were before.  We asked Jim Humbert to reduce their alarm limits to 1.8 LPM, to correspond to our low warning limit.
<!-- Author: Tom, Miguel -->


<!-- Date: Fri Oct 10 21:21:55 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12964&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12964  width='300' align='top'></a></td><td><h3>Temperature Trip Limits</h3> 
This plot is for future reference. It shows the trip limits loaded to the PLC via Ifix.</td></tr></table>
<!-- Author: Miguel M / Tom J -->


<!-- Date: Fri Oct 10 22:12:53 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12965&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12965  width='300' align='top'></a></td><td><h3>Step in temperature</h3> 
After the shutdown (10/6 - 10/10) the ISL cooling sensor  <b>CCW2L6_3OUT</b> shows an increase of temperature from about 7 C to 20 C. Also during check-out, a number of bias pinkies mostly in ISL west appeared; we think that it is an effect of the higher temperature in the circuit. We tried to aleviate the problem by increasing the flow. In order to do this, we adjusted the e-valve from 50 to 60%, the flow increased from 2.1 to 2.9 lpm; but the drop in temperature is only 0.6 C after 35 minutes.<br></td></tr></table>
<!-- Author: Miguel M / Tom J / Sasha S -->
<!-- Comment: Fri Oct 10 22;29;35 comment by...Tom -->
<pre>We raised the flow in the CCW4 input manifold, which corresponds to this cooling line (in the crazy mapping between manifolds and cooling lines).</pre>

<!-- Comment: Fri Oct 10 23;10;29 comment by...MM / TJ -->
This line had this problem before; see <a href=
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=400&frame=2&anchor=17:28:13'>
entry</a>.




<!-- Date: Sun Oct 12 19:12:29 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12966&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12967  width='300' align='top'></a></td><td><h3>Store 6486, Init. lumi 98.7 x 10^30 ub^-1 s^-1</h3> 
The long awaited store is in! TevMon was in warning because 
<a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2008&action=view&page=858&frame=2&anchor=17:48:39.145'>  
LOSTP was negative</a>. The SciCo (Morris) adjusted the  
offset to fix this. Then silicon was in. The SVTSpyMon and  
SVXMon plots look OK. 
</td></tr></table>
<!-- Author: Satyajit -->
<!-- Comment: Sun Oct 12 19;17;54 comment by...Satyajit -->
Sorry for the wrong unit! Shouldn't have 10^30.


<!-- Comment: Sun Oct 12 21;14;03 comment by...ronmoore -->
TEVMON doesn't care if LOSTP is negative.  (It is often negative later in HEP stores.)  TEVMON does check if the LOSTP RMS is too low, i.e. it is not updating, like it seemed to be here.
<!-- Comment: Sun Oct 12 22;22;40 comment by...Satyajit -->
Thanks for the correction! In this case (where RMS is too low due
to the ACNET variable not updating) do we consider alternative
variables?

<!-- Date: Sun Oct 12 19:20:05 2008 -->
ACE notified that IMon is stuck. Restarted it.
<!-- Author: Satyajit -->
<!-- Comment: Sun Oct 12 22;19;36 comment by...Satyajit -->
I was paged again that IMon is hung w/ out of memory error. The
ACE had to kill it and start afresh. Unfortunately the log file
contains nothing useful about these hangs/crashes.

<!-- Comment: Sun Oct 12 22;35;14 comment by...Mark -->
Was there a mess of channel info/plot windows open? A lot of times SciCos don't close them; they just open a new one for each pinky...
<!-- Comment: Sun Oct 12 22;51;55 comment by...Satyajit -->
No, at least in the recent cases I have made sure with the ACE that
there weren't many windows open. In the previous case IMon had
crashed with Java exceptions.
<!-- Date: Sun Oct 12 19:22:22 2008 -->
<H3>ISL bias pinkies</h3> 
After turning the detector ON we have several ISL bias pinkies, 
as expected. Some of the jumps are quite high, e.g. 400 uA vs a 
nominal of 200 uA. Let's hope these clear out soon.  
<!-- Author: Satyajit -->







<!-- Date: Sun Oct 12 23:07:10 2008 -->
<h3>Monitoring Summary</h3> 
 
No runs to check this week
<!-- Author: Mark -->


<!-- Date: Sun Oct 12 23:28:13 2008 -->
Was paged because of a new yellow ladder in SVXMon. The ladder 
e032 returned no data in Run 268158. In the later Runs it's doing  
fine.  
<!-- Author: Satyajit -->
<!-- Comment: Sun Oct 12 23;34;51 comment by...Satyajit -->
This is e052!


<!-- Date: Mon Oct 13 05:31:49 2008 -->
Got a page around 4:00am about pinkies showing up w/o pop-up 
windows! Also, one pinky showed current drop close to zero. 
Asked the ACE to Hockerize crate #5.  
<!-- Author: Satyajit -->


<!-- Date: Mon Oct 13 07:45:40 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12968&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-12969  width='300' align='top'></a></td><td><h3>Store 6487, init. lumi. 82.6 ub^-1 s^-1</h3> 
Silicon included from the beginning. SVTSpyMon and SVXMon 
plots look OK. 
</td></tr></table>
<!-- Author: Satyajit -->


<!-- Date: Mon Oct 13 08:35:33 2008 -->
Very wide variations in AVDD/DVDD/Bias currents for e052. The 
previous Hocker didn't help. Hoeckerized again. The currents are 
back to normal. Would keep an eye on this guy.  
<!-- Author: Satyajit -->
<!-- Comment: Mon Oct 13 08;36;08 comment by...Satyajit -->
This is PS crate #5.


<!-- Date: Mon Oct 13 09:32:58 2008 -->
<h3>Week Summary</h3><p> 
<h4>General</h4> 
<ul> 
 <li>Shutdown activities Monday through Friday</li> 
 <li>Power outages on Monday and Friday</li> 
 <li>Generator swap done</li> 
 <li></li> 
 <li></li> 
</ul> 
 
<h4>Monday</h4> 
<ul> 
 <li>Power outage from 7:00 to 7:20 am.</li> 
 <li>COT work began with silicon OFF and coolant temparatures at 
     their intermediate values (SVX:-4, ISL:10 degC)</li> 
 <li><a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=426&frame=2&anchor=16:21:18'>COT 
    dewpoints</a> cause an interlock trip.</li> 
 <li><a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=426&frame=2&anchor=17:05:18'>SVX 
     BHW3 flow</a> recovered after the power outage!</li> 
 <li>Stayed at the intermediate temperatures overnight</li> 
</ul> 
 
<h4>Tuesday</h4> 
<ul> 
 <li>VNODE2 snapshots are now available online 
     [ <a href="http://www-cdfonline.fnal.gov/java/snapshot/ShowImage.jsp?id=5fd1eadd">1</a> ] &nbsp; &nbsp; 
     [ <a href="http://www-cdfonline.fnal.gov/java/snapshot/ShowImage.jsp?id=5fd1eade">2</a> ] </li> 
 
 <li><a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=426&frame=2&anchor=12:55:55'>SVX PCW, 
     ISL PCE DPT meters checked</a></li> 
 <li><a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=426&frame=2&anchor=13:59:26'>Vacuum 
     pump maintenance</a></li> 
 <li><a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=426&frame=2&anchor=20:12:34'>Evolution 
     of dewpoints</a> during COT bore work</li> 
</ul> 
 
<h4>Wednesday</h4> 
<ul> 
 <li>COT access done. <b>Survived the COT bore access!</b></li> 
 <li>Cooled down to nominal (SVX:-6, ISL:10 degC) values, ladders 
     taken out of HWDB due to e-valve failure are reinstated and  
     <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=426&frame=2&anchor=15:25:02'>detector 
     checkout</a> came out fine!</li> 
</ul> 
 
<h4>Thursday</h4> 
<ul> 
 <li><a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=426&frame=2&anchor=13:46:22'>New 
     air-cooled chiller</a> work.</li> 
 <li><a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=426&frame=2&anchor=13:53:20'>E-valve 
     electronice replacement</a></li> 
 <li>Detector warmped up (SVX:-5, ISL:10 degC) in anticipation of 
     the Life Safety Test and the DAQ/DB outage. 
 <li><a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=426&frame=2&anchor=19:31:27'>PLC 
     code change implemented</a></li> 
</ul> 
 
<h4>Friday</h4> 
<ul> 
 <li><a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=426&frame=2&anchor=18:19:47'>Power 
    supply access</a> met with mixed success. IB4W5L0 out.  
    New loadbox tried out!</li> 
 <li>Generator swap done. Few <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=426&frame=2&anchor=20:04:24'>surprises!</a></li> 
 <li><a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=426&frame=2&anchor=21:21:55'>New 
     temperature trip limits</a></li> 
 <li><a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=426&frame=2&anchor=22:12:53'>Step in 
     CCW2L6_3OUT temperature</a> after the shutdown</li> 
</ul> 
 
<h4>Saturday</h4> 
<ul> 
 <li>Tevatron startup activities.</li> 
</ul> 
 
<h4>Sunday</h4> 
<ul> 
 <li>Store 6486, Init lumi 98.7e30. LOSTP negative delayed the start. 
 <li>IMon hung/crashed a few times.</li> 
 <li>ISL bias pinkies after the shutdown. Recovering..</li> 
</ul> 
 
<h4>Monday</h4> 
<ul> 
 <li>Store 6487, Init. lumi. 82.6e30.</li> 
 <li>No monitoring summary.</li> 
</ul> 
 
<p>
<!-- Author: Satyajit -->