<!-- Title: 06:53:04  Wed Feb 20 2008 -->
<!-- Crewchief: John Strologas (NM) -->
<!-- Op1: Dale Stentz (NWU) -->
<!-- Op2: Phil Schlabach (FNAL) -->
<!-- Op3: Andrew Kubik (NWU) -->
<!-- Op4: the many gone -->
<!-- Op5: but not forgotten -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
New page was needed, so here it is.<br><Br> 
 
John's MS HV configuration <a href= 'http://www-cdf.fnal.gov/internal/people/links/JohnStrologas/cmx/miniskirt.eps'>miniskirt.eps</a> 
<br><Br> 
Dale's Cable/iFIX wedge/Physical wedge table<br> 
<table style="background-color: rgb(0, 0, 0); width: 750px; text-align: left;" border="1" cellpadding="2" cellspacing="2"> <tbody> <tr> <td style="background-color: rgb(102, 0, 204); color: rgb(255, 255, 255); text-align: center; vertical-align: middle;">quadrant<BR> 
</td> <td style="background-color: rgb(102, 0, 204); color: rgb(255, 255, 255); text-align: center; vertical-align: middle;">slow control<BR> 
ifix channel<BR> 
</td> <td style="background-color: rgb(102, 0, 204); color: rgb(255, 255, 255); text-align: center; vertical-align: middle;">cable<BR> 
channel<BR> 
</td> <td style="background-color: rgb(102, 0, 204); color: rgb(255, 255, 255); text-align: center; vertical-align: middle;">old wedge/<BR> 
<span style="color: rgb(255, 255, 255); font-weight: bold;">ifix wedge</span><BR> 
</td> <td style="background-color: rgb(102, 0, 204); color: rgb(255, 255, 255); text-align: center; vertical-align: middle;">actual <BR> 
wedges<BR> 
</td> </tr> <tr> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">NW<BR> 
</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">1<BR> 
</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">4<BR> 
</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">0-1<BR> 
</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">0-1<BR> 
</td> </tr> <tr> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">NW<BR> 
</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">2<BR> 
</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">5<BR> 
</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">2-3<BR> 
</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;"><span style="font-weight: bold;">2</span><BR> 
</td> </tr> <tr> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">NW</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">3<BR> 
</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">6<BR> 
</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">4-5<BR> 
</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">4-5<BR> 
</td> </tr> <tr> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">SW</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">4<BR> 
</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">6<BR> 
</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">6<BR> 
</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;"><span style="font-weight: bold;">6+10</span><BR> 
</td> </tr> <tr> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">SW</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">5<BR> 
</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">5<BR> 
</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">8-7<BR> 
</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">7-8<BR> 
</td> </tr> <tr> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">SW</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">6<BR> 
</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">4<BR> 
</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">9-10<BR> 
</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">9<BR> 
</td> </tr> <tr> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">SW</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">7<BR> 
</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">3<BR> 
</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">11-12<BR> 
</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">11-12<BR> 
</td> </tr> <tr> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">SW</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">8<BR> 
</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">2<BR> 
</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">13-14<BR> 
</td> <td style="background-color: rgb(255, 204, 204); text-align: center; vertical-align: middle;">13-14<BR> 
</td> </tr> <tr> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">NW</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">9<BR> 
</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">2<BR> 
</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">21<BR> 
</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;"><span style="font-weight: bold;">21+3</span><BR> 
</td> </tr> <tr> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">NW</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">10<BR> 
</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">3<BR> 
</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">22-23<BR> 
</td> <td style="background-color: rgb(153, 255, 153); text-align: center; vertical-align: middle;">22-23<BR> 
</td> </tr> <tr> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">NE<BR> 
</td> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">1<BR> 
</td> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">4<BR> 
</td> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">0-1<BR> 
</td> <td style="background-color: rgb(255, 255, 153); font-weight: bold; text-align: center; vertical-align: middle;">1<BR> 
</td> </tr> <tr> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">NE<BR> 
</td> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">2<BR> 
</td> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">5<BR> 
</td> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">2-3<BR> 
</td> <td style="background-color: rgb(255, 255, 153); font-weight: bold; text-align: center; vertical-align: middle;">2<BR> 
</td> </tr> <tr> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">NE<BR> 
</td> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">3<BR> 
</td> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">6<BR> 
</td> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">4<BR> 
</td> <td style="background-color: rgb(255, 255, 153); font-weight: bold; text-align: center; vertical-align: middle;">3-4<BR> 
</td> </tr> <tr> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">SE<BR> 
</td> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">4<BR> 
</td> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">6<BR> 
</td> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">7<BR> 
</td> <td style="background-color: rgb(153, 255, 255); font-weight: bold; text-align: center; vertical-align: middle;">7-8<BR> 
</td> </tr> <tr> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">SE</td> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">5<BR> 
</td> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">5<BR> 
</td> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">8-9<BR> 
</td> <td style="background-color: rgb(153, 255, 255); font-weight: bold; text-align: center; vertical-align: middle;">9<BR> 
</td> </tr> <tr> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">SE</td> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">6<BR> 
</td> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">4<BR> 
</td> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">10-11<BR> 
</td> <td style="background-color: rgb(153, 255, 255); font-weight: bold; text-align: center; vertical-align: middle;">10<BR> 
</td> </tr> <tr> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">SE</td> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">7<BR> 
</td> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">3<BR> 
</td> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">12-13<BR> 
</td> <td style="background-color: rgb(153, 255, 255); font-weight: bold; text-align: center; vertical-align: middle;">11-12<BR> 
</td> </tr> <tr> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">SE</td> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">8<BR> 
</td> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">2<BR> 
</td> <td style="background-color: rgb(153, 255, 255); text-align: center; vertical-align: middle;">14<BR> 
</td> <td style="background-color: rgb(153, 255, 255); font-weight: bold; text-align: center; vertical-align: middle;">13-14<BR> 
</td> </tr> <tr> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">NE<BR> 
</td> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">9<BR> 
</td> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">2<BR> 
</td> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">21<BR> 
</td> <td style="background-color: rgb(255, 255, 153); font-weight: bold; text-align: center; vertical-align: middle;">21-22<BR> 
</td> </tr> <tr> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">NE<BR> 
</td> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">10<BR> 
</td> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">3<BR> 
</td> <td style="background-color: rgb(255, 255, 153); text-align: center; vertical-align: middle;">22-23<BR> 
</td> <td style="background-color: rgb(255, 255, 153); font-weight: bold; text-align: center; vertical-align: middle;">23+0<BR> 
</td> </tr> </tbody> </table><BR>


<!-- Date: Wed Feb 20 07:01:22 2008 -->
CMX west 6 (physical 6+10) was tripping off when it was being ramped up from stand-by following a previous access.  Trips are (to be confirmed) on the anode and it never makes it to full voltage.  <br><Br>The plan is to use the Control Access today to make repairs.  First guess is that there was an issue at the wedge 6-7 overlap but we haven't confirmed that the SW arch was pulled and it seems unlikely that the pcal02 work (which roughly the same phi wedge area) was related although it is a good lead at the moment.  Not impossible that someone "interacted" with wedge 10 (Silicon).  Basically we will have to debug from scratch and hope that no chambers have gone belly up.
<!-- Author: Dale and John -->
<!-- Comment: Fri Feb 22 10;10;38 comment by...Dale and John -->
<a href=
'/cgi/elog/elog.pl?nb=2008&action=view&page=152&frame=2&anchor=14:31:46.812'>Access Summary</a><br><Br>
More details:  Testing showed that wedge 10 was fine and testing 6 (from the cables at 10) showed trips as the voltage was ramped up to around 5.0kV.  The current draw as normal until it would suddenly (but slow enough to be seen on the drogie) go full scale and trip off.  Testing at 6 was slow due in part of us trying to be overly cleaver in testing the last stack.  Eventually, we tested the 1st five stacks and then the first 11 preamp boards and got the anode to hold (5.4kV w/ ~1 uA or 5.7 w/ 2 uA).  <br><br>
A slight degression: in order to test the last board we wanted to remove any and all unessary compoents.  For example, the last two  wires of stack 5 (to upper-right most chambers) are physically momented on wedge 7.  Because of the noise there, we elected to remove the preamps but we have kept the HV on.  In addition, wire 4 (the would-be chamber below preamp 11) does not exist.  The anode wire was "cut" and kapton taped.  We effectively repeated the same action the the last two wires and, additionally, removed the cathode from thoes chambers as well.  In doing so we recovered to SHV-to-jumper connections (they were needed to jump the gap between 6 & 7) and also the conecting cable.  We also undid the relavent grounds (one less thing to get tangled up).  In short, pre-amp 11 of wedge 6 had only one active channel and we made sure the HV reflected this reality before testing further.
<br><br>Testing the last pre-amp we got the same behavior of trips where as the rest of the wedge was able to hold (w/ wedge 10) when powered by the CAEN.  We removed the last pre-amp (no anode HV) and removed its only radeka.  We also removed the signal/LV cable and removed the anode conection between the board and the chamber.  Note that we did not undo the cathode (thus the only chamber of this preamp that exists on the structure of wedge 6 is still powered all be it completely isolated on the anode).<br><br>
Other stuff: The problem could have been how the preamp was resting on the steel frame work of the arch (the bottom of the board "leans" heavly on the steel on the lower left-hand corner).  Because this was a special preamp, replacing it (especially for just one channel) wasn't a realistic option.  Note that there was no signs of problems that could have been related to the IMU being moved in and out.  Nothing was damanged and our "pre-isolation" to test the one remaing channel removes the possiblity that the problem deals with one of the overlaping componets.  The anode leads from the preamp board were cut off from the horizontal SHV patch pannels (cables 1&2) and kapon taped off.  We didn't try to recover thoes (pink) anode SHV cables for fear of messing of something with the compoents that jump from wedge 7 to 6 (cables 3&4).  In recovering the cathode jumpers for wedge 6 across the gap, our intial attemp likely would have cause a great deal of destruction if the SW arch were to be pulled.  We beleive that the cable should unfold correctly now and also doesn't hang so low to be effected by IMU movement.  That being said, we should have techs and ops aware that there might be potenial issues.  In truth, the configuration is potentially better but it is always hard to visualize until the wires are actually streached out.


<!-- Date: Sat Feb 23 17:53:55 2008 -->
CMX NE 18-19 (a 20 degree section of the NE MS) started to trip off Friday evening, <a href= 
'/cgi/elog/elog.pl?nb=2008&action=view&page=159&frame=2&anchor=18:35:24.042'>see e-log entry here</a>.<br><br> 
The wedge is currently off.  The cathode definately looks bad on this one.  John was able to get it somewhat stable only to find that it went in to a more prononced tripping state.  I found that it would reliable (?) trip at 400V and was only stable at ~200 V.  Just now, I was able to get it up to 1.5 kV w/o tripping and with a nominal current draw.  Note that turning on the anode (e.g. to 1.5 kV) doesn't help or hurt.  Ramping up (the cathode) past 2.0 kV made it trip and now I cannot get it back past 400 V again.  Looks like a something actually went bad.  History suggests that its a chamber that went bad although this is the HV channel that uses the two spare HV cables (these are the 'new' ones that go from the 1st floor directly to the miniskirt without stopping at a patch pannel).  <br><br> 
John has submitted the access request.  If the detector is in this state on an access we should be able to get it down to 5-degrees fairly quick (or deduce that the cable is bad, however unlikely that is). Note that we should (just to be sure) switch the HV cables at the CAEN to make sure the problem isn't there (this is doublely-unlikely but you never know...).  
<!-- Author: Dale -->
<!-- Comment: Sat Feb 23 18;39;05 comment by...Dale -->
The wedge is reconnected back to the CAEN.  Possible access owl-shift Sunday.<br><br>Current pre-access HV configuration plus a reminder on the the CAEN cable names:
<pre>  HV Channel:     NE iFIX 18-19            NE iFIX 20 
============================================================ 
physical area:  18:08-47 + 19:00-23    19:24-47 + 20:00-07
drift cable:    SPARE HV NE 1          NE CM-NE-DRIFT 1 
anode cable:    SPARE HV NE 3          NE ANODE 7 [original] 
============================================================
</pre>
Thus iFIX 18-19 (currently) is the first four 5-degree sections.

<!-- Comment: Sun Feb 24 13;07;23 comment by...Dale and John -->
<a href=
'/cgi/elog/elog.pl?nb=2008&action=view&page=163&frame=2&anchor=00:28:35.444'>Summary of control access for NE iFIX 18-19.</a><br><br>
We discovered before the access that the HV channel held stand-by and full voltage without tripping and with normal current draw.  This despite having been tested 5 hours before and being unable to hold above 300 V.  Nevertheless, we resolved to divide the 20 degree section so that we had a 10/20 degree split (rather than the 20/10 that existed before).<br><Br>
The various cathode SHV cables and connections looked fine, i.e. no sign of foul play.  We first tested the default 15/15 setting (it held fine) before switching to the 10/20 as planed.  Note that we elected to promply test the wedges rather then optimize the SHV cables (swapping cables at the boundary of the two channels often has the effect of putting a long cable where a short one would be better and vise-versa).<br><br>
Currently the iFIX 18-19 channel is the one which, if it tripped off, we could afford to live with (mark the data good).

<!-- Comment: Sun Apr  6 23;15;21 comment by...Dale -->
<a href=
'/cgi/elog/elog.pl?nb=2008&action=view&page=291&frame=2&anchor=22:19:16.187'>NE 18-19 trips off</a>.  Started a new run without the 10 degree channel.<br><br>
The trips were on the cathode.  Trying on the droggie, we get a simular result as before: trips at very low voltage (~100 V) on ramp-up.  I tried it several times and even put voltage on the anode (1kV) to see if that would help.  Eventually it didn't trip and I ramped it up without issue to 3.0kV.  It has been holding for +5 minutes with a steady current draw of 0.26 microA.<br><br>I will try to re-ramp it from 0 to 500V several times to see if it goes back to its previous behavior.


<!-- Comment: Sun Apr  6 23;32;53 comment by...Dale -->
<a href=
'/cgi/elog/elog.pl?nb=2008&action=view&page=291&frame=2&anchor=23:30:00.751'>The wedge is back in</a>.  No issues on ramping/holding HV - turned it on during the current run.  The "CMX iFIX HV" inhibit will mark when we ramped it up.


<!-- Date: Thu Apr 24 20:49:26 2008 -->
SW 16-17 started to trip on the drift (also some trips on CMP and near-by 15).  <a href= 
'/cgi/elog/elog.pl?nb=2008&action=view&page=345&frame=2&anchor=19:43:20.193'> 
See shift log for more details</a>.<br><br> 
Below the shift log entry above I put in the humidy plots and it looks like we passed the trip threshold.  The good news is that it looks like we are coming down.  It was ~45% rel. humidity and a dew-point around 55 degrees at the peek.  The increse was sudden: 15% in under an hour.  Note that the change is very much in line with the outside weather.<br><br>I ramped it up just fine (3.0 kV) but after checking back it tripped and now the current draw ramps in that stupid way (i.e. 100V w/ .3mA and 200V w/ .6mA).  I plan to play with it some more although time is likely the best solution at this point.
<!-- Author: Dale -->
<!-- Comment: Thu Apr 24 21;07;36 comment by...Dale -->
It still trips at ~400V on the cathode.  I don't think there's much to be done now although I think it unlikly that it is really "broken".  I'll leave instructions to the effect that it should be ramped up every 2-3 hours to get it to recover.

<!-- Comment: Fri Apr 25 10;16;05 comment by...Dale -->
So far, this store hasn't had any muon trips and SW 16-17 is at full voltage w/o issue.  See updated plot and Phil's note about low (not full on) HVAC fans <a href=
'/cgi/elog/elog.pl?nb=2008&action=view&page=347&frame=2&anchor=08:30:52.683'>here</a>.

<!-- Comment: Fri Apr 25 15;40;59 comment by...Dale -->
Phil's comment in the <a href=
'/cgi/elog/det-system-elog.pl?nb=operations&action=view&page=187&frame=2&anchor=12:13:42'>ops e-log</a>:
<pre>
It looks like summer has arrived. About 1500h yesterday the storms started to move in. The humidity in the hall went from low 30s to 50s where the set point that controls it is. (Note this sensor isn't actually the one we regulate on which is (AC_AT_RP5_RAH), but that one isn't logged in the slow controls database.) This probably threw things out of whack in the hall causing the various cmp/x trips last night.</pre>
<pre> 
During the shutdown, we found that we were able to keep the collision hall dryer without as much effort as we thought. We had decided then that we would make this change permanent, but hadn't yet. This morning we reduced the relative humidity target from the 52 to 40 deg. Note that this is still above where we typically run during the winter (or even yesterday). One can see the hall responded quickly.</pre> 


<!-- Date: Wed Apr 30 07:56:56 2008 -->
There was an access today for D0 and the COT.  The SW arch needed to be pulled so I made sure to be present when it was.  Unfortinately, the ground cables (which connect preamps to chambers that are physically located on the other wedge) was all ready too tight around the anode (vertical) pannel on the SW CMX arch.  I relieved the strain and all way good until I noticed that the anode lead from the patch pannel broke off from the preamp board.  <br><br>Seeing that the alterative was that the entire pannel would have bought the dust, I'm happy that I went in but upset that the anode lead came off so easy.  This should corspond to wire 1 of stack 0 of wedge 7 or YMon #= 1 in west wedge 7.  Also note that this channel looked sick/dead from YMon (looking at #261518).  It is not impossible that it was already broken although I think this to be unlikely although I'm sure this connection has (via its location) cause tagle issues in the past.<br><br> 
Recovery prospects: slim but possible.  Someone would need to solider at the board - lots of issues that can will be hard to resolve.
<!-- Author: Dale -->
<!-- Comment: Wed Apr 30 08;14;41 comment by...Dale -->
Additional thoughts: HV holds fine (current draw at ~1.2 microA), the anode lead was simply tucked away after applying kapton tape.


<!-- Date: Sat May 10 18:18:54 2008 -->
Nearly forgot about the latest NE 18-19 (suspect 10 degrees) tripping session (starting May 6th):<br><br> 
<a href= 
'/cgi/elog/elog.pl?nb=2008&action=view&page=381&frame=2&anchor=23:13:51.582'>Page with instructions + plot below.</a>  The wedge didn't make a smooth recovery (humidity didn't drop).  <a href= 
'/cgi/elog/elog.pl?nb=2008&action=view&page=384&frame=2&anchor=16:51:10.588'>Removed from CAEN here</a> but then <a href= 
'/cgi/elog/elog.pl?nb=2008&action=view&page=385&frame=2&anchor=01:45:28.547'>recovered before the start of the next.</a>
<!-- Author: Dale, John, Phil -->


<!-- Date: Sun May 11 12:03:36 2008 -->
<a href= 
'/cgi/elog/elog.pl?nb=2008&action=view&page=394&frame=2&anchor=03:49:06.033'>NE 18-19 started to trip off again</a>.  This time I didn't even bother to have them end the run but rather just noted the trigger inhibit time stamp (2008.05.11 03:46:42 @ #698391).  Given the two recent run ins with this channels, I've decided to wait until the HVAC values start to fall back into place.<Br><BR> 
Currently the dew point/relative humidity are on there way done but  aren't a full degree/percent below where the wedge started to trip (50 degrees and 45%, respectively).  
<!-- Author: Dale -->
<!-- Comment: Sun May 11 13;29;22 comment by...Dale -->
update:<br>looks like I came in too soon, the wedge is tripping at around +300V on the cathode (same as before).  I can slow ramp it as high as ~500V but it trips very quickly.  Even at ~300V it will trip off randomly given enough time.<br><br>
The dew point plot appears to have miss lead me - there was a global disconitunity (where is a good firefox browser w/ spell check when you need one...) in the HVAC/slow control varibles.  I assumed that the plots where to continue to decrease but it appears that there was a (real) major drop and then another rise.  I can tell we are "close" to the wedge being ready but the values are heading the wrong way now.

<!-- Comment: Sun May 11 15;38;54 comment by...Dale -->
Updated dew point:<br>
<a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-499&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-499' width='300'></a>

<!-- Comment: Mon May 12 23;45;25 comment by...Dale -->
<a href=
'/cgi/elog/elog.pl?nb=2008&action=view&page=399&frame=2&anchor=21:12:46.467'>Finally recovered.</a>  Details to follow.

<!-- Comment: Tue May 13 07;00;12 comment by...Dale for John -->
<a href=
'/cgi/elog/elog.pl?nb=2008&action=view&page=400&frame=2&anchor=04:40:08.853'>Details here</a>.


<!-- Date: Fri May 16 23:29:12 2008 -->
<a href= 
'/cgi/elog/elog.pl?nb=2008&action=view&page=411&frame=2&anchor=22:40:38.823'>Another NE 18-19 trip-off</a>.<br><br> 
It tripped a few times on ramp up (~300V) but, while tweaking the anode, it held and I was able to bring it up to 2.9kV w/o issue.<br><br> 
I've posted the dew point plot (follow the link to the shift elog).  It was going up but is much lower than what it was Tuesday-Wednesday.  Also the realtive change doesn't look so bad either (wasn't the start of a large spike over a single hour).<br><br> 
Voltage is still holding - will try to recover with the TOF.
<!-- Author: Dale -->
<!-- Comment: Fri May 16 23;59;53 comment by...Dale -->
It is back in - <a href=
'/cgi/elog/elog.pl?nb=2008&action=view&page=411&frame=2&anchor=23:21:14.978'>link</a>.  As I note in the link, this trip seemed more random then before although I recall recovering channel's like this before (circa 2004-05).


<!-- Date: Sat May 31 12:10:02 2008 -->
<a href= 
'/cgi/elog/elog.pl?nb=2008&action=view&page=455&frame=2&anchor=10:53:53.414'>NE 20</a> trips multiple times - I realized my mistake <a href= 
'/cgi/elog/elog.pl?nb=2008&action=view&page=455&frame=2&anchor=11:57:22.169'>here</a>.<br><br> 
I allowed myself to be tricked into thinking that east 20 was the smaller 10-degree section rather than 18-19.  This was very sloppy mistake on my part which was confounded by the special rule we have for being able to run without 18-19.<br><br> 
The wedge is off (there were other trips in the previous shift) - no doubt due to the relatively high humidity (thankfully the value is flat).  Given that the wedge was generally able to recover (i.e. it resets ok, it was just tripping to fast) I like to bring it back but with the rel. humidity around 46% (dew point 52 degrees) it is unlikely to get better anytime soon.  We will aim to test around the day-evening shift transition.
<!-- Author: Dale Stentz -->


<!-- Date: Fri Jun 6 22:12:06 2008 -->
<a href= 
'/cgi/elog/elog.pl?nb=2008&action=view&page=474&frame=2&anchor=21:15:50.958'> 
SW 16-17 trips</a> taking 20 degrees out of the SW miniskirt.<br><br> 
On the cathode and simular to NE 20 the conditions (HVAC wise) were largely stable but with realtively high humidity.<br><br> 
Off for 40 minutes, on for 10 minutes with ~.5mA draw at 2.8kV, testing now - will restart with a new run assuming it doesn't trip (hasn't yet).
<!-- Author: Dale -->
<!-- Comment: Fri Jun  6 22;43;31 comment by...Dale -->
<a href=
'/cgi/elog/elog.pl?nb=2008&action=view&page=474&frame=2&anchor=22:17:33.459'>Back in</a> without issue.


<!-- Date: Sun Jun 8 23:42:08 2008 -->
<a href= 
'/cgi/elog/elog.pl?nb=2008&action=view&page=480&frame=2&anchor=23:22:15.902'>SE 16-17</a> et tu, Brute.<br><br> 
Cathode doesn't hold at 2.8kV, semi-stable at 1.0kV, stable at 800V.  Will inch it up.
<!-- Author: Dale -->
<!-- Comment: Sun Jun  8 23;47;19 comment by...Dale -->
with anode at a static 1.5kV, cathode up to 2kV ok with .33mA draw.  Will ramp down and up a few times - first two slow ramps ok, next one I went quick (~2-3x then iFIX would) and it tripped around 900V.  Back to where we started...

<!-- Comment: Mon Jun  9 00;06;13 comment by...Dale -->
HVAC conditions: flat since the previous entry, SE humidity is 45-46% while dew point is 52 degrees.  It was realtively cool when I left my apartment and started to rain as I was driving.<br><Br>
Inched it to 3.0kV (.5mA) - will keep it here but will have to think about ramping it up a few times.<br><br>
On the 5th ramp it tripped after getting to full voltage but was able to go back without tripping on the 6th.  7th try also ok - will try to see how ifix/CAEN treats it (will try to minimise downtime in running of course).


<!-- Comment: Mon Jun  9 00;28;24 comment by...Dale -->
<a href=
'/cgi/elog/elog.pl?nb=2008&action=view&page=481&frame=2&anchor=00:17:48.270'>Well, I tried</a>, the store was lost just as I was about to ramp up.  After words the wedge ramped to stand-by (end-of-store) without issue and to full voltage afterwords.  Current draw is 0.5mA.  I ramped up the other miniskirts, and will likely keep them there for some time (nothing in the main injector and only 50 in the RR).


<!-- Date: Thu Jun 12 02:02:20 2008 -->
<a href= 
'/cgi/elog/elog.pl?nb=2008&action=view&page=490&frame=2&anchor=00:11:06.631'>"regular" trips in NE 18-19</a> - it is still in and holding in... for now.<br><br>This is the 10 degree section that has the ~300V failure mode.  However, it is acting like a "regular" trip in the sense that it recovers/resets ok.  Because of this, I've instructed the ace to keep on resetting - it appears to be paying off (no trips in the last 40 minutes).  
<!-- Author: Dale -->


<!-- Date: Sun Jun 29 21:28:35 2008 -->
<a href= 
'/cgi/elog/elog.pl?nb=2008&action=view&page=543&frame=2&anchor=20:51:32.667'>NE 18-19 trips</a> - this time it is the continous trips at low HV.<br><Br> 
It is tripping on the cathode at voltages ~300V.  It is not even stable at 200V.  Slow ramping is ineffective - will likely have to run this way for the rest of the night.  Note that run 264015 has NE18-19 on for part of the time (see the trigger inhibit via the run summary page for details).  New aces are being "broken-in" still otherwise I would have opted to just start a new run.  
<!-- Author: Dale -->
<!-- Comment: Sun Jun 29 21;32;57 comment by...Dale -->
Additional note: Dew point at 52 while Realtive Humidity is 43%.  Values are somewhat erractic over the last 48 hours but largely flat (values are consistant with the average over the same period).

<!-- Comment: Mon Jun 30 10;31;21 comment by...Dale -->
The wedge is still crap.  Looks like anything above 250V will eventually trip.  Maybe stable at 200V.  In this configuration, it can never be slowly ramped up to a large value - 500V max before it auto-trips off.<br><br>
<a href=
'/cgi/elog/elog.pl?nb=2008&action=view&page=545&frame=2&anchor=08:14:59.441'>Phil already disconnected the HV cables</a> and given its current state I have to aggree.  The plan will be to run without this wedge for the next store (maybe stores).<br><br>
I will continue to test it for a bit longer at 200V (anode at 500V).  I have no strong evidence one way or another that keeping such a small voltage will help so I'll switch it off but leave it connected to the droggie for now.


<!-- Comment: Mon Jul  7 18;51;31 comment by...Dale and John -->
Rather lax report on the access back on July 2nd - <a href=
'/cgi/elog/elog.pl?nb=2008&action=view&page=551&frame=2&anchor=14:35:38.872'>see e-log report here</a>.<br><br>
We tested the first 5 degrees (wedge 18, stack 1) and it tripped with the same behavior as when it was 10 degrees.  We tested the remaining 25 degrees <i>using the same cables/ifix channel (NE 18-19)</i> and it held fine.  We believe that these two tests along with the previous knowledge that the problem does not follow the CAEN suggest that the problem is inside the 5 degrees and is not an artifact of the cables.
<br><br>
We left the 1st physical 5-degree section by itself on ifix NE 18-19 and put the remainder on NE 20.  Testing the working section yielded expected current draws (~0.1 uA and 630 uA, respectively).<br><Br>
We have been testing it off and on.  John was working on inching it up to be stable at and around 400-500V which Dale quickly mucked up when the trips on NE 20 became problematic (see entry below).  The plan will be to tested on the order of once every store or two.  It has not shown any signs of recovery so it may be down for the count.  We have submitted an access request to move steel and to fix the problem.
<!-- Comment: Tue Jul  8 09;44;45 comment by...Dale and John -->
<br><br>
Correction: by wedge 18, stack 1, we of course mean stacks 1-2.<br><Br>
We are going in to work on the 5-degree section assuming we get our steel moved.  We are still tripping so we look to be in good shape to fix this guy.
<!-- Date: Mon Jul 7 03:23:37 2008 -->
<a href= 
'/cgi/elog/elog.pl?nb=2008&action=view&page=565&frame=2&anchor=02:24:16.877'>NE 20 trips</a> - testing now after 30 minutes off - at 3k it is holding fine (+600 microA because its 25 degrees)<br><br> 
Will wait a bit and then try it with the CAEN/iFIX.
<!-- Author: Dale -->
<!-- Comment: Mon Jul  7 03;42;53 comment by...Dale -->
<pre>working just fine - been in for 15 minutes - the current run is being marked bad for CSX and thus there no reason to start a new run but it looks like it is ready to go.  No other trips else where.</pre>

<!-- Comment: Mon Jul  7 04;32;48 comment by...Dale -->
<a href=
'/cgi/elog/elog.pl?nb=2008&action=view&page=565&frame=2&anchor=03:59:21.700'>More details here</a>.  Note that the humidity has come down (was pushing 51% and a dew-point 56 degrees) so we are likely in the clear... for now





<!-- Date: Wed Jul 9 03:08:32 2008 -->
We had the opportunity to fix the isolated first two physical stacks of wedge 18.  We had a few hurdles such as the CSX work and arriving at different times but things worked out in the end and the steel was moved out and ready when we began.  <a href= 
'/cgi/elog/elog.pl?nb=2008&action=view&page=569&frame=2&anchor=11:48:16.969'>See shift summary here.</a><br><br> 
The problem was indeed a chamber (not any of the jumpers nor an artifact of something physically shorting the chamber).  The removed wire 4 (east wedge 18 YMon #12 or 18:1:4 [wedge:stack:wire]).  Note that wire 2, which is also on the far 'left' side of the NE miniskirt above 4, was previously by-passed and we believe that this was a causality from when some water got  splashed on the miniskirt.  With this chamber by-passed (radeka removed) the section held HV fine.<br><Br> 
We took the additional step of hooking up the HV in a fashion that may help with trips that had been occurring on iFIX NE 20.  The first 20 degrees are now on iFIX 18-19 while the last 10 are iFIX 20.  Given the recent history and assuming (perhaps incorrectly) that there is only one 'semi-bad' component, a trip on iFIX 18-19 suggests that the 3rd-4th 5-degrees are suspect.  Trips on iFIX 20 without issues on 18-19, would confirm the problem was in the last two 5-degree sections of the NE ms. 
<br><br> 
Currently, with the exception of the NW ms all iFIX channels with the larger inner section (e.g. 16-17 & 18-19) are 20 degree sections.  Thus the smaller ones (15 & 20) we can run without if need be without declaring the data bad except for the NW where,  as John has noted, we would be hosed either way.  We provide an update of the various miniskirt HV configurations below.  Each '[]' represents 2 physical stacks (a 5-degree section).  Note that the digram is good for thinking in an "iFIX manner". However, the west side is of course physically "reversed" with the wedge numbers increasing from right-to-left. 
<pre>Quad  iFIX 15    iFIX 16-17 
 SW    [][]       [][][][]  
 SE    [][]       [][][][] 
 
Quad  iFIX 18-19    iFIX 20 
 NW     [][][]      [][][] 
 NE    [][][][]      [][]</pre>In other news, we believe that the CSX problem has been addressed (with the bad PMT/channel removed).
<!-- Author: John and Dale -->


<!-- Date: Sun Jul 13 05:07:11 2008 -->
<a href= 
'/cgi/elog/elog.pl?nb=2008&action=view&page=583&frame=2&anchor=04:11:41.182'>NE 21-22</a> trips off during ramp-up to a new store - the iFIX wedge east 21 appears to be tripping on the cathode<br><Br> 
Putting 2.1kV on the drift, draws 0.5mA and it appears to be stable even without anything on the anode.  Above 2.2kV the current seems to jump a bit.  Putting something on the anode (2kV) seems to cut away the current wandering but once I hit 2.4kV on the drift it seems to want to trip off on the order of 20 seconds or less.  Trip time decreases when I go above.  This is made harder due to the fact that the channel is actually two (well perhaps only 1 and 5/6) wedges and draws +0.6mA.  <br><br> 
When it trips, the current does appear to physically go off scale (it trips fast but sometimes you can see the spike).  Will test it at 2.1kV and then try to inch it up if it holds there for 5 minutes.  The anode is at 2.0kV for "support".  
<!-- Author: Dale -->
<!-- Comment: Sun Jul 13 05;24;58 comment by...Dale -->
2150V looks ok for a few minutes.  At 2300V it trips in under a minute.  Looking at the current draw, it looks like a 5-10 microA "shift" (as if looking for a short) before it relaxes and this happens almost repeatively.  The anode (if one changes the scale enough) shows the resulting effect - need less to say it is a product of the drift and not the other-way-around.<br><br>
I've floated the idea of having an access for this guy to the sci-co.  I'll first switch the cables at the back of the CAEN to check it is not the pod (very unlikely I wager).

<!-- Comment: Sun Jul 13 05;31;05 comment by...Dale -->
It's not the CAEN.  Submitting access soon...

<!-- Comment: Sun Jul 13 06;18;58 comment by...Dale -->
WRT above, why would I need to check the CAEN?  Perhaps I'm overy parnoid or perhaps it is too eairly...<br><Br>
I'm testing via iFIX and chaning the 'on' settings.  I've confirmed that 2.1kV is fine but it looks like 2.2kV is where it breaks (trips on rampup despite the 100V minor differnece).  Testing 2.15kV - ok... 2175V ok so far...

<!-- Comment: Sun Jul 13 06;28;35 comment by...Dale -->
Got it up to 2200V for awhile and try to push it to 2250V, it tripped 10 seconds after it got there.  I will reset the values on the CAEN to normal and go back to the droggie.

<!-- Comment: Tue Jul 15 15;42;26 comment by...Dale -->
Resolution:
<br><Br>I submitted an access request and sure enough the store was aborted and we asked for a 1.5 hour access.  Please <a href=
'/cgi/elog/elog.pl?nb=2008&action=view&page=584&frame=2&anchor=09:17:41.034'>see shift summary here</a>.<br><br>
Wedge 21, stack 0, wire 2 was the bad guy (it shows up as expected as the only dead channel in YMon wedge 45 [24+21], channel 2 [3rd up from bottom].  All things considering it was a relatively quick search.  As is often the case with testing the arch, most of the time is spent removing the covers.  In this case, the problem was in an easy access area and the covers that had to be removed in the search went off and on relatively easily.  It also helped that it was at ground level and that the HV source connected to 21 and then to 22 and not the other way around.<br><br>
The chamber had its grounding wire on the cathode damaged such that it no longer was connected to the cathode face plate.  It did not show any signs of direct physical abuse.  One hypothesis is that the cable was simply "snagged" by someone passing by there.  Suspects exist (including this author) but the wire/chamber "stick out" the farthest at the start of NE 21 so one might wonder why it had not been damaged sooner.
<br><BR>
As suggested by the above, there is every reason to think that the chamber would function ok if the grounding was re-soldered back on.  The wire is even plenty long to do such a thing.  The grounding wire and the anode/sense wire from the preamp board are "tied together" with kapton tape a-plenty.  The recovered cathode jumpers looked to be in good shape.  The behavior of the chamber is effectively the same as noted in the previous entries; it shouldn't hold for long at and above 2.2kV.


<!-- Date: Tue Jul 15 15:53:29 2008 -->
NE 20 is causing trouble <a href= 
'/cgi/elog/elog.pl?nb=2008&action=view&page=590&frame=2&anchor=14:24:42.941'>see entry in the e-log here</a>.  This was happening at the start of a store where there were COT and Silicon issues, so we opted to run without for a while.  Phil encounter some odd behavior which seems to suggests a major screw up on the last time we reconfigured after successfully fixing the problem in wedge 18, stack 1.  John is testing this hypothesis and seeing if the chamber is ready for action.  Dale will post a YMon plot below:<br> 
<a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-501&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-501' width='300'></a>  
<br>
<!-- Author: Dale, John, Phil -->
<!-- Comment: Tue Jul 15 17;22;59 comment by...Dale and John -->
John has confirmed what Phil saw and that, taken together with the above YMon plot, suggest that the anode and cathode are not linked together.  It 5-degree configuration looks something like this:
<pre>iFIX NE #    18-19      20
Cathode     [][][][]   [][]
Anode       [][][]   [][][]</pre>
Although it is clear that this must have happened on the access to fix the NE miniskirt, it is very strange indeed that this happened.  What is done is done, and, more importantly, it is moot for the time being as John has recovered the HV and returned things back to the shift crew.  We will submit an access request to Ops to switch this back correctly.  Note that if 18-19 trips, it will almost certainly trip off the anode in 20 - we will be mindful of this.

<!-- Comment: Wed Jul 16 11;03;25 comment by...Phil -->
<pre>Note that probably if NE20 trips, you will have to turn off NE18-19 to clear the CMU hardware trigger inhibit until you get a chance to unplug NE20 anode...</pre>

<!-- Comment: Wed Jul 16 11;05;55 comment by...Phil -->
<pre>Or you could turn NE20 to standby (if you can) and mask the CMX software trigger inhibit. This would be so CMU would still be protected.</pre>

<!-- Comment: Fri Jul 18 12;38;51 comment by...John and Dale -->
<a href=
'/cgi/elog/elog.pl?nb=2008&action=view&page=599&frame=2&anchor=12:37:07.668'>It is fixed</a> - the anode matches the cathode now.


<!-- Date: Mon Jul 21 07:07:04 2008 -->
The plan for today is to take a stab at the last two stacks in the keystone wedge 5 (NW 4-5 in iFIX).  <a href= 
'/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=11&frame=2&anchor=14:50:41'>Link to the realvent entry/summary before.</a><br><br> 
The trip ought to be on the anode; high current draw with a trip rate which starts at 5kV.  Whether it is still there or not remains to be seen - first order of business will be to open up the covers and test each section.  Only thing stopping us now is the steel being open - hopefully in 30 minutes the chance will arise.
<!-- Author: Dale and John -->
<!-- Comment: Tue Jul 22 03;47;59 comment by...Dale, Phil, and John -->
correction: as seems to be the case with this wedge, we always confuse it with the keystone.  In fact, the problem is with the very end of wedge 4 (not 5) as suggested above.<br><br>
Here is the basic summary of the work done: <a href=
'/cgi/elog/elog.pl?nb=2008&action=view&page=608&frame=2&anchor=11:40:58.528'>link</a>.  We were able to figure out that preamp board 08 (which matches to stack 4, wires 0-3) was the bad guy.  Testing (see below) suggests the preamp board was the problem and not any chamber.  Our initial attempt to replace the board with a spare proved unsuccessful - likely the board itself was bad.  We decided to keep this section without the anode, effectively by-passing the first 4 channels in the 2nd to last stack of wedge 4 (the cathode was left, "as is")
<br><br>We actually got the anode to spark across the preamp board.  The exact problem is less clear.  Dale observed sparking in-between the resistors leading to the blue capacitor (which then ran off into the lead for the anode pin/wire).  However, sparking (when the anode connection was removed) seemed to come literally out of and through the blue capacitor in the upper right and corner.  Note that this is where the green reinforcement clamp was and the board was very much in solid/strong contact.  As suggested above, removing the anode lead from the board and taping it off had no effect so there is no reason to suspect any of the chambers are physically bad.  Also note that, there were no issues with the cathode once the board (anode HV) was removed from the chain.  That is, the funny business about the cathode acting odd (see the original entry's link) did indeed seem related to the anode.  Old current draw (iFIX 4-5) was 630; new is 760 uA as expected.<br><br>
Prospects for recovery are good.  John will post the needed details.  Testing the preamp board and being able to deduce a real problem from a manufactured one is a bit difficult.  Having a 10-degree ms section would help but in principle we can put 5kV on the board and that should be enough to test it.  The only bad news is that the iFIX channel has tripped twice (for sure the first was on the anode).  Perhaps this is an artifact of being off for +6 months, or perhaps there was another anode problem that had a longer time scale...



<!-- Date: Mon Dec 29 14:16:55 2008 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-502&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-502  width='300' align='top'></a></td><td>I raised software current limits today per this spreadsheet</td></tr></table>
<!-- Author: Phil -->


<!-- Date: Sun Feb 8 16:14:35 2009 -->
Spring time for miniskirts: <a href= 
'/cgi/elog/elog.pl?nb=2009&action=view&page=116&frame=2&anchor=14:59:57.232'>trip on SW 16-17</a> which is four 5 degree sections.  John had it removed; we are running with out it and with CMX marked bad.<br><br> 
Drift is solid up to 2.2 kV when it quickly trips off.  Putting a matching voltage on the anode (~2kV) doesn't help.  After holding it steady on the cathode at 2.1kV, moving it to 2.2 the current becomes shakey with 0.4 mA draw but then settles down. <br><br> 
I'm working on upping the voltage by 100Vs; up to 2.6kV with 0.45 mA.  Will try to re-ramp up to this a few times to see if it is really stable.  Now it is up to 2.9kV and I've gone up and down several times without incident.  Will hold for a bit and then inform the shift crew if it looks ready 
  
<!-- Author: Dale -->
<!-- Comment: Tue Feb 10 10;39;14 comment by...Dale -->
<a href=
'/cgi/elog/elog.pl?nb=2009&action=view&page=117&frame=2&anchor=16:43:08.087'>Back in</a> in the evening shift.



<!-- Date: Fri Feb 13 13:03:12 2009 -->
Phil observed that the NE 21 wedge was oscillating.  Looking at the scope shows that most of the wedge is noisy but is uglier at the low (bottom) end of the wedge.  Wedge 22 looked fine and the few miniskirt channels in 20 (same TDC as 21) were solid.<br><br> 
Turning the wedges on full, we see healthy cosmics in 22 and some near the high phi side of 21 but full-blown oscillation domination in the first few stacks.  The oscillation apeared to still be present with the HV off.<br><Br> 
I need to try to better locate where it is worse (at the level of the stack).  I've captured to scope pics to be uploaded later (1st @ stand-by, 2nd @ full HV in NE21).  I will play with the HV a bit more but this may be a losing battle.
<!-- Author: Dale -->
<!-- Comment: Fri Feb 13 13;22;53 comment by...Dale -->
With the HV on, almost every preamp board looks bad.  The two good ones (healthy cosmics) are 21-08 & 21-10.  These would be the last two "top" (inner) pre-amps.  As noted above, the condition doesn't change much with the HV off (other than the 8 good channels being off/dead as expected).<br><br>
Note that iFIX NE21 is actually all of wedge 21 and most of (5/6) of 22.  The effect of turning iFIX NE22-23 (that is 23-0) on/off/stand-by is indepedent of the condition in wedge 21.


<!-- Comment: Fri Feb 13 13;57;07 comment by...Dale and John -->
We're setting the wheels in motion for an access.  The plan is for John and a warm body to pull out LV/signal cables until they fine the guilty party.  The hard part will be the back in forth comunication.  Dale will stay in the counting room and look via the scope.  We will prep the access with having some spare (old) radekas but likely the best plan is to find the bad pre-amp board and recover the rest of the wedge and fix it later (hopefully Dale will be trained by then for CA).

<!-- Comment: Fri Feb 13 16;59;18 comment by...Dale and John -->
<br><a href=
'/cgi/elog/elog.pl?nb=2009&action=view&page=131&frame=2&anchor=15:27:01.436'>Shift Summary of Access</a><br><br>
John and the ace made quick work at removing the oscillation.  Dale  seemed to observe the signal return to normal upon pulling the first cable.  With pre-amps 00 and 01 (stack 0) off, there were normal cosics in all other stacks.  Putting the cables back in (one-by-one) there wasn't any return of oscillations nor any suspect noisy channels.  <br><br>Ramping the HV had no effect and a short cosmic run showed that the whole of CMX was ok.  We both think that the problem is unlikely to easily return and that, give the location, it shouldn't be stright foward to fix if need be.


<!-- Comment: Fri Feb 13 21;37;27 comment by...Dale -->
<br><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-503&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-503' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-504&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-504' width='300'></a><br><br>
Left plot is with the oscillations at stand-by while the Right is after the "fix".