<!-- Title: 13:26:35  Fri Apr  1 2005 -->
<!-- Crewchief: Dale Stentz (NWU) -->
<!-- Op1: Phil Schlabach (FNAL) -->
<!-- Op2: Andrew Kubik (NWU) -->
<!-- Op3: John Strologas (NM) -->
<!-- Op4: Ingyin Zaw (HU) -->
<!-- Op5: the many gone -->
<!-- Op6: but not forgotten -->
<!-- Notes: -->
<h2>Welcome to the CMX e-log.</h2> 
Seems like a <i>perfect day</i> to make a new e-log page (+ the other one was getting too big).


<!-- Date: Mon Apr 4 15:01:06 2005 -->
Since I had to stop the HVMon++ to mark CPR/CCR as "do not use" I decided to change the HV ramp settings. 
<p>  Seeing thoes dead time blocks in the store summary due to CMX trips made me feel a tad guilty.  I raised the ramp rate up to 100 V/s (was 50).  The ramp rate down I got a bit greedy on: 300 V/s (was 100).  As Phil pointed out ramping it down too fast causes HV to be overvoltage due to the limitation of the <i>RC</i> of the chambers.  CMX may very well behave like CMU when the voltages are rampdown.  My hope is that we can have the same iFIX behavior with 100 (200) V/s ramp up (down) settings.  If things get too screwy we'll go back to the orginal settings and live with things the way they were. 
</p> 
My shift elog entry is  
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=281&frame=2&anchor=08:44:05.665'> 
here</a>.
<!-- Author: Dale [1.0] -->


<!-- Date: Mon Apr 4 16:07:59 2005 -->
Some minor CMX problems over the weekend - basically things recovered.  See  
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=278&frame=2&anchor=14:50:42.070'> 
link here</a>.
<!-- Author: Dale [.25] -->


<!-- Date: Tue Apr 5 08:38:17 2005 -->
Via the trip log: 
<pre> 
Wed Mar 02 18:31:08 2005  CMX  H TRP  Crate # 5  Channel # 37 
Fri Mar 04 13:37:59 2005  CMX  H TRP  Crate # 4  Channel # 28 
Fri Mar 04 13:37:59 2005  CMX  H TRP  Crate # 4  Channel # 24 
Tue Mar 08 17:10:37 2005  [quench?  17 channels] 
Wed Mar 09 09:31:48 2005  [quench?  6 channels] 
Fri Mar 11 22:07:17 2005  CMX  H TRP  Crate # 6  Channel # 35 
Fri Mar 11 22:07:19 2005  CMX  H TRP  Crate # 6  Channel # 39 
Fri Mar 11 22:14:25 2005  CMX  H TRP  Crate # 5  Channel # 37 
Fri Mar 11 22:14:25 2005  CMX  H TRP  Crate # 5  Channel # 33 [5] 
Mon Mar 14 10:51:54 2005  CMX  H TRP  Crate # 6  Channel # 39 
Mon Mar 14 10:51:55 2005  CMX  H TRP  Crate # 6  Channel # 35 
Wed Mar 16 16:34:27 2005  CMX  H TRP  Crate # 5  Channel # 37 
Wed Mar 16 16:34:27 2005  CMX  H TRP  Crate # 5  Channel # 33 [6] 
Thu Mar 17 11:15:25 2005  [quench?  21 channels] 
Fri Mar 18 01:34:25 2005  CMX  H TRP  Crate # 5  Channel # 36 
Fri Mar 18 01:34:25 2005  CMX  H TRP  Crate # 5  Channel # 32 [2] 
Fri Mar 18 15:33:04 2005  [quench?  19 channels] 
Mon Mar 21 21:44:21 2005  [quench?  10 channels] 
Tue Mar 22 09:27:26 2005  CMX  H TRP  Crate # 5  Channel # 33 [3] 
Tue Mar 22 09:33:46 2005  CMX  H TRP  Crate # 5  Channel # 37 
Tue Mar 22 09:33:46 2005  CMX  H TRP  Crate # 5  Channel # 33 
Tue Mar 22 13:05:09 2005  CMX  H TRP  Crate # 5  Channel # 33 [3] 
Thu Mar 24 16:23:07 2005  CMX  H TRP  Crate # 5  Channel # 33 [1] 
Thu Mar 24 20:45:57 2005  CMX  H TRP  Crate # 5  Channel # 33 [2] 
Thu Mar 24 21:34:11 2005  CMX  H TRP  Crate # 6  Channel # 39 
Thu Mar 24 21:34:14 2005  CMX  H TRP  Crate # 6  Channel # 35 
Thu Mar 24 21:50:35 2005  CMX  H TRP  Crate # 6  Channel # 39 
Thu Mar 24 21:50:36 2005  CMX  H TRP  Crate # 6  Channel # 35 
Sun Mar 27 02:45:24 2005  CMX  H TRP  Crate # 5  Channel # 37 
Sun Mar 27 02:45:25 2005  CMX  H TRP  Crate # 5  Channel # 33 [2] 
Sun Mar 27 11:15:04 2005  CMX  H TRP  Crate # 5  Channel # 36 
Sun Mar 27 11:15:04 2005  CMX  H TRP  Crate # 5  Channel # 32 
Sun Mar 27 13:51:37 2005  CMX  H TRP  Crate # 5  Channel # 37 
Sun Mar 27 13:51:37 2005  CMX  H TRP  Crate # 5  Channel # 33 
Sun Mar 27 13:55:15 2005  CMX  H TRP  Crate # 5  Channel # 37 
Sun Mar 27 13:55:15 2005  CMX  H TRP  Crate # 5  Channel # 33 [1] 
Sun Mar 27 14:02:08 2005  CMX  H TRP  Crate # 6  Channel # 39 
Sun Mar 27 14:02:08 2005  CMX  H TRP  Crate # 6  Channel # 35 
Mon Mar 28 10:42:25 2005  CMX  H TRP  Crate # 5  Channel # 33 [2] 
Mon Mar 28 12:01:17 2005  CMX  H TRP  Crate # 6  Channel # 39 
Mon Mar 28 12:01:17 2005  CMX  H TRP  Crate # 6  Channel # 35 
Mon Mar 28 12:06:44 2005  CMX  H TRP  Crate # 6  Channel # 35 [3] 
Wed Mar 30 12:23:17 2005  CMX  H TRP  Crate # 5  Channel # 33 [1] 
Wed Mar 30 17:18:35 2005  CMX  H TRP  Crate # 5  Channel # 37 
Wed Mar 30 17:18:35 2005  CMX  H TRP  Crate # 5  Channel # 33 
Wed Mar 30 17:21:05 2005  CMX  H TRP  Crate # 5  Channel # 33 
Sat Apr 02 11:53:08 2005  CMX  H TRP  Crate # 6  Channel # 35 
Sun Apr 03 13:46:26 2005  CMX  H TRP  Crate # 6  Channel # 38 
Sun Apr 03 13:46:27 2005  CMX  H TRP  Crate # 6  Channel # 34 
Sun Apr 03 13:49:05 2005  CMX  H TRP  Crate # 6  Channel # 38 
Sun Apr 03 13:49:05 2005  CMX  H TRP  Crate # 6  Channel # 34 
Sun Apr 03 14:13:54 2005  CMX  H TRP  Crate # 6  Channel # 39 
Sun Apr 03 14:13:54 2005  CMX  H TRP  Crate # 6  Channel # 35 
Mon Apr 04 08:08:05 2005  CMX  H TRP  Crate # 5  Channel # 37 
Mon Apr 04 08:08:05 2005  CMX  H TRP  Crate # 5  Channel # 33 
Mon Apr 04 16:45:33 2005  CMX  H TRP  Crate # 5  Channel # 36 
Mon Apr 04 16:45:33 2005  CMX  H TRP  Crate # 5  Channel # 32 [1] 
Mon Apr 04 18:40:58 2005  CMX  H TRP  Crate # 6  Channel # 39 
Mon Apr 04 18:40:58 2005  CMX  H TRP  Crate # 6  Channel # 35 [9] 
Tue Apr 05 02:28:39 2005  CMX  H TRP  Crate # 6  Channel # 39 
Tue Apr 05 02:28:39 2005  CMX  H TRP  Crate # 6  Channel # 35 
Tue Apr 05 03:53:07 2005  CMX  H TRP  Crate # 5  Channel # 36 
Tue Apr 05 03:53:08 2005  CMX  H TRP  Crate # 5  Channel # 32 
</pre> 
Details to follow
<!-- Author: Dale -->


<!-- Date: Tue Apr 5 16:00:08 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-418&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-419  width='300' align='top'></a></td><td>Humdity over the last ~24 hours.  Details to follow.</td></tr></table>
<!-- Author: Dale -->
<!-- Comment: Thu Apr  7 14;08;48 comment by...Dale [.75] -->
There was only one trip during the last surge at the end.
<a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=284&frame=2&anchor=12:59:54.294'>
It happened around 13:00</a> about where the thing maximises. It recovered and everything else was quite...for a while.<br><br>I've looked resently and the new relative humidty and dew point values are about where they are when the maximum flatens out.  It's a good sign that nothing tripping around the clock but then again we still do have peroids of trouble.<br><br>
The thing to take away is that the threshold of 45 degrees/40% seems moot at this stage.  Sure having the bad section in the SW isolated may help but I would have thought othere areas would be trippping.  Extra monitoring can certainly help. Could there also be other HVAC factors to consider?
<!-- Date: Wed Apr 6 15:45:20 2005 -->
Woe is me.  I'm about a day behind on entires.<br><br><a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=287&frame=2&anchor=15:11:12.957'> 
Here's the most resent epsoide</a>.
<!-- Author: Dale [2.25] -->
<!-- Comment: Thu Apr  7 11;10;30 comment by...Dale [2.0] -->
Here's the prequel:
<p>
<a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=282&frame=2&anchor=18:55:17.170'>As noted in the shift e-log</a>.  The key information which is what I have observed before and later when I looked at it was "We tried to recover it three times, but the 'bottom' one (FIELD) trips immediately.".  I saw this before with the NE 20 ms and also with <a href=
'/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=8&frame=2&anchor=14:21:24'>
the SW miniskirt</a> as I noted in the previous e-log page.
</p>
I worked my magic by tripping off the wedge at <u>low</u> voltage (maybe just 100 V or so - basically trips off as soon as anything is applied).  5-10 seconds of this and it suddenly held.  At that point I could ramp the wedge up with ease.  It held and I even turned if off all the way and turned it back on after waiting a minute - ramped and held fine.  Wedge was put back in <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=282&frame=2&anchor=21:37:20.700'>
as noted here</a>.
<!-- Comment: Thu Apr  7 11;15;07 comment by...Dale -->
It seems like the <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=287&frame=2&anchor=15:11:12.957'>
3 wedge problem</a> was clearly not just a random problem.  Humdity may certainly be a factor but it's not obvious the problems are 100% the same.  In any case, the solution seems easy enough although the shift crew will have to page me when ever a wedge goes bad.
<!-- Date: Thu Apr 7 10:56:25 2005 -->
From the 4/6/05 muon meeting: 
<p>Here's Phil's <a href='http://www-cdf.fnal.gov/internal/WebTalks/Archive/0504/050406_muon_detector/05___other_1_4_acceptancevsdeadtime.pdf'> acceptance vs. # trips/shift plots</a> in pdf format.  We can conclude that 3 trips in a shift for a (15 degree) wedge is the optimal(??) limit - e.g. 4 trips we loose more in dead time then we make up with additional acceptance.  For a 30 degree wedge 5-6 is the highest we should take.</p><p> 
Here's my <a href='http://www-cdf.fnal.gov/internal/WebTalks/Archive/0504/050406_muon_detector/03___other_1_2_bad_runs.html'>CMX bad runs</a> web page.  Bottom line: data that's bad due to miniskirts being off is ~4.2 pb^-1.  Keystone/beam losses/no low voltage goes into my "really bad" CMX catagory - 1.5 pb^-1 but this number hasn't been growing.  The BAD-BAD-BAD (not recoverable!) CMX catagory is Trigger/TDC problems which comes in at 4.7 pb^-1.</p> 
George Ott suggested getting a cheap-o humdity sensor and sticking down into the abyss.  Would give us better data on what's really going on.
<!-- Author: Dale [1.5] -->
<!-- Comment: Sat Apr  9 22;11;16 comment by...Dale [.25] -->
The one thing I've yet to calculate is the amount of deadtime inbetween trips.  This is stright-forward but getting this in terms of luminosity is very non-trival.
<br><br>
The more interesting thing to note is that trip time on CMX appears to be on the 1.5 minutes time scale (a bit better than the 4 minutes I remember).
<!-- Date: Fri Apr 8 08:31:45 2005 -->
2 hour access starting at ~10:30.  The plan: segment the rest of the NE miniskirt (wedge "19").<br><br> 
Off to make the necessary preperations + get the needed SHV cables and 'T's.
<!-- Author: Dale -->
<!-- Comment: Fri Apr  8 13;30;05 comment by...Dale and Phil -->
<a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=293&frame=2&anchor=12:09:35.233'>Done and done</a>.
<p>
Now all of the NE miniskirt is resegmented.  We also moved things around in the SE miniskirt.  See the table where we've use  <b>wedge:stack</b> to define in which HV section each physical section is at.  Bascially the SE is back to "normal" aka 15 - 15 degrees wedges (Dale thinks there were trips in SE 16-17 but not SE 15) and NE is now 20 - 10 (before that it was 15 - 15 and before that 10 - 20).
</p>
<table
style="background-color: rgb(102, 51, 51); width: 600px; text-align: left; margin-left: auto; margin-right: auto;"
border="2" cellspacing="2" cellpadding="2">
<tbody>
<tr>
<td
style="vertical-align: top; background-color: rgb(255, 255, 255); text-align: center;">iFIX
HV wedge<br>
</td>
<td
style="vertical-align: top; width: 18%; text-align: center; background-color: rgb(255, 255, 255);">SE
15<br>
</td>
<td
style="vertical-align: top; width: 18%; background-color: rgb(255, 255, 255); text-align: center;">SE
16-17<br>
</td>
<td
style="vertical-align: top; width: 18%; background-color: rgb(255, 255, 255); text-align: center;">NE
18-19<br>
</td>
<td
style="vertical-align: top; width: 18%; background-color: rgb(255, 255, 255); text-align: center;">NE
20<br>
</td>
</tr>
<tr>
<td
style="vertical-align: top; background-color: rgb(255, 255, 255); text-align: center;">before
access <br>
</td>
<td
style="vertical-align: top; width: 18%; background-color: rgb(255, 255, 255); text-align: center;">15:5
- 17:0<br>
</td>
<td
style="vertical-align: top; width: 18%; background-color: rgb(255, 255, 255); text-align: center;">17:1
- 17:5<br>
</td>
<td
style="vertical-align: top; width: 18%; background-color: rgb(255, 255, 255); text-align: center;">18:1
- 18:4<br>
</td>
<td
style="vertical-align: top; width: 18%; background-color: rgb(255, 255, 255); text-align: center;">18:5
- 20:0<br>
</td>
</tr>
<tr>
<td
style="vertical-align: top; background-color: rgb(255, 255, 255); text-align: center;">after
access</td>
<td
style="vertical-align: top; width: 18%; background-color: rgb(255, 255, 255); text-align: center;">15:5
- 16:4<br>
</td>
<td
style="vertical-align: top; width: 18%; background-color: rgb(255, 255, 255); text-align: center;">16:5
- 17:4<br>
</td>
<td
style="vertical-align: top; width: 18%; background-color: rgb(255, 255, 255); text-align: center;">18:1
- 19:0<br>
</td>
<td
style="vertical-align: top; width: 18%; background-color: rgb(255, 255, 255); text-align: center;">19:1
- 20:0<br>
</td>
</tr>
</tbody>
</table>

<!-- Comment: Fri Apr  8 13;49;23 comment by...Dale [4.] -->
<br>



<!-- Comment: Fri Apr 22 12;56;00 comment by...Dale -->
I'm clearly wrong here: The SE is as shown but the NE I have setup like 15-15 not 20-10 as was done.  Thus,<br><br>
after access NE 18-19 = 18:1 - 19:2 (and NE 20 = 19:3-20:0)
<br>A new table for the NE miniskirt HV channels given 
<a href=
'/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=9&frame=2&anchor=15:01:58'>below</a>.
<!-- Date: Sun Apr 10 16:11:32 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-420&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-421  width='300' align='top'></a></td><td>Relative humdity went up by a factor of 2 and dew point by about 15 degrees since last night. 
<br><br><a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=299&frame=2&anchor=12:19:53.672'>A few trips here</a> and  
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=299&frame=2&anchor=13:33:08.436'>more  trips here</a>.  My instructions  
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=299&frame=2&anchor=16:01:27.591'> 
here</a>.  Thing to ask silicon folk: would a massive swing in humdity and other HVAC conditions have caused their silicon cooling problems?  Both happened about the same time .  See silicon info <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=299&frame=2&anchor=13:24:48.804'> 
here</a>.</td></tr></table>
<!-- Author: Dale -->
<!-- Comment: Sun Apr 10 16;18;20 comment by...Dale [.5] -->
See info on the other 3 wedge problem <a href=
'/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=9&frame=2&anchor=15:45:20'>
above</a>.


<!-- Date: Sun Apr 10 16:24:20 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-422&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-423  width='300' align='top'></a></td><td>More eye candy - Question for HVAC/ops: why is the humdity fine yesterday when it was almost as warm, etc.?  Is there any way to slowdown or proactively effect these rapid rises?</td></tr></table>
<!-- Author: Dale [.25] -->


<!-- Date: Sun Apr 17 21:28:49 2005 -->
My weekend fix... 
<p><a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=319&frame=2&anchor=06:28:56.199'>1 trip in the SW iFIX 16-17</a> (the 25 degree section).  Then a triplet in the NE iFIX 18-19 20 min later as <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=319&frame=2&anchor=06:44:22.825'> 
described here</a>.  They were having L2 type problems so I told them to try again in 10 minutes if they still hadn't started a run.  </p> 
They did and <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=319&frame=2&anchor=07:05:34.387'> 
it worked</a> but I was inbound anyway.  <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=319&frame=2&anchor=07:30:53.823'> 
Here's my thoughts and an HVAC plot</a>.<br><br> 
A few more trips in the next shift as noted <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=320&frame=2&anchor=10:03:18.893'> 
here</a>, <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=320&frame=2&anchor=14:15:44.705'> 
here</a>, and <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=320&frame=2&anchor=15:13:02.820'> 
here</a>.  A differnt spot each time and one reset fixes it.  The HVAC conditions were not much better by then - a bit lower but some hourly fluctuation.  
<!-- Author: Dale [1.] -->


<!-- Date: Wed Apr 20 07:54:01 2005 -->
NE iFIX 18-19 triped <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=327&frame=2&anchor=17:57:39.299'> 
yesterday</a>.<p>It appears to be the only wedge which doesn't recover after being reset right after the trip but has been brought back in as quick as 10 min.  In both cases, we were a bit lucky with L2 and L3 problems cause a long delay that allowed us to recover.  In this case, as like before, the wedge seems happy until the rel. humdity cycles down and then back up again.  I think ~40% rel. humdity or ~45 degrees dew point is the pseudo-threshold (note: when it recovers it doesn't seem to care about the damn humdity - actually appears more sensitive [as do the other ms] to rapid changes in the HVAC conditions).</p> 
I've now seen enough evidence to suspect that <a href= 
'/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=9&frame=2&anchor=08:31:45'> 
via our reconfiguration above</a> that the 4th 5 degree section or 18:5 and 19:0 (wedge:stack) might be the bad guy.  Before the majority of trips came from NE iFIX 20.  I'll make up an access request in the very near future.
<!-- Author: Dale [.75] -->
<!-- Comment: Wed Apr 20 15;41;50 comment by...Dale [.25] -->
<br>I've submitted the access request to reconfigure NE ms.  Also requested an access to segment the HV on the lower phi part of the SE ms (a much lower prioity at this point).<br><Br>The realtive humdity is steady just below 40% and CMX is acting fine...


<!-- Date: Thu Apr 21 15:01:58 2005 -->
<b>Control Access report on the NE and SE miniskirts:</b> 
<br><br> 
Details to be posted below.
<!-- Author: Dale and Phil -->
<!-- Comment: Thu Apr 21 15;18;26 comment by...Phil and Dale [2.25] -->
<p>From the <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=332&frame=2&anchor=15:10:36.016'>
shift elog</a> - basically we resegmented the NE miniskirt so that it was 25 - 5 and did the HV segmentation for the lower phi part of the SE miniskirt.</p>  I'll add a table with what HV maps to what physical stacks.  Basically if NE iFIX 20 goes bad (trips due to humdity spike) this will signal that we found the bad 5 degree section.  He'll then be in the same boat as SW iFIX 15 which is also narrowed down to just 5 degrees.  This <i>might</i> help by itself (the SW hasn't been tripping as often + it recovers on reset) but the real goal here is to be able to turn off this section for a store if it should start to go bad.

<!-- Comment: Thu Apr 21 21;20;43 comment by...Dale [.5] -->
<bR><Br>Lots of tom foolery in the evening shift as noted <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=333&frame=2&anchor=19:57:40.937'>
here</a> and <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=333&frame=2&anchor=20:58:09.514'>
here</a>.  I'm at a loss for words...


<!-- Comment: Fri Apr 22 13;01;26 comment by...Dale [.25] -->
<p>Here's the new HV configuration for the NE miniskirt:</p>
<table
style="background-color: rgb(102, 51, 51); width: 500px; text-align: left; margin-left: auto; margin-right: auto;"
border="2" cellspacing="2" cellpadding="2">
<tbody>
<tr>
<td
style="vertical-align: top; text-align: center; background-color: rgb(255, 255, 255); width: 20%;">iFIX
HV wedge<br>
</td>
<td
style="vertical-align: top; background-color: rgb(255, 255, 255); text-align: center; width: 30%;">NE
18-19 [ms 30]<br>
</td>
<td
style="vertical-align: top; background-color: rgb(255, 255, 255); text-align: center; width: 25%;">NE
20 [ms 10]<br>
</td>
</tr>
<tr>
<td
style="vertical-align: top; background-color: rgb(255, 255, 255); text-align: center; width: 20%;">before
access <br>
</td>
<td
style="vertical-align: top; background-color: rgb(255, 255, 255); text-align: center; width: 30%;">18:1
- 19:2
</td>
<td
style="vertical-align: top; background-color: rgb(255, 255, 255); text-align: center; width: 25%;">19:3
- 20:0
</td>
</tr>
<tr>
<td
style="vertical-align: top; background-color: rgb(255, 255, 255); text-align: center; width: 20%;">after
access</td>
<td
style="vertical-align: top; text-align: center; width: 30%; background-color: rgb(255, 255, 255);">18:1
- 19:0 + 19:3 - 20:0<br>
</td>
<td
style="vertical-align: top; background-color: rgb(255, 255, 255); text-align: center; width: 25%;">19:1
- 19:2<br>
</td>
</tr>
</tbody>
</table>

<!-- Comment: Fri Apr 22 17;04;41 comment by...Dale [.5] -->
<br>two birds of a feather flock together:<br><br>
CMX NE 20 (the isolated 5 degree section) tripped off (once; then recovered) the same time CMP did (it wasn't as lucky...).  See <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=335&frame=2&anchor=14:38:51.529'>
shift entry</a>.  <P>This happened after we had the wire chambers at stand-by and then turned thing back on after MCR gave us the all clear.  The humdity was steady and low - the fact that these sections (CMP and CMX) seem to go together is either real (i.e. there is a common effect to both of them) or just consisdence.  The CMP problem is with the anode and they suspect it's the HV board - this seems the likely option to me but we'll see if we start having any more of these double trips now that we have the bad 5 degrees by himself.</p>


<!-- Date: Tue Apr 26 14:35:29 2005 -->
NE 20 (the suspect isolated 5 degrees) trips durning a rather hetic time.  First noted <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=347&frame=2&anchor=09:42:45.770'> 
here</a> and by the aces <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=347&frame=2&anchor=11:14:08.410'> 
here</a>.  
<p>I was in route at the time for B0 to see what was up with CMP.  After everything settled down I turned on NE 20 after it been holding at stand-by since before I arrived.  As I expect it came up just fine and has been holding ever since.  The HVAC dew point / humdity look flat and fine - the wedge just decided to trip. 
</p> 
The dumb thing was that I broke my own rule - I changed CMX HV (for NE) while taking data.  Such a silly move.  HV was ramped up before there was a pcal10 problem - the HRR and when for sure CMX was on is noted in the table here: 
<br><br> 
<table style="background-color: rgb(0, 0, 0);" border="1" 
cellpadding="4" cellspacing="2"> 
<tbody> 
<tr> 
<th style="background-color: rgb(255, 255, 0);">RunNumber</th> 
<th style="background-color: rgb(255, 255, 0);">TransitionTime</th> 
<th style="background-color: rgb(255, 255, 0);">RunTime, <i>seconds</i></th> 
<th style="background-color: rgb(255, 255, 0);">Transition</th> 
<th style="background-color: rgb(255, 255, 0);">At Event</th> 
</tr> 
<tr align="center"> 
<td colspan="5" rowspan="1" 
style="vertical-align: top; background-color: rgb(255, 255, 0);">---<br> 
</td> 
</tr> 
<tr> 
<td style="background-color: rgb(255, 255, 0);" align="center"><a 
href="http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196988">196988</a></td> 
<td style="background-color: rgb(255, 255, 0);" align="center"><a 
href="http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&amp;action=view&amp;page=Tue_Apr_26_2005_Day&amp;anchor=11:21:06.708">2005.04.26 
11:44:09</a></td> 
<td style="background-color: rgb(255, 255, 0);" align="right"><span 
style="color: rgb(255, 0, 0);">[<span style="font-weight: bold;">CMX 
has been on</span>]</span> 1396</td> 
<td style="background-color: rgb(255, 255, 0);" align="left">HALT<br> 
</td> 
<td style="background-color: rgb(255, 255, 0);" align="right">174105</td> 
</tr> 
<tr> 
<td style="background-color: rgb(255, 255, 0);" align="center"><a 
href="http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196988">196988</a></td> 
<td style="background-color: rgb(255, 255, 0);" align="center"><a 
href="http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&amp;action=view&amp;page=Tue_Apr_26_2005_Day&amp;anchor=11:21:06.708">2005.04.26 
11:45:08</a></td> 
<td style="background-color: rgb(255, 255, 0);" align="right">1455</td> 
<td style="background-color: rgb(255, 255, 0);" align="left">RECOVER</td> 
<td style="background-color: rgb(255, 255, 0);" align="right">174105</td> 
</tr> 
<tr> 
<td style="background-color: rgb(255, 255, 0);" align="center"><a 
href="http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=196988">196988</a></td> 
<td style="background-color: rgb(255, 255, 0);" align="center"><a 
href="http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&amp;action=view&amp;page=Tue_Apr_26_2005_Day&amp;anchor=11:21:06.708">2005.04.26 
11:45:18</a></td> 
<td style="background-color: rgb(255, 255, 0);" align="right">1465</td> 
<td style="background-color: rgb(255, 255, 0);" align="left">RUN</td> 
<td style="background-color: rgb(255, 255, 0);" align="right">174105</td> 
</tr> 
</tbody> 
</table> 
<br> 
I made some offical instructions and posted them near the HV summary page in the control room.  They are noted <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=347&frame=2&anchor=13:49:10.504'> 
here</a>.
<!-- Author: Dale [1.25] -->


<!-- Date: Tue May 3 15:03:29 2005 -->
There an access tentatively planed for next Thursday (Cinco de Mayo). 
<p> 
The plan would be to open up one of the bad 5 degree section (likely the one on the SW) and try to "guop" it with this new RTV spray.  I'll test the new stuff (via asking Dervin) on one of the 10 degree ms and see how it works.  If it looks useable and pratical to use we'll go for it.  The working assumption is that we need to remove the preamp board but not the anode HV wires, etc.  If we have to take out the full guts as well to put the stuff on it'll be a bit too much and tight to get out in 4 hours.  I'll post my findings asap. 
</p> 
Need to make sure that the miniskirts get back in the same spot (+/- 1 mm) to appease Victoria and that nothing gets broken (e.g. nobody damages the detector).
<!-- Author: Dale [1.] -->
<!-- Comment: Thu May  5 07;14;02 comment by...Dale [.5] -->
The RTV stuff is a bit unworkable in my oppion - we may just try using corna dope.  The plan is to work on the SW miniskirt on the isolated 5 degrees and try to make the cathode more robust (make a seal around the drift jumper connection.  We'll make everything else up as we go.


<!-- Date: Thu May 5 14:53:17 2005 -->
<b>Access summary for the <i>Cinco de Mayo</i> access:</b><p> 
Details to follow.  Shift E-log entry is <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=374&frame=2&anchor=13:20:18.444'> 
here</a>. 
</p>
<!-- Author: Dale [7.5] -->


<!-- Date: Thu May 5 16:01:33 2005 -->
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=374&frame=2&anchor=15:02:15.730'>This is a first</a>.<br><br>All the miniskirts trip together.  Suspects in order of less crazy to super crazy:<br> 
(1) The CMU CAENs [but no problems in CMU]<br> 
(2) TOF guys - putting in a new CAEN [differnt rack/cannot mess up all channels at same time]<br> 
(3) Bad grounding [except the MS are grounded to everything else/why both sides]<br> 
(4) Plug sorceing - huge increase radiation flux... [never happened before plus the physics makes no sense]<br> 
<br>  
<!-- Author: Dale [.5] -->
<!-- Comment: Thu May  5 17;34;55 comment by...phil -->
<pre>was this coincident with search and secure? it was supposed to happen at 1500h.</pre>


<!-- Comment: Fri May  6 12;36;52 comment by...Dale [0.] -->
<br>Should take up Phil comment and ask ops when things were finished - also look at the HVAC again to pick up any presure correlation.<br>  I would put the door closing above all the others (perhaps tie with 1) I listed above - crazy but it's much better then my ideas.
<!-- Date: Thu May 5 18:25:45 2005 -->
More <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=375&frame=2&anchor=16:21:56.307'> 
trouble</a>... 
<bR> 
<br>CMX iFIX NW 20 (around where the CMP work was done) start to trip.  <i>Very simular</i> to the problem seen before: trips on the cathode at very low voltage - looks like it won't hold anything - successive trips "solves" the problem - able to ramp HV through with normal current responce.  I got it at 3.0 and 5.6 kV holding strong.  Putting back with the CAEN/iFIX.
<!-- Author: Dale -->
<!-- Comment: Fri May  6 12;32;52 comment by...Dale [1.0] -->
Highly unlikely this is related to the CMP work but it's something to think about.  My <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=375&frame=2&anchor=17:47:02.120'>
comments in the shift e-log</a> say that the action above was a success.



<!-- Date: Tue May 10 16:00:42 2005 -->
I took a peak at the humdity levels in B0 - dew point was just above 50 degrees with rel. humdity at 42%.  It looked like it had been constant for a long time.  No trips seen in CMX for several stores now - we'll wait and see what happens as the outside becomes more and more summer like.
<!-- Author: Dale [.25] -->
<!-- Comment: Tue May 10 20;39;00 comment by...Dale -->
Jinxed myself.  Took out iFIX SW 15 as noted <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=390&frame=2&anchor=20:25:25.776'>
here</a>.  More details to come.

<!-- Comment: Wed May 11 12;50;16 comment by...Dale [.15] -->
This happened at the very tail end of run <a href='http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197763'>197763</a>.  Sadly the run was ended (a bit of mis-comunication on my part and the shift crew) so it was silly not to wait until we ended the run.  Details:
<pre>
CMX wedge 15 off after:
  2005.05.10 20:23:23 	19073274 	86985 	null

Back to Activate:
  2005.05.10 20:23:35	87000	RUN	19073274

Ended run:
  2005.05.10 20:24:15	87040	END	19079086
</pre>
<br>
Note that the rel. humdity and such actually went down and back up [it correlates to the back up part] - not that sudden of a movement but proof that things were changeing.  I think I remember a slight dip (when it started to go down) when I checked before.  Sigh...

<!-- Comment: Wed May 11 15;10;47 comment by...Dale [.5] -->
The next store (4136) the same wedge had trouble but the shift crew took the correct action and took it out for the night.  See <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=390&frame=2&anchor=23:58:24.766'>
here</a>.  I got it back up in the same run infact.  Details are given below:
<h3><a
href="http://www-cdfonline.fnal.gov/java/cdfdb/servlet/Inhibits?RUN=197808">Trigger
Inhibit</a> Events during Run <a
href="http://www-cdfonline.fnal.gov/java/cdfdb/servlet/RunSummary?RUN_NUMBER=197808">197808</a></h3>
<table>
</table>
<table style="background-color: rgb(0, 0, 0);" border="1"
cellpadding="4" cellspacing="2">
<tbody>
<tr>
<td
style="vertical-align: top; font-weight: bold; text-align: center; background-color: rgb(255, 255, 0);">InhibitTime</td>
<td
style="vertical-align: top; font-weight: bold; text-align: center; background-color: rgb(255, 255, 0);">AtEvent</td>
<td
style="vertical-align: top; font-weight: bold; text-align: center; background-color: rgb(255, 255, 0);">RunTime<br>
<i>seconds</i></td>
<td
style="vertical-align: top; font-weight: bold; text-align: center; background-color: rgb(255, 255, 0);">Components</td>
</tr>
<tr align="center">
<td colspan="4" rowspan="1"
style="vertical-align: top; background-color: rgb(255, 255, 0);">- - -<br>
</td>
</tr>
<tr>
<td style="background-color: rgb(255, 255, 0);"><a
href="http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&amp;action=view&amp;page=Tue_May_10_2005_Eve&amp;anchor=23:47:33.921">2005.05.10
23:57:26</a></td>
<td style="background-color: rgb(255, 255, 0);" align="right">61826</td>
<td style="background-color: rgb(255, 255, 0);" align="right">516</td>
<td style="background-color: rgb(255, 255, 0);">CMU01 East Trip</td>
</tr>
<tr>
</tr>
<tr>
<td style="background-color: rgb(255, 255, 0);"><a
href="http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&amp;action=view&amp;page=Tue_May_10_2005_Eve&amp;anchor=23:47:33.921">2005.05.10
23:57:31</a></td>
<td style="background-color: rgb(255, 255, 0);" align="right">61826</td>
<td style="background-color: rgb(255, 255, 0);" align="right">519</td>
<td style="background-color: rgb(255, 255, 0);">CMX HV:iFix CMU01
East Trip</td>
</tr>
<tr>
</tr>
<tr>
<td style="background-color: rgb(255, 255, 0);"><a
href="http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&amp;action=view&amp;page=Tue_May_10_2005_Eve&amp;anchor=23:47:33.921">2005.05.10
23:57:43</a></td>
<td style="background-color: rgb(255, 255, 0);" align="right">61826</td>
<td style="background-color: rgb(255, 255, 0);" align="right">531</td>
<td style="background-color: rgb(255, 255, 0);">CMX HV:iFix</td>
</tr>
<tr>
</tr>
<tr>
<td style="background-color: rgb(255, 255, 0);"><a
href="http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&amp;action=view&amp;page=Tue_May_10_2005_Eve&amp;anchor=23:47:33.921">2005.05.10
23:57:51</a></td>
<td style="background-color: rgb(255, 255, 0);" align="right">61826</td>
<td style="background-color: rgb(255, 255, 0);" align="right">540</td>
<td style="background-color: rgb(255, 255, 0);">null <span
style="color: rgb(255, 0, 0);">[CMX SW 15 is off]</span><br>
</td>
</tr>
<tr align="center">
<td colspan="4" rowspan="1"
style="vertical-align: top; background-color: rgb(255, 255, 0);">- - -<br>
</td>
</tr>
<tr>
<td
style="vertical-align: top; background-color: rgb(255, 255, 0);"><a
href="http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&amp;action=view&amp;page=Wed_May_11_2005_Day&amp;anchor=08:15:32.781">2005.05.11
08:11:41</a></td>
<td
style="vertical-align: top; background-color: rgb(255, 255, 0);">7976720<br>
</td>
<td
style="vertical-align: top; background-color: rgb(255, 255, 0);">30171<br>
</td>
<td
style="vertical-align: top; background-color: rgb(255, 255, 0);">CMX
HV:iFix <span style="color: rgb(255, 0, 0);">[ramp up]</span><br>
</td>
</tr>
<tr>
<td
style="vertical-align: top; background-color: rgb(255, 255, 0);"><a
href="http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&amp;action=view&amp;page=Wed_May_11_2005_Day&amp;anchor=08:15:32.781">2005.05.11
08:12:38</a></td>
<td
style="vertical-align: top; background-color: rgb(255, 255, 0);">7976720</td>
<td
style="vertical-align: top; background-color: rgb(255, 255, 0);">30228</td>
<td
style="vertical-align: top; background-color: rgb(255, 255, 0);">null <span
style="color: rgb(255, 0, 0);">[CMX is all on]</span><br>
</td>
</tr>
</tbody>
</table>


<!-- Date: Thu May 12 19:11:45 2005 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-424&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-424' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-425&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-425' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-426&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-426' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-427&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-427' width='300'></a></center>
<!-- Author: Dale -->
<!-- Comment: Thu May 12 19;19;25 comment by...Dale [.5] -->
Another record store and anode currents (same ones as before) go over the software current limit as noted <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=396&frame=2&anchor=17:30:54.988'>
here</a>.
<br><br>
Above are the currents for CMX west 9-10 [west channel 6] (first 2) and east 2-3 [east channel 2] (last 2) for store 4121 (intial lum = 128 E30).  The 2nd and 4th are the luminosity vs. current plots.  I'll come up with limits via a fit for the highest ones such that it can take 150 E30.  Will likely adjust all the wedges up but in realilty only ~8 channels need the full boost.  Full analysis to come shortly.


<!-- Date: Thu May 12 22:17:00 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-428&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-429  width='300' align='top'></a></td><td>I could give a ton of details but the bottom line is that this is enough to tell me that I only need to raise the current values by 20 microA to reach safe-haven at 150.  I could just as easily take all the wedges to a nice 75 microA.  The trip limit I think is at 150 (I think I could take down the average current trip rate if its also at the value).  We'll have to worry about this when we get to intial luminosities of ~325 E30 err... else we probably don't have to worry about it.  <br><br> 
I'll come up with an offical game plan soon but I will almost certainly change the warning values for the few wedges that are the most "open" to 70 or 75.</td></tr></table>
<!-- Author: Dale [1.] -->
<!-- Comment: Fri May 13 07;20;09 comment by...Dale [.25] -->
There <i>might</i> be a quadratic term in there but when I made the fit it didn't seem to make a big deal (curve is dominated by the lower region and the lum^2 fit still does not hit the last few points.  Catalin correctly pointed out the Lucio used a quadratic fit for the CMP.  In any event I decided to increase the limit to 75 microA for wedges (on the east side as well!) 0-4 and 7-11 or however, the HV pairing works.  I'll double check that the rest are  at 50 but I think I have a fair amount of breathing room on the rest.

<!-- Comment: Fri May 13 08;35;31 comment by...Dale -->
<table border=0><td><tr><td bgcolor=#e0e030>Changed Settings on HVMon++:<p>
Ramp-down voltage is 200 V/s not 300 V/s [ramp-up is unchanged] for both field (cathode/drift) + sense (anode).<br><br>
Changed hardware (high) current limit for stand-by to 50 microA on the sense wires (some were at 150 which seemed crazy to me).<br><br>
Changed software (high) current limits for stand-by for all sense to a uniform 25 microA (one or two were 40, 50, etc.).
</p>
Most importantly as suggested above I changed the software (high) current limits for "on" such that: East 0-3 + 8-11 have 75 uA as do West 0-5 + 7-10.  I let the miniskirts be screwy and kept their limits at 40 uA while all the other arches have limits of 50 uA (several were 40 uA).
<br><br>
Assuming I remember to write / re-read everything the settings should go into effect shortly.</td></tr></td></table>

<!-- Comment: Fri May 13 11;05;14 comment by...Dale [.5] -->
Everything checks out - new values are loaded and a SRS shows that everything has been updated.  Noted in the shift e-log <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=398&frame=2&anchor=08:39:33.004'>
here</a>.


<!-- Comment: Wed May 18 13;08;30 comment by...Dale -->
The changed HV ramp settings are in relationship to the work done <a href=
'/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=9&frame=2&anchor=15:01:06'>
above</a>.
<!-- Date: Wed May 18 12:53:42 2005 -->
A rather odd occurance: 
<p>While at stand-by CMX NE iFIX 20 tripped as noted <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=413&frame=2&anchor=10:36:31.150'> 
here</a>.  The ace recovered without any problems.  I checked it out and it was (as I would have guess) on the cathode but there was no obvious corelation to conditions in the hall (we were out of control access for sometime by then but no beam in the TeV as D0 was still in theirs).  I turned the wedge all on for a few minutes - normal current draw no obvious signs of trouble (I then turned it back to stand-by).</p>As everything came up fine I think this means things are ok in the short term but that one of thoes chambers has a real potenial problem to go bad.  As this is the isolated section I reminded the ace about my special instructions for this wedge (he seemed well aware).<br><br> 
In other news SW iFIX wedge 16-17 (5 out of the 6 5-degree sections for the SW MS) has tripped twice in the last 4 stores.  It recovers right away - I'm content to live with this for now.
<!-- Author: Dale [.5] -->



<!-- Date: Tue May 24 11:09:41 2005 -->
<b>Summary for the <i>SW:ROTS Day</i> Access:</b><p> 
Simular to <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=374&frame=2&anchor=13:20:18.444'>before</a> we made an access last Thursday to corna dope the isolated 5 degrees on iFIX wedge 15.  Shift summary is given  
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=416&frame=2&anchor=12:05:02.913'>here</a>.  Basically things went as planed - no fallout.  However, SW iFIX 16-17 [i.e. all the other 5 degree sections] is slowly starting to trip at a "real" time scale of (best guess) 2/day if left full on.</p>
<!-- Author: Phil and Dale [.25] -->


<!-- Date: Tue May 24 14:37:39 2005 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-430&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-431' ></a><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-432&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-433' ></a></center>
<!-- Author: phil -->
<!-- Comment: Tue May 24 14;39;45 comment by...Phil -->
<pre>Analysis of the frequency of trips in sw miniskirt 30 (not the 5 deg. section). It's getting worse. I've concluded from a very sophisticated analysis of the data that it's the cathode causing the problem :-)</pre>


<!-- Date: Fri May 27 13:22:02 2005 -->
Result of my DQM work:<p>I asked that runs 193889 and 193890 be marked bad (<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=440&frame=2&anchor=13:08:11.611'>done and done</a>).  This correlates with the CMX entry <a href= 
'/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=8&frame=2&anchor=23:17:09'> 
here</a> and the shift elog one <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=133&frame=2&anchor=00:11:57.480'> 
here</a> where SW iFIX 15 (or wedge "16") went bad and had to be turned off. 
</p> 
Other DQM stuff:  one stack in west wedge 10 is normally hot - sometimes it will "cool" and give a simular occupancy to the rest of the stacks in that area.  Most of the time, as the refrence was done with the stack 25% higher than it "should" be this will not cause a problem but the runs where this doesn't happen we get a false negative.  Also note that high luminosities imply that the number of entries in the CMX dX histogram will be too low when normalized to the total number of CMX stubs realtive to a (lower lum) refrence.
<!-- Author: Dale [4.] -->


<!-- Date: Tue May 31 10:35:00 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-434&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-435  width='300' align='top'></a></td><td>updated</td></tr></table>
<!-- Author: Phil -->


<!-- Date: Thu Jun 2 14:16:18 2005 -->
A (~2 hour) access is plan for Friday morning (6/3).  I sent the access request and made the necessary preparations.  The plan is to reconfigure the SW miniskirt HV.   
<p> 
How we actually configure it will depend on what "works" by the time we get things scouted out as far as what will fit where.  One optimal scheme is to hook up the highest phi (bottomness) 5 degree section to the NW miniskirt.  Then connect the isolated 5 degrees section to <u>1</u> next door 5 degree section and keep the other 3 sections (necessarly split up) together as a 15 degree section. 
</p> 
Some advantages to this is that if the section connected to the NW starts to trip we already go things down to 5 degrees.  If the 10 degree section with the (currently isolated) suspect section trips we'll likely conclude that we may also have the problem down to 5 degrees.  Of course if it is in the 15 degree section we search as normal (at most two more accesses).  Using the NW miniskirt is optional but I think it can be done and taken advantage of.  Having the section with the already isolated 5 degree section be bigger than 10 degres is asking for trouble.
<!-- Author: Dale [.5] -->
<!-- Comment: Mon Jun  6 09;42;31 comment by...Phil and Dale [1.5] -->
We put the offical summary in the <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=461&frame=2&anchor=09:14:49.560'>
shift e-log</a>.  <p>The one thing which we (Dale) completely forgot was that the higher phi of the SW miniskirt (i.e. the lower part next to the NW miniskirt - wedge "17") was not segmented.  It was possible to hook up the NW miniskirt to one of the 5 degrees sections that were segmented on SW "16" but it was thought this was a bit to complicated and required extending several of the SHV cables.</p><p>We opted insteed for a 20-10 configuration.  We took the middle 5 degrees of wedge "16" (the 5 degrees section next to the isolated 5 degree section.  This was kept as iFIX 15.  We than took the lowest phi 5 degree section which includes the last and only stack from wedge 15 proper and kept it combinded with the 15 degrees of wedge "17" for iFIX wedge 16/17.</p>
The weather got really hot and humid over the weekend yest no signs of trips so far.  For now e will wait and see.  It's certainly possible that we may have made things better but the jury is still out of that.  For now we wait.
<!-- Date: Fri Jun 3 15:27:00 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-436&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-437  width='300' align='top'></a></td><td>Some weird HVAC activity.  The dip in the middle is when we had the access.  The peridic structure is a bit too odd for me.  Hopefully it's harmless.</td></tr></table>
<!-- Author: Dale [.5] -->



<!-- Date: Mon Jun 6 13:01:40 2005 -->
The NW wedge "19" (iFIX 20) starts to trip.  My intial thoughts are posted <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=470&frame=2&anchor=11:25:20.642'> 
here</a>.  Bottom line: we need an access as it looks like the cathode really went (it will barely hold at 350 V).  The anode appears fine (ok up to 2 kV).  I'll add some additional thoughts below but the next priority is to submitt an access request.
<!-- Author: Dale Stentz -->
<!-- Comment: Mon Jun  6 13;59;20 comment by...Dale [2.0] -->
Update:<br>The request has been sent.  Note that I thought of all the possible causes - none fit the bill.  For example, The CES NW bottom has been tripping but there's no reason to think that they really share the same "conditions", etc.  More to the point the humdity looked dead flat all be it at a realtively level (dew point was above 50 degrees if I recall correctly).  While it's possible to suggest the humdity was a "long term factor" it's clear the cathode was just not as robust as it should be and had a hardcore failture rather than the come and go problems we have seen else where in the miniskirt.  <br><br>The good news is that if the wedge can't hold 400 V (it barely even makes it to 350 V) it should be easy to fine.  Might even be a sparker/snaper.  Phil and I both agree that we should at least segment the section which has the bad 5 degrees (and if time and opputurinty allows do the others in the wedge as well).


<!-- Comment: Wed Jun  8 05;49;02 comment by...Dale [.5] -->
I've looked at this HV channel a few more times.  Before the 8:00 meeting I noticed that it seem to be stable at a higher voltage (say 500 V) but would still trip if taken higher.  Most recently I notice two differnt behaviors:
<p>
1.  Basically as stated before I was able to get the wedge to trip by ramping it up "as normal" but that it was getting more stable around the 1 kV mark.  I didn't think much of this as I could still get it to trip consistantly at a threshold voltage much lower than the operational "on" voltage. [I need to continue this later...]
</p>
<!-- Date: Tue Jun 7 18:32:46 2005 -->
NW 18-19 now shows signs of trouble - I got it on the drogie and it was holding at 3.0 kV (5.6 kV) on the cathode (anode) - it tripped off when I checked it a few minutes later.  This is as the ace described; e.g. would recover but then trip ~5 mintues later.  More details to follow...
<!-- Author: Dale [.5] -->
<!-- Comment: Tue Jun  7 18;42;56 comment by...Dale [.5] -->
<br>For got to link to the elog entry - <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=474&frame=2&anchor=17:44:59.195'>here it is</a>. <br><br>I did some quick ramp ups - holds fine and can go up without issue on both cathode and anode.  This is definately not like the problem on the NW iFIX wedge 20.  The trip log says that the trips happen at the same time on both.  Suspect the cathode although this is the wedge we had anode troubles on last summer.  Had it on the default "HV on" values - moved it up by 200 V to the values it tripped at above.  Will likely test just the cathode if it trips again.
<br><br>
While talking to the ace on the phone I looked at the humidity and (although high +42%) it was steady varying by +/- 1 to 2% max.  I'll also take a look at the actual B0 tempatures.<br><br>
Tripped again at 3.0/5.6 - trying just the cathode.

<!-- Comment: Tue Jun  7 18;56;03 comment by...Dale [.25] -->
Tripped - looks like it's the cathode.  Try again and see what time constant we have...
<br><br>
Temp. steady for the day but actually the ICICLE/iFIX humdity display shows the same structure <a href=
'/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=9&frame=2&anchor=15:27:00'>
as noted before</a>.  Too soon to rule anything out but I don't see this being a problem in its own right.  Nevertheless this is definately odd.  The period of the most resent event (i.e. from 6:00 this morning to now) was 19-20 minutes.  

<!-- Comment: Tue Jun  7 19;03;47 comment by...Dale [.25] -->
Via the trip log and my rough timeing the time constant is about 5 minutes.  I don't want to have to shift crew try anything over the night although it is possible that it might recover.  Best thing to do is to come in really eairly before the 2 hour access and do all my playing then.  I'll leave both of wedges unplugged for now.



<!-- Date: Wed Jun 8 06:11:24 2005 -->
NW "19" is still tripping at 400 V - should be a lock to find for the access.  Testing "18" which holds on rampup on the cathode.  Held for 5 minutes - I like to test the SHV to jumper (the ones that Dervin had made up which are in use in the keystone/NE MS/SE MS) with the drogie on one of the 10 degree sections before I use them (why did I canibalize all my other ones? - seemed like the thing to do at the time).  Note we are already at the end of store - I'll have the shift crew baby-sit "18" [iFIX 18-19] while I go to test.
<!-- Author: Dale [.5] -->
<!-- Comment: Wed Jun  8 11;59;03 comment by...Phil and Dale [4.25] -->
<h3>Access Summary</h3>
<a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=476&frame=2&anchor=11:10:00.493'>
Entry made here</a>.  A few minor details to follow...

<!-- Comment: Mon Jun 13 15;04;44 comment by...Dale and Phil [.75] -->
Better late then never.
<p>
We had two false starts - one that could have been avoided but was actually harmless as far as getting the access done and another which costed us some time in the hall despite our prepration.
</p><p>
The 2nd case was that I test the SHV to jumper via a chamber on the 10 degree miniskirts in the assembly hall.  It held at 3.0 kV and looked good to go.  Durning the access (when all 15 degrees were together) we could barely hear the trip somewhere inbeween the lowest and 2nd lowest 5 degrees section.  We preped the middle one to have the cover taken out but then decided to go after the 1st 5 degree section of HV wedge "19".  Long story, we had one "holding" and too many trips.  We suspected it was the SHV to jumper and had to undo everything.  It turned out that the problem was actually in the middle 5 degree section.
</p><p>
After we found the bad jumper, the slightly more "intense" problem came at +2 hours into the access when the system seem to hold conistantly (the first time we got a weird set of trips on the cathode when testting the whole wedge with the droggie) when tested by us but then tripped (quickly) on the ANODE upstairs.  By reconfiguring the HV we were able to show that the only logical rational was that the cables were switched at the CAEN.  We came out of the access and saw this to be the case - switched them back - everything came up fine.</p>
Note that the actual problem with the jumper is odd only because it had been in use and "not touched" for such a long time but then decided to fail.  Dale has seen this once when working on the 10 degree miniskirt sections.  In any case, this was easy to figure out and had the obvious advantage of us not loosing a chamber due to by-passing.


<!-- Date: Tue Jun 14 14:15:42 2005 -->
There were two trips on CMX SW iFIX wedge 16-17.  This would be the 20 degree section consisting of wedge "17" and the first (lowest) section in phi.  Two trips was enough to convince us to do something about it.  Elog summary is  
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=494&frame=2&anchor=13:00:48.052'> 
here</a>.  We reconfigured it to be still 20/10 but with the 20  (+10) degree section continous.  If iFIX 15 trips we'll suspect the lowest 5 degree section in phi.  Otherwise, it might point to the un-segmented wedge of "17".
<!-- Author: Phil and Dale [1.25] -->


<!-- Date: Wed Jun 15 15:29:10 2005 -->
The fun never ends... 
<p> 
SVX cooling work was being done on the valves near the NE miniskirt.  We have observed in the past there being ice on thoes valves (state the obvious: they are cold) and a rather crudely placed blue bucket + rag to get any condensation drops.  Two sets of interations caused trips on (first) the NE iFIX miniskirt 20 (this would be the isolated 5 degree section) and then today when the rest of the miniskirt had it's troubles. 
</p><p> 
The first eposoide was odd and almost certainly was not caused by "human contact" etc.  There were no signs of damage and its the first 5 degrees of wedge "19" so it is well isolated.  Documented thoughts + event in the <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=494&frame=2&anchor=15:38:36.429'> 
e-log</a>. 
</p><p> 
The 2nd case there was definately human activity (eye-witnesses [as before] + I caught them on the video camera!) around the NE miniskirt.  Where as the first case there's no sign of "change" there was a very differnt enviroment on the second time around.  See <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=497&frame=2&anchor=10:41:14.919'> 
e-log entry</a>.  I suspect that when they moved the standing fan to the NE from the SE and (according to others) chiped off the ice that both the tempature and water levels in the local area changed.  I also noted that it was also possible that a trip or two was caused by some contact with the fan-trunck being moved back into possition.</p> 
I'm not too sure what to take away from these cases.  No harm, no foul.  Note that the 2nd caused a cooling trip for silicon.  Read all about it in the <a href= 
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=253&frame=2&anchor=12:36:22'> 
silicon e-log</a>
<!-- Author: Dale [4.] -->


<!-- Date: Wed Jun 29 12:55:58 2005 -->
Bill added some ACNET varibles with the iFIX information - potenially very useful. <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=539&frame=2&anchor=12:39:49.580'>See link here.</a>
<!-- Author: Dale -->


<!-- Date: Fri Jul 1 11:09:33 2005 -->
CMX had about 8 trips over the same number of hours yesterday (for example  
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=542&frame=2&anchor=12:15:22.641'> 
see here</a>).  No obvious conection to the HVAC when I looked at it although they did start happening a few hours after the rain/thunderstorm passed in the morning.  Was fine at the end of the store (some "hot flashes" - increase in temp. in the col. hall durning the evening shift but by then all the trips were gone).
<!-- Author: Dale [.5] -->
<!-- Comment: Tue Jul  5 09;50;41 comment by...Dale [1.] -->
<br>
Things may have quieted down but there appears to be a problem on NE 18-19 (the 25 degree section of the NE miniskirt).  If trips approch 1 an hour that will be too much (we start to lose too many muons realtive to the muon acceptance that we would be gaining by when on+running).  I'll submit a low level access request if things don't make an obvious improvement.  The NE is 25-5 split so the first thing to try would be what we did on the SW - try 20-10 and search from there.  Possible that fans might help.  Note it is humid outside but the HVAC seems very steady dew point and humdity wise.

<!-- Comment: Mon Jul 11 12;27;00 comment by...Dale [3.] -->
The trips for NE 18-19 continue:  <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=557&frame=2&anchor=08:28:34.748'>
July 5th</a>, <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=567&frame=2&anchor=18:00:03.538'>
July 8th</a>, and 
<a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=570&frame=2&anchor=19:23:11.187'>
the following day</a> (was taken out for ~2 hours).  Needless to say this is piss-poor performace and even averaging over a whole stores the trip rate is still high on average.  Luckly (?) there was some down time in the TeV - see the access summary below.


<!-- Date: Mon Jul 11 12:51:53 2005 -->
<b>CMX access report from control access on 7/11</b><p> 
We made an access to go after the NE miniskirt.  The plan was to reconfigure it so that it was 20-10 degree sections and to see what happens (e.g. what trips) after that.  The shift e-log summary is given <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=575&frame=2&anchor=09:45:08.028'> 
here</a></p><p> 
As it turns out, we got lucky.  Recall that NE iFIX 20 is in fact stack 1 and 2 [i.e. the 2nd and 3rd stacks] from wedge 19.  See the <a href= 
'/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=9&frame=2&anchor=15:01:58'> 
above table</a> for the previous HV reconfiguration.  Also keep in mind that we had corona dope this isolated 5 degrees and that it had be realtively quite if you igore the short supervised access (<a href= 
'/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=9&frame=2&anchor=15:29:10'>see here for example</a>) for quite a while.  We choose to segment the next higher 5 degree section with the suspect one to get our 20-10 configuration.</p><p> 
As already suggested, we got luck and iFIX wedge 20 started to trip.  Dale was hessitate to uncombine and possibly split up the "bad apples", but after 3 trips in ~20-25 minutes it was clear that the truely bad section was likely the one that we just added.</p> 
We went back in and now stacks 3+4 of wedge 19 are isolated on iFIX 20 while the rest is on iFIX 18-19.  Sure enough we saw at least two trips on iFIX 20.  There's still a small probablity that the section we re-incorperated into the 25 degree piece may go bad.  History (short term anyway) and precident factor that we corona dope several of the end-chamber cathode HV boards suggest that this section is much more reliable and that splitting the one section was the prudent thing to do.
<!-- Author: Phil Schlabach and Dale Stentz [5.] -->
<!-- Comment: Fri Jul 15 06;05;19 comment by...Dale -->
First non-beam related trip (actually first two) was (were) on iFIX NE 18-19 - the 25 degree section.  We'll see how it handles itself as the day progress.  
<a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=586&frame=2&anchor=04:12:45.419'>
Trips are sited here.</a>


<!-- Date: Fri Jul 15 12:55:07 2005 -->
Repeated trips on 18-19 but not the NE but the NW.  See play out  
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=587&frame=2&anchor=11:06:41.500'>here</a>.  It's very humidid outside but the HVAC looks flat on on the relative humdity.  Running fine now no idea what happened as we had no trouble ramping it up by-ourselves.
<!-- Author: Dale and Andrew [1.] -->
<!-- Comment: Fri Jul 15 14;38;08 comment by...Dale [.5] -->
Lots of problems deciding the run bits for CSX and RunSectionFlag as noted <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=587&frame=2&anchor=13:59:47.137'>
here</a>.  Make sure they mark run 201040 good.

<!-- Comment: Fri Jul 15 14;38;51 comment by...[contiune] -->
<i>(good for everything except CMX)</i>

<!-- Comment: Mon Jul 18 13;53;40 comment by...Dale -->
Update:
<br><br>A few more trips over the weekend (<a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=589&frame=2&anchor=03:43:43.615'>for example</a>) but things were basically steady for most of Saturday and Sunday.  The fact that the problem seems to be pointing at the NW miniskirt (realtively quiet except for when one of the jumpers decided to fail) does means we need to be on our toes.  I've asked Dervin to make us 6 more SHV to drift HV jumpers.  We'll use these (and the same number of SHV cables) to segment the NW miniskirt (it is unsegmented).  We should only need 4 but I like to have spares and also there's one wedge not segmented on the SE (15) which may need to be done.  If we had a long access I like to exchange the "old" ones we had on the east side that use to be the ones we used for normal tracking down new trips.  I'll likely send in a low priority access request with the condition that we need to have the new jumbers+cables ready.


<!-- Comment: Tue Jul 19 15;58;01 comment by...Dale [.25] -->
Access request submitted.


<!-- Date: Thu Jul 28 13:23:58 2005 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-438&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-438' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-439&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-439' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-440&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-440' width='300'></a></center>
<!-- Author: Dale -->
<!-- Comment: Thu Jul 28 13;32;28 comment by...Dale -->
<p>Ok, so the 1D occupancy (middle) looks good - nice and flat nothing crazy.  The trigmon plot (as discovered by Phil) looks a bit more odd the west side - namely why isn't it more flat like the occupancy suggest.  I think normally the geometry is simular to this but "top" (5-7) wedges look much to low.  </p>
The real thing that causes me consern is east 12-13 - something is  skrewed up bad.  The first 2 and last 2 stacks are dead - or so it appears.  Investigating further...

<!-- Comment: Thu Jul 28 14;03;40 comment by...Phil -->
This "breaks" between runs 201362 and 201371. The former was stopped because Larry wanted to power cycle the SE arch. This was followed by some to-and-froing to get proper calibrations. The next physics run was 201371. There is <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=602&frame=2&anchor=14:19:13.106'>
note</a> shortly thereafter saying a new version of the consumers was installed specifically saying there were new features in YMon.   As Dale says there are hits coming in, I'll guess it wasn't Larry's fault.

<!-- Comment: Thu Jul 28 15;46;17 comment by...Phil, John, Dale [2.5] -->
We made an entry in the <a href=
'/cgi/elog/fixlist.pl?nb=fixlist&action=view&page=1&frame=2&anchor=14:57:01.415'>CO Fix-List</a> (overkill really) and sent an email to Consumer+YMon experts.


<!-- Comment: Mon Aug  1 07;35;08 comment by...Dale [.5] -->
Here is an example of another trigmon plot when things didn't look right (this relates to Phil orginal consern in the first trigmon plot not the problem in east 12-13).
<!-- Date: Mon Aug 1 07:14:56 2005 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-441&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-441' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-442&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-442' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-443&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-443' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-444&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-444' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-445&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-445' width='300'></a></center>
<!-- Author: Dale -->
<!-- Comment: Mon Aug  1 07;21;50 comment by...John, Andrew, Dale -->
<h3>The good, the bad, the ugly:</h3>
Plots are from the following runs and will be explained in better detail in comments to follow: 201212 (bad), 201351 (good), 201362 (going bad), 201371 (bad), 202041 (good).  Bottom line is that the last one is from a resent run and when we left on Friday (and after Eric had made the TDC switch) all was good.  See 
<a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=630&frame=2&anchor=17:29:00.231'>
here</a> for a few details in the shift e-log.  We'll update the CO Fix-list and send out the needed emails later today.



<!-- Comment: Mon Aug  1 15;11;29 comment by...John and Dale [4.] -->
<p>Email sent to Victoria about reconstruction side of the issue and general email to TDCers and YMoner.</p>
Details:<p>
Run 201212 [first plot] goes back to July 17th.  Andy documented back to run 201133 but we may want to go back before July to see if the problem existed before.  [2nd plot] everything looks good.  Run 201361 looked good but the next run [3rd plot, #201362] clearly has higher occupancy in the middle 3rd.  This was before the end of a store.  It was fully bad by run 201371 [4th plot] and was fixed and put back to normal on Friday (see above link).  The last plot is from a run at the end of this weekend.
</p>
Thoughts:<p>
Let's just state the obvious: 1+1/3 of a wedge was dead for some period of time (some luminosity that we will estimate soon).  2/3 of a wedge was working at some level and all the "wrong" hits likely are lost after L3 and/or aren't constructed as a stub.  The scheme for how the TDC gave the error we can logically write out but the details and cause are unknown.</p><p>The TDC must pass a "word" for the channel in binary.  First note that YMon # != TDC # but that the mapping inside a stack is closed: (YMon# % 8) == (TDC# % 8).  Logically we see that either bit 16 OR bit 64 are on (but not both) but bit 32 is always off.  Possible combination are therefore 001XXXX and 100XXXX.  Higher possiblities might happen - we don't know.  For example, 110XXXX >= 96 which normally would never happen but the total wedge occupancy is lower so maybe there are numbers being thrownout.
</p>
<!-- Comment: Mon Aug  8 18;42;59 comment by...Dale, Phil, and Andrew -->
Our search shows that run 201155 was bad while 201154 was good (July 17th) we should search before this to make sure the rest of July and all of June, etc. was good.  Note that Phil and Dale quickly talked to Eric about the TDC board - statement of the obvious - something is wrong with it as it fails the TDC tests.  If they could trace where one trigger word goes realtive to another that would be the best case.  I may ask Jianrong if this is possible.
<!-- Comment: Tue Aug  9 19;53;39 comment by...Andy -->
<pre>I checked many (~25) of runs with significant data that occured during June and July before run 20115.  In all runs I checked, the plots looked fine, so it appears this problem started with run 20115.</pre>
<!-- Date: Mon Aug 1 14:36:36 2005 -->
Last night a trip on the NE miniskirt occured while the Muon3 PC went through its normal Sunday night HB troubles.  Aces were on top of things but it was about the worse case senerio you can have for such things.  The problems and issues are documented well <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=636&frame=2&anchor=22:09:27.095'> 
here</a> and just below that entry <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=636&frame=2&anchor=23:05:09.160'> 
here</a>.<br><br>The issue that I can fix without any trouble for the future (i.e. I cannot get a new PC, stop the backup from occuring, make it so CMX will never trip, fix the inhibit channels so CMX miniskirt trip is not confused with CMU trip, etc.) is to put the password out into the open.  With the instructions not there (for restarting HVMon++) and the password virtually hidden by the NIM crate above it, I added the username [cdf-muon3], domain [fnal], and password [*********] and taped it to the left-bottom on the monitor for all to see.
<!-- Author: Dale [1.] -->
<!-- Comment: Tue Aug  2 09;43;46 comment by...Dale w/ Phil -->
<br>Per Computer Division's policy we've decided to remove the login information.  All it means is that the expert needs to make sure the information is passed to the shift crew when the PC gets restarted.  This was hard in the above sited condition as I told the DAQ ace the password (the monitoring ace was working on the PC) but realized that in a hurry the password being written out would be better.  Rather than say "look for the password at..." I could have just told them what it was.  In the pass (and perhaps in the future) I may not remember the password correctly and cause more time to be wasted - there's no optimal solution here except to get the PC to backup inbetween stores rather than Sunday before midnight.  Actually, it would make more sense to have the backup be around noon on the weekday - plenty of experts on hand and just as likely to have a store going on (perhaps even less likely).


<!-- Date: Sun Aug 7 07:42:50 2005 -->
CMX SW 16-17 (this would be the large "good" section of the SW miniskirt has gone crazy with trips.  Might have been started by the humdity but no sure why it's bad now. 
<br><br> 
Elog entires: <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=654&frame=2&anchor=21:50:43.611'> 
first here</a>, a <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=654&frame=2&anchor=21:54:20.554'>humdityplot</a>, <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=655&frame=2&anchor=00:16:41.548'> 
goes bad in the owl shift</a>, <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=655&frame=2&anchor=01:15:34.356'> 
we take the wedge out</a>, <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=655&frame=2&anchor=02:24:00.590'> 
no luck with recoveries</a>. 
<br><br> 
I've got it up to 3.0 kV on the drift for about 10 minutes (a "record" at this point).  The trip log shows anode/cathode trips so I suspect the cathode as the normal suspect but I should raise the anode all the way up as well (currently at 2.0 kV for HV "balance").  
<!-- Author: Dale [1.] -->
<!-- Comment: Sun Aug  7 07;54;52 comment by...Dale -->
@ 3.0 and 5.6 kV - no trips yet.  I thought I noted that the current was ~600 microA on the drift (when I ramped it up with the ace and also when I first ramped it up on the drogie) but now its drawing more like 500 microA.

<!-- Comment: Sun Aug  7 08;01;14 comment by...Dale [.25] -->
it's held long enough to try again with the shift crew.  Hooking it back up...

<!-- Comment: Sun Aug  7 08;23;34 comment by...Dale [.25] -->
<a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=656&frame=2&anchor=08:16:32.214'>Works so far</a> this is very suspect.  Either I really did "condition" the wedge or was lucky that it went good when I came in.  There's a possiblity for an access - we might do a reconfiguration but ideally I like the wedge to still be tripping.


<!-- Comment: Mon Aug  8 07;02;08 comment by...Dale [1.5] -->
More headaches on SW 16-17:<p>
<a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=657&frame=2&anchor=19:04:59.730'>
Trips start here in the evening shift</a>.  Before I left to run I thought we were in the clear as the humdity was going down - unfortinately yesterday's problem had the same pre-condition.  Amanda (ace) notices possible cyclic behavior.  See <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=657&frame=2&anchor=19:59:04.572'>
ACNET plot</a> or 
<a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=657&frame=2&anchor=20:04:02.956'>
ICICLE plot</a> this is also when we started a new run with CMX bad after repeative trips.</p><p> Expert comes in, lays hand on the detector doesn't and doesn't trip.  We put it <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=657&frame=2&anchor=21:11:32.655'>
back in and start running</a>.</p><p>The major observation (as noted in the previous link) which seems consistant with my comment above is a clear floating/dancing/shakeing of the current on the cathode.  The break down of cause and effect is less than obvious.  A jumper could be bad somewhere (as was the case on the NW miniskirt) or perhaps one chamber is really feeling the effect of the humdity.</p>
The plan:  Do some background research on the last work on the SW miniskrit and why it's in a 20-10 configuration.  Likely write up an access request after the 8:00 meeting: 2 hours if it tripping and we want to go after it or <30 minutes if we want to re-segment the HV.


<!-- Date: Mon Aug 8 07:38:09 2005 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-449&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-449' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-448&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-448' width='300'></a></center>
<!-- Author: Dale [1.] -->
<!-- Comment: Mon Aug  8 07;50;14 comment by...Dale -->
On the left is the cathode current draw for iFIX SW 15 - on the right SW 16-17.  Clearly 15 is normal and borning - no current changes.  The situtation is very differnt on the iFIX west 16-17 cathode: slight rise at 16:00 on Saturday - goes to 600 microA before it trips so often that we turn it off.  The situation yesterday is much more pronounced.  Flat as a board before the humdity went up then the current went all over.  Resently it has been tripping at the ~30-45 minute time scale with a current closer to 600 rather than 500 microA.
<br><br>I just motivated enough to think about using ACNET to monitor this current.  The thing I don't know is if the system will like the .5 mA current rather than a ~5 microA draw.  I asked George Watt if he knew off hand of the limitation - didn't know the details off hand (he suggested that I not touch it if it is still working!).

<!-- Comment: Mon Aug  8 08;37;28 comment by...Phil -->
<pre>I don't think it will work.</pre>


<!-- Comment: Mon Aug  8 10;10;40 comment by...Dale -->
Aye, just looking at the signal output it is 1 V/ 50 microA highly suggestive that it isn't configured for that.  Wishful thinking on my part...
<!-- Date: Mon Aug 8 08:41:09 2005 -->
I got iFIX SW 16-17 on the droggie.  The current draw is about 550 microA (2.8 kV w/ 1.5 kV on the anode) but does look stable - no spikes or shakeing.  I'm going to increase it to 3.0 kV and (if it continues to look ok) leave it on the droggie that way while I write up the access request.  Rumor is that a evening access today is very possible - of the 4 hour variety.
<!-- Author: Dale [1.5] -->
<!-- Comment: Mon Aug  8 09;04;03 comment by...Dale [.25] -->
current has been dead on .6 mA for the last 20 minutes.


<!-- Comment: Mon Aug  8 10;13;07 comment by...Dale -->
Solid like a rock - no trips.  I don't suspect the CAEN HV pods as the problem as the problem but it wouldn't hurt to switch just to make <b>100% sure</b>.  Going to talk to the SciCo

<!-- Comment: Mon Aug  8 12;56;14 comment by...Dale -->
It's <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=659&frame=2&anchor=10:29:51.936'>
back in</a>.  The current on "15" is drawing .5 mA and looks solid and steady.  No trips so far - should take a look at the HVAC to see what is up.  I like the humdity to fall around 16:00 with an increase by ~19:00.  By this time the store will be over and an access will follow at ~19:30.



<!-- Date: Tue Aug 9 09:04:56 2005 -->
Access report was put in the <a 
href="/cgi/elog/elog.pl?nb=2005&amp;action=view&amp;page=660&amp;frame=2&amp;anchor=22:10:51.277"> 
shift e-log</a> - here are the details to the ~2 hour access on the SW 
miniskirts: 
<p>After the toroid had been moved out we attempted to remove the 
middle 
cover of SW "17" while testing the 5 degree section already isolated 
but originally apart of iFIX wedge 16-17. To avoid confusion let us 
reference the section of the miniskirt by which 5 degree section it is. 
It is easiest to think left-to-right so the counting is 1, ..., 6 and 
we denote "5-degree section" via [#] (of course phi goes from 
right-to-left so that section [6] is the low phi section which contains 
stacks 10 and 11 of wedge 15. 
</p> 
<p>Nomenclature aside, [4] was being tested (just with the cathode) 
while [2]'s cover was being removed. [4] held without any 
problems and 
drew the expect current at a steady level.&nbsp; Considering how we 
need to 
segment and test each new 5 degree section [1-3], we went ahead and 
segmented wedge "17" into 3 5-degree sections as normal.&nbsp; We 
tested 
each section for a few minutes; first testing the cathode @ 3.0 kV and 
then adding the anode all the way up to 5.6 kV (this is a sanity check 
to be 100% sure that we got the boundary right). All section held 
for a 
couple of minutes and none of them had an obvious current draw problem 
on the cathode or the anode.<br> 
</p> 
<p>With no trips we decided to do some long testing by effectively 
dividing the tripping wedge (i.e. iFIX wedge 16-17 = [1-4]) into 
two. 
Thus [1-2] and [3-4] had their own cathode and we decided (because it 
was easy to do) to hook up one common anode for both wedges. The 
split 
pieces held and both drew about ~250 microA. We tested for about 
20 
minutes but no trips or problem otherwise. 
</p> 
<p>At that point, we recovered [2] and prepared to segment the SW 
miniskirt as we wanted. (The configuration was [1-4] = iFIX 16-17 
while [5-6] = iFIX 15 as noted <a 
href="http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=cmx&amp;action=view&amp;page=9&amp;frame=2&amp;anchor=14:15:42">here</a>)&nbsp; 
There were a couple of schemes for this. 
Basically we suspect that section [4] is more likely to be good than 
[1-3]. In addition, [6] is also thought to be good ([5] was the 
original motivation for splinting it into the wedge). We 
configured it 
so that [1-2, 6] = iFIX 16-17 and [3-5] is iFIX 15. (<u>Special 
note</u>: we 
switch back the cables that hook up to the CAEN - thus the <a 
href="http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=cmx&amp;action=view&amp;page=9&amp;frame=2&amp;anchor=08:41:09">above 
work switch</a> 
was undone.) While being a non trivial configuration a trip which 
acts the same way as before (<a 
href="http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=cmx&amp;action=view&amp;page=9&amp;frame=2&amp;anchor=07:42:50">see 
last comment above</a>) on iFIX 15 tells us the likely problem is 
on [3]. Conversely a trip on iFIX 16-17 gives a strong indication 
of 
problems with [1-2]. Either way it will take at least one for 
access 
and reconfiguring to get it down to the true bad section.</p> 
Once we got everything connected the CMX turned on with the expect current draws, etc. via the shift crew and now we wait to see if the problem is still there or not.  Note that Dale pointed the big wheel mounted fan realtively close (to where it was before) and tried to give it a good angle to blow under the IMU.  Note that it is outside of the IMU screws (what moves it).  Dale thinks it might be better to get this one in the middle where it has a clearly line of sight.  The fan is heavy as heck and getting over the screw isn't so trival.  The fans do seem to have a net positive effect but then again it isn't an exact science either.
<!-- Author: Dale and John [6.] -->



<!-- Date: Wed Aug 10 07:56:22 2005 -->
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=664&frame=2&anchor=00:11:52.444'>SW 15</a> trips in the owl shift.  <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=664&frame=2&anchor=00:17:14.628'>It was out</a> - <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=664&frame=2&anchor=01:35:58.369'>put back in</a> - <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=664&frame=2&anchor=07:22:43.520'>now back out</a>.<p> 
Trips on the field due to current being to high (same symptom as before).  I got the droggie on it - here's what I see: high current draw, very non-linear.  Trips when current gets to 80% full scale.  Normally trips this way before I can get it to 2.4 kV.  Very unstable at 2.1 kV.  Current is not erradic but rather seems to shift up in down and have a gental (+/- .01 mA) fluctuation on top on the shifting.  I saw currents as high as .800 mA with voltages just above 2 kV. 
</p> 
The bad news is that the problem has gone away (I let it set at 2.1 kV where it held and stabalized).  Current draw is solid at .4 mA - right where it ought to be.  More testing to come...
<!-- Author: Dale -->
<!-- Comment: Wed Aug 10 08;57;54 comment by...Dale [2.5] -->
Update:<br><br>Still holding when I left at ~8:15.  I tried to trip it to see if it would go bad but of course it didn't.  The key is that "under the right conditions" (what ever they are) the current will become unstable and then it should trip off.<br><br>
Note that iFIX 15 being bad means that [3-5] are bad (this would be  16:2-5 + 17:0 in the wedge:stack numbering scheme).  [3-4] were apart of the wedge that was tripping before.  We suspect (don't know) that [4] is good or has been isolated with [5] and known to be good.  This suggests that [3] is bad ([5] which has been been a known problem area before was the section working before and the trips seem to be just like before).  <br><br>
Best case is it trips and we fix it (or if time doesn't allow) we isolate the bad 5 degree section.  If no trips/problems pop up then the conserveative option is to put [3] and [4] together by-themselves. [1-2, 5-6] would then be the other (20 degree) section.  We'll debate this before we go in.  With ~4 hours we got a good chance of at least getting partally lucky and isolating down to 5 degrees.


<!-- Date: Wed Aug 10 14:37:42 2005 -->
<pre> 
IFIX wedge 15 was tripping before the beginning of the access.  It was turned on just before the access and tripped again (a couple of times).  This IFIX wedge corresponds to 5-degree sections #3-5 of the SW CMX miniskirt wedges.  The first test we performed at the beginning of the access was to isolate section #3 and test it by itself, turning the HV from the control room.  This section held well for about 10 minutes.  Next test was performed on sections #3 and #4 separately (which are the main suspects) and they both held HV for about 20 minutes.  The executive decision was made to reconfigure the SW miniskirt the standard way (#1-3 and #4-6), since in  
principle one of the #3 and #4 could be the problematic one.  We are testing this configuration right now. 
 
In the meanwhile, we worked on the NW miniskirt wedges, and we segmented the HV, so that it can be applied to each 5-degree section separately.  We had to remove 3 of the covers to perform this.  One jumper cable could not be completely removed, and we just isolated it with kapton tape.  We subsequently tested each 5-degree section to ensure proper current levels.  After the segmentation testing, we returned to the standard HV configuration (split in the middle). 
 
An unfortunate event was that the interlock was dropped as we were exiting.  It is unclear why, since we followed standard procedure. 
 
 </pre>
<!-- Author: Dale and John -->
<!-- Comment: Wed Aug 10 15;16;14 comment by...Dale and John [7.] -->
<a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=665&frame=2&anchor=15:12:04.291'>Summary in shift e-log</a>.


<!-- Date: Thu Aug 11 14:05:12 2005 -->
Now for something completely differnet...<p>On Wednesday I was messing around and saw the correlation between HVAC conditions (rel. humdity being my varible of choice).  The problem was in the proton abort gap losses (B0PAGC) on <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=663&frame=2&anchor=18:03:23.507'> 
Tuesday</a>.  I noticed a HVAC "glitch" that was a minor fluctionation (nothing that would pass our threshold and cause trips) but seem to occur around the same time.  I ploted the realtive humdity vs. the abort gap losses and <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=664&frame=2&anchor=07:24:20.347'> 
this was the result</a>.</p><p>  Others became intrested in this (Marcel, et. al.) and Rick smartly looked at the effect via the collision hall temps.  His <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=665&frame=2&anchor=11:54:01.608'> 
plot</a> shows 1) the bottom temp. is a bit lower than the top and 2) top is ~flat while the bottom shows a strong correlation with the abort gap losses.  Also shown is the SVT verticle postions.</p><p>  Most resently Ron put <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=668&frame=2&anchor=13:02:06.247'>a plot</a> of D0 quad movements with respect to proton abort gap.  The only minor relation all this abort gap stuff has is that it is related to the HVAC conditions in the halls and this certainly effects us.  In any case, I think of this in reverse - looking at such an actnet varible (which is posted every hour) might be an easy way to gage problems in the future.</p> 
I mention in passing that temp. change is likely much more responcible for problems but since Ron's plot shows HVAC problems   at D0 and we just got an emial from the D0 folks about humdity(?) related trips this might be a good thing for them to check out - i.e. do they see the same correlation (no need for direct relationship here) between humdity and abort gap losses and the like?
<!-- Author: Dale [0.] -->


<!-- Date: Mon Aug 15 06:51:59 2005 -->
Basically a quiet weekend for CMX.  Here's the trip log 
<pre> 
Fri Aug 12 12:51:26 2005 H TRP  Crate # 4  Channel # 23 [+19] 
Sat Aug 13 13:17:57 2005 H TRP  Crate # 6  Channel # 36 [+32] 
Sat Aug 13 15:39:54 2005 H TRP  Crate # 5  Channel # 33 
</pre> 
Details to follow...
<!-- Author: Dale [.25] -->
<!-- Comment: Wed Aug 17 08;03;04 comment by...Dale [.25] -->
<a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=685&frame=2&anchor=00:32:20.169'>3 trips in NW iFIX 20.</a> No obvious signs of trouble on the SW (one trip so far).


<!-- Date: Mon Aug 22 07:27:04 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-450&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-450  width='300' align='top'></a></td><td>I'll post comments below.  Trips started around 22:20 when the humdity (B0RHNE) was just starting to come down.</td></tr></table>
<!-- Author: Dale [.25] -->


<!-- Date: Wed Aug 31 16:07:13 2005 -->
<h3>Resolution TDC problem 7/17-1/29</h3> 
<a href= 
'/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=9&frame=2&anchor=07:14:56'>In reference to the above</a>, we have checked the amount of data and the TDC folk have run their tests.<br><br>  We find that there are 31 runs that have the problem which have a combined (online) luminosity of 20.84/pb.  This takes into account the period where there were good runs [201361, 201351, and 301349].  We used the run summary page for the calculation by checking "AnalysisRun" and asking for runs with at least 9/nb with "physics" and "beam data".  8 runs had over 1/pb and were double checked via Chuck's "<a href='http://www-cdfonline.fnal.gov/cgi/internal/mon/cdfdoc/ConsumerGifs/anyCanvasList.pl?run=201155&consumer=YMon'>every YMon plot in the world</a>" web page and showed the problem (2/3 of the TDC empty with higher occupancy in the "middle" of the wedges).<br><br>Eric and Jianrong for their part concluded that the board failed their test every-which-way.  More to the point,  the timing in addition to the channel mapping may have been bad as well.  No hope to logically reconstruct what was going on.  Our conclusion is that the data should be marked bad for CMX for these runs listed below. 
<pre> 
201155  201212  201362  201371  201373  201375  201376  201377  201396  201451 
201543  201607  201625  201643  201644  201656  201657  201658  201660  201672 
201673  201674  201707  201716  201795  201800  201822  201883  201947  201948 
201963 
</pre>
<!-- Author: Dale, John, Andrew, and Phil -->
<!-- Comment: Mon Oct  3 09;28;43 comment by...Dale -->
[the date of the title above should be 7/17-7/29]
<p>
This (the above list) should be the inclusive list - I only required that the ACE sent the run to the production farms - no requirement on good run componets.
</p>
Just to be safe let me list the other runs where there was at least 1 L1 accept:
<pre>
201175 201177 201178 201182 201350 201352 201353 201355 201356 201374
201710 201711 201712 201713 201714 201797 201955 201956 201957 201958
201959 201962</pre>
 total live lum = ~153 /nb
<p>
DQM spotted the problem of the bad TDC in runs 201155+201212.  However, it
also caught a run from May 22(!) which has the same problem (YMon and the
DQM plots make this clear that this is the exact same problem with the
same TDC, etc.).
</p><p>
We will continue our investigation but for sure runs 
<pre>198426  198428</pre>
should be marked bad for CMX.  I like to know what caused the TDC to go
bad (and then recover) but unless there is something obvious in the
e-log we won't worry about this so much as finding all the bad runs.  My
hope is that DQM will catch all of these events but it may not.
</p>


<!-- Date: Tue Oct 4 15:53:23 2005 -->
Via run 205070 I see that west wedge 19 - YMon# = 20-23 is dead.  This is the preamp board for stack 2 - wires 4-7.  I suspect this may have been caused by a recent access (likely from us but maybe by others).  It was there in July (via run 200309).  Signal cable or LV may be unpluged.  Worse case is that the fuse blew - need to repace the preamp board then.  We aren't going to have another damn hole so we will fix this when we get a chance.
<!-- Author: Dale -->
<!-- Comment: Thu Oct  6 09;51;49 comment by...Dale -->
Also seen in TDC (of course) channels 18-19 + 22-23 [TDC = channel+48].  We should figure out when this happened.


<!-- Date: Thu Oct 6 14:47:10 2005 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-451&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-452  width='300' align='top'></a></td><td>worst case current draw to 200e30</td></tr></table>
<!-- Author: Phil -->
<!-- Comment: Thu Oct  6 14;52;26 comment by...Phil -->
<pre>As BD is trying to stuff as many pbars into the recycler as they can, I looked at current vs. luminosity to see what we might expect out to 200e30. (BTW, last night it looked to me like they had a potential from the stored pbars of 190e30ish.) I tried to choose a worst case channel, and for sure I've chosen one with high current draw. Current limits are 75/150 for software/hardware limits in CMX, 40/50 on wire 11 of CMP and 200/200 for BMU. I'd say we could get some software overcurrents any time in CMP/X, but the hardware limits are fine.</pre>


<!-- Comment: Fri Nov  4 07;21;03 comment by...Dale -->
When there's a down time we need to adjust the limits.  The last fit <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=9&frame=2&anchor=22:17:00'>
I made</a> gives (as it should) the same result as Phil's - namely that lum = 200 E30 gives an expected current draw of 90 uA on the highest drawing wire.  As reported the current limits are 75 for the "exposed" wedges and 50 for the rest of the arches (an odd 40 for the miniskirt).  Might just go for limits of 100 and 75 which will keep things under tab until ~215.  Hardware limits will stay as they are.
<!-- Date: Thu Oct 6 14:54:59 2005 -->
<pre>Yesterday I set the trip time parameter for West 9/10 & 11/12 to 1 to see if they might let us ride out these lost p spikes due to the orbit distortion from the D0 quad role. I doubt that it will help, but it might. I did this via the settings page APPLY button rather than SAVE so they are only stored temporarily. And it occurs to me that it would be a cleaner experiment if I only did one of the two...</pre>
<!-- Author: Phil -->
<!-- Comment: Thu Oct  6 14;57;07 comment by...Phil -->
<pre>&lt;a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&action=view&page=833&frame=2&anchor=11:35:59.103'&gt;
link&lt;/a&gt;
</pre>

<!-- Comment: Thu Oct  6 14;57;46 comment by...Phil -->
<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&action=view&page=833&frame=2&anchor=11:35:59.103'>
link</a>


<!-- Date: Fri Oct 7 11:36:37 2005 -->
Here's an access report via the <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=839&frame=2&anchor=10:58:45.794'> 
shift e-log</a>.  Bottom line is that the problem with the 4 channels has been fixed and a short term solution for the HV cables that jumper between wedges 6-7 on the SW CMX has been imposed.  I'll write up an access request so that we are there the next time the SW arch needs to be pulled.
<!-- Author: Dale Stentz -->
<!-- Comment: Thu Oct 13 12;56;32 comment by...Dale and John -->
We sent an "automatic" access request to ops so that they know to consult us and that we will be happy to oversee the movement of the SW arch out (we'll have to cut one cable tie that connects to the signal cables; closer to the keystone portion).


<!-- Date: Thu Oct 13 13:42:31 2005 -->
Rain drops keep falling on my... 
<p>On Monday the Solnoid had a fast dump (<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=848&frame=2&anchor=12:22:39.734'>see here</a>) and this caused a leak onto wedge 4  
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=848&frame=2&anchor=15:10:35.121'>as first noted here</a>.  We figured out what was happening and left our <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=849&frame=2&anchor=16:38:52.316'>instructions</a>. 
The water must have run also to the BMU and for awhile CMX east 2-3.  Eventally the east side forward detectors were <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=849&frame=2&anchor=22:17:21.358'> 
turned off</a>.  The solution was of course to wait for the detector to dry off (no hurry as we weren't taking any real data for 2 days).</p> 
Post dry peroid and turn on things look basically ok.  I took a look at the 2D occupancy and the wedge is actually healthy (the LV didn't blow out).  Still there looked to be two new dead channels at YMon 32+33 (stack 4, wires 0 and 1).  I've since looked and see that the wire 0 channel looks fine but 33 is still dead (note that near-by YMon = 31 is a known dead channel as are all the others).  Might be possible that the preamp board got splashed a bit.  In any case, we should check this wedge out when we get a chance.
<!-- Author: Dale, John, Phil -->
<!-- Comment: Wed Nov 23 13;28;25 comment by...Dale and John -->
The problem with east 4 ymon 33 went away before we could look at it.  It's fine for example in run 207079 so it must have recovered in the time in between the two times (a bit less than a month).  Seems that looking at wedge 4 now would really only be to look at potenial water damange which we kind of did already when we look at it via the cyro platform.
<!-- Date: Thu Oct 13 20:51:33 2005 -->
<pre>More the the trip timer experiment:</pre> 
On Tuesday morning (I think it was), while I was screwing around trying to see what would hold voltage after the rain, some of the usual small loss spikes which recently we've associated with quadrupole rolls, started tripping off W9/10. There were a handful in the course of say a half hour. So I decided to try setting the trip time to 1 again - it had been washed out by one of the restarts due to the work to swap to the new pc - and this time I di d the clean experiment. I set the trip timer to 1 on 9/10 and didn't on 11/12. Later in the day, 11/12 went a couple of times, but 9/10 did not. Nine/ten are always the first to go, so I think the trip time thing worked. Or at least it's worth pursuing in spite of my earlier <a href= 
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&action=view&page=833&frame=2&anchor=11:35:59.103'> 
pessimism</A>. I've reset the trip time for 9/10 to 0, because what I'd like to do is repeat the experiment. The next time we get one of these situations, set 9/10 trip time to 1. I hope that we will clearly see a bunch of trips followed by none before MCR tunes the beam. I am willing to incur a little deadtime to prove this.</pre> 
<pre>What the trip time parameter does is current limit the channel for t*100ms (I think) before tripping. We had quite a bit of success with this wrt main ring losses in Run 1, but they were long slow showers instead of quick spikes which is why I didn't think it would work. It looks like I should have tried it sooner.</pre>
<!-- Author: Phil -->


<!-- Date: Thu Oct 20 10:44:42 2005 -->
<pre>A few news items: 
&lt;ul&gt; 
&lt;li&gt;Bill Badgett marked all the runs with the bad tdc bad yesterday.&lt;/li&gt; 
&lt;ul&gt; 
&lt;li&gt;201716 was not marked bad, my guess because it was already globally bad and the query probably didn't catch it. I'd guess we don't care.&lt;/li&gt; 
&lt;li&gt;201963 was missed which is the last one, this one needs to get fixed.&lt;/li&gt; 
&lt;/ul&gt; 
&lt;li&gt;The southwest miniskirt is dead from some point in 205891; the trigger inhibit starts complaining at 0536h.&lt;/li&gt; 
&lt;/ul&gt;</pre>
<!-- Author: Phil -->


<!-- Date: Thu Oct 20 10:45:17 2005 -->
A few news items: 
<ul> 
<li>Bill Badgett marked all the runs with the bad tdc bad yesterday.</li> 
<ul> 
<li>201716 was not marked bad, my guess because it was already globally bad and the query probably didn't catch it. I'd guess we don't care.</li> 
<li>201963 was missed which is the last one, this one needs to get fixed.</li> 
</ul> 
<li>The southwest miniskirt is dead from some point in 205891; the trigger inhibit starts complaining at 0536h.</li> 
</ul>
<!-- Author: formated this time -->
<!-- Comment: Thu Oct 20 10;46;26 comment by...Phil -->
<a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=9&frame=2&anchor=16:07:13'>
link</a> to the tdc problem


<!-- Date: Wed Oct 26 13:13:02 2005 -->
I made a quick access with Stefano and Craig to address the HV cables that jumper between the SW arch and keystone.  Fairly strightforward and the overlap region looked good during the pull out and pull in. 
<br><br> 
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=896&frame=2&anchor=09:52:08.652'>Link to e-log entry</a>. 
<br>
<!-- Author: Dale -->


<!-- Date: Wed Oct 26 22:08:28 2005 -->
<pre>I asked Steve Hahn to include the two new Relative Humitity sensors in the slow control logger and also in the HVAC historical data in the iFix.  He actually did the latter immediately and he had to talk to somebody else about the slow control. 
The numbers are: 
 
FIX32.CDF_S2.AC_AT_RP3_RAHE.F_CV 
FIX32.CDF_S2.AC_AT_RP4_RAHW.F_CV 
 
(The description in the HVAC of the slow control slides 
is NE Miniskirt and NW Miniskirt respectively) 
 
PS.  Tried to find the curves in the iFix Historical Data, 
but I couldn't.  Could somebody check this out again tomorrow?)</pre>
<!-- Author: John and Dale -->
<!-- Comment: Wed Nov  2 14;16;43 comment by...Phil -->
It looks like these were added to the ICICLE list <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/elog.pl?nb=2005&action=view&page=914&frame=2&anchor=15:35:09.618'>
yesterday</a>.



<!-- Date: Fri Nov 4 09:03:07 2005 -->
I've updated the CMX software current limits as noted  
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=923&frame=2&anchor=08:25:25.286'>here</a>. 
I'll add the details later but basically I changed all the wedges that were 75 to 100 microA and the ones that were 50 to 75 [I think the ones that alarmed were actually the lower set].  The miniskirts I kept at 40 micoA - we can relook at these current draws but I don't think they could be past 20 microA since there is the BMU + the floor. 
<br><br> 
As noted above and elsewhere the hardware limits at 150 microA - if and when the inst. luminosity get past 300 E30 (!!!) we will have to make some adjustments.  We'll likely want to keep a nice "buffer" between the software and hardware limits anyway which will force us to make the change then. 
<br><br> 
HV wedge channel setting will be posted below in due time.<br>
<!-- Author: Dale -->
<!-- Comment: Mon Nov  7 10;55;50 comment by...Phil -->
<pre>I also think it's the lower current limit channels that alarmed. Looking the slow controls data, the currents barely got into the 60s on the worst channels.</pre>


<!-- Date: Thu Nov 10 07:55:50 2005 -->
A whole bunch of trips  
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=927&frame=2&anchor=19:12:05.656'> 
happened on Saturday Nov 5</a>.  The basic understanding of what happened is that the humdidity went up and east wedge 4 (which had had gotten a bit wet after the cyro dump a while back) and then things started tripping (wedge 4 had to be taken out for me to look at.  The correlation with the humdity plots I put in was fairly strong when I put them in and perhaps even better now that I look back on it. 
<br><br> 
The resolution of the matter along with the humdity plots are <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=927&frame=2&anchor=20:11:46.150'> 
here</a>.  Bottom line is that as the humdity went down I was able to work in wedge 4 and all the other wedges stablized.
<!-- Author: Dale -->


<!-- Date: Mon Nov 14 07:20:02 2005 -->
CMX SE 15 may have a bad cathode - not sure what is wrong.  See  
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=952&frame=2&anchor=00:21:40.323'>latest saga</a> and when it tripped at stand-by in the <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=951&frame=2&anchor=21:09:46.540'>shift before</a>.<br><br> 
Access request submitted (at the very least we will resegment).  Cause is not humdity at least that seems the farthest removed option - the realtive humidity was steady+flat around 30%.  We should check out the other humdity varibles just to make sure everything was as peaceful as it seemed.<br><br> 
Nature of the trips: The first time when we were in shotsetup the wedge tripped off at stand-by and seemed to trip imdiately after recovery (via dicussion with ace).  When I looked at it, indeed it tripped very quickly on the cathode.  The droggie said the same thing but in more detail.  Current draw was very high and non-linear (but steady).  Looked like a short somewhere.  I wrote the wedge off as dead but it somehow recovered after I had informed the shift crew of its demise.  It seems that the wedge went bad again after the next store was in.  The nature of the problem didn't seem to repeat itself - the ace said that the 2nd tripped occured a few mintues after the 1st.  Furthermore he was able to set it at stand-by.  See links above for how it played out - bottom line is that it is back in but I don't see why it wouldn't give us trouble in the near future.   
<!-- Author: Dale -->
<!-- Comment: Wed Nov 16 14;15;23 comment by...John and Dale -->
So far so good - the problem has not returned.  We're thinking it was a one time event although it certainly looked like a cathode had gone bad.  Just to be on the safe side we followed through with our access request and re-segmented so that iFIX 15 is the first 1- degrees of the two wedges while iFIX 16-17 are the 4 others.  Current draws check out (~275 vs ~525 microA, respetively).  Shift elog summary is <a href=
'/cgi/elog/elog.pl?nb=2005&action=view&page=959&frame=2&anchor=10:52:27.934'>
linked here</a>.<br><br>
We also tried to look at wedge 4 on the NE via the cryo platform - actually would have worked ok but would have had to lay down and out the platform a bit to get a good look at things.  The real problem was all the cables and all the other stuff that made getting to the east corner harder than expected.  No obvious water damage although we didn't get a good look at the preamp board as we wanted (all the covers are on in that section).



<!-- Date: Wed Nov 23 13:31:32 2005 -->
East wedge 4 went crazy for a bit - <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=980&frame=2&anchor=10:35:59.408'>see here</a>.  Details to follow in the near future.
<!-- Author: Dale -->
<!-- Comment: Wed Nov 23 14;50;56 comment by...Dale -->
<br>When I ramped it up a few times on iFIX the current would jump into the +100 uA around 1 kV.  The trip log shows only anode trips and certainly the cathode looks well behaved.
<br><br>
Intially when I looked at it I saw the current changing radically once steady voltage had been applied (I think I had it on the 100 uA scale and could see it around 10 uA, bobbing up and down).  However, with the scale on 100 uA I got it to full voltage and along the way the thing settled down.  At 5.6 kV the anode drew 0.2 uA but still looked like the anode was shakeing (maybe +/- 0.05).  I kept it on while I searched for suspects (I didn't find any).
<br><br>
Now it's fine so we just play happy unless it acts up again.  The only bad thing is that we are in a shutdown period and the voice alarms (on Vnode 1 which is not working) are inactive so the aces are more likely not to notice this.


<!-- Date: Fri Dec 16 08:31:42 2005 -->
Akram and fellow CSX expert replace a discriminator at described <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=1035&frame=2&anchor=17:55:18.698'>here</a> 
Runs after 208706 should be ok.  The discriminator was discribed as "noisy'.  The interesting thing is that this section corsponds to the location of CMX west wedge 14 - the bottom of the SW arch.  the top group has effiencies here on the order of 5% and we have been unable (until now perhaps) to figure out the exact cause.  More details will likely be slow to come (for one Akram has left the country).
<!-- Author: Dale Stentz -->
<!-- Comment: Fri Dec 16 09;15;56 comment by...Dale -->
A few more details: the bad discriminator was in slot 7 of crate 3 in the 1RR12E rack.  Details on the NIM setup and the semi-bizzaro way counters are mapped to the discriminator (physical requirement on the meantimes requirements and how they work) I will obmitt here.  The realvent details are that the first 8 channels of the discriminator corspond to internal counters 62-69 which corsponds close to wedges 12-13 and the last 4 channels (the 4 others are blank as they do not related to a physical wedge) are for external counters 69-72 which do indeed map onto SW wedge 14.  Given that Eric James and company had just about totally removed the possiblity of a track/XTRP related trigger problem, it is a safe bet to assume that this was the problem.  We should bug them to see if they can show that the problem was localized to the lower half of the discriminator or someother realvent feature.


<!-- Comment: Thu Jan  5 20;25;26 comment by...Dale -->
<br>Looks like I got the logic wrong which justfies some of my conserns above.  As Eric explains a brom at the meantimer setup should propergate to the readout (thus YMon plots) and to the trigger/match box setup etc.  Thus the problem cannot really be explained by a CSX problem <i>per se</i> but really must be further down the chain when only the trigger could go bad.<br><br>
Update - here's a entry from Eric on a change is just made:<pre>
Earlier this afternoon, I changed the gate widths for forming 
CSX scintillator trigger primitives for channels 0-23 in the  
TDC in slot 14 of b0musc00 and channels 48-71 in the  
TDC in slot 15 of b0musc00.  The change was to widen  
these gates to be consistent with that used for west CSX  
miniskirt channels.  It is hoped that the change will recover 
some observed inefficiencies in CMX triggers (particularly  
for west CMX wedge 14).</pre>
Also see <a href=
'/cgi/elog/elog.pl?nb=2006&action=view&page=15&frame=2&anchor=16:24:47.011'>link  given here</a>.  This should be taken in context with Eric's orginal change - <a href=
'/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=7&frame=2&anchor=15:35:28'>
linked in this CMX e-log entry</a>.  The realvent details as sighted by Phil in the link are<pre>
[T]he gate was lengthened [by 52.5 ns] in MUSC00 slots 14: ch 24-47 (csx&msk), ch 48-71 (msk) 15: ch 0-23 (msk), ch 24-47 (csx&msk)
</pre>
I'll try to get an update on the situation and if Eric has found anything new.
<!-- Date: Fri Dec 16 09:57:12 2005 -->
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=1034&frame=2&anchor=11:52:22.801'> 
CMX (also BMU) TDCs</a> were hot as noticed by Phil on 12/11.  This issue was taken care of but then it became apparent that west wedge 8 was oscillating (<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=1035&frame=2&anchor=16:13:56.799'>link</a>).  Ingorning the possible causes for now, our action was to move CMX to off for a while and then turn it on (<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=1035&frame=2&anchor=17:34:05.918'>link</a>).  Long story short the oscilations were not present when ran cosmics with the full detector on ( 
<a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=1035&frame=2&anchor=17:39:32.842'>link</a>). 
<br><br> 
Later (while taking cosmics with the detector at stand-by) the oscilations returned.  The <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=1037&frame=2&anchor=08:32:15.862'> 
link here</a> gives an older plot but they were certainly there in the owl and eairly day shift.  A quick note <a href= 
'/cgi/elog/elog.pl?nb=2005&action=view&page=1037&frame=2&anchor=11:40:41.459'>here</a> discribes that turning on the HV removes the oscilations.<br><br> 
This behaviour has been seen before: <a href= 
'/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=5&frame=2&anchor=23:58:55'> 
link to Nov 2003 oscilations</a>.  Here's the 2D occupancy plot of what they looked like then <a href= 
'/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=5&frame=2&anchor=23:48:04'> 
link</a>.<br><br> 
Note that (as I had incorrectly recalled before) the current oscialtion issues the the ones linked above are separte from the oscilations in wedge 10 which causes us to put in the modified radekas in stacks 1 and 2 (details are noted  
<a href= 
'/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=5&frame=2&anchor=06:01:41'> 
here</a>).  The problems with SW 10 acutally occured just before the wedge 8 troubles (hence my comment "they're back"). 
<br><br> 
In any event, the oscilations always went away with the HV on and  eventally disapeared without my intervention by our part.
<!-- Author: Dale -->
<!-- Comment: Fri Dec 16 10;01;52 comment by...Dale -->
Note that there ahve been some peroids where the detector was at stand-by but there were no oscilations.  More often then not the are there.  Here's the 2d occupancy + muliplicity plot for the current cosmic run.<br>
<br><a href='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-453&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-453' width='300'></a>



<!-- Date: Wed Jan 25 07:52:46 2006 -->
Yesterday there were a few trips on NE 20 as we were closing up the east IMUs.  Actions taken are noted <a href= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=71&frame=2&anchor=09:05:15.179'>here</a>.  Note that the trips were on the cathode but the global HVAC varibles say things were fine (I thought it was a bit on the warm side when I went down).  Did not trip after I went down - we are at stand-by waiting for beam later today.  I don't expect any problems with CMX (or the other detectors) coming out of the shutdown.  Note that cosmics have shown things are in good working order (no obvious new 4 channel gaps).  
<!-- Author: Dale -->


<!-- Date: Sun Jan 29 18:42:29 2006 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-454&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=cmx&action=view&page=-454  width='300' align='top'></a></td><td>Dewpoint and all the other humdity varibles appear to be coming back down.  It stopped raining around 10:00 (1-29) but clearly the HVAC was kicking in doing its job.  Note that this set off trips in NE 20 (also had troubles at the end of the shutdown above).  The resolution is <a href= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=84&frame=2&anchor=19:41:57.870'> 
here</a> - various other comments are above such as <a href= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=84&frame=2&anchor=17:17:52.552'> 
here</a>. 
</td></tr></table>
<!-- Author: Dale -->