<!-- Title: 11:01:38  Mon Jan 29 2007 -->
<!-- Crewchief: Ignacio (MPC) -->
<!-- Op1: Jose (SPL) -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
Let's hope for another eventless week...


<!-- Date: Mon Jan 29 11:39:18 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10692&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10692  width='300' align='top'></a></td><td>Several pinkies appear at the same time a prescale change happened. Two level one triggers, bit 33 and bit 60 went in 
producing a step in all bias and DVD currents in SVX only (ISL and L00 did not show any change).  Ulrich checked that no temperature had changed.</td></tr></table>
<!-- Author: Ulrich/Jose/Ignacio/Roberto -->
<!-- Comment: Mon Jan 29 12;00;23 comment by...Ulrich -->
I was first suspecting the SVX cooling, but there was no indication that the SVX chiller temperatures changed significantly. They went up a little bit this morning around 7:30, so it's worth keeping an eye on the situation, but the change is not large enough to explain what we see.
<!-- Date: Mon Jan 29 11:55:20 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB1W4L1 
<li>problem n: 1907 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: Went just above the limit as a prescale changed.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>52.66</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>150.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B1W4&layer=L1">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Mon Jan 29 11:58:44 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB2W2L1 
<li>problem n: 1908 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: Went just above the limit as a prescale changed.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>45.9914</td><td>100</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100</td><td>150</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B2W2&layer=L1">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Mon Jan 29 12:01:38 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB2W7L0 
<li>problem n: 1700 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: Went just above the limit as a prescale changed.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>70.0</td><td>153</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>150.0</td><td>200.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B2W7&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Mon Jan 29 12:05:47 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W0L1 
<li>problem n: 1909 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: Went just above the limit as a prescale changed.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>46.7</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>150.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B3W0&layer=L1">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Mon Jan 29 12:07:51 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB4W4L3 
<li>problem n: 1910 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: Went just above the limit as a prescale changed.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>54.45</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.00</td><td>150.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B4W4&layer=L3">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Mon Jan 29 12:10:26 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W5L3 
<li>problem n: 1760 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ballarin 
<li>description is: Went just above the limit as a prescale changed.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>50</td><td>125</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>125</td><td>200</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B3W5&layer=L3">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Mon Jan 29 12:12:31 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W5L1 
<li>problem n: 1911 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ballarin 
<li>description is: Went just above the limit as a prescale changed.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>52.71</td><td>100</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100</td><td>150</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B3W5&layer=L1">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Mon Jan 29 12:13:36 2007 -->
<h3>Resin exchange required</h3> 
After we had warmed up the Silicon for the access last Thursday, the SVX coolant conductivity went up to 5 μSi/cm this morning. We will have to request a resin exchange whenever there is a good opportunity.
<!-- Author: Ulrich -->

<!-- Date: Mon Jan 29 12:14:35 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W8L0 
<li>problem n: 1701 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ballarin 
<li>description is: Went just above the limit as a prescale changed.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>101</td><td>150</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>150</td><td>200</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B3W8&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Mon Jan 29 12:16:44 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W9L0 
<li>problem n: 1912 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ballarin 
<li>description is: Went just above the limit as a prescale changed.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>47.88</td><td>100</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100</td><td>150</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B3W9&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Mon Jan 29 13:29:36 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB4W11L0 
<li>problem n: 1913 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: Went above 100 mA limit this morning following its monotonic increase as usual.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>47.6</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>150.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B4W11&layer=L0">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Mon Jan 29 13:37:15 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10693&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10693  width='300' align='top'></a></td><td>The monotonic increase of  B4W11L0 IBias.</td></tr></table>
<!-- Author: Ignacio -->


<!-- Date: Mon Jan 29 13:39:53 2007 -->
Since it is Monday, I will bump up IMON limits for a few yellow BIAS limits.
<!-- Author: Ignacio -->

<!-- Date: Mon Jan 29 13:42:30 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB4W9L0 
<li>problem n: 1914 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: Prevently increase the limit, since it is very close to cross it.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>40.6135</td><td>100</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100</td><td>150.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B4W9&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Mon Jan 29 13:45:59 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB4W7L0 
<li>problem n: 1915 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: Increase IMON limit prevently.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>40.69</td><td>100</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>150.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B4W7&layer=L0">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Mon Jan 29 13:48:24 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB2W7L4 
<li>problem n: 1916 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: Increase IMON Bias limit prevently.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>62.8</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>150.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B2W7&layer=L4">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Mon Jan 29 13:50:32 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W7L1 
<li>problem n: 1917 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: Increase IMON Bias limit prevently.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>49.6</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>150.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B0W7&layer=L1">report</a></ul>
<!-- Author: Problem DB -->


<!-- Date: Mon Jan 29 13:54:16 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W3L0 
<li>problem n: 1846 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: I will be a bit more generous and go up to 150 instead of 110 mA.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>100.0</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>110.0</td><td>150.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B0W3&layer=L0">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Mon Jan 29 13:57:34 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10694&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10694  width='300' align='top'></a></td><td>This is the only ladder of the above which seems to increase faster lately. Keep an eye on it.</td></tr></table>
<!-- Author: Ignacio -->
<!-- Comment: Mon Jan 29 17;41;24 comment by...Jen -->
This is the power supply that we swapped <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=337&frame=2&anchor=12:54:01'>last Thurs</a>, that explains the sudden jump up a few days ago.  Should behave normally from now on.
<!-- Date: Mon Jan 29 14:00:15 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W7L1 
<li>problem n: 1917 
<li>Status: needs_observation
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: The bias current increase seems to be accelerating in this ladder. Keep an eye on it.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>100.0</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>150.0</td><td>150.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B0W7&layer=L1">report</a></ul>
<!-- Author: Problem DB -->





<!-- Date: Mon Jan 29 14:04:37 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB3W4L1 
<li>problem n: 1918 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: Increase IMON bias limit prevently.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>45.0</td><td>100.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>100.0</td><td>150.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B3W4&layer=L1">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Mon Jan 29 17:50:23 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: husemann
<li>Description of Problem: As the next store is expected to be a big one, Dave Waters has tested a new trigger table, which deals with some high-rate CMUP triggers for luminosities above 250e30. We ran the table successfully without and with Silicon at 37e30. The table will stay in as the default for the next store (which is expected to be big due to the imminent Main Injector shutdown for the NuMi magnet swap).
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=118">Incident Report</a></ul>
<!-- Author: Incident DB -->


<!-- Date: Mon Jan 29 18:08:02 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB0W7L1 
<li>problem n: 1917 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: Jen points out <a href= 'http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=xe&page=338&time=13:57:34'>here</a>  that the reason for the increase is the PS  swap.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>??</td><td>??</td></tr>
<tr><td><b>Lower</b></td><td>??</td><td>??</td></tr>
<tr><td><b>Upper</b></td><td>??</td><td>??</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B0W7&layer=L1">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Mon Jan 29 19:32:35 2007 -->
This morning there was another short communication glitch with a 
CAEN crate that did not need a hockerization; it recover by itself. 
However IMON crashed and had to be restarted. 
<!-- Author: Jose/Ignacio -->


<!-- Date: Mon Jan 29 19:35:22 2007 -->
There was an electron lense trip just  
  <a href= 
'http://www-cdfonline.fnal.gov/cgi/internal/elog/elog.pl?nb=2007&action=copyentry&page=87&time=19:21:02.419'> now</a>. The backgrounds came back to the previous, flat behaviour very fast. On top of  the the electron lense variable (T:LICOLI1) switch  and AGIGI2, there was a broad spike in the proton abort gap counters, that only reached 3500 Hz, i.e. only ~2 above the flat background. The rest of the background variables remained constant.
<!-- Author: Ignacio/Jose -->


<!-- Date: Tue Jan 30 08:17:43 2007 -->
We caused a loss of some lumi at the beginning of the store 5201, that was a new record lumi of 278 E30. A couple of ISL ladders 
tripped on overvoltage (f450). As I went into the control room, Ulrich called in suggesting to hockerize, which I did from the IFix panel. Two L00 wedges  tripped once after hockerization but the recovery was pretty quick. 
 
Another thing is that I arrived late, scrapping only took 11 min. After talking to Jose I have asked to change the note on the blackboard to page silicon after the begin of pbar loading.
<!-- Author: Ignacio -->


<!-- Date: Tue Jan 30 08:20:47 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10697&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10697  width='300' align='top'></a></td><td>Store 5201 looks OK in SVT and SVXMon. It was a record (but see above).</td></tr></table>
<!-- Author: Ignacio -->


<!-- Date: Tue Jan 30 10:13:49 2007 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10698&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10698' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10699&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10699' width='300'></a></center>
<!-- Author: Mark -->
<!-- Comment: Tue Jan 30 10;15;37 comment by...Mark -->
From this morning's store, it is nice to see that we run with a L2A rate of 850 Hz with deadtime of 10% at 235e30.


<!-- Comment: Tue Jan 30 11;31;24 comment by...Ulrich -->
So it seems that the trigger table we tested yesterday did its job.  The main change was to remove CMUP triggers without stereo confirmation at high luminosities.
<!-- Comment: Tue Jan 30 11;48;53 comment by...Mark -->
Looking back at yesterdays main elog entry, I think the new trigger table is only different above 260e30, so we probably won't know until we have another big store (and we don't have any problems at the beginning).  
<!-- Date: Tue Jan 30 10:20:09 2007 -->
<center><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10700&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10700' width='300'></a><a href='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10701&button=yes' target='xxx' onclick='jswindow()'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10701' width='300'></a></center>
<!-- Author: Mark -->
<!-- Comment: Tue Jan 30 10;21;59 comment by...Mark -->
From store 5194 we have points at higher luminosity and see deadtime of 30% with L2A rate of 1000 Hz at 255e30.


<!-- Comment: Tue Jan 30 12;05;31 comment by...Mark -->
As can be seen from our previous <a href=
'/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=328&frame=2&anchor=17:19:02'>
event size studies</a>, we should be hitting ~20kB at 260e30.  According to Jonathon Lewis's studies, this would allow for a L2A rate up to 1200 Hz while still keeping deadtime below 5%, so we still look pretty good.  
<!-- Date: Tue Jan 30 10:25:37 2007 -->
Shift crew issued a <a href= 'http://www-cdfonline.fnal.gov/cgi/internal/elog/elog.pl?nb=2007&action=copyentry&page=89&time=10:01:49.901'> HRR</a> because of a AVDD pinkie in (the chip came back after the HRR) B3W7L2
<!-- Author: Ignacio -->
<!-- Comment: Tue Jan 30 14;56;15 comment by...Ignacio -->
Happened twice more.


<!-- Date: Tue Jan 30 17:57:52 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10704&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10704  width='300' align='top'></a></td><td>SB1W8K4 AVDD pinkie inmune to HRRs as it happened last  
week.</td></tr></table>
<!-- Author: Ignacio -->
<!-- Comment: Tue Jan 30 17;58;50 comment by...Ignacio -->
<a href=
'http://www-cdfonline.fnal.gov/cgi/internal/elog/elog.pl?nb=2007&action=copyentry&page=70&time=07:13:23.515'>
Then</a>,  it  recovered just after terminate or by the impact of Oscar presence. Let's wait and see.


<!-- Comment: Tue Jan 30 18;22;22 comment by...Jose/Ulrich/ignacio -->
We took the opportunity of a crate sheffarding to recycle the wedge: the chips came back gracefully.
<!-- Date: Tue Jan 30 18:04:51 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10705&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10705  width='300' align='top'></a></td><td>IB1W0L3 bias line 1 has gone pink. The history justifies bumping up the limits.</td></tr></table>
<!-- Author: Ignacio -->

<!-- Date: Tue Jan 30 18:11:33 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB1W0L3 
<li>problem n: 1510 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: The current has reached the limit, I give it a little room.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>165.0</td><td>180.0</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>180.0</td><td>200.0</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B1W0&layer=L3">report</a></ul>
<!-- Author: Problem DB -->




<!-- Date: Tue Jan 30 21:00:31 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10706&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10706  width='300' align='top'></a></td><td>Cooling trip temperatures</td></tr></table>
<!-- Author: Ignacio -->


<!-- Date: Tue Jan 30 22:15:41 2007 -->
After the cooling flow came back we  started recovering the detector while data was being taken without the silicon. Since  IFix was broken we had powered up the detector with the PSGUI. 
After a while the temperature in E1.1 surpassed 21 degrees so it became  clear that we had to exclude one of the compromised ISL ladders (f451). We aborted  the run that was about to start to exclude the ladder. This took longer than expected since we were not successful generating the CAENVolts.dat at first because we were usesing   a username that was not svxii.  However the  inhibits did not clear even though the PSGUI showedthe detector  to be ON.  At this point we reboot svxinter to have a clean start, and sent the silicon to standby from the PSGUI 
followed by "HV On" from Ifix (that at this point was working). Then, data taking resumed. 
<!-- Author: Jose/Ulrich/Ignacio -->
<!-- Comment: Tue Jan 30 22;21;20 comment by...Jose, Ignacio, Ulrich -->
Not that this was trouble enough, we also had a problem with SVX B5W4 which was all pink because of a CAEN readback problem. We managed to first hockerize the wrong crate (#16). When we finally hockerized the right crate (#10), everything came back fine.
<!-- Date: Tue Jan 30 22:17:39 2007 -->
<h3>ISL Cooling Trip</h3> 
The problem started this morning around 8:30, when the flow to ISL CCE1 dropped suddenly by 0.2 psi. Flows usually don't drop by themselves. The pressures did not change at all when the flow decreased. On the other hand, this is our prime suspect for a problem with the ISL cooling. We will have to investigate further why this happened. One thing would be to visually inspect the connections on the manifold where the flow-meter is situated.  
<p> 
The problem was only realized during the evening shift because of an iFix warning. Usually we would not touch a valve during a store, but given that this store will be kept for another day or so, we decided to act instead of risking a trip on low flow. We opened up the e-valve from 75% to 77%, which brought up the flow a little bit. The pressure was still below 12 psi at that point. So we decided to try 80%, which pulled an ISL East flow trip on h igh pressure. Dr. Tripmann should have known better! As many of the temperatures are quite high, at least one of them surpassed 25°C shortly, pulling a temperature übertrip, which also took out SVX and L00.  
<p> 
The flows were re-established quickly (and without the help of the cooling expert!). Something is still weird in the system, so that many valve settings are off from their default values now: 
<ul> 
<li>CCE1: 78% instead of 75% 
<li>CCE2: 72% instead of 75% 
<li>CCE4: 80% instead of 85% 
<li>PCE: 75% instead of 80% 
</ul> 
 
 When we switched on the only ladder which was included on the compromised cooling line (f451), the temperature increased above 22°C. This did not happen with a steep increase as before but very slowly. After we took out f451, the temperature is back to 20.9°C.
<!-- Author: Jose, Ignacio, Ulrich -->
<!-- Comment: Wed Jan 31 09;40;38 comment by...She who disagrees with the entry -->
<b>"...without the help of a cooling expert..."</b>. Interesting. I could have sworn that I was on the phone with Jose last night for 15 minutes talking about how to re-establish flows to ISL....
<!-- Comment: Wed Jan 31 11;24;07 comment by...A non cooling expert -->
As it was mentioned in the 8am meeting, instructions from Jennifer were precise and fundamental for recovering the trip.
<!-- Comment: Thu Feb  1 10;26;47 comment by...Happy Jennfer :-) -->
Thank you :-) That makes me feel all warm and fuzzy inside :-)
<!-- Date: Tue Jan 30 22:23:21 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB4W5L1 
<li>problem n: 1900 
<li>Status: needs_work
<li>Type: PC_Temperature
<li>Action: Removed_hdwdb
<li>entered by: husemann 
<li>description is: After a self-induced ISL cooling trip, the temperature in ISL CCE1.1 is too high to power this ladder, so we removed it from the HWDB. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B4W5&layer=L1">report</a></ul>
<!-- Author: Problem DB -->




<!-- Date: Tue Jan 30 22:26:33 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ignacio
<li>Description of Problem: The cooling trip is described <a href= 'http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=338&time=22:17:39'> here</a>.The detector recovery is in this <a href= 'http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=338&time=22:15:41'> entry</a>.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=119">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Wed Jan 31 09:03:27 2007 -->
During a trigger test this morning, when the run control  
was going through partition, we got a spontaneous switch off with associated hockerization (Crate # 2). The ace mentioned the same happened yesterday during an <a href='http://www-cdfonline.fnal.gov/cgi/internal/elog/elog.pl?nb=2007&action=copyentry&page=89&time=11:05:29.462'>HRR</a>. 
So it seems  not all cases of CAEN spontaneous switch off can be attributed to  SEU.
<!-- Author: Ignacio -->
<!-- Comment: Sat Feb  3 04;44;41 comment by...The Old Yale -->
Interesting though, that the power supplies do not know anything about a "PARTITION" command nor an HRR.
<!-- Date: Wed Jan 31 11:39:45 2007 -->
 <a href='http://www-cdfonline.fnal.gov/cgi/internal/elog/elog.pl?nb=2007&action=copyentry&page=92&time=11:07:50.378' 
>Autohockerization</a> of crate five as an HRR was going on. The previous error logger entry: 
<pre> 
(RC)   11:02:54 Run -> ACTIVE 
</pre> 
The IFIX alarm: 
<pre> 
Wed Jan 31 11:03:53 2007        SVX_PS_SMTH_OFF SVX power supply off but expected to be on 
</pre> 
...and a Tevatron abort. Shift crew did post quench checkout.
<!-- Author: Ignacio/Jose -->


<!-- Date: Wed Jan 31 14:01:43 2007 -->
The kernel on b0dasvx03 has been successfully updated, for testing purposes we have IMON and PSGui running on that pc. We will keep both processes running for a while before updating b0dasvx01 and b0dasvx02.
<!-- Author: Jose -->
<!-- Comment: Thu Feb  1 08;44;25 comment by...Jose -->
As we are not running IMON in the standard pc, ProcMon is now complaining about IMON missing. We asked the ACEs to ignore this check up to the new kernel test is done.
<!-- Date: Thu Feb 1 08:41:53 2007 -->
Two changes has been made in the DAQ this week. 
 
<ul> 
<li> The <i>Silicon Resonator</i> message was appearing too often in the error logger and in some times making more difficult to discover the important information. The messages: 
<pre> 
(MLE) b0svx03:Messenger:Jan 30, 2007 09:57:20 AM->Silicon Resonator: Sl 10 Ch 9 -> e334: Bytes: 00082 e360 
(MLE) b0svx09:Messenger:Jan 30, 2007 09:57:20 AM->Silicon Resonator: Sl 20 Ch 3 -> f823: Bytes: 00780 
(MLE) b0svx07:Messenger:Jan 30, 2007 09:57:20 AM->Silicon Resonator: Sl 13 Ch 3 -> fa23: Bytes: 00780 
</pre> 
 
has been moved to a new files (also a copy of the <i>Resonances</i> goes to these files). 
<pre> 
/cdf/onln/data/cdfdaq/rcLog/errmon/log/errorfile233260_resonator.log 
</pre> 
 
<li> We had some problems with IMON crashing and nobody noticed it up to some hours later. To prevent this to happen again, IMON is now in implemented in ProcMon. IMON has been also modified to save a file: 
<pre> 
~svxii/imon_dev/logfiles/LastUpdate.log 
</pre> 
with the last time of the update. This file is used by ProcMon to check if it is running. 
</ul> 
 
Thanks to Bill, Donatella, Jonathan and Ray for all the work done and suggestions.  
  
<!-- Author: Ulrich and Jose -->
<!-- Comment: Thu Feb  1 11;20;03 comment by...Ankush -->
<p>Q: With the resonator log file...do you just record the resfinder message or a set of RC messages before/after ? <br>
ie:- did the GB fire or was this a regular HRR ?</p>


<!-- Comment: Thu Feb  1 11;28;12 comment by...Ulrich -->
We are getting the SRC error message (FERML_SRC_FATALITY) in both logs and the resfinder message only in the resonator log.
<!-- Date: Thu Feb 1 09:22:41 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: husemann
<li>Description of Problem: L00 West power supplies tripped on low flow during resin exchange.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=120">Incident Report</a></ul>
<!-- Author: Incident DB -->


<!-- Date: Thu Feb 1 09:36:02 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: husemann
<li>Description of Problem: Yet another cooling trip due to the resin exchange. This time it took out East ISL.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=121">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Thu Feb 1 09:56:22 2007 -->
<h3>Cooling Work: Resin Exchange and E-Valve Tuning</h3> 
We requested a resin exchange this morning. During the course of this, we had two cooling alarms due to low-flow trips which took out the PS enables, first in L00 West, then in ISL East. I have spoken with the cryo tech who did the resin exchange (KC) about the procedure, and it seems to be fine. I wonder if the system is just a little touchy or if there is a real flaw in the procedure. 
<p> 
While East ISL was off, I re-tuned the e-valves for East ISL which were left in non-standard positions after the self-induced trip on Tuesday. All all back to their default, except for CCE1, which is at 80% (default: 75%). 
<p> 
I have also taken the opportunity to remind the shift crew and the ops manager in training about the procedures for cooling alarms. In both cases, only half of the subsystem in alarm went off, which, according to our training, would require them to hit the crash button.
<!-- Author: Ulrich -->
<!-- Comment: Thu Feb  1 10;14;10 comment by...Ulrich -->
As it is the first of the month, I have also rebooted SVXIICON.





<!-- Date: Thu Feb 1 13:51:54 2007 -->
<h3>Access Summary</h3> 
We took advantage of the supervised access to do some DAQ/Detector work in the collision hall.  We swapped in FIB (S/N) 122 for f54/5 in b0fib07 slot 11.  This pair of wedges gives occasional pipeline out of sync errors at the beginning of runs, so we expect it is a FIB or FTM problem.  The wedges run fine after the change.  We also did permanent light level measurements in the collision hall and on the first floor: 
 
<TABLE border=1> 
<tr><th>Bit<th>Collision Hall (uW)<th>First Floor (uW) 
<tr><th>OBDV<td>464<td>308 
<tr><th>0<td>520<td>320 
<tr><th>1<td>792<td>480 
<tr><th>2<td>664<td>368 
<tr><th>3<td>712<td>448 
<tr><th>4<td>624<td>404 
<tr><th>5<td>648<td>396 
<tr><th>6<td>186<td>356 
<tr><th>7<td>640<td>440 
</table> 
 
 
<!-- Author: Ignacio, Roberto, Takayuki, Jose, Ulrich, Mark -->

<!-- Date: Thu Feb 1 15:12:02 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB5W4/5All 
<li>problem n: 1689 
<li>Status: needs_observation
<li>Type: Chip_capid
<li>Action: Swap_FIB
<li>entered by: mhartz 
<li>description is: We swapped out the FIB because we were seeing capid errors at the beginning of some runs.<table border=1>
   <tr>
    <td>
     <b>IN:</b>
    </td>
    <td>
     <a href="http://b0sili01.fnal.gov/cgi-bin/fibdb/DBreport.pl?serial=122">122</a>
    </td>
    <td>
     <a href="http://b0sili01.fnal.gov/cgi-bin/fibdb/DBreport.pl?prep=560598">560598</a>
    </td>
   </tr>
   <tr>
    <td>
     <b>OUT:</b>
    </td>
    <td>
     <a href="http://b0sili01.fnal.gov/cgi-bin/fibdb/DBreport.pl?serial=107">107</a>
    </td>
    <td>
     <a href="http://b0sili01.fnal.gov/cgi-bin/fibdb/DBreport.pl?prep=550269">550269</a>
    </td>
   </tr>
  </table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B5W4/5&layer=All">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Fri Feb 2 09:29:43 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10707&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10708  width='300' align='top'></a></td><td>Tonigh's pinkie.</td></tr></table>
<!-- Author: Ignacio -->
<!-- Comment: Fri Feb  2 09;30;33 comment by...Ignacio -->
It was back green in the morning.

<!-- Date: Fri Feb 2 09:33:43 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ignacio
<li>Description of Problem: See <a href='http://www-cdfonline.fnal.gov/cgi/internal/elog/elog.pl?nb=2007&action=copyentry&page=97&time=03:47:19.633'> main elog</a>  and pinkie <a href='http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=338&time=09:29:43'> plot</a>.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=122">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Fri Feb 2 09:43:23 2007 -->
<h3> b0dasvx01 and b0dasvx02</h3> 
This morning the OS was upgraded by REX for b0dasvx01 and b0dasvx02.  We were moving IMON and PSGui from server to server. 
The machine where the RTServer runs was rebooted at around 78:50. We asked Koji to restart it.
<!-- Author: Ulrich/Jose/Ignacio -->
<!-- Comment: Fri Feb  2 09;44;11 comment by...Ignacio -->
<pre>Nowe everything is running where it should.</pre>


<!-- Date: Fri Feb 2 09:56:00 2007 -->
<h3>An open partition that does not close? </h3> 
In the proccess of turning on the ladder that trips  (e3a4) when the muon wall opens, we closed a rc without releasing the partition. We could not book the crates nore realese the partition. Bill told us how to realease the zombie, just restart the resource manager from RC: 
Proxy->ResMgr 
 
Stop and Start ont he first line (ResMgr_Prd)  
<!-- Author: Ulrich/Jose/Ignacio -->
<!-- Comment: Fri Feb  2 10;02;35 comment by...Ignacio/Jose/Ulrich -->
After the RTServer came back, we had to restart b0fib00, b0fib02, b0fib04  and b0fib06 because the messanger was not running. After that, we recovered e3a4 and all detector  is in standby.


<!-- Date: Fri Feb 2 10:06:19 2007 -->
Dervin informs us that he wants to replace a fan in one of the crates in the rack 11R21F, where the VRB crates b0svx00 amd b0svx01 live. He is doing it.
<!-- Author: Jose/Ignacio -->
<!-- Comment: Fri Feb  2 11;06;52 comment by...Ignacio -->
Fan job is finished.

<!-- Comment: Fri Feb  2 11;07;59 comment by...Ignacio -->
Coldstart running OK.

<!-- Date: Fri Feb 2 15:58:45 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ignacio
<li>Description of Problem: As we were doing tests for the TurboHRR that involved using  the SRC trigger and will be described later we managed to trip a good part of ISL and L00.  The proccess of recovering involved first rebooting b0svx06,b0svx07,b0svx08 and  b0svx09 and resetting al the VRB boards on those crates as we were getting FIFO errors on the CPU consoles.   Recovering L00 was a bit more cumbersome, and required several wedge recycles. 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=123">Incident Report</a></ul>
<!-- Author: Incident DB -->
<!-- Comment: Fri Feb  2 15;59;54 comment by...Sebastian/Jose/Ignacio/Saty -->
<pre>Coldstart runnig OK, we'll go for cosmics as soon as possible.</pre>


<!-- Comment: Fri Feb  2 17;10;16 comment by...Sebastian -->
The tripped was caused by me disconnecting the TS input to the SRC, sorry.
<!-- Date: Fri Feb 2 16:01:34 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10709&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10709  width='300' align='top'></a></td><td>Temperatures during the trip.</td></tr></table>
<!-- Author: Ignacio -->


<!-- Date: Fri Feb 2 16:28:34 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10711&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10711  width='300' align='top'></a></td><td>The flow that made us trip  earlier in the week has suffered another step doww from 1.91 lpm to 1.68 lpm.</td></tr></table>
<!-- Author: Ignacio -->





<!-- Date: Fri Feb 2 17:32:33 2007 -->
<h3>Testing turboHRR</h3> 
 
After tripping the silicon (that was me, Sebastian) we talked to Bill about not seeing the L1A in tstsvx1, while trying to remap the cross points he found that he had put tstsvx1 offline back in July, after he put it back to online we got the L1A.<br><br> 
We also see HALT, RECOVER and RUN coming through taxi into the SRC, and the SRC works fine through the transitions (using RC in 2src_test configuration), the next step is to aske Steve as to why this was not enabled.<br>
<!-- Author: Sati/Ignacio/Jose/Sebastian -->
<!-- Comment: Sat Feb  3 10;50;46 comment by...sebastian -->
I compared the turboHRR with normal HRR and both take the same time, about 3seconds (in the teststand). 
<!-- Comment: Sat Feb  3 11;11;23 comment by...s -->
See LA signals <a href=
'http://www-cdfonline.fnal.gov/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=338&frame=2&anchor=11:08:08'>
here</a>. TurboHRR does nto change the taxi signals, but does the crate cpu still send the transitions to the SRC via the backplane?

<!-- Date: Fri Feb 2 17:46:12 2007 -->
<h3>Another Drop in CCE1 Flow</h3> 
As last Tuesday, we have observed a 0.2 lpm drop in the ISL CCE1 flow. At the same time the COT face pressure went down to around 11.5 psi. So we decided to play with the flows again. We switched off all of ISL as a precaution and cranked up the CCE1 valve from 80% to 85%. To equalize the pressure, we also opened CCE3 from 75% to 80%. The pressure is now around 13 psi, and the flow is 1.9-2.0 lpm. Also this time, it is not clear why the drop occured.
<!-- Author: Jose, Ignacio, Ulrich -->


<!-- Date: Fri Feb 2 19:22:25 2007 -->
<h3>Cosmics</h3>
We run cosmics. Everything looks basically as before, which is reassuring.  The ladders in the fib we exchanged yesterday show errors until 50 events are integrated, then comes back green. 
 
<!-- Author: Jose/Ulrich/Ignacio -->



<!-- Date: Sat Feb 3 10:42:00 2007 -->
the b0dasvx01 and 02 machines refuse to open X windows.
<!-- Author: sebastain -->
<!-- Comment: Sat Feb  3 10;46;26 comment by...s -->
this happened last week too. imon is running fine though.



<!-- Comment: Sat Feb  3 11;16;04 comment by...Ulrich -->
This happened both before and after the software update (unless somebody did the update before telling us). You may run into problems opening new windows, such as the history of an IMON current. The only known solution so far is stopping all programs and logging out.
<!-- Date: Sat Feb 3 10:51:28 2007 -->
<pre>Since the upgrade of b0dasvx01 and b0svx02 yesterday the X server does not allow to access the windows on the screen remotely. 
So the Screenshot  part of MPCSetup.sh  is broken. I do not know if anything else would be affected. 
 
Probably connected are the following errors you can  see when you log  to an online machine as svxii: 
 
X connection to localhost:10.0 broken (explicit kill or server shutdown). 
X connection to localhost:10.0 broken (explicit kill or server shutdown). 
X connection to localhost:10.0 broken (explicit kill or server shutdown). 
<svxii@b0dasvx01> 
 
 
These correspond to these lines in .cshrc: 
 
xhost +b0dasvx01 
xhost +b0dasvx02 
xhost +bodasvx03</pre>
<!-- Author: Ignacio -->
<!-- Comment: Sat Feb  3 11;17;05 comment by...Ulrich -->
Are you sure this is due to the upgrade? It could also be due to the problem Sebastian reported above.
<!-- Comment: Sat Feb  3 11;27;22 comment by...Ignacio -->
I am pretty sure the last time the screenshot worked was Feb 2 04:22 AM (see the web page), i.e. a few hours before the upgrade. To my knowledge it has always worked before.
<!-- Comment: Sat Feb  3 11;31;25 comment by...Ulrich -->
That's actually not true. The last time the X server connection was lost, the screenshot didn't work either, therefore my previous comment.
<!-- Comment: Sat Feb  3 11;37;07 comment by...Ignacio -->
<pre>OK, in that case we better log off and restart everything before
shot setup. Does anyone know how to restart the Xserver without 
having to log off? </pre>
<!-- Comment: Sat Feb  3 12;03;11 comment by...Ulrich -->
No good idea so far. <tt>/var/log/XFree86.0.log</tt> has a lot of entries like:
<pre>
AUDIT: Fri Feb  2 22:06:08 2007: 13993 X: client 24 rejected from IP 131.225.236.32 port 38476
AUDIT: Fri Feb  2 22:06:08 2007: 13993 X: client 24 rejected from IP 131.225.236.32 port 38477
AUDIT: Fri Feb  2 22:06:08 2007: 13993 X: client 24 rejected from IP 131.225.236.32 port 38478
AUDIT: Fri Feb  2 22:06:08 2007: 13993 X: client 24 rejected from IP 131.225.236.32 port 38479
AUDIT: Fri Feb  2 22:10:01 2007: 13993 X: client 24 rejected from local host
AUDIT: Fri Feb  2 22:13:35 2007: 13993 X: client 24 rejected from IP 131.225.236.32 port 39094
AUDIT: Fri Feb  2 22:13:35 2007: 13993 X: client 24 rejected from IP 131.225.236.32 port 39095
AUDIT: Fri Feb  2 22:13:35 2007: 13993 X: client 24 rejected from IP 131.225.236.32 port 39096
AUDIT: Fri Feb  2 22:13:35 2007: 13993 X: client 24 rejected from IP 131.225.236.32 port 39097
AUDIT: Fri Feb  2 22:14:59 2007: 13993 X: client 24 rejected from IP 131.225.236.32 port 39219
AUDIT: Fri Feb  2 22:14:59 2007: 13993 X: client 24 rejected from IP 131.225.236.32 port 39220
AUDIT: Fri Feb  2 22:14:59 2007: 13993 X: client 24 rejected from IP 131.225.236.32 port 39221
AUDIT: Fri Feb  2 22:14:59 2007: 13993 X: client 24 rejected from IP 131.225.236.32 port 39222
AUDIT: Fri Feb  2 22:20:01 2007: 13993 X: client 24 rejected from local host
AUDIT: Fri Feb  2 22:30:01 2007: 13993 X: client 24 rejected from local host
</pre>
I have googled a little and found explanations like: the hostname may have changed in the middle (DNS problem), or the .Xauthority file is not correct. Here is the current xauth settings:
<pre>
<svxii@b0dasvx01>xauth list
b0dasvx01.fnal.gov/unix:10  MIT-MAGIC-COOKIE-1  09c63db22db54a96a29646ff1f68b1c6
</pre>
Any ideas? I suggest we fire off a helpdesk ticket for the problem.
<!-- Comment: Sat Feb  3 12;20;46 comment by...Ulrich -->
I'm writing a helpdesk ticket right now. Another thought: could it be that the attempt to do the snapshot triggers the problem. That would also explain why it happens simultaneously on both b0dasvx01 and b0dasvx02.
<!-- Comment: Sat Feb  3 12;49;48 comment by...Ignacio -->
<pre>I just arrived to B0, I will log off and restart the programs. When did the problem started to happen? Does it match with the use of the MPCscript (from about the middle of September)?
</pre>
<!-- Comment: Sat Feb  3 12;55;44 comment by...Ulrich -->
The first time I saw the problem was on January 25, 2007. Having thought about the problem again, I don't think it is the screen shot. b0dasvx03 is unaffected by the screen shot and shows the same problems.
<!-- Comment: Sat Feb  3 13;15;59 comment by...Ignacio -->
<pre>I log off from the 3 machines and restarted IMON and PSGui. </pre>
<!-- Comment: Sat Feb  3 15;41;42 comment by...Ulrich -->
Another observation: After X was restarted, the xauth list is much longer (at least when I'm logging in from home via b0gateway):
<pre>
b0dasvx01.fnal.gov/unix:10  MIT-MAGIC-COOKIE-1  2fff88dcf67a1ddbcacdc9062d7d0f26
b0dasvx01.fnal.gov/unix:0  MIT-MAGIC-COOKIE-1  4937b1ac3a053baf12306f781adcff10
b0dasvx01.fnal.gov:0  MIT-MAGIC-COOKIE-1  4937b1ac3a053baf12306f781adcff10
b0dasvx02.fnal.gov/unix:0  MIT-MAGIC-COOKIE-1  1fc966f7dfd618dd2a353349016f951e
b0dasvx02.fnal.gov:0  MIT-MAGIC-COOKIE-1  1fc966f7dfd618dd2a353349016f951e
b0dasvx03.fnal.gov/unix:0  MIT-MAGIC-COOKIE-1  43f709ca522463e05e630baf2acc7667
localhost.localdomain/unix:0  MIT-MAGIC-COOKIE-1  43f709ca522463e05e630baf2acc7667
localhost.localdomain:0  MIT-MAGIC-COOKIE-1  43f709ca522463e05e630baf2acc7667
b0dasvx03.fnal.gov:0  MIT-MAGIC-COOKIE-1  43f709ca522463e05e630baf2acc7667
localhost.localdomain:0  MIT-MAGIC-COOKIE-1  43f709ca522463e05e630baf2acc7667
b0gateway.fnal.gov/unix:12  MIT-MAGIC-COOKIE-1  d46cc37e78855f1212bf75ce2b7f4475
b0dasvx03.fnal.gov/unix:10  MIT-MAGIC-COOKIE-1  ae51dec373ff982f4591ac1ead4a1d79
b0dasvx02.fnal.gov/unix:10  MIT-MAGIC-COOKIE-1  7fd3079743e3c012515a82ff057a0a96
</pre>
<!-- Comment: Sat Feb  3 18;02;38 comment by...Ignacio -->
<p>Well, it happend again.  The .Xauthority file was empty. It 
growed to:</p>
<pre>
xauth list
b0gateway.fnal.gov/unix:23  MIT-MAGIC-COOKIE-1  7a5c745804f35aa96da5ecbb307a1562
</pre>
when I logged into b0gateway from b0silki01. IMON has crashed, it stopped updateing at 17:13, abot 55 min ago. ProcMon box was red.
<!-- Comment: Sat Feb  3 18;38;28 comment by...Ignacio -->
<pre>I logged out but svxii could not log in again. It turned out 
that we are over quota. I have deleted a few ps files, we should make a serious clean up.
Disk quotas for user svxii (uid 13170):
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
b0home:/vol/home
                4728800* 4096000 4915200           35870  131072  131072
b0spool:/vol/data
                42393052  52428800 52428800           90379  4294967295 4294967295
b0spool:/vol/spool
                      0  10240000 10240000               5  262144  262144
</pre>
<!-- Comment: Sat Feb  3 19;50;57 comment by...Ignacio -->
I made 800M by  copying the directories "plots_to_be_deleted" and "files_to_be_deleted" to b0sili01:/data01/PSQuickPlot. 
Also deleted a couple of large old data dump files, the files under the mozila cache and a core.
<pre>
Disk quotas for user svxii (uid 13170):
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
b0home:/vol/home
                  3661M   4000M   4800M           31852    132k    132k
</pre>
I saw another 300M disappear, I guess someone else was cleaning in parallel....We should probably get rid of some of the imon versions.
<!-- Comment: Sat Feb  3 19;56;22 comment by...Ulrich -->
Ahh, that would explain it. It turns out that the worst offenders are the svxdaq directories. For both SVX and ISL, the data directories contained ASCII files with the output of more than one year's worth of the svxdaq error parser. I have created tarballs for  both in <tt>/cdf/onln/data/svxii/daq/backup/</tt> and deleted them in the original directories (except for those which belong to pronko or mhartz). This has freed up some 1.5 GB of space, so that we are way below quota now.
<!-- Date: Sat Feb 3 11:08:08 2007 -->
<center><a href='/elog/notebooks/silicon/Sat_Feb__3_2007_110706.ps'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10713' ></a><a href='/elog/notebooks/silicon/Sat_Feb__3_2007_110748.ps'><img src='/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10715' ></a></center>
<!-- Author: Sebastian -->
<!-- Comment: Sat Feb  3 11;09;55 comment by...s -->
no difference in taxi signals with or without turboHRR. Will turboHRR change the taxi signals, probably not, so is the SRC still getting the transition throught the backplane? probably yes...

















<!-- Date: Sat Feb 3 21:14:24 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ignacio
<li>Description of Problem: See <a href='http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=338&time=10:51:28'> here</a>
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=124">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Sun Feb 4 01:44:12 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10716&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10717  width='300' align='top'></a></td><td>Store  5208 came in with 140 E30 lumi. Triggers look OK. 
SVXMon shows red on the exchanged FIB until 165 events were recorder as happened during the cosmics.</td></tr></table>
<!-- Author: Ignacio -->


<!-- Date: Sun Feb 4 01:48:48 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10718&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10718  width='300' align='top'></a></td><td>TevMon was updating but got  somehow stuck. New TeVMon was in safe state. I checked the rates on ACNET and were low and flat, so I went ahead.  Jose came in afterwards and relaunched TevMOn.</td></tr></table>
<!-- Author: Ignacio -->

<!-- Date: Sun Feb 4 01:59:13 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: SB5W6L4 
<li>problem n: 1919 
<li>Status: fixed
<li>Type: BIAS
<li>Action: IMON_limits
<li>entered by: ignacio 
<li>description is: Increase bias limits of this Micron ladder.<table border=1>
<tr><th>IMON LIMITS</th><th>Old</th><th>New</th></tr>
<tr><td><b>Nominal</b></td><td>86</td><td>110</td></tr>
<tr><td><b>Lower</b></td><td>0.5</td><td>0.5</td></tr>
<tr><td><b>Upper</b></td><td>110</td><td>150.</td></tr></table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=SVX&wedge=B5W6&layer=L4">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Sun Feb 4 03:46:29 2007 -->
<h3> b0fib07 slot 11 strikes again</h3> 
<p> 
The exchanged fib went red again and I was paged. Also there are frequent automatic HRR produced by around 150 chips out of sync: 
</p> 
<pre> 
34'50" b0server03.fnal.gov:ConsumerErrorRe:Feb 4, 2007 03:34:50 AM->Runtime Error 4, Event 3765, RunNum 233500: SvxMon Halt Recover Run: Pipeline out of synch in 150 silicon readout chips . -->   
 Additional Information:  
 
 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!  
  
 Pipeline out of synch in 150 silicon readout chips .  
 AUTO HRR will be issued  
</pre> 
 I guess this is somehow related (the SVX fib board serves 88 chips). I have checked the strip occupancies from one wedge and seems reasonable, so I the data is not complete garbage.  I have asked the shift crew to restart the run and it came green again after a while. I think it is only a fraction of the events that are out of sync that cause the problem. I told them to repeat the procedure once if the problem comes back. If it repeats twice I asked them to ignore it. I could  rebot the FIB but I do not really think this is going to change anything. We may have to go in again to exchange/shake the FIB board a bit...
<!-- Author: Ignacio -->
<!-- Comment: Sun Feb  4 13;05;13 comment by...Ignacio -->
Ok, I was not very bright tonight. It is an ISL ladder  f54/55.
So I think it serves the 160 chips. Roughly the number that appears in the error logger.
<!-- Date: Sun Feb 4 12:11:09 2007 -->
<h3> CAEN PS Converter</h3> 
It has been know for long by some that our beloved  CAEN PS Converter list two wedges incorrectly. IB0W1 is read by a FIB board in fib03 (and was listed in fib00) and powered by PS crate # 2. 
IB5W0 is read by fib07 (was listed in fib04) and powered by PS crae # 10. Jose have done the hard work of  correcting this some time ago but validation was missing. I have compared the 
output by crate and by FIB. Since it was correct, I have updated the link in the detector web page (I leave a link to the old one in case of problems).
<!-- Author: Ignacio -->



<!-- Date: Sun Feb 4 14:19:32 2007 -->
They lost an <a href='http://www-bd.fnal.gov/cgi-mcr/elog.pl?nb=2007&action=xe&page=94&time=14:02:24'> RF</a>. LostP and Agigi2 went nuts, both the ace and TevMon catched it. Silicon was sent to standby. After 5 minutes, backgrounds came down and Silicon is safe so we reset the permit while a no silicon run is being taken.  RFSUm was 1.25 before they lost the station (I though it was always 1), now is 0.83.  While I was writing this, the abort gap antiprotons showed large spikes, permit revoked.  
<!-- Author: Jose/Ignacio -->
<!-- Comment: Sun Feb  4 14;25;06 comment by...Ignacio -->
See hourly plots starting  from <a href='http://www-bd.fnal.gov/cgi-mach/machlog.pl?nb=cdf06&action=xe&page=26&time=14:13:59'> here </a> 

<!-- Comment: Sun Feb  4 14;39;08 comment by...Ignacio -->
Beam is <a href='http://www-bd.fnal.gov/cgi-mcr/elog.pl?nb=2007&action=xe&page=94&time=14:34:22' >  lost</a>. Jose discussing with Koji, Koji with MCR for an access to fix that FIB.

<!-- Date: Sun Feb 4 16:52:37 2007 -->
Entered in Silicon database: <ul>
<li>Ladder: IB5W4/5All 
<li>problem n: 1689 
<li>Status: needs_observation
<li>Type: Chip_capid
<li>Action: Swap_FIB
<li>entered by: mhartz 
<li>description is: It seems that the rate of pipeline out of sync errors actually increased with the new FIB, so we put the old one back in.<table border=1>
   <tr>
    <td>
     <b>IN:</b>
    </td>
    <td>
     <a href="http://b0sili01.fnal.gov/cgi-bin/fibdb/DBreport.pl?serial=107">107</a>
    </td>
    <td>
     <a href="http://b0sili01.fnal.gov/cgi-bin/fibdb/DBreport.pl?prep=550269">550269</a>
    </td>
   </tr>
   <tr>
    <td>
     <b>OUT:</b>
    </td>
    <td>
     <a href="http://b0sili01.fnal.gov/cgi-bin/fibdb/DBreport.pl?serial=122">122</a>
    </td>
    <td>
     <a href="http://b0sili01.fnal.gov/cgi-bin/fibdb/DBreport.pl?prep=560598">560598</a>
    </td>
   </tr>
  </table> 
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/pdb/DBreport.pl?detsys=ISL&wedge=B5W4/5&layer=All">report</a></ul>
<!-- Author: Problem DB -->



<!-- Date: Sun Feb 4 17:06:15 2007 -->
<h3> FIB re-swap </h3> 
The FIB we swapped in on Thursday had been tested in the teststand and after the swap using svxdaq. When the store came in on Saturday night, the FIB started to give problems of pipeline out of synch issuing an HRR each time. This problem appeared more than 20 times during the run 233500.  
<br> This afternoon the Tevatron lost an RF station and a bit later the beam was lost. We asked for an access to re-swap back the old FIB (122 → 107) that was given problems just occasionally.  
<br> Mark and Ignacio went in and put back in FIB 107 (tested again today in the teststand). Once the FIB was in it was tested using SvxDaq and looked fine. After that we took several cosmics runs (restarting the run) and the problem seemed not to be there (at least as bad as before). 
 
<!-- Author: Mark, Ignacio, Jose -->

<!-- Date: Sun Feb 4 17:06:42 2007 -->
Entered in Incident database: <ul>
<li>Submitted by: ignacio
<li>Description of Problem: We went in to re-swap fib board b0fib07 slot 11. Report of the access  can be found <a href='http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=338&time=17:06:15' > here</a>.
<li>See <a href="http://b0sili01.fnal.gov/cgi-bin/idb/DBentry.pl?incident_number=125">Incident Report</a></ul>
<!-- Author: Incident DB -->



<!-- Date: Sun Feb 4 17:30:12 2007 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10719&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=silicon&action=view&page=-10719  width='300' align='top'></a></td><td><h3> Another cooling step </h3> 
The flow went down from 1.93 lpm to 1.72 lpm  at  4:27 PM, while  we were taking cosmics. The preassures are 8.4 (DPT) and 11.9(P_COT). It will probably hit the warning limit. I will instruct the ace to page silicon cooling if it goes to 1.6 lpm.</td></tr></table>
<!-- Author: Ignacio/Jose -->
<!-- Comment: Sun Feb  4 21;55;35 comment by...Ulrich -->
Seems that there are ISL warnings on and off, partly because of the low flow, and partly because of the high pressure in other cooling lines (the system got out of balance due to the drop in flow). Let's hope we can ride it out over night and get the techs involved tomorrow.

<!-- Comment: Mon Feb  5 07;30;30 comment by...Ulrich -->
ISL is now in warning consistently. I'll try and adjust valve settings again. Keep fingers crossed that nothing trips on the way.

<!-- Comment: Mon Feb  5 07;37;26 comment by...Ulrich -->
I have opened the valve to 90%, and now the flow is back to 1.9 lpm, at a COT face pressure of 12.6-12.9 psi and a differential pressure of 9.1 psi. We are losing headroom...

<!-- Comment: Mon Feb  5 08;31;14 comment by...Ignacio/Jose -->
We switch off ISL and L00 for coooling investigations by Ulrich.


<!-- Date: Mon Feb 5 09:27:04 2007 -->
<h3>Summary of the week</h3> There was a little bit of everything:<br> 
<b>Monday</b>  
 
<li> 11 bias pinkies in coincidence with a prescale change. Looking back in history it seems it is common (meaning it has happend before)  a small step in bias when L1 tracking triggers kick in. I sistematically increase IMON limits of ladders at the limit from 100 to 150 mA. </li> 
 
<b>Tuesday</b> <li>    Store 5201 , record store of 278 E30 but we lost a few minutes of peak lumi at the beginning, that was a new record lumi of 278 E30. A couple of ISL ladders from f450 tripped on overvoltage and had to hockerize. I arrived 10 min late because scrapping takes now 10 minutes (instead of the usual 20-25 min). I asked to be called at the beginning of pbar loading.</li> 
<li> Self-inflicted <a href='http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=338&time=22:17:39' 
> ISL cooling trip</a>  managed to trip also SVX after a short while. This is the first time the flow in the compromised line  line hit the warning limit 1.7 lpm. Lumi loss 200 nb-1. f451 had to be exclude back. </li> 
<b>Wendsday</b> <li>  <a href='http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=338&time=09:03:27'> Autohocker</a> seems to be reacting on RC transitions, while it shouldn't. Keep observing this.</li> 
<li> Start update of OS in b0svx03, NVIDIA driver issues.</li> 
<li> Updates in the ErrorLogger  
<a href='http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=338&time=08:41:53'> code</a></li> 
<b>Thursday</b> <li> Resin exchange in both circuits triggered two <a href='http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=338&time=09:56:22'> trips</a>. Valves are back at default appertures. </li> 
<a href='http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=338&time=13:51:54'> </li> 
<li>Access</a> to exchange ISL FIB in b0sili07 slot 11, train future MPC and permanent light level measurements in hall and splitter rack.</li> 
<b>Friday</b> <li>  End upgrade of OS. Everything runnning where it should (keep new PSGui running). X server errors seems to be a red herring for a out of quota issue. Became clear in Saturday, we did a bit of <a href='http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=338&time=10:51:28'> cleaning</a>. </li> 
<li> Unclosed partition 
<a href='http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=338&time=09:56:00'> trick.</a></li> 
<li> VRB fan rack exchange. No effect on us. </li> 
<li>  Turbo HRR investigations <a href='http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=338&time=15:58:45'>trip </a>ISL when fiber carring TSI was taken out to compare signal to that in test stand. It turned out it was a HWDB issue that Bill had to solve. <a href='http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=338&time=17:32:33'>Now</a> the teststand is able to test TurboHRR implementations. </li> 
<b>Saturday</b> <li> Xserver/Quota. </li> 
<b>Sunday</b> <li>  Store 5208, small lumi of  140E30. Everything OK. (Old) TevMon was in funny state and had to restarted. New TeVMon worked fine. <ahref='http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=338&time=14:19:32'>RF </a>station switched off and 1/2 later the store was lost</li> 
 
<li> Fib was giving problems from the begeniing of the store but SVMon cleared after 165 events as it had happend iduring cosmics. The problem reappeared, SVMOn started issuing automatic HRRs at a rate of 1 per 1/2 hour. <a href='http://www-cdfonline.fnal.gov/cgi/internal/elog/det-system-elog.pl?nb=silicon&action=copyentry&page=338&time=17:06:15'>Access</a> to replace back old FIB. Hopefully we are back with only the old problems </li> 
 
<b>Monday</b> <li> Cooling investigations, ISL and L00 are prevently turned off. </li>
<!-- Author: Ignacio -->