<!-- Title: 17:22:36  Mon Nov 27 2006 -->
<!-- Crewchief: Willis+Peter -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>Week of 27 November 2006</pre>


<!-- Date: Mon Nov 27 17:33:43 2006 -->
<p>Store 5095 contines from yesterday. We were down about 40&#39; about 
   an hour into the store due to a combination of problems. It started with 
   the silicon CAEN SY527 crate 14 spontaneously switching off, followed 
   by problems in the silicon b0fib06 and b0svx02 crates. Running went 
   smoothly until about 1200 hrs. At that time, we had decided to do test 
   runs, but we were unable to run. This was tracked down to some issues 
   with the online <a href= 
'/cgi/elog/elog.pl?nb=2006&action=view&page=898&frame=2&anchor=16:03:14.275'> 
   file server</a>. It appears to be better after being rebooted, but not 
   perfectly so. We need to keep monitoring it. 
</p> 
<p>Store 5096 starts at 1705 hrs, with an initial luminosity of 189e30. 
   We started up without any problems. 
</p>  
<!-- Author: Willis+Peter -->


<!-- Date: Tue Nov 28 17:57:44 2006 -->
<p>Store 5096 continues to spin at 1750 hrs. We have recorded 5.6 pb-1 at 
   88% efficiency. Running on this store has been relatively smooth. This 
   store is to be kept spinning until 0800 hrs, 29Nov06. This is much 
   longer than usual because the pbars in the Recycler had to be dumped 
   this morning, due to "smoke" at <a href= 
'http://www-bd.fnal.gov/cgi-mcr/elog.pl?nb=2006&action=view&page=898&anchor=125353'>MI-20</a>. 
</p>  
<!-- Author: Willis+Peter -->


<!-- Date: Wed Nov 29 17:38:30 2006 -->
<p>Store 5096 spun until 0805 hrs, when it was normally terminated. We 
   recorded 6.7 pb-1 at 88% efficiency. This was a good store with fairly 
   smooth running. During the extended low luminosity running, we 
   interspersed silicon d-mode calibrations, CSL, and XFT tests. At 
   1319 hrs, 29Nov06, the COT inner two super layer dE/dx was turned off. 
   This is now peramanent. 
</p> 
<p>Store 5097 starts at 1049 hrs, with an initial luminosity of 172e30. 
   We have been running fairly smoothly from the start. At about 1242 hrs, 
   there was a power glitch on the ComEd power lines. We did not suffer 
   any ill effects, but it appears the TeV RF TRF4 went off a few minutes 
   later, and this induced a luminosity drop of about 10e30. At 1737 hrs, 
   we have recorded 2.3 pb-1 at 86% efficiency. 
</p>  
<!-- Author: Willis+Peter -->


<!-- Date: Thu Nov 30 18:43:27 2006 -->
<p>Store 5097 continued to spin until 1000 hrs, when it was normally 
   teminated. We recorded 4.8 pb-1 at 90% efficiency. This was a smooth 
   store, with only one run. D0 went in on a 1 hr controlled access to 
   replace a bad calorimeter power supply. We stayed out. 
   Store 5099 starts at 1430 hrs, with an initial luminosity of 184e30. 
   We started running quickly and smoothly. However, about 1.5 hrs into 
   the store, a TDC in b0cot16 failed, giving DTOs. Power cycling the crate 
   did not help, so the TDCs had to be taken out the fast readout (spymon) 
   mode. It was an hour before we started running again. 
</p>  
<!-- Author: Willis+Peter -->


<!-- Date: Fri Dec 1 17:40:50 2006 -->
<p>Store 5099 continues to spin at 1730 hrs. We have been running smoothly 
   since yesterday&#39;s COT TDC problem. There was a 0.5 hr down time 
   at 1415 hrs due to what appears to have been memory corruption in a 
   TF (track finder) board in the b0svt06 (displaced silicon vertex 
   trigger) crate. CSL end of store commissioning tests began about 
   1500 hrs. These commissioning tests are system robustness 
   tests to failure modes. The data taken is still good, and  
   only the efficiency is reduced. 
     
  The termination of store 5099 has been postponed from 1600 hrs 
   when the Linac 750 keV buncher tripped off at 1500 hrs. As of 1735 hrs, 
   we have recorded 5.4 pb-1 at 84% efficiency for store 5099. 
</p>  
<!-- Author: Willis+Peter -->


<!-- Date: Sat Dec 2 19:08:11 2006 -->
<p>Store 5099 is terminated normally at 1909 hrs, 01Dec06. We recorded    5.4 pb-1 at 83% efficiency. Store 5100 begins at 2116 hrs, 01Dec06, with    an initial luminosity of 175e30. At about 2229 hrs, 01Dec06, the T:RF8    RF station tripped, but it did not affect running at that time. At 0107 hrs, 
   b0pcal02&#39;s 5V supply tripped, and we were down about 0.5 hrs because 
   of that. The shift crew had to be guided through a crate power cycling 
   and reminded that b0pcal02 has SMXR cards. Store 5100 is lost at 0414 hrs: 
   a T:QDE2 (power supply?) faulted and aborted the beam. We had no effects 
   from this abort. 
</p> 
<p>There was no beam for all of the day shift. There was a 1 hr D0 access, 
   work on T:RF8, and the investigation of the T:QDE2 abort. The  T:QDE2 
   problem was a bad power supply in a saftety system ESS interface 
   chassis at E3. Store 5013 starts at 1859 hrs, with an initial luminosity 
   of 104e30. We appear to be running okay. 
</p>  
<!-- Author: Willis+Peter -->


<!-- Date: Sun Dec 3 21:20:16 2006 -->
<p>Store 5103 continued from yesterday but was lost on an abort 
   at 1143 hrs. The first indication was that the TeV C0 shunt  
   pulled the abort. but in the end the HWL at B3 a quench 
   protection circuit was given the blame. We had our own problems 
   during this store as well. Our solenoid quenched and fast 
   dumped about 0100 hrs due to some problem in the Cryo area 
   <a href= 
'/cgi/elog/det-system-elog.pl?nb=cryo-gas&action=view&page=2128&frame=2&anchor=03:11:11'> 
   UPS system</a>. We were back up at 0526 hrs. We recorded  
   2.2 pb-1 at 79% efficiency. 
</p> 
<p>Store 5105 starts at 2057 hrs, with an initial luminosity 
   of 184e30. All systems are up, and we seem to running okay. 
</p>
<!-- Author: Willis+Peter -->


<!-- Date: Sun Dec 17 20:12:28 2006 -->
<pre>The following is the downtime summary for the week of Nov. 27 
 
The past week (starting with store 5095 and ending with store 5105 @L=80) 
we were ~85% efficient taking 26.1 pb-1 of 30.1 pb-1 delivered. 
 
Here are the top ten culprits and lumi lost as far as I can gather: 
(of course, lost lumi depends on the length of the downtime AND the 
instantaneous lumi during the downtime ...) 
 
0) Deadtime                                      1890 nb-1 
 
1) Solenoid trip (those darn UPSs)                395 nb-1 
 
2) TDCs in COT16 (never understood)               321 nb-1 
 
3) Plug HV: PCAL02 crate supply trip              145 nb-1 
            PCAL PC and Mon crashes               146 nb-1 
 
4) CSL testing (stop and start runs)              278 nb-1      
   (maybe we need a better/more realistic CSL torture ...) 
 
5) SVX HV : CAEN switch off (and RC hangs)        200 nb-1 
            Trips                                  55 nb-1 
 
6) XFT hardware   -&gt; new wire list 
                  -&gt; new run -&gt; DAQ problems      185 nb-1 
 
7) NFS problems                                   120 nb-1 
                                       
8) CMP trip (solution = more fans?)                67 nb-1 
 
9) Shepherding PCAL crates (what is up??)          60 nb-1 
   (usually they DTO, stick in HALT, and must be shepherded twice!) 
  
 
Observation: No more SVT09 problems since the crate was abandoned! 
(Will there be a Post Mortem ...?) 
 
One more observation - htdc00,imu00,imu01 crates have a lot more 
Done Timeouts than all the other TDC crates combined. 
Is this where the lesser TDCs end up or is there some other problem(s) 
here ... can we afford to abandon 3 more crates?   :) 
</pre>
<!-- Author: Greg -->