<!-- Title: 14:00:11  Tue May 11 2004 -->
<!-- Crewchief: RR -->
<!-- Op1: &nbsp; -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<pre>CDF Operations for the week of May 11th</pre>


<!-- Date: Tue May 11 14:01:12 2004 -->
<pre>Sorry for the tardy inputs....  I was out west trying to do a 
little fly fishing in rivers still better suited to white 
water kayaking....</pre>
<!-- Author: RR -->


<!-- Date: Tue May 11 14:02:55 2004 -->
<pre>Status  May  7 - 10 
 
COT gas recirculation is still in a testing mode. Pumping gas through a  
dummy volume has shown that the system has a wide stability range.  
However, time ran out on Friday before a test with the actual COT could  
be performed. Tests with the COT on the recirculation system should  
begin Tuesday, with full integration expected by Wednesday. When  
integrated, the COT will be operated with all super-layers at full voltage. 
 
Friday, Store 3475 Average Luminosity 46.8 E30 
Saturday, Store 3477 Average Luminosity 57.1 E30 
B0 Luminosity is bigger by 7  9 % than D0 Luminosity 
 
Early Friday morning the primary database computer reports errors in  
writing to its disks. A disk failure is confirmed in the afternoon which  
causes a halt to data taking. By late evening, the failed disk is  
replaced by a duplicate of its mirrored disk, however, one file and its  
backups are corrupted. Early Saturday morning  the file is regenerated  
from scratch, however Level 3 calibration files are not recovered  
properly. Abnormal files are then removed. Data taking resumes at the  
end of the Owl shift Saturday. 
 
It took the combined efforts of 12 experts, 17 hours to return to data  
taking, though only 9 hours of collisions were lost. In the event of a  
primary disk failure, a mirrored disk does not appear to guarantee a  
quick return to data taking. For more details, see Bill Badgetts E-log  
entry of Saturday May 8, 07:38:16. 
 
</pre>
<!-- Author: Carl -->


<!-- Date: Tue May 11 18:39:20 2004 -->
<pre>COT Gas Recirculation Status 
 
The engineers had a very successful day.  For the first 
time, we were able to recirculate gas to the COT.  The system 
worked very well.  We were able to recirculate at 160 scfh. 
Add that to the 40 scfh of fresh gas and you get up to the  
desired 200 scfh design goal for this phase of the project. 
 
There is still some work to do and a few things to understand. 
We did see the oxygen go up in our exhaust line.  Normally we 
measure 11ppm o2.  During the recirculation test, that went up  
to 51ppm.  That by itself is not a problem as long as it does  
not keep going up.  The interesting note is that within a few 
minutes of turning off the fast flow, the 02 ppm dropped to 
its original value as if the 02 never entered the chamber 
but just the exhaust header (which is not under vacuum). 
 
Tomorrow we need to add a button to allow an operator to switch 
from fast flow to single pass flow with a button -- to be 
used if there is some stability problem.  The program may also 
execute this sequence if it identifies a problem.   Once this 
is done, we will be ready to flow 24x7.  </pre>
<!-- Author: RR -->


<!-- Date: Tue May 11 18:41:25 2004 -->
<table><tr><td><a href=/cgi/elog/det-system-elog.pl?nb=operations&action=view&page=-1&button=yes target='xxx' onclick='jswindow()'; align=top><img src=/cgi/elog/det-system-elog.pl?nb=operations&action=view&page=-2  width='300' align='top'></a></td><td>IFIX display while flowing gas in fast recirculation 
mode into the COT during a two hour test today.  Looks  
GOOD!</td></tr></table>
<!-- Author: RR -->


<!-- Date: Wed May 12 11:16:32 2004 -->
<pre>Status -- Wednesday May 12th 
 
We had reasonably smooth running on the evening and owl shifts 
taking data.  We had problems with L3 that caused some time 
and a problem with PCAL02 as well.  At low luminosity, Chris 
Neu spent was allowed to work on the SVT system -- we are trying 
to provide AD with an on-line B* measurement at CDF.   
This was the first time he tested his new code with the beam. 
The results looked promising but need to be validated off-line. 
IF good, we still need to write the code to get the info 
into ACNET where it can be used to tune on. 
 
Store was dropped at 5:30am.  Cryo did some wet engine work 
on the tevatron.  Currently doing tev studies.  The goal is to 
install the new low beta optics -- an attempt to optimize the 
lum at both collision points. 
 
CDF is taking opportunity of this downtime to fix the DB  
computing hardware.  We are also rebooting part of the on 
line system and the engineers are continuing to work on COT 
gas recirculation. 
 
We expect shot setup sometime on the evening shift.  The stack 
should be at 200ma by then.  Not sure how agressive they will be 
with pbars given the new optics.  Nothing has been decided yet.</pre>
<!-- Author: RR -->


<!-- Date: Wed May 12 20:23:47 2004 -->
<pre>First shot went in with new low beta optics to try to improve 
luminosity at both collision halls.  They shot from a stack of  
180ma.  They using about 140ma of the stack.  They ended up  
using about 8400e9 protons at low beta and 1020e9 pbars.  
The initial luminosity was 50.4 at CDF and 46 at d0.  In 
other words, still about a 10% discrepancy between the  
two halls. 
 
Further poking around and I see large "scallops" in the  
horizontal and vertical (less) emittances for both protons 
and pbars.  Dampers?!  Given this, I don't think we can say 
much about whether the new optics were an improvement or not. 
We will have to wait for the pro's to do the analysis. 
Losses were pretty good this store. 
 
We had a little rocky startup but are taking data fine now! </pre>
<!-- Author: RR -->


<!-- Date: Thu May 13 16:50:09 2004 -->
<pre>Status -- Thursday, May 13th 
 
We took data on the last shot quite smoothly.  Efficiency  
was about 85%.  Most of the problems can be attributed to 
poor beam conditions where we had to wait for tuning or 
rescraping.   
 
COT work was completed today for recirculation.  Final sign-off 
is tomorrow morning.  Then we will start recirculation 24x7  
after one remaining test.  Once we start the pump, we will turn 
the COT HV all on full, change the XFT trigger masks back 
to single miss, and switch trigger tables to PHYSICS_2_05_v1. 
We will move forward from there. 
 
The previous store ended at 8am due to a failure of a TeV 
power supply.  We have been in studies all day looking at  
emittance issues and damper checkout.  We expect shot  
setup to begin within the hour.  Current stack is 175ma 
</pre>
<!-- Author: RR -->


<!-- Date: Fri May 14 12:15:41 2004 -->
<pre>STatus -- Friday, May 14th 
 
The TeV quenched last night at the end of studies while 
trying to abort the beam.  Nominal blame was too much  
in the abort gap.  this is a bit surprising since it was 
only 23khz.  The abort was not clean.  CDF did see ~0.04rad/sec 
at the time of the abort.  Not alot but &gt;0 and clearly timed 
with the abort firing. 
 
The next shot went in around 10pm last night with an initial 
luminosity of 53e30.  D0 was about 10% lower.  In spite of 
all the studies, there was still an emittance blowup for the  
pbars at 980.  This is not understood.  Changes will be  
implemented for next store to try to correct this.  The other 
numbers for the store look good -- indicating that if this 
problem is solved, we (and D0) should each expect a 10%  
improvement in lum for a given set of input protons and pbars! 
Kudos to the AD for this work! 
 
CDF had a rough startup at the beginning of the store. 
There was a problem with Level3. Converter 10 and Scanner 
Manager were hanging.  It was necessary to reboot them 
and clean up Level3 and EVB to fix the problem.  The same  
thing happened yesterday but with a different converter node. 
We don't understand this.  We believe SCRAMM net is involved. 
We will physically clean the fibers to see if this helps.   
We also had problems with our ACNET interface and a few other 
issues all at startup.  Since then, we have been running quite 
well. 
 
Plan for weekend is stack and store.  Next week will be the 
same -- probably with 2-3 shifts of tev studies similar to this 
week.  The TeV delivered 10.6 pb this week (friday to friday). 
Excellent.  They need to average 9.5 pb delivered each week 
between now and August 23rd to stay on the "red" curve. This  
seems very tough unless we can improve pbar stacking. 
 
June 1st is the target date for mixed mode operation of the  
pbar complex.  The hope is by then, the recycler will be used 
to store a fraction of the pbars used in each shot -- say 
20-30%.  Some of the buckets in a shot will be filled from 
the recycler and the rest from the accumulator.  This does 
two things -- it forces the recycler folks into an operations 
mode faster and better structures their work.  It should also 
allow us to shoot from slightly higher total stacks because 
we can offload some pbars to recycler and thus take advantage 
of that cooling. 
 
COT recirculation will begin this afternoon in ernest! 
 
</pre>
<!-- Author: RR -->


<!-- Date: Fri May 14 16:04:53 2004 -->
<table border=0><td><tr><td bgcolor=#e0e030><pre>COT STATUS!!! 
 
COT is now RECIRCULATING!!!  XFT masks has been changed back 
to nominal single miss.  COT HV for ALL superlayers back to  
nominal.  Now running a trigger table appropriate for the 
full experiment -- namely physics_2_05[1,475,455] 
 
The run number is 182629 
 
The plan is to stay this way for some number of stores while 
we evaluate the impact on aging.  Stay tuned here for details 
as we gain experience with it. 
 
THANKS to Del Allspach, Bob Sanders, and Rich Schmidt, and  
Warren Bowman for this system 
 
</pre></td></tr></td></table>
<!-- Author: RR -->


<!-- Date: Sat May 15 11:11:16 2004 -->
<pre>Status on Saturday, May 15th 
 
Ran well!  COT was integrated at 4pm with all HV on.  WE took data throughout the night without problems at about 95% effic. 
The store was dropped at 8am this morning.  As soon as 
the store went down, we had a problem with CANWT2 VME power  
supply.  Tried power cycling etc to no avail.  Requested an 
emergency 2 hour access and was granted it by MCR (thanks!) 
WE swapped in a new power supply and all is well.  We are 
in good shape for the next store now. 
 
We had to make one adjustment this morning in the COT  
recirculation pump.  The oxygen level in the COT looks to be  
stable around 135ppm.  We will keep a close eye on the pump 
to see how much TLC it needs to run with high efficiency. 
 
</pre>
<!-- Author: RR -->
<!-- Comment: Sat May 15 11;12;08 comment by...RR -->
<pre>Thanks to Dervin and Lew for coming in and helping me
swap that thing!</pre>


<!-- Date: Sun May 16 08:46:10 2004 -->
<pre>SUNDAY, May 16th 
 
 
The Tevatron shot went in on Saturday afternoon.  ITs initial 
luminosity was disappointing given the input protons and 
pbars.  The losses were also quite high.  The initial lum 
was 54e30.  About 25 minutes into the store, there was a  
massive quench.  6 houses quenched as well as both sets of 
low beta's.  CDF pulled the rad abort and we saw 140rad on 
our silicon.   Investigation of the silicon is still ongoing 
as to permanent damage. 
 
As the time of this writing, MCR claims that after an extensive 
investigation, no cause of the quench could be identified.  MCR 
has decided to put another shot in from a 100ma stack.  That 
happened this morning with again loss problems at the start of 
the store.  I objected to this store -- rather I would prefer to see more effort expended on understanding the quench.   
 
The silicon detector is currently off.  I have asked for a  
meeting with the lab people today to figure out how to move 
forward.  I don't want to stay off -- but its prudent given the 
poor documentation of the quench in the elog's and the severity 
of the incident and the current lack of understanding. 
 
</pre>
<!-- Author: RR -->
<!-- Comment: Sun May 16 10;05;13 comment by...RR -->
<pre>Just spoke with Jeff Appel on the phone.  HE will help with
getting the follow-up done on the quench.  Hope to have a 
meeting this afternoon with appropriate parties.  Will keep
people informed.</pre>


<!-- Date: Sun May 16 11:46:42 2004 -->
<pre>The Sunday from hell continues.... 
 
The oxygen level in the cOT took a substantial jump at 1030 
this am.  This means we have a leak on the low pressure 
side of the pump.  I don't understand it yet.  Its not at the 
chamber face because the baggie parameters look fine.  Switched 
back to single pass mode.  Engineers en route -- we will start 
leak checking here to see whats up 
 
At the same time as above -- the tev lost the store.  D0  
vacuum valves closed.  Store lost.  Stack at 60ma at the moment. 
 
Spoke with Jerry Annala about yesturdays quench investigation.  The problem of the quench was an abrupt horizontal change of the beam orbit.  Not sure why.  Looks like a corrector element 
tripped off.   HE still wants to go through the snapshot manager 
to look at further details but nothing else obvious about the 
trigger of this event.  The corrector elements have historically 
been masked off from pulling the abort.  At 10am on Sunday, 
Jerry had them unmasked so if one trips again, it will pull the 
abort and hopefully not cause the kind of havoc seen yesturday. 
We have been trying to get this for a while -- so this is a good 
thing.  It remains to be seen how many false aborts are caused 
by this step. 
 
PLAN FOR NEXT STORE 
 
Still under construction.  Depends alot on the COT status.  If 
we can get the problem fixed and turn the COT on, then we will 
leave silicon off for the first hour.  If beam conditions allow after the first hour, we will go back to full running.... 
 
 
 
</pre>
<!-- Author: RR -->


<!-- Date: Sun May 16 18:16:01 2004 -->
<pre>Spent the afternoon looking at the COT problem.  Bottom line is  
there is a leak in the gas system.  We did a number of tests 
and are pretty confident the leak is on the return side  
of the plumbing -- most likely the west side donut.  The  
oxygen level is still coming down.  ITs now around 400 ppm. 
 
We did a procedure 13 which looks at the integrity of the COT 
plumbing.  What we measure is the smallest amout of gas flow 
necessary in order to get some return flow.  Normally, that value is about 5scfh.  We measured it today to be about 9scfh. 
 
Since we know we have a point leak somewhere, our procedures 
tell us that we have to leave the chamber off.  So, we will 
do that.  I will probably do another leak rate to make sure 
I did not make a mistake. 
 
If you want all the gory details, you can look at today's  
cryo log.  
 
http://www-cdfonline.fnal.gov/cgi-bin/det-system-elog.pl?nb=cryo-gas 
 
We will turn the silicon on for a few hours toward the end 
of the store to complete the checkout from Saturday's incident. 
 
We will be requesting a lengthy access tomorrow to try to make 
repairs.</pre>
<!-- Author: RR -->