Analysis of Tevatron 16 House Quench on December 5, 2003

D. Still, Fermi National Accelerator Laboratory\*, Batavia, IL, 60510
USA

# Introduction {#introduction .INTRODUCTION}

On December 5, 2003 at 10:35:41 the Tevatron suffered a 16 house quench
during the beginning of a Tevatron study period. In addition to the 16
house quench, the D49 collimator target and E03 1.5m collimator were
damaged by proton beam Figure \## & ##. The C19 spool also suffered
damage at the #1 pin of the conning tower as a result of larger
cryogenic pressures induced from the quench Figure ##. The initial cause
of the large quench was due to the CDF Roman Pot 3 reinserting itself
back into the beam after it had been issued 2 retract commands from the
Tevatron sequencer (C48). The pot has a control problem that if the
limit switch is "hit" the pot frequently will spontaneously run itself
back into the beam. This is exactly the scenario that initiated the 16
house quench on December 5, 2003. A detailed analysis of the quench,
explanation for damaged components and recommendations for changes are
discussed.

# Detailed understanding of quench 

## Sequence of Events that Lead to Quench

1\) When the pots were retracted, it appeared that not all of them had
moved. Subsequent inspection showed that Pot #1 had an LVDT that
occasionally would stick when moving in the OUT direction. It cannot be
verified but it is suspected that this LVDT incorrectly reported that
Pot #1 was not fully retracted, causing the file to be manually sent
again in an attempt to move it.

2\) Sending the retract file a second time caused the pots to move an
additional distance past the desired park position until they contacted
their limit switches. Pot #3 then entered its failure mode and ran back
inward at high speed. (Later recreation of the failure suggests a
velocity exceeding 1200mils/second or .03mm/msec).

3\) Pot #3 encountered Proton beam halo first on the way into the even
though Pot #3 comes the pbar side of the beam. Our only explanation for
this is that the proton halo is bigger than the pbar beam plus halo at
A48. From past history CDF and Pot colleagues have complained of proton
halo swapping the pot detector from the pbar side. Evidence for pot #3
creating proton loss first can be deduced from the T44 loss plot
Figure 1. D49 losses start to increase first due to the fact that this
the location of the D49 proton collimator target which is the closest
horizontal and vertical aperture to the beam. There are no F48 losses
which would be consistence with pbars losses on the F49 pbar collimator
target. This would be the closest aperture to the pbar beam.

![](media/image1.jpeg){width="3.064583333333333in"
height="2.466666666666667in"}

**FIGURE 1.** T44 loss buffer plot of losses at various locations during
the 16 house quench.

4\) The A48U cell quenches immediately roughly 16-14 msec before the
Tevatron abort fired. This can be seen on Figure 2 which is a plot A4,
D4, E1 and F1 QPM over sample buffer. The instant A48U quenches the 5
main bus dipoles in that quenching cell begin to loose current and field
at a conservative estimated rate of

100Volts = L(DI/Dt)

where L = .232 H

DI/Dt = 500Amps/sec or **.5Amps/msec**.

5\) Once the current began to decay in these 5 dipole magnets in the
A48U cell Figure 3, the beam orbit began to change around the ring.
Calculations in Table 1 and 2 estimates that the horizontal orbit at D49
would begin to move toward the D49 horizontal collimator target edge at
a rate of -.36mm/msec or **-3.92mm in 9 msec**. The estimate is that the
proton beam would be approximately 3mm from the target. Locally at A48
the beam would be moving to the radial outside away from the incoming
pot 3.

![](media/image2.jpeg){width="3.069291338582677in"
height="2.6070866141732285in"}

**FIGURE 2.** Voltage plots from QPM's over sample buffer depicting
quenches at A4, D4, E1 and F1. These quenches are very fast and quench a
large portion of super conduction bus. A48U quenches approximately
10-12msecs before Tevatron abort loop is pulled. (Courtesy of D. Wolff
and EE Support)

6\) At the point when the primary proton beam reaches the D49 tungsten
target edge, N. Mokhov estimates that in \~ 20-30 turns the beam would
drill a hole in the 5mm tungsten wing and continue to circulate beam
once a hole was created. Damage of the D49 target is shown in Figure 3.

7)Once the hole in the D49 target was created, proton beam could travel
to the next limiting horizontal aperture which would be the E03 1.5
collimator. The beam then etched a horizontal line in the stainless
steel collimator horizontal surface Figure 7.

8\) The abort system then fired the A0 abort kickers some 12-16 msec
after the quench of A48U at which time there is no evidence that any
beam; proton or pbar went to the abort blocks at A0.

![](media/image3.jpeg){width="3.0381889763779526in"
height="1.441732283464567in"}

**FIGURE 3.** Drawing of the A4 Tevatron bus at the quenching cell A48U.
Due to 5 dipoles quenching very fast the current is these dipoles is
decreasing conservatively at 500Amps/sec creating a distortion in the
Tevatron horizontal orbit.

9\) The resulting quenches due to beam loss around the ring system then
fired the A0 abort kickers some 12-16 msec after the quench of A48U at
which time there is no evidence that any beam; proton or pbar went to
the abort blocks at A0

## Questions to Answer

1)  One question to answer is why there is proton halo bigger than the
    pbar beam and halo to the horizontal inside at A48. This may easily
    be explainable but there is some uncertainty at this point.

## 

## Common Quench Data

Table 1 provides conditions surrounding the quench. The houses that
quenched during this event were A2, A4, B1, B2, B4, C1 ,C4 , D1, D4, E1
,E2 ,E3 ,E4 ,F1 and lowbeta houses B0 and D0. Figure 4 provides a
Tevatron ring wide loss profile from the beginning of the quench to the
end. Table 2 provides the T67 abort input timestamps for events.

  ------------------------------ ----------------------------------------
  **Table 3. Common facts about  
  16 house quench**              

  Proton Intensity               6600 E9

  Pbar Intensity                 321 E9

  Store number                   3083

  Time of Quench                 10:35:41 Dec 5, 2003

  Energy                         980gev

  Lattice                        Lowbeta at Collisions
  ------------------------------ ----------------------------------------

![](media/image4.png){width="1.3895833333333334in"
height="1.1604166666666667in"}

![](media/image5.png){width="1.4118055555555555in"
height="1.1791666666666667in"}![](media/image6.png){width="1.5347222222222223in"
height="1.2819444444444446in"}

**Figure 4.** Progression of Tevatron ring wide losses during the
quench. These are frames 21, 14 and 10 respectively. 1 frame is 2 msec.
Frame 21 is before any loss.

  ------------------------ -------------------- --------------------------
  T67 abort event          time stamp (secs)    delta time from first
                                                event (msec)

  b0 blmba                 51.376               0

  d1 blm                   51.376               0

  f4 blm                   51.379               3

  b2 blm                   51.379               3

  b4 blm                   51.379               3

  c1 blm                   51.379               3

  d0 blm                   51.379               3

  d4 dipole                51.38                4

  e2 blm                   51.38                4

  e3 blm                   51.38                4

  c1 blm                   51.38                4

  e1 dipole                51.381               5

  a2 blm                   51.381               5

  a4 dipole                51.381               5

  b0 dipole                51.381               5

  d0 qpm                   51.384               8

  d4 qpm                   51.393               17

  e1 qpm                   51.393               17

  f1 qpm                   51.393               17

  a4 qpm                   51.393               17

  b4 qpm                   51.393               17

  c4 qpm                   51.393               17

  b0 qpm                   51.394               18

  b1 qpm                   51.394               18

  d0q2                     51.394               18

  a2 tecar                 51.402               26

  e1 qfe1                  51.403               27

  e2 qde2                  51.403               27

  f4 qdf4                  51.403               27

  a4 qfa4                  51.403               27

  b2 dipole                51.403               27

  b0q2                     51.404               28

  b0q3                     51.404               28

  b0q5                     51.404               28

  e2 qpm                   51.41                34

  e3 qpm                   51.41                34

  a2 qpm                   51.41                34

  b2 qpm                   51.41                34

  c1 qpm                   51.41                34

  d1 qpm                   51.41                34

  d0q3                     51.411               35

  b4 dipole                51.412               36

  d1 dipole                51.432               56

  f1 dipole                51.433               57

  b1 dipole                51.437               61

  e3 dipole                51.454               78

  e2 dipole                51.456               80

  e4 qpm                   51.46                84
  ------------------------ -------------------- --------------------------

# Damage to Accelerator Components

## D49 Collimator Target Damage

The D49 collimator target is used as the primary beam scatter for
conducting halo removal in the Tevatron. In order to provide reduction
of halo losses at CDF and D0 detectors during a store this collimator is
typically placed 5σ from the beam. It will be the closest device next to
the beam horizontally and vertically during a store. The target has a
5mm tungsten wing that halo goes through in order to cause scattering of
the halo. In this case, the tungsten wing was what was damaged (Figure
##). The wing is designed to replaced by unscrewing to bolts and replace
with a new tungsten wing. Efforts were made to replace this wing but the
wing developed a crack during installation and the unused target at D171
had to be removed from the beam line, replaced with a straight section
of pipe and replaced at D49. The Tevatron will run with out the D171
target. Also 12 newly machined tungsten wings have been manufactured as
replacements for future use.

![](media/image7.wmf){width="1.7897637795275592in" height="1.9in"}

**Figure 5.** Tevatron collimator halo removal system. Red collimators
indicate the primary proton set that were damaged.

![](media/image8.jpeg){width="3.066666666666667in"
height="2.3in"}![](media/image9.jpeg){width="3.066666666666667in"
height="2.3in"}

**Figure 6.** Damage to D49 collimator target during the 16 house
quench. This is a tungsten wing used to scatter protons as part of a 2
stage collimator system for halo removal for CDF and D0 experiments.
Tungsten melting point is 3400 C. Size of exit hole at maximum is 2.8mm.
Size of Tungsten target is 5mm. This hole was created in 10-20 turns of
980gev proton beam after being mis-steered at A48 due to quenching
magnets.

## E03 1.5m Collimator Damage

![](media/image10.jpeg){width="3.0965277777777778in"
height="3.377083333333333in"}

**Figure 7.** Damage to E03 1.5m collimator during the 16 house quench.
This is a secondary collimator downstream of the D49 target the catches
halo from the target. After 10-20 turns there was a hole drilled in the
D49 target allowing proton beam to continue to next aperture restriction
which is the E0e collimator. The length of damage is \~ 5-10 inches long
at 1-2mm deep.

## C19 Spool Conning Tower Damage

Damage to the C19 spool piece mainly involved a correction element feed
through called the conning tower. This is where the warm correction
element bus connects to the cold spool. The style that failed during the
quench is of the "old" style and probably was already loose at the time
of the quench. Large pressures induced by the quench resulted in 1 pin
of 12 being pushed out of the conning tower. A couple of efforts to
repair the pin while at 80K failed requiring the C1 house to be warmed
up to room temperature to replace the conning tower.

![](media/image11.jpeg){width="3.066666666666667in"
height="2.3in"}![](media/image12.jpeg){width="3.0555555555555554in"
height="2.2916666666666665in"}

**Figure 8.** Picture of a helium leak on the C19 spool conning tower.
The #1 pin and later more pins are leaking helium. After many efforts to
repair the conning tower it was decided to warm up the C1 house and
replace the C19 spool.

# CDF Roman pot details

The B0 pots operate on a legacy motion control system originally used in
a past incarnation of these detectors. Minimal documentation existed to
allow understanding of the internal workings of the motor controllers.
There had occasionally been problems with Pot #3 being found near its IN
limit switch during shot setup (the sequencer checks to make sure they
are out of the aperture before allowing beam injection.). On one
occasion on 3/17/03, the pots were retracted for a study before beam was
aborted and high losses were observed soon after. It was determined that
pot #3 had run all the way to the IN limit as soon as it reached the OUT
limit. The interface card was ultimately changed and the problem could
no longer be intentionally reproduced, at least for some weeks. Pot #3
would still infrequently be found to have gone back in after store
termination, and as a stopgap measure, the sequencer retract file was
modified to park the pots at positions which were out of the aperture
but short of the limit switches, since all observed instances of
misbehavior appeared to be related to contacting the OUT limit switch.
This change was made during the Fall shutdown and appeared to function
as expected for two weeks prior to the quench.

## Pot 3 Motion Test Following the Quench

Sequencer retract file was sent to retract the pot, and was then sent
again after the pot had stopped. This should have closely duplicated the
command sequence that occurred just prior to the quench. The pot
continued to the outer limit switch and then rapidly ran back in. This
failure could not be duplicated again to get a better idea of the speed
or final position the pot attained, but from the plot it appears that
the pot traversed an inch in under three seconds. (IN= +)

## Pot 3 Damage

There was no damage viewed on the Pot 3 upon local investigation after
the quench. Figures \## and \## are pictures of the Pot before the
incident but would look similar at the time of investigation. There are
still questions surrounding how far pot 3 went into the beam but there
is much evidence and high confidence that pot interacted with proton
beam halo and did not interact with core proton beam.

![](media/image13.jpeg){width="3.057638888888889in"
height="2.2930555555555556in"}![](media/image14.jpeg){width="3.057638888888889in"
height="2.2930555555555556in"}

**Figure 9.** Picture of CDF Roman pot before Dec 5,2003. Top picture is
detector that is inserted into beam; lower plot is housing that is
perpendicular to beam pipe. The pot is inserted horizontally from the
radial inside on the antiproton side of the helix. (Courtesy of CDF )

![](media/image15.png){width="3.066666666666667in"
height="2.452777777777778in"}

**Figure 10.** This is a plot of pot 3 upon recreation of the failure
mode at the time the pot is driven back into the beam. This is on a fast
time scale in order to estimate the speed driven back in. The speed is
1200 mils/sec

![](media/image16.png){width="3.066666666666667in"
height="2.452777777777778in"}

**Figure 11.** This is a plot of the motion of pot 3 in an attempt to
recreate the failure of retracting the pot and then it inserting itself
back into the beam.

# Recommendation for changes

1\) T44 and T67 ACNET applications have already been modified in order
to provide better analysis tools to understand this type of event more
clearly and accurately.

2\) There were some initial thoughts that the dipole correction elements
in the spools at D4, A4 and E1 were the first devices to quench causing
the corrector outputs to begin to decay and move the orbits globally
until the abort kickers would fire. Initial concerns centered on the
fact that at collisions the dipole correctors are masked from pulling
the abort on T67. This would mean that more time was given to orbit
decay from dipoles tripping off and potentially causing more problems
with beam loss, likely a quench , until the A0 abort kicker could fire.
In this case, the unmasking of the dipole correction elements from the
abort would not have helped since the quench at A48U developed so fast
and involved a good portion of quenching superconductor changing the
current in 5 main bus dipoles not correction elements. Discussions have
taken place with in the Tevatron Department as to whether this would be
better practice in the future of unmasking the dipole correction
elements from the abort but no conclusion has been reached.

3\) Having the QPM'S respond to faster quench from the over sample
buffer could be an option but this would have to be discussed and
implemented with the EE support department.

4\) Better loss monitor protection could be another option for aborting
fast quenches. Currently during stores the loss monitors are masked from
the abort system due to the fact that there is only high field loss
limit. It is know for sure that it would be impossible to ramp with
these loss limits but a more complicated system could be proposed
incorporating loss limits for different Tevatron states. This would need
more planning and effort.

# Conclusions

The CDF pot 3 reinserted itself back into the beam after it had been
given commands to retract. This initiated large proton and pbar beam
losses at A48. These losses produced a fast quench involving many
superconducting magnets at the cell A48U. These magnets loosing field
produced a horizontal orbit change at the D49 collimator target setting
up conditions to damage this collimator and its corresponding secondary
collimator E03 with beam. It is unfortunate that there was damage to the
2 collimators requiring replacement but on the other hand the
collimators took the damage that potentially could have damaged another
accelerator component.

# acknowledments

The data, analysis and conclusions presented in this report come from
many people and departments in order to carefully and accurately
describe the events of the 16 house quench on December 5, 2003. I would
like to thank and acknowledge all those who contributed. The list is of
personnel is long but these are the main contributors: J. Annala, B.
Hanna, T. Johnson, D. Wolff, B. Flora, N. Mohkov, S. Drozhdin, B.
Hendricks, CDF Roman Pot personnel and Mechanical support.