"Ring-of-Welding" and Calorimeter Noise Characteristics

# D0Note 4286 draft

##### October 31, 2003

## H. T. Diehl,^\[1\]^ M. Arov,^\[2\]^ A. Askew,^\[3\]^ D. Bauer,^\[4\]\ ^Y. Scheglov,^\[5\]^ M. Cooke,^\[3\]^ D. Edmunds,^\[6\]^ J. Najdzion,^\[1\]^ S. Sengupta ^\[7\]^ 

## 

\[1\] Fermi National Accelerator Laboratory, Batavia, IL 60510

\[2\] Northern Illinois University, DeKalb, IL 60115

\[3\] Rice University, Houston, TX 77005

\[4\] Indiana University, Bloomington, IN 47405

\[5\] Petersburg Nuclear Physics Institute, Russia

\[6\] Michigan State University, E. Lansing, MI 48824

\[7\] Florida State University, Tallahassee, FL 32306

### Abstract

The evening shift on July 29, 2003 encountered strong noise in the
calorimeter trigger and precision readout. With help from the Cryo
Operator and experts lingering in the vicinity, the source of the noise
was identified as a welding machine operated on DAB3. This note provides
some details of the discovery of the source of the noise, what it looked
like in the control room monitoring tools and in the reconstructed data,
and some implications about the detector. It also provides common
examples of calorimeter noise not associated with the welding.

\[1\] Introduction: The Story

The evening shift on July 29, 2003 started off smoothly. We had
inherited store 2828 from the previous shift with a luminosity
approximately 24e30/cm^2^/s. Global run number 179331 was proceeding
smoothly, producing good quality data at normal trigger rates.

By 17:00 we had noticed calorimeter hot cells in the L1 trigger examine
plots. The noise seemed to be concentrated across all phi at eta = -11
and -12 according to the "hot cell finder". The GM plots didn't show
particular excess energy in any histograms. Logbook notes \[1\] from the
previous shift mention some calorimeter noise in run 179330. They
attributed it to a malfunctioning BLS board and noted that the board
noise was not seen in the L1 calorimeter trigger Examine. We continued
to attribute the problem to the BLS but, confused about how the noise
could be in the trigger, began paging calorimeter experts in search of
advice. We ended run 179331 at 17:20, reloaded pedestals and started run
179332.

By 18:11 the noise had gotten worse. It appeared in several control room
monitors. The vertex Examine showed an unusual number of events with no
L3 tracks. The monitor that shows the L1 trigger rate for each trigger
(Trigmon) showed some unusually sharp spikes in the rate. Figure \[1\]
shows the trigger rate (Hz) for two low E~T~ threshold EM triggers. The
first is CEM(2,6)\_ncu. The second is CEM(2,3)CEM(1,9)\_ncu. One can see
the sharp, brief spikes in the rate.

![](media/image1.jpeg){width="4.677083333333333in"
height="2.5729166666666665in"}

Fig. \[1\]. Trigger rate for two low E~T~ EM triggers. The vertical axis
is the rate (hz). The horizontal axis is the time, in minutes, since
(then) present. The gap centered on --47 minutes is the gap between runs
179331 and 179332.

Shortly after that we obtained new calorimeter trigger Exmine plots.
They showed hot EM triggers concentrated at approximately eta = -7 and
at no particular phi. However, the hadronic triggers seemed unaffected.
Figure \[2\] shows calorimeter trigger Examine plots. By this time the
control room event display monitor was demonstrating frequent events
with tall stripes of calorimeter noise at about the same place in the
calorimeter as the trigger noise and across all phi. There was large
missing E~T~ as well.

![](media/image2.jpeg){width="1.8756944444444446in"
height="1.6881944444444446in"}![](media/image3.jpeg){width="1.8743055555555554in"
height="1.6784722222222221in"}

Fig. \[2\]. Selections from the Trigger Examine in run 179332. Note the
hot spot in EM Trig eta and the lack of a corresponding hot spot in HAD
Trig eta.

At about this time an expert (D.E.) arrived in the control room and he
informed the shift crew that the noise was a long-standing serious
problem called "Ring-of-Fire". He told the shift crew that the problem
had appeared on many occasions during Run II but that the cause was
unknown to any experts. It would eventually go away after a few hours.
The shift decided that it was worthwhile to put some effort into
uncovering the source of the problem because the data being collected
appeared useless.

The shift contacted the D0 cryo operator (J.N.) and inquired about
unusual conditions in the calorimeter controls which might be the source
of the "Ring-of-Fire". He arrived in the control room a few minutes
later, prepared to show the shift crew how to understand the calorimeter
cryogenic monitoring tools. "However," he said, "You might want to know
that there is welding going-on in DAB3". Light dawns over Marblehead! By
this time it was 18:58 and the welder was due to leave by 19:00. It
happened he had turned off his machine and was preparing to go when we
found him via the building page.

We arranged a quick test. He fired the welding machine back up. We
watched the L1 trigger rate display shown in part in Fig. \[1\].
Whenever he squeezed the welding machine trigger the spikes in the L1
rate instantly appeared. Then disappeared. It was the initial trigger
that caused the noise, not the subsequent welding arc. We thanked the
welder. He left. We started a new run, number 179333. We had no further
problems that evening.

We renamed the effect "Ring-of-Welding".

Subsequent tests verified the observation \[2\]. Welding was
discontinued for the time being at D0. The hypothesis was developed that
the building ground was not isolated from the detector ground. The
"Ring-of-Fire" transmission path was demonstrated to be through the
calorimeter liquid-Argon temperature monitoring probes \[3\]. During the
fall 2003 shutdown, it was determined that the two grounds were not
separated \[4\]. Investigations are continuing today to locate the
short. It is not known at this time whether-or-not the "Ring-of-Welding"
noise would have occurred if the grounds were isolated. I suppose we
will find out in time. The appendix lists the dates and times of welding
that occurred at D0 during 2003 \[5\].

\[2\] Ring-of-Welding in RECO

We analyzed run 179332 with RecoCert version p14.03.01 \[6\]. We found
hot regions in the calorimeter corresponding to that which we saw during
data taking. Figure \[3\] shows calorimeter hits in run 179332 (black)
compared to 179333 (red) showing the "Ring-of-Welding" hits. Figure
\[4\] shows the eta distribution of "all" and "good" s-cone electrons in
run 179332 (black) compared to run 179333 (red). One notes the spike at
eta approximately -1.2, in correspondence with the hits, in the plot
showing "all" electrons. The spike is gone when one applies electron
selection criteria. The distribution was flat in phi in both cases, with
and without the electron selection. Figures \[5\] and \[6\] show the
distribution of "all" jets and "good" jets in those two runs. Figure
\[7\] shows the missing transverse energy in those two runs. The effect
of the "Ring-of-Welding" is evident in run 179332.

![](media/image4.wmf){width="3.8604166666666666in"
height="4.120833333333334in"}

Figure \[3\]. IETA (top-left) and IPHI (top-right) of calorimeter cells
in runs 179332 (black) and 179333 (red). Note that the "Ring-of-Welding"
run has excess hit cells in the EC's but that the distribution is flat
in phi. Cell energy in two dimension (bottom-left). Note that at IETA
approximately --12 there is a light-brown colored excess spanning phi.

![](media/image5.wmf){width="1.9979166666666666in"
height="2.4631944444444445in"}![](media/image6.wmf){width="1.7673611111111112in"
height="2.095138888888889in"}

Figure \[4\]. The eta distribution of All simple-cone electrons (left)
and good simple-cone electrons (right) in runs 179332 (black) and 179333
(red).

![](media/image7.wmf){width="5.99375in" height="6.904861111111111in"}

Figure \[5\]. All Jets found with simple cone of radius 0.7 in run
179332 (black) and run 179333 (red). The "Ring-of-Welding" run has more
jets at high-pT particularly clustered at \|eta\| \~ 1.2. The EMF of
those jets is smaller than normal. Perhaps the noise was in the hadronic
sections as well as EM, but that those cells weren't part of the
hadronic trigger.

![](media/image8.wmf){width="5.99375in" height="7.0368055555555555in"}

Figure \[6\]. "Good" Jets found with simple cone of radius 0.7 in run
179332 (black) and run 179333 (red). The "Ring-of-Welding" run has
approximately the same number of jets at high-pT and the cluster at
\|eta\| \~ 1.2 is gone. The EMF of those jets is more normal. The eta
distribution continues to show a difference.

> ![](media/image9.wmf){width="3.920138888888889in"
> height="4.499305555555556in"}

Figure \[7\]. Missing E~T~ total and components in run 179332 (black)
and run 179333 (red). The "Ring-of-Welding" run has more events with
high MET, particularly in the tail and many may be off the plot. The
strongest indication of the difference is that the good run has more
events at low MET, for these histograms are normalized to each other.

\[3\] Examples of Other Calorimeter Noise Sources

We have used RecoCert to investigate calorimeter noise that has occurred
at other times during RunII. In this section we show two other examples:
a bad ADC in run 175645, and uncalibrated pedestals after a BLS power
supply was changed prior to run 180301.

\[3.1\] Run 175645 -- a bad ADC

Run 175645 was recorded on April 12, 2003. The CalMuo shifter noted
\[7\] the large number of hits in a particular part of the calorimeter
and that there were an unexpected number of events with missing E~T~
larger than 60 GeV. Figure \[8\] shows a Lego plot event display picture
recorded by the shifter. The hot cells stand in a line of constant phi
and the missing E~T~ is large.

![](media/image10.jpeg){width="2.30625in" height="1.90625in"}

> Figure \[8\]. Event display (Lego) plot from run 175645. The hot cells
> are evident as is the large missing transverse energy (yellow spike).

![](media/image11.wmf){width="5.055555555555555in"
height="5.522916666666666in"}

Figure \[9\]. All calorimeter cells RecoCert histograms from run 175645.
Notice the lump in the "P~T~ Sum" and "E Sum" figures in the top row.
These are due to the energy deposited in the hot spot at phi \~ 4.4 to
4.6, corresponding to a 16-channel ADC.

![](media/image12.wmf){width="3.3916666666666666in"
height="3.9569444444444444in"}

Figure \[10\]. The missing transverse energy distribution in run 175645.
The lump centered on \~ 30 GeV in the x-component is the effect of the
bad ADC. Presumably there's a similar lump offscale someplace on the
y-component. The beginning of the lump can be seen in the total MET plot
at 100 GeV.

We ran RecoCert to check the effect of the line of hot cells. Figure
\[9\] shows the hits in the central calorimeter. Notice the lump in the
"P~T~ Sum" and "E Sum" figures in the top row. These are due to the
energy deposited in the hot spot at phi \~ 4.4 to 4.6, corresponding
\[8\] to a single 16-channel ADC. An ADC is 8-pairs of IETA channels
spanning more than one row of IPHI (6 pairs of cells per row). All of
the cells of EM3 had the same large energy. Figure \[10\] shows the
missing transverse energy distribution. N. Parua replaced the ADC during
the next available access, which occurred on that same day \[9\]. The
problem was solved.

\[3.2\] Run 180301 -- Pedestals need updating after BLS supply exchange

A calorimeter BLS supply failed on August 19, 2003 \[10\] and was
replaced that evening \[11\]. Run 180301 was recorded on August 20, 2003
after the BLS supply was replaced but before a stable set of new
pedestals were available \[12\] or before a new calibration was done
\[13\]. We studied this run with RecoCert and compared the run with
180329, taken after the next calibration \[13\]. Figure \[11\] shows all
central calorimeter cells for run 180301 and run 180329. The hot spots
are evident in the IETA and IPHI plots. Figure \[12\] shows "all" jets
in run 180301 (black) vs. run 180329 (red). The eta and phi plots and
the correlated eta-phi plot show an excess of jets corresponding to the
excess of hits shown in the previous figure. Study of the "good" jets
histograms indicated that the jet selection criteria could not remove
all of the effects of the extra noise. Substantial jet excesses were
seen in the same phi and eta distributions. Figure \[13\] shows the
missing transverse energy distributions. There was a 5 GeV shift in the
uncalibrated run.

![](media/image13.wmf){width="3.9479166666666665in"
height="4.205555555555556in"}

Figure \[11\]. Calorimeter all central cells for run 180301 (black) and
run 180329 (red). The hot spots are evident in the IETA and IPHI plots.

![](media/image14.wmf){width="4.94375in" height="5.861805555555556in"}

Figure \[12\]. All jets in run 180301 (black) vs. run 180329 (red). The
eta and phi plots and the correlated eta-phi plot show an excess of jets
corresponding to the excess of hits shown in the previous figure.

![](media/image15.wmf){width="3.8291666666666666in" height="4.41875in"}

Figure \[13\]. Missing transverse energy and components in run 180301
(black) and run 180329 (red).

\[4\] Summary and Discussion

The evening shift on July 29, 2003 discovered the cause of severe noise
in the calorimeter and trigger named "Ring-of-Fire". We renamed it
"Ring-of-Welding". We characterize it and some of its effects on the
reconstructed events in this note. This kind of calorimeter noise was
not supposed to be able to occur because it was thought that the
detector and building grounds were isolated from each other. They are
not and investigations have been underway in the shutdown this fall.

"Ring-of-Welding" is a serious form of calorimeter noise, but it is not
the only one. We have showed evidence of two other kinds of calorimeter
noise in this note. One is the result of a bad ADC. The other the result
of a delay in obtaining calorimeter pedestals after a BLS supply was
changed. While the effects aren't the same as in "Ring-of-Welding" data
they are serious enough to question the data's quality.

We expect that this is not a comprehensive list of causes of calorimeter
noise. We know and appreciate that calorimeter experts are working hard
to identify and suppress them.\
A worthwhile contribution aimed at assisting them might be to make a
systematic quantification or classification of "bad runs" as scored in
the "Run Quality Database" \[15\]. Perhaps the causes can be
accounted-for in the logbook or by experts.

#### Appendix: Welding at DZero in 2003

NOTE:

Times shown are approximate. Some notes were recorded from memory days
after the fact, and may be inaccurate. Actually welding usually not
continuous for periods delineated. Set-up, leak checking and
superinsulation activities are necessarily interspersed with weld times.

[Date]{.underline} [Times]{.underline} [Location]{.underline}

7/29/03: 1530-1900 including "Ring of Fire Test" Weld Shop

7/28/03: 1530-1800 "

7/24/03: 1530-1800 N-305 "

7/23/03: 1530-1800 N-305 "

7/17/03: 1530-1845 Separator Field joints Refrigerator Area

7/16/03: 1530-1845 " "

7/15/03: 1530-1845 " "

7/14/03: 1530-1845 " "

7/8/03: 1530-2000 Separator Field joints, inner lines "

7/7/03: 1530-2000 Separator Field joints, inner lines "

7/3/03: 1530-2020 " inner lines "

7/2/03 1530-2000 " inner lines "

7/1/03: 1530-2000 Subassemblies and field joints Weld shop & Refer Areas

6/30/03: 1530-2000 Subassemblies & field joints " "

6/26/03: 1530-2000 Subassemblies Weld shop

6/25/03: 1530-2000 Subassemblies Weld shop

6/24/03: 1530-2000 Subassemblies Weld shop

6/23/03: 1530-2000 Subassemblies Weld shop

6/4/03: 1300-1530? (half day could have been morning) Weld shop

4/28/03: 0730-1530 Separator subassembly Weld shop

4/14/03: 0730-1530 " "

4/11/03: 0730-1530 " "

3/18/03: ? Weld relief tree to separator vessel "

3/3/03: 0730-1530 VLPC Test LN2 Rtn U-tube "

2/28/03 ?(45 min of welding ) "

2/17/03: 0730-1530 VLPC Test "

1/22/03: 0730-1530 VLPC Test Phase sep. jumper "

1/17/03: 0730-1530 VLPC Test "

1/16/03: 0730-1530 VLPC Test "

1/15/03: 0730-1530 VLPC Test "

1/14/03: 0730-1530 VLPC Test "

1/6/03: 0730-1530 VPLC Test "

#### Acknowledgements

We thank the experts, particularly Dean Schamberger, Robert Zitoun,
Nirmalya Parua, V. Shary, Laurent Duflot and others for their work in
maintaining and operating the calorimeter and Harry Melanson and others
for writing and maintaining RecoCert.

B. Vachon was co-daq-expert on shift. Her nagging the shift to
understand the problem was important in getting to the cause of the
noise.

References

\[1\] D. Kaefer, CALMUO Entry 227292, D0 Control Room Logbook, July 29,
2003.

\[2\] R. Rucinski, Communication to operations email list on or about
July 31, 2003. The tests were carried out by B. Barger, D. Edmunds, D.
Schamberger and a welder. The results were independent of welding
machine and location of the machine in the building.

\[3\] D. Schamberger, public and private communications.

\[4\] D. Denisov, D. Schamberger, and R. Zitoun, public and private
communications.

\[5\] The complete contents of the appendix comes from B. Barger via R.
Rucinski.

\[6\] RecoCert is a product written so as to look at the quality of the
output of the reconstructed data. It's maintained by Harry Melanson and
is available for use by anyone interested.

\[7\] L. Duflot, CALMUO Logbook entries 193929, 193947, 193968, and
193978.

\[8\] D. Schamberger and R. Zitoun, private communication.

\[9\] L. Duflot, CALMUO Logbook entry 194033.

\[10\] V. Shary, CALMUO Logbook entry 233940.

\[11\] V. Shary, CALMUO Logbook entry 233993.

\[12\] L. Duflot, CALMUO Logbook entry 234170.

\[13\] V. Shary, CALMUO Logbook entry 234300.

\[14\] V. Shary, CALMUO Logbook entry 234393.

\[15\] G. Bernardi keeps track of JET/MET run quality in the "Run
Quality Database" located at URL
http://d0db.fnal.gov/qualitygrabber/qualQueries.html.