# Presentation: 905050

## Magnet Reliability in the Fermilab Main Injectorand Implications for the ILC

- Michael A. Tartaglia
- Fermilab Technical Division
- Magnet Systems Department

- Accelerator Technology Seminar

- 11/1/2007

##    Motivation 

- ILC Magnet Technical Systems
**Reference Design Effort and Report **
- For reliable (>75%) operation, *High Availability* is a necessity and a concern for all systems
- Availability group is reserving contingency of 10%
*Magnets* were allocated *unavailabililty* of 5%*0.15 *=0.0075*
- >13000 Magnets in the 5 ILC areas, ~6800 Water Cooled
- Magnet Availability Goal:        MTBF ~18 . 106 hours
-   			 assuming:        MTTR ~ 8 hrs
**Can this level of reliability be achieved w/o extraordinary cost?? **
- In the RDR text we said we thought this should be possible “by applying best modern engineering practices, ensuring adequate quality control of materials and procedures during fabrication, and use established guidelines for operation within reasonable environmental limits (water T, T, flow)”

- Accelerator Technology Seminar

- 11/1/2007

##    Motivation, continued 

-    Motivation, continued

- RDR Magnet Count, by ILC Area and Style
- CONCEPTUAL DESIGNS	    [ ILC-REPORT-2007-001 ]

- 		N= 13190 magnets/135 styles
- NC= 10872  {6873 lcw, 3999 air},  SC= 2318

- Accelerator Technology Seminar

- 11/1/2007

##    Motivation, continued 

- Availability Calculations (a simplified example)
- 	A = uptime fraction for one magnet
- 	MTBF = Mean Time Between Failures
- 	MTTR = Mean Time To Repair
- 			MTTR (~10h) << MTBF (~106h)
- 	A	= MTBF / (MTBF+MTTR)
- 			1- A ≈ MTTR / MTBF
- 	A = availability for system of N (=13000) magnets
- 		= (A1)(A2)(A3)...(AN) = AN (assuming all equal)
- 		= (1 - 0.0075) 	*by decree*
- 	MTBF = MTTR/(1 - .0075)1/N  ~ (8)/(5.8 10-7) ~ 14 106h
- 		if water cooled dominate, N ~7000  MTBF ~ 7.4 106h
- Availability Group Assumptions? (details not in RDR)
- 18 million hours implies ~17000 magnets (early ILC design)

- Accelerator Technology Seminar

- 11/1/2007

##    Motivation, continued 

- Main Injector may provide some useful data points
**Constructed, Operated with “modern approaches”**
- RDR studied Magnet Availability of existing machines,
-    esp. SLAC: wide MTBF range ~0.5 to 12 106 hr
- Parts procured from industry; Design, Assembly at FNAL
**Hundreds of Magnets operated for 9 years**
- Room Temperature Air and Water Cooled Styles
- Both New and Refurbished magnets
**Magnet Technology Conference Publication**
- Timely opportunity to document the FMI magnet performance
- Attempt to draw some conclusions relevant to the ILC

- Accelerator Technology Seminar

- 11/1/2007

## FMI Magnet Reliability Study

- MT20 Publication (5K07)
**Product of a team from various groups at FNAL**
- Magnet Systems Department in TD
- J. Blowers, D. Harding, O. Kiemschies (formerly Ops Dept in AD), MT, S. Rahimzadeh-Kalaleh (SIST),
-    J. Tompkins (ILC Magnet Tech.Sys. area leader)
- Main Injector Department in AD
- D. Capista
- Special thanks to people we interviewed
- {B. Mau, D. Augustine, R. Slazak B. Brown, D.Morris, others}
**FNAL Pre-print  **FERMILAB-CONF-07-443-TD** **
- reviewers suggested minor revisions; not yet implemented
*(deadline is Nov 9 – there is still time to make changes)*

- Accelerator Technology Seminar

- 11/1/2007

## Reliability Study, continued

- Main Injector Magnets **	**
**New Construction**
- Incorporate lessons learned into design & fabrication
- Analysis of failed MR quad magnet [N.Chester memo, 1990]
- FMI Dipole Design Considerations [PAC, FNAL-Conf-91/127]
- Prototype Dipole Construction, Meas [IEEE TransMag1992]
- B2/B3 dipole reliability: thermal stresses [MTF-93-004,9]
**Magnet Cooling and Bus design:** [PAC, FNAL-Conf-95/145]
- New C-Magnet design reviews
- Concise Process Discussion: [FMI Tech. Design Handbook]
**This could be a separate seminar**
**Reworked Main Ring Magnets**
- hipot, leak check – all passed (coil/impreg rework not required)
- Replace ceramic insulators
- Support and Vacuum tube changes

- Accelerator Technology Seminar

- 11/1/2007

## Reliability Study, continued

**Reliability Study, continued**

- Main Injector Magnets
- 	 [ from FMI Tech.
- 	Design Handbook ]
-  Start of Operation:
- 		9/13/98
- Operating Efficiency:
- 		76.8%
- Study through 8/1/07
- 	fdt = 77856 hours

*Recycler NOT included*

- (WQB: 10968 h)

- Accelerator Technology Seminar

- 11/1/2007

## Reliability Study, continued

- Water
- Cooled
- Air
- Cooled

- Accelerator Technology Seminar

- 11/1/2007

## Reliability Study, continued

- Methods
**Research records on magnet “failures”**
- Existing Failure Catalogs
- AD lists of changed magnets (Augustine, Brown)
- TD lists of suspicious magnets (Blowers)
- Electronic Searches
- MCR and elog
- Machine elogs
- TD “Onbase” Device Service Records
- TD Accelerator Support web pages
- Interview Key Personnel
- AD, TD operations/MI/magnet experts
- TD factory/repair technicians
**Worry: did we miss anything?? ***(appeal to audience)*
- How reliable are our records, memories, searches?
- 		(esp. wrt correctors: would we know? Run without?)

- Accelerator Technology Seminar

- 11/1/2007

## Reliability Study, continued

- Methods
**Research records on magnet “failures”**
- Existing Failure Catalogs
- AD lists of changed magnets (Augustine, Brown)
- TD lists of suspicious magnets (Blowers)
- Electronic Searches
- MCR and elog
- Machine elogs
- TD “Onbase” Device Service Records
- TD Accelerator Support web pages
- Interview Key Personnel
- AD, TD operations/MI/magnet experts
- TD factory/repair technicians
**Worry: did we miss anything?? ***(appeal to audience)*
- How reliable are our records, memories, searches?
- 		(esp. wrt correctors: would we know? Run without?)

- Accelerator Technology Seminar

- 11/1/2007

## Reliability Study, continued

- Methods
**Compare Lists/resolve discrepancies **
- Understand details of each event
- Eliminate “MI” failures not in the “study region”
- e.g. 	PS/Bus fault, not magnet;
- 		magnet in shared tunnel, not “MI” or beamline;
**Definition of “failure”: Magnet must be changed**
- Hipot (short to ground) – 1000V for main bus; not *correctors*
- Inductance (turn-to-turn short)
- Overheating (loss of coolant accident)
- Water Leaks
**Magnet Change may occur during a shutdown/ maintenance period (to prevent failure during operation)**
- Would not count against “availability” in ILC (e.g.,LCW leak)

- Accelerator Technology Seminar

- 11/1/2007

## Reliability Study, continued

- Results
**Mean Time To Repair **
- Careful study of elog to find cases where **magnet change** was the only activity preventing MI operation:
- 7 cases  <17.8 hours> from problem to startup
- What’s typically involved?
- Diagnose problem (call in experts, assess problem)
- Conduct Safe Access procedures (travel to site)
- Field diagnosis, transport crews & equipment
- Replace and align device
- Conduct checkout and startup procedures
- 8 hours is unlikely for replacement in ILC
- in Main Ring, it was said to be ~12 hours
- may be faster to find/diagnose (~individually powered)
- Long distances to sites; Stage magnets for installation?

- Accelerator Technology Seminar

- 11/1/2007

## Reliability Study, continued

- Results
**Failure Rates (limits), by Style and Failure Mode**
- MTBF = Nmagnets * Toperation / Nfailures
- Note: we don’t distinguish T “live” and T “down”
- 90% CL Lower limits from Poisson Statistics
- Np=a (N large, p small, finite event probability a)
- f(x) = e-aax/x!
- If Observe x=0, what is a for f(0)=0.1?  a=2.3 (90%UL)
- numerical calc/lookup required for higher observed x:
- 	x={0,1, 2, 3, 4}	a90%UL={2.3, 3.89, 5.32, 6.68, 7.99}
- 				a95%UL={3.0, 4.74, 6.30, 7.75, 9.15}

- Accelerator Technology Seminar

- 11/1/2007

## Reliability Study, continued

- Results

- Numbers
- Of Failures
- &
- MTBF in
- Hours

- Accelerator Technology Seminar

- 11/1/2007

## Reliability Study, continued

**Reliability Study, continued**

- FMI Quad and Dipole Failures  vs Time

- New
- quad

- old
- quad

- old
- dipole

- ?

- Accelerator Technology Seminar

- 11/1/2007

## Failure Mode Discussion

_***Failure Modes: Water Leaks***_

- Accelerator Technology Seminar

- 11/1/2007

## Failure Mode Discussion

_***Failure Modes: Water Leaks***_
*Are a great concern* – penetrate cracks in insulation and cause electrical failures (which stop machine)
- Monitored constantly during operation, and vigilence during tunnel accesses (but don’t usually stop machine)
- Repairs (and level of) *made as needed* (judgement call)
- Generally occur at ***external braze joints*** and ***manifold connections***, but may be encapsulated in epoxy
- Careful design, technique, tests, minimum # internal joints in new construction: paid off !
- No Hose bursts have occurred (well beyond 5 year lifetime)
- apparently no hose degradation from (p) radiation
- ILC will have same design/fab/operations concerns *10
- How to keep quality@low cost across many vendors?

- Accelerator Technology Seminar

- 11/1/2007

## Failure Mode Discussion

_***Failure Modes: Overheating***_
- Failure to cool properly (design, operation) can result in
- Thermal stresses (T)/ insulation cracking
- Epoxy and insulation degradation
- Softening of braze joints, water leaks
- Personnel hazards
- FMI { Tin ~35 oC, Tmax~10 oC } are good guidelines
- ILC considering Tmax of 25 oC for some magnets
- FMI temperature switches not interlocked to power supplies
- one magnet burned by operating for hours without LCW
- Recommendation for ILC is to interlock (flow ? and) temperature switches to PS
- sensor reliability, costs, EDIA, issues

- Accelerator Technology Seminar

- 11/1/2007

## Failure Mode Discussion

_***Failure Modes: Electrical Shorts***_
- FMI Main Dipole and Quadrupole Magnets must withstand 1000 V to ground Hipot test
- High ramp rates and large inductance, many bussed magnets
- Routinely done before/after access/maintenance to find and prevent problems during operation
- 		   (most problems are related to power supplies and bus)
- 		   surprisingly, failures are detected following OK operation
- Hipot (short to ground) & inductance (turn-to-turn short =1) failures require a magnet change
- Failures have mostly been of old and new quads;
- one old dipole
**root cause is combination of compromised insulation + Water **

- Accelerator Technology Seminar

- 11/1/2007

## Failure Mode Discussion

_***Failure Modes: Electrical Shorts- cont’d***_
- NO FAILURES of TRIMs, HOCs (air cooled)
- 	can _get wet too_, perhaps less, but operate at lower {I x V}
- 		new and reworked styles have been very reliable (not LEP??)
- 		lower hipot requirements (in fact, it’s not done)
- 		experience less thermal and Lorentz stress
- 		failures generally do not stop operation (but would be noticed)
- Air-cooled styles should not be a problem in ILC
- The challenge for ILC magnets is to build many reliable styles of water cooled magnets (from a global vendor pool)
- Individual powering (mostly) and ~ DC operation
- Hipot requirements may not be as severe (not known)**	**
**Mechanical stresses from Lorentz force cycling reduced**

- Accelerator Technology Seminar

- 11/1/2007

## Reliability Study, continued

**New Quad Failures: Early Investigation**
- Autopsies revealed cracks in epoxy at quad end
*Conclusion (theory?):*_ Design change_ (from MR IQB design) likely to have introduced a problem
- _G10 sheet insulation was added_ to improve insulation in “end plate” region (from MR quad failure study in 1990)
- Coils were insulated and impregnated *with the Steel Core*
- 	into a monolithic unit (to avoid differential expansion)
- Epoxy impregnation was incomplete around the G10 sheets, not able to see this
- Non-uniform stresses led to epoxy cracking in the ends
- Failure mode of FMI reworked quads ?
- Autopsies not done (lower priority, resource limitations)

- Accelerator Technology Seminar

- 11/1/2007

## Reliability Study, continued

**New Quad Failures: Early Investigation**

- Accelerator Technology Seminar

- 11/1/2007

## Reliability Study, continued

**New Quad Failures: None “lately”**
- Perhaps lucky (infant morality?)
- not all quads may have suffered this problem
- Compare rate of old Quads: avg the same!
- too soon to tell... (need time: 2.3*MTBF)
**New FMI Dipoles: No Failures**
- Coils are insulated & impregnated independent of the Steel Core
- One anchor, with features to limit Lorentz expansion
- Recent WQB and IQC,D construction follows the new dipole design – no longer bonded to the Cores
**However, the old dipole design DID bond coils to core**
- No WQB failures
- only 7 x 1.5 years in operation

- Accelerator Technology Seminar

- 11/1/2007

## Reliability Study, continued

**ILC vs FMI Magnets summary**
- ILC still largely conceptual designs
- Iterating to reduce number, increase similarity of styles
- Failure Modes & Effects Analyses need to be performed
- FMEA study of one FNAL quad style is planned
- “Implications” are speculative – a lot of variables!
- ILC mostly individually ~DC powered magnets
- 1356 dipoles, 4165 quads, 1352 “other” water cooled
- Reduced Lorentz stresses ?
- Electrical failures of LCW-cooled may be less severe
- Diagnosis of failure may be easier (lower MTTR)
- Build in hipot / fault detection capability?
- Lower voltage to ground requirement ?
- Failure of air-cooled trims, HOCs may stop the machine
- e.g., needed to center the beam in focusing elements

- Accelerator Technology Seminar

- 11/1/2007

## Reliability Study, continued

- Conclusions
**It takes years to find out if you have a problem with reliability! ****(Even with large numbers of magnets)**
**“****very good” air cooled FMI magnet reliability**
**No electrical failures in 42.7 million hours**    >18.5 106 h (90%LL)
**“****good” water cooled FMI magnet reliability**
- Internal water leak rates are low
- Electrical Failure Rates vary –
**New dipoles 26.8 million hours**: MTBF  >11.6 106 h (90%LL)
**Old Dipoles: MTBF=17.1 million hours**  > 4.4 106 h (90%LL)
- Old and New Quads: MTBF = 2.0 106 h
**Time will tell if new quads are really better**
- This may be a place for further study (FMEA)

- Accelerator Technology Seminar

- 11/1/2007

## Reliability Study, continued

- Conclusions, continued
**Assess ILC MTTR estimates: 16 h  (not 8 h) ?**
**Magnets Combined Need **A ~ .9925
- 4165 Quads @ 2.0 106h gives **A**** ~ .**967  (.984)
- 1356 Dipoles @17 106h gives **A**** ~ .**99879  (.9994)
- What will superconducting magnet reliabilities be (Hera?)
- how much of the availability budget they will eat
- This will be challenging
**Other Caveats: **
- Effects of different Radiation? protons, vs synchrotron
- How to maintain the design/fabrication/QA expertise? (many experienced FMI-era engineers have retired)
- How to maintain quality across the machine?
- 	(“magnet systems group” is an RDR entity)

- Accelerator Technology Seminar

- 11/1/2007