# Presentation: 674894

## Data on Tevatron Complex Reliability

- Elliott McCrory
- 25 February 2004

## Tevatron Reliability Data

- Czarapata:
- General aspects of our reliability
- “Once is an accident; twice is a problem”
- How to improve reliability
- Here:
- Numerical analysis of downtime data
- What is the reliability of each system?
*Is it changing?*
*What is the best way to characterize it?*

- Data On Tevatron Complex Reliability – McCrory, AD/Integration

## Talk Outline

- Downtime/failure data sources
- Temporal characteristics of failures
- Focus on
*Controls*
*Simple example*
*Tevatron*
*More complex, but very interesting and relevant*
- Mean Time Between Failure (MTBF)
- Downtime Duration
- Average downtime per week
- Time Between Stores

- Time Between Stores

- Data On Tevatron Complex Reliability – McCrory, AD/Integration

## Sources of Downtime Data

- Downtime Logger
- ACNET Console Application D18
*Data entry by Operations Dept personnel*
*Data entered:*
*Downtime beginning (Time stamp) & duration*
*Category*
*Description*
*~60 categories*
*E.g., LRF, BPS, TQUEN, POWER, etc.*
- This analysis
*1 Jan 2003 to 9 Sept 2003 and 30 Dec 2003 to Now*
*Exclude data from Fall 2003 shutdown*
*More details in supplemental materials*
- SDA

- Data On Tevatron Complex Reliability – McCrory, AD/Integration

## Failures: Temporal Accounting

- MTBF
- Examined for each Major System
- Show some charts from Tevatron
- Look in detail at one system: _Controls_
*54 Failures*
- Downtime Duration
- Average downtime per week
- Tevatron reliability is improving

- Data On Tevatron Complex Reliability – McCrory, AD/Integration

## MTBF For Each System

- MTBF, hours (Log Scale)

- Data On Tevatron Complex Reliability – McCrory, AD/Integration

## Average Downtime Duration

- Downtime Duration, Minutes (Log Scale)

- Data On Tevatron Complex Reliability – McCrory, AD/Integration

## Controls Failure Intervals

- MTBF = 112.2 hours,
-           RMS =113.2

- This is an exponential
- distribution with *<t> =** **σ** *
- Recall: Random Queuing Problem

*Data from 1 Jan 2003 to 1 Feb 2004*

- Data On Tevatron Complex Reliability – McCrory, AD/Integration

## Controls Failures Intervals

- This is a Random Process.
*0.0089 failure probability/hour;*
*Up Time = 99.11%/hour*

- Data

*   **F(t) =** N **λ* *e **–**λ** t*
*<t> = 112 hrs*
*     **σ** = 113 hrs*
*λ** = **1/112.5** = 0.0089*

*Other systems show similar characteristics: *
*Failures are Random w.r.t. time*

- Data On Tevatron Complex Reliability – McCrory, AD/Integration

## Tevatron Reliability

- Something else is going on

*Measure **<t> = **σ** =* *26 hours.*
*Assuming** <t> = **σ** = 1/ **λ** *
- ∴* **λ** = **0.038*

- Data On Tevatron Complex Reliability – McCrory, AD/Integration

## Tevatron as Two Machines

- “Low Beta”, Interval between failures;
*<t> = **σ** =* 38 hours, λ = 0.026

- “Low Beta” interval recently
- became longer.

- “Not Low Beta”, Interval between failures;
*<t> = **σ** =* 18 hours, λ = 0.056

- Data On Tevatron Complex Reliability – McCrory, AD/Integration

## Tevatron: MTBF vs. Time

- Days since 1/1/2003

- Running average MTBF, minutes

- Running average over 120 days

- Running average over 30 days

- Nov/Dec 2003
- Shutdown

- See Supplemental Materials
- for data on other systems

- Good

- Bad

- Data On Tevatron Complex Reliability – McCrory, AD/Integration

## Conclusions on MTBF

- All Systems are characterized by random failures
- Exponential “λ” distribution
- Tevatron behaves as two machines
- Tevatron at low beta
*Uptime probability ≈ 0.974 / hour*
*Could be as high as 0.985 / hour, now*
- Tevatron at all other times
*Uptime probability ≈ 0.944 / hour*
- Analysis Continues...
- Impact on operations
- Complicated!
- Ref: Czarapata
- Used in Operations Model

- Data On Tevatron Complex Reliability – McCrory, AD/Integration

## Average Downtime/Week

- Average Downtime Per Week, Hours

- Averaged over the 30 days
- ending 23-Feb-04

- Averaged over the 120 days
- ending 6-Sept-03

- Data On Tevatron Complex Reliability – McCrory, AD/Integration

##  Average Downtime Fraction

-  Average Downtime Fraction

- Days since 1/1/2003

- Running average Downtime fraction

*Tevatron*

- Bad

- Good

- See Supplemental Materials
- for downtime fraction
*F(t)* for each system

- Data On Tevatron Complex Reliability – McCrory, AD/Integration

## Time Between Stores in Tevatron

- Time Between Stores, Hours

- dN / dT

- 45

- Shot
- Setup

- Failures/Studies/Accesses

- Shutdowns

- Data On Tevatron Complex Reliability – McCrory, AD/Integration

## Conclusions

- Downtime/failure data
- Collected by operations & in SDA
- Reliability of systems calculated
- All systems are characterized well by random failures
- Tevatron behaves as two different machines
*“**Low Beta” is more reliable than otherwise*
*And it is better now than last year*
- Impact on operations is complicated
*Using Operations Model to sort this out*

- Data On Tevatron Complex Reliability – McCrory, AD/Integration