RELIABILITY ASPECTS AND SYNCHROTRON LIGHT SOURCES
Emanuel Karantzoulis
ELETTRA, Sincrotrone Trieste, Italy
Abstract
Availability and Reliability definitions and aspects arepresented. Their importance for x-ray sources is discussed
and emphasis is given to the failure analysis wherereliability serves as a tool. X-ray sources availability andtheir component failure are presented for different
machines.

1 INTRODUCTION
Reliability is considered as a very important aspect of life
including any enterprise such as financial, scientific,
commercial or other. We often speak about a machine asreliable: "I have a reliable car" meaning that it doesn’t
break down often. Or, news people talk about an "usually
reliable source". In both cases, the word reliable means"dependable" or "trustworthy." In research, the term
reliability means "repeatability" or "consistency". A
measurement is considered reliable if it would give thesame result over and over again (assuming that what weare measuring isn't changing!). To explore in more detail,define a measure X, that might be a person's score on aachievement test, a measure of severity of illness or ameasure of break downs of a machine. It is the value of X
(numerical or otherwise) that is observed in the study.
Now, to see how repeatable or consistent an observationis, the measurement has to be repeated many times.
While we observe a score for what we're measuring, weusually think of that score as consisting of two parts, the'true' score or actual level and the 'error' while measuring
it for the ith time (the error can be due to the observation)and therefore Xi=T + ei It is important to note that theXi score is observed and not the true (T) or error (e)scores. The measure X is reliable if by repeating
measurements the resulting scores are pretty much thesame and the reliability in the layperson terms might bedefined to be the ratio R= T/X Since T=X-e R=1-e/X
and if e-> 0 R->1 or 100% whereas if e/X->1 (no T part )R=0. The reliability therefore has a value from 0 to 1 orfrom 0 to 100%.
In general for X1,2 sets ofmeasurements R=covariance (X1,X2) / sd(X1) * sd(X2)

2 RELIABILITY ASPECTS
2.1 Overview
Reliability theory is a mathematical frameworkcomprising methods models and ideas in order to predict,estimate, understand and optimize the lifespandistribution of systems (living or not) and their________________________________________

karantzoulis@elettra,trieste.it

components.
Started around the time of World War II, Reliability of asystem refers to its ability to operate properly according
to specified standards. Reliability is described by the
Reliability function (also called survival function) S(t)which is the probability that a system will work throughtime t. This function evaluated at time x is just theprobability P that the failure time T, is beyond time t.
Thus the reliability function is defined as
S(t)=P(T>t)=1-P(T<t)=1-F(t) where F(t) (failure function)
is a standard cumulative distribution function in theprobability theory. In general reliability for systems isdefined via the failure rate λ(t) (also called hazard rateh(t) ) which is the relative rate for reliability function
decline and it is expressed in fits (10-9 failure/hour).
dS (t )dF (t )d (log e S ( x ))S (t )dt=−=−
λ (t ) =S (t )dt
dt
For constant failure rate

S ( t ) = e − λtReliability is the probability that a system will work in agiven time interval under specified conditions..
Availability represents the time a system meets its
specification. A system must be designed for highavailability when continuous service is important.
Failure is defined as F( t) = 1 - S( t) and a system can beclassified as Non-repairable (e.g. satellites ) or
Repairable (e.g. car, particle accelerator ).
Some other useful definitions are:Mean Time To Failure (MTTF) (in hours) is defined as
1/λ and S(t) = exp(- 1) = 0.37 therefore it is the time thecomponent has a probability of failure of 0.63 or
63%.Mean Time Between Failures (MTBF) is the mean
number of time during which all parts of a system
perform within specified limits, for a given time interval.
Mean Down Time (MDT), the average time a system isunavailable due to a failure
Mean Time To Repair (MTTR) is the sum of corrective
maintenance time divided by the total number of failuresduring a given time interval.
Redundancy is the existence of more than one way(component) for accomplishing a given task. In generalcomponents may be connected as follows:
In Series (cheap - repairable) all n components must workin order for the whole system to work S=S1*S2*S3*…Snor S=ΠSi and λ = Σλi
In parallel (expensive – non-repairable or highredundancy - living ) (Active or Standby), the system

30

will work as long as at least one component works R=1(1-S1)*(1-S2)*(1-S3)*…*(1-Sn)
thus R=1-ΠFi and1/λ∼Σ1/λι.
Mixed connection used for cost and reliability
optimization.

Cumulative number of failures

25

20

15

10

5

2.2 Controlling Components Aging
The reliability function of non-aging systems(components) is described by the exponential distribution:λ(t)=λ=const è S(t)=S0 exp(-λt)
This failure law describes “lifespan” distribution of atomsof radioactive elements, beam lifetime in accelerators etc.
If failure rate increases with time we have an aging
system (component) that deteriorates with age. If failure
rate decreases with time indicates that the system was not
set up correctly or not all its subsystems were functioning
properly.
In reality system failure rates may contain both non-aging
and aging terms:
λ(t)=N exp(-b t) + A + R exp(a t)with parameters N A R and a,b >0, where N and b are
age dependent terns that indicate decrease with time (e.g.
infant mortality), term A is aging independent term thus it
is due to external causes like accidents, mishandlings etc.
whereas R and a are the true aging terms. Their values
depend amongst other on how the system components areconnected. Such a function could be like in Figure 1 using
N=1, b=0.005, A=10, R=10-8 and a = 0.0046 :

0

-5
-20

0

20

40

60

80

100

120

140

Cumulative TBF

Figure 2: Cumulative number of failures vs. cumulative
TFB (time between failures)

2.3 Fully repairable systems
They have a behaviour similar to non aging system for
which
λ(t)=λ=const and S(t)=S0 exp(-λt)
However in repairable systems under continuous
supervision, λ is a quasi-constant and reliability is notalways a pure exp function. The cumulative number of
failures oscillates around the line that indicates constantfailure rate. Also plots of failure rates do not follow one
bath tube diagram but are a combination of many suchgraphs. In this case the failure rate can be represented asa polynomial or even as a power function too.
ELETTRA CNOF
2000
y = -87.253 + 0.0621x R= 0.99679

youth

aging

1 1 .2

Total CNOF

1500

1000

11

500

failure rate

1 0 .81 0 .60

Useful life

1 0 .4

0

5000

4

1 10

4

1.5 10
2 10
Ac TBF4

4

2.5 103 10

4

4

3.5 101 0 .210-1 0 0 0

0

1 000

2 00 030 00

4 00 0

50 00

Figure 3: Cumulative number of failures for ELETTRA
vs. cumulative TFB

tim e ( h o u r s )Since failure rates indicate the state of health of thecomponents of a system the existence of any trends is ofparticular importance since if the failure rate increases
with age we have an aging component that deteriorates. In
order to see these trends one usually plots the cumulative
number of failures against the cumulative time between
failures (TBF). These graphs help to predict the behaviourof a system. Maintenance scheduling, spare partprocurement and early detection of trends are essential inrunning a facility and planning for reliability
improvement.

1000
y = 0.33396 * x^(0.77535) R= 0.99477

800

600
Ac. MTBF

Figure 1: bath tube model

400

200

0
0

5000

1 104

1.5 104
2 10 4
Ac TBF2.5 10 4

3 10 4

3.5 10Figure 4: Cumulative MTBF for ELETTRA vs.
cumulative TFB

3.1 Up-time statistics
where the TFB is from 1995 until now (about 35000
hours). Assuming a power relationship for MTBF one canestimate the evolution of the system. The fit gives
MTFB(t)=0.334* t0.77535
and it is evident that the machine runs better now than in
the beginning

X-ray availability
X-ray availability/storms excluded













$36

3 RELIABILITY AND AVAILABILITY IN
ACCELERATORS(/(775$(65)

Figure 5: X-ray sources availability for 2001
These are relatively new issues, concerning more X-ray
sources and spallation sources (for different reasons). X
ray sources are are multi user storage rings that produce
synchrotron radiation. There reliability has an increased
importance exactly due to their nature i.e. many users
work in parallel and plan their experiments long in
advance (usually over one year). The demand for beamtime is still unsaturated and new x-ray sources are
emerging ( e.g.
Soleil , Diamond ). In theseestablishments work typically 600 - 6000 users per year
for about 5000 hours of machine time for user operation.
Usually a machine can break as long as it has highAvailability, otherwise will loose its users. One candistinguish between:• High reliability -> the machine breaks seldom• High Serviceability -> the machine can break more
often but has short recovery time.
As a rule of thumb many short failures are tolerated better
than a long one. Thus Availability comprises such issues
as reliability, serviceability and others.
The availability qualitatively depends on• Budget (personnel, redundancy of equipment,
maintenance)• Operations(operators,
experts/on
call,
troubleshooting, modes of operation )• Management (personnel, equipments, statistics)• Planning (new installations, coordinating shut
down work, careful control before start up )and is measured with the usable up-time (=beam on withagreed quality) and the MFD (mean fault duration).
Additional information is given by: CNF (Cumulative
number of failures), MTBF,
Failure rates (Faults / hour ) and MTTR (mean time torepair)$36(/(775$


(65)











$Y DLODELOLW\PLQ$Y DLODELOLW\PD[WRWDO

Figure 6: Maximum, minimum and average availability
since 1995 (APS since 1997)



07%)



0)'



)['D\



UVX
RK






1.2



0.8

1.2



$36

(/(775$(65)

Figure 7: X-ray sources Mean Time Between Faults(MTBF) , Mean Fault Duration (MFD) and Faults per
Day (FxD) for 2001

07%)PLQ07%)PD[




K






In the next, 3 x-ray sources (APS 7GeV, ELETTRA 22.4GeV, ESRF 6 GeV) will be considered for the only
reason that they seem different since use 3 different
modes of operating. Thus APS has full energy booster
and top-up with many individual power supplies, ESRF
full energy booster and ELETTRA injects from a linac at
1 GeV while the machine operates for the users at 2 or 2.4
GeV (25% of user time).

$36(/(775$(65)

Figure 8: Maximum and minimum Mean Time between
Faults since 1995 (APS 1997)
All considered machines had a high reliability >95% in
2001. Extending the statistics over many years one seesthat in general the average lays above 93% . Looking at

3.2 Where to look for improving - Equipment
failures$36FRPSRQHQWVWDWLVWLFV8787


H 

87

LP
W
Q 
Z
R 
G
 
I

R

 

IS
I
Un Oth
e
id
en r
tif
ie
d

O S&
pe
A
ra
tio
ns
O
A
Ph G
ys
ic
s
ID
-F
E

PS
on
tro
at ls
er
/M
Va E
cu
um
C

W

D
R
ia
gn F
os
tic
s



Figure 9: Three years of component failure analysis for
APS(65)FRPSRQHQWVWDWLVWLFV87H 8787

LP
W
Q
Z
R
G
I
R

FE
TA
KE
O
N
D
TR
IA
G
O
N
L
BE OS
AM TI C
TU S
N
IN
BE
G
AM
LI
IN
N
AC
ST
AB
IL
IT
Y
ST
O
R
M
M
IS

AN
H

U

M

SR

C

F
R

-C

SR

SR

VA
C
O
O
LI
N
G
S
R
PS









Figure 10: Three years of component failure analysis for
ESRF(/(775$FRPSRQHQWDQDO\VLV87









8787

SR

In

st

VA P S
C
U
ru
m UM
en
ta
tio
n
M
ag
R
ef net
R
i
s
l
l
ad
/d
io
el
Fr
a
eq y
ue
nc
In
se Co ynrtio n tr o
W
D ls
at
ev
er
ic
/A
es
ir
C
om
L
pr ina
c
/C
St
oo
or
m
lin
s/
g
ne Elect- i
nt tric
er
it
ru y
pt
io
M
n
is
ha s
nd
Be lin
am g
l in
es
O
th
er

% of down time

MTBF, MFD and Faults per Day, for 2001 APS and
ELETTRA seem to be close however ESRF does factor of
two better. Seen all statistics since 1995 (APS 1997)
whereas the minimum mean time between faults is in theorder of 15-20 hours the maximum is still by 30% higher
for ESRF (a very good achievement). It is useful toobserve that 95% is practically a non stop mode. For 5000
hours of user operation, 250 hours is down time. A MFD
of 1.2 h means that the machine in a year had 208
failures. (It was reported that Spring8 and ESRF are now
approaching 99% meaning 50 hours of downtime in ayear or about 50 faults). It is very important however thatthe faulty 5% should be smoothly distributed through theyear (otherwise it might happen that some users do notwork at all since 5% of 5000 is about 10 days) the figureof merit being MFD and it should be small. ESRF has it
0.8 meaning more faults at 95% but faster recovery.

Figure 11: Three years of component failure analysis for
ELETTRA
Table 1: Max. % of down-time for x-ray sources’subsystems% of down- ALS
ELETTRA ESRF
time
PS
>15
>15
>5
RF
>15
>5
>20
controls
>10
>10
>5
VAC
>35
<5
>25
Cooling
<5
>10
>15
BL
>5
>5
>5
other
>10
>5
<5
Amongst the many different reasons that a subsystem canfail there may be a correlation to the energy, size or
quantity of units and the operating mode of the source.
Thus ESRF and APS both high energy storage rings
suffer from failures of the rf system, whereas fromfailures of the power supplies suffer APS (too many) and
ELETTRA (ramping to the final energy). This mode ofoperating is prone to faults during refills since the main
ring power supplies and all other systems are stressed. Infact over 50% of failures happen during this period. The
control system failures appear independent being almost
the same for all machines. Human error can contributealso in a non-direct manner. For both ELETTRA and
ESRF the evident mishandling is around the 5% of thedown-time relatively low, no statistics exist for theindirect mishandling. “Other” is something to beeliminated, ELETTRA works towards this direction andno unspecified fault is tolerated. Finally all machines
suffer from the users, the percentage is low >5% for alltree machine and peaks when new lines are installed.
Storms and other external disturbances reduce the uptime
at about 2%. ELETTRA suffers from storms and network
micro-interruptions (20-25% of its down-time) with MFDof less than 2 hours. However although one could discuss
whether it is worth spending some millions $ to get
continuity generators for just a 2%, many forget thatthese interruptions can provoke major equipment failures3.3 Detecting trends
Trends may be revealed plotting the CNOF (cumulative
number of faults) of a subsystem against CTBF
(cumulative time between faults). In general repairable
systems have a quasi stable failure rate but trends are wellvisible.

89Y = M0 + M1*x + ... M8*x + M9*x
M0

0.032202

M1
M2

1.1005e-05-2.3887e-09

M3

2.2861e-13

M4-8.9871e-18

M5

1.2164e-22

R

0.48394

0.16
y = 0.012334 * x^(0.16359) R= 0.38406

200

0.14

y = -5.7232 + 0.005048x R= 0.99383

0.12

ring F/H

150

ps CNOF

100

0.1

0.08

0.06
50

0.04

0.02
0

0

5000

4

1 10

4

1.5 1042 10

4

2.5 103 10

4

4

3.5 10Acc. TBF
-50
0

5000

1 10

4

4

1.5 102 10

4

2.5 10

4

3 10

4

4

3.5 10Ac TBF

Figure 12: CNOF vs. CTBF for the PS of ELETTRA for
35000 hours of operation
This has to be well distinguished from stochastic (random
external ) disturbances that have similar trends and cangreatly influence the failure behaviour of the system, likethe graph below.

Figure 14: ELETTRA failure rate vs. CTBF for the for35000 hours of operation and its polynomial and exp fits.
Fit coefficients on the graph
External disturbances can be dominant for ELETTRA andwhen this occurs no good prediction can be achieved

8

y = -0.42245 + 0.0043325x R= 0.98551

9

Y = M0 + M1*x + ... M8*x + M9*x
M00.014244
M1-5.1202e-06

150

0.03

0.025

M2
M3
M4
M5

6.9239e-10-3.8023e-14
9.3404e-19-8.6022e-24

R

0.44707

0.02

Storms F/H

Storms CNOF

100

50

0.015

0.01

0.005

0

-0.005
0

5000

1 10

4

4

1.5 102 10

4

4

2.5 1043 10

4

3.5 10

0
0

5000

1 10

4

1.5 10

4

2 10

4

4

2.5 103 10

4

4

3.5 10Acc. TBFAc TBF

Figure 13: CNOF vs. CTBF for the storms and external
disturbances at ELETTRA for 35000 hours of operation
It is very important to be able to distinguish between thetwo cases.

3.4 Modeling and predicting trends
This may be achieved plotting and fitting the failure rates(failures / hour) or MTBF against CTBF. One may use
polynomial, power or exp fits.

Figure 15: Storm and external disturbance failure rate vs.
CTBF for the for 35000 hours of operation
From Fig. 14 we see that at about the end of 2002 a small
increase in failure rate occurs that it is not inconcomitance with external disturbances. The prediction
and trends are correct since new equipments are installed(a 3rd harmonic super conducting rf system) and while it iscommissioned the failure rate increases.
On the other hand other sub-systems reveal a tendency to
not create problems in the close future as the RF system

89Y = M0 + M1*x + ... M8*x + M9*x
M00.001585
M1
7.7781e-07

0.03

0.025

M2
M3
M4
M5

-1.808e-10
1.7883e-14-6.7509e-19
8.4535e-24

R0.32730.02

RF F/h

0.015

0.01

0.005

0

-0.005
0

5000

1 10

4

4

1.5 10

2 10

4

4

4

2.5 103 10

4

3.5 10Acc. TBF

Figure 16: RF failure rate vs. CTBF for the for 35000
hours of operation
Fitting by parts exp functions tendencies can be better
observed. Below the PS failure rates between 5 000 – 15000 hours is plotted and a good exp fit is obtained
0.02
y = 0.44021 * e^(-0.00051888x) R= 0.72636

PS F/H

0.01

0.005

0

-0.005
6000

8000

1 10

4

1.2 104

5 ACKNOWLEDGEMENTS
Many thanks to L. Hardy for numerous suggestions andhis trust revealing the ESFR statistics.

6 REFERENCES

0.015

4000

the aim of any operations group. Availability heavily
depends on budget but not only. Actions that appear ofadministrative nature like seriously programming any
activity, regular operations meetings, good statisticalanalysis not allowing for unspecified events, experienced
and good operators team, good personnel management
can help to gain a 2-3% in up-time. Maintenance andespecially the preventive one is very important too (butenters into the budget question) however there thereliability analysis can greatly help in prioritizing workand optimizing costs.
Machines that do not do (or have) “exotic things” (likeenergy ramping, individual PSs for each magnet, many
modes of operation) are prone to work better. Machines
are always in evolution and during major evolution steps
up-time suffers because not all aspects are known andusually not well analyzed. Availability can suffer byseemingly irrelevant actions. Thus careful planning andrigorous decisions on installations and innovations should
not be taken without discussing with the Operations.

4

1.4 104

1.6 10Acc. TBF

Figure 17: PS failure rate vs. CTBF for the period
between 5000-15000 hours

4 CONCLUSIONS
Availability concepts can help to make accelerators more
efficient. Early detection of trends and weak points are
essential in running a facility. It also helps to optimize theavailability services keeping costs low. For example
intervention speed, on-call service, maintenancescheduling, service speed, spare parts procurement andavailability, etc. can be optimized so that the usual
accelerator operating mode “run it until it breaks” (being
cheap) can also be efficient.
X-ray sources have in general high availability
levels i.e. above 92% Three examined sources had morethan 95% in the last years keeping also MFD low which is[1] L. Gavrilov and N. Gavrilova “ The reliability Theoryof Aging and Longevity”, J. Theor. Biology (2001)
213,527-545[2] A. Hurtel,” Product Assurance and management ofrisks in ESA spatial projects”, CNRS/ IN2P3/ LAL,
Accelerator Reliability Workshop, 4 February 2002,
ESRF.
[3] C. Piaszczyk, “Understanding Accelerator
Reliability”, Advanced Science & Technology Center,Northrop Grumman Corporation, Bethpage, NY (PAC
1999)
[4] L. Hardy, “Procs of Acc. Reliability Workshop”
Grenoble, Feb 2002