From: karl@ddsw1.mcs.com (Karl Denninger)
Subject: Re: 500,000 hour MTBF!!!
Date: Sun, 13 Dec 1992 04:34:12 GMT
Organization: MCSNet, Chicago, IL

In article <5csFVB1w164w@hub.parallan.com> danj@hub.parallan.com (Dan Jones) writes:
>karl@ddsw1.mcs.com (Karl Denninger) writes:
>
>> A little suspiciously?  More like with much suspicion.  Most current MTBFs
>> are little more than marketing drek.  I've dealt with installations where
>> there were several hundred drives, and not one of those installations met
>> MTBF claims across the population.
>
>This is an area where it is difficult to get hard information. It interests
>me, because MTBF is the basis for making the cost trade-off between RAID
>and striping. I remember making my first reliability calculations in the 
>days of 40,000-50,000 hr MTBFs for disk drives and there was no question 
>that some form of protection was needed. Nowadays, it is a more difficult
>sell due to the amazing ten fold improvment in drive spec'ing.
>
>Garth Gibson was only able to refer to two disk reliability examples for 
>his dissertation on RAID, "Redundant Disk Arrays: Reliable, Parallel Disk 
>Storage." Both installations, one DEC and one IBM, showed about a 50,000
>hour MTBF for disk units. Interesting; but one, not a very large sample and 
>two, these were fairly complex disk units. Unfortunately, takes a 
>significant economic incentive to keep good records.
>
>So, Karl, do you want to write a paper? I'd be glad to look over your first
>draft. :-)

As an example, at one site I am very familiar with they had something like
300 PCs in the building and the failure rate on a particular CONNER drive in
those machines (all one brand, all one model, etc) was something like one
per week on average over a one-year time period.

That's 50,400 powered-on hours, assuming continuous operation, per failure,
or an MTBF of ~50,000 hours.  Considering that many of these machines were
getting more like 20-30 power-on hours per week the real numbers are
considerably worse.  I am being charitable.

I believe the published numbers were either 100,000 or 200,000.  In any
event we didn't even get close to those numbers. 

However, one must realize that 50,000 hours MTBF >sounds< real good until
you have a couple of hundred machines!  Then it is a TERRIBLE statistic from
a user support perspective!  This one particular location had a stack of
failed hard disks that was over 4 feet high, and we were tossing a good
number of 'em.  The stack was a rather impressive sight in the techie area
though ;-)

Over large populations (100+) you start to see some interesting REAL
MTBF numbers.  When one gets 300-500 disks in a facility you find out that 
even with outrageously >good< MTBFs (REAL ones) you don't get that far before 
one croaks.  Given 50,000 hours across 300 machines in a week you can see
just what you're battling here.  

Now, this is the reason that REGULAR backups are EXTREMELY important.  In a
large installation I'd go so far as to say you're nuts not having every
critical business-related file on a central file server where you can be
CERTAIN it is backed up.  Otherwise you >will< have real users losing real
data with an impact on your business weekly.  This is NOT fun, and is the
best justification I've seen for a managed network operation.

Consider that if you need 1GB of storage for 50 machiens you can either buy
50 20MB fixed disks or one 1GB disk.  IF the one 1GB disk fails you're in
trouble (you could always mirror it in a server to guard against this), but 
one of the 50 WILL fail 50 times as often, MTBFs being equal!

Considering that disk drives typically have a failure curve which looks like
this:

F
A	    X
I	    X
L	    X
U X        X
R  X       X
E  X      X
S  X	  X
   XXXXXXX
T   I    M    E

You see the real problem.  That particular population I was familiar with
had the machines for a little over a year when I got involved with them; as
a result, they were in the trough.  I don't even want to think what the
failure rate looks like now, 2 years later.....

I also used to service two major Chicagoland accounts for PC repairs.  One 
had over 300 PCs installed, the other about 150.  My actual >measured<
MTBF across those populations NEVER exceeded 100,000 hours, even with the
better PC drives.  Surprisingly enough, the only real variable was that 
the cheaper drives were severely impacted by power cycling, leading me to
recommend 24-hour power as a possible way to lower repair costs given that
many of these machines were purchased with inexpensive fixed disks.  It 
proved out; their MTBF went up ~50% if the systems weren't powered on and 
off daily (power off only on weekends, and every system gets a quality 
surge protector WITH verified proper grounding to guard against power
problems).

Now, the reason for the "500,000 hour" claims:

	The makers are, of course, counting on you being in that trough!  
	Sure, you'll see 500,000 hours across a large population when you're 
	in there with some current model units.  But as the systems age you'll
	come out of the trough, and those failures you did not expect WILL 
	happen.  You will not be prepared.  Trust me.
	
The failures at end of life-cycle tend to be catastrophic (bearing seizures, 
spindle motor failure, lost servo tracks, insane electronics overwrites
data, etc) rather than the "soft" R/W failures that are more frequent 
mid-life-cycle.

One other interesting point -- I used to handle a line of systems, the
TELEVIDEO "TeleCAT 286" systems, that racked up a rather impressive >system
level< MTBF.  Out of over 100 shipped I had >one<, count it -- ONE --
system failure (not including disk drive or keyboard) past infant mortality, 
and I believe two inside infant mortality.  This includes the power supply
and everything inside the case, and is over a 2+ year time period.  That's 
an ACTUAL >measured< MTBF of 584,000 hours.  As it spanned the three year
amortization time of the units (most computer gear is depreciated over 3
years) I consider this to be a quite accurate and useful number.

Damn impressive; the best I've seen before or since.  BTW, the MTTR (mean 
time to repair) on those failures was < 30 minutes in all cases but one -- 
the one out of warranty failure was caused by a cockroach (!) infestation 
which precipitated a short circuit in the logic (cockroaches have to go to 
the bathroom too).  Yes, it was disgusting :-)

Discounting the cockroach infestation the MTBF was 876,000 hours.  I think
you can see why both I and my customers loved those machines.  To this day 
I haven't seen documented numbers replicating this kind of reliability from
ANY computer system line.

--
Karl Denninger (karl@ddsw1.MCS.COM, <well-connected>!ddsw1!karl)
Data Line: [+1 312 248-0900] Anon. arch. (nuucp) 00:00-06:00 C[SD]T
Request file: /u/public/sources/DIRECTORY/README for instructions
Newsgroups: comp.periphs.scsi