From: eajjs@cbnewsf.cb.att.com (john.sottile)
Subject: Re: Startus vs Sequoia vs Tandem (was Re: WHATZIT?)
Organization: AT&T
Date: Wed, 14 Oct 1992 06:58:04 GMT

In article <1992Oct14.001620.1548@mdivax1.uucp> robinson@mdd.comm.mot.com (Jim Robinson) writes:
>Vasu Nugala (vasu@anolis.bnr.usu.edu) wrote:
>>In article <1992Oct13.152416.19562@access.usask.ca>, choy@skorpio.usask.ca (I am a terminator.) writes:
>>|> In article <1beg0iINNpes@bigboote.WPI.EDU>, beuf@wpi.WPI.EDU (Richard J Fuller) writes:
>>|> |> A Stratus is a Fault-Tolerent Computer that is on the ofder of
>>|> |> $500,000 - $5,000,000 and is most likely to be found in a 
>>|> |> bank, a mail order house, or an airline (stuff like that)
>>|> 
>How about Tandem? Anybody have any comments on it in comparison to Stratus
>and Sequoia. We have a product running on Sun SPARCstation that we will
>probably be wanting to port to a fault tolerant UNIX environment. Thus, I
>would be interested in hearing comments from anyone who has used the UNIX
>versions of these systems. We have *no* interest in guardian and other
>proprietary fault tolerant OSes.
>

Note to the Reader:
  This message is long and describes my experience with Tandem's S3 Integrity
Fault Tolerant Computer.

I have been "fooling" around with Tandem's S2 and S3 Integrity.  A UNIX(r)
System V R 3.1.1 OS with fully fault tolerant hardware and OS
configuration.  In our configuration, we have 2 pair of mirrored 1GB Drives
(4 drives), duplicated SCSI Controllers, Triple CPU (23-30 MIPS), all buses
are also duplicated, 96 MB of memory (64 per cpu, 2 32MB "global memory cards,"
reliable ethernet, and duplicated batteries.  

The Main Cabinet is about 2 feet high, 2 feet wide and 3 feet long.  The 
Main Cabinet houses 2 power supplies, 2 IOP (I/O Processors), 
2 SCSI Controllers (1 Differential, 1 Single Ended), 3 CPUs, 
2 32MB memory cards, 2 ethernet contollers, DCC (AC to DC Convertors), 
and 2 batteries.  We have only 1 Mass Storage Cabinets (MSC), although 
you can have more.

The MSC that we have houses 4 1GB drives (2 mirrored pairs) which I can 
mirror all partitions or selected partitions onto the mirror, 
1 150 QIC Cartridge Tape Drive, 1 DAT 1.3GB Tape Drive and 
1 Empty Bay (7 Bays total).  

The Operating Systems is SVID 3 (System V Interface Definition) 
Compliant with Berkeley Extensions (sockets included).  Tandem has "hardened"
the OS Kernel to ensure integrity at the OS Level.  We run NFS, TCP/IP,
Oracle(r), etc on the Tandem.  We had the option to install X, OSI, etc
but chose not to since we really didn't need them.  This machine (and several
like it) commincate with SUN Sparcs (1, 1+, IPC, IPC, 2), 80386/80486,
AT&T 3B15s, etc over ethernet with the TCP/IP protocol Stack.

We were trialing the machine with our application here in California and 
were fortunate to test the fault tolerant nature of the whole package.  Here's
what we found:

Simplex = Runs on one (the other) component

User Action          Machine Reaction                  User Impact
-----------          ----------------                  -----------
Pull 1 CPU           Goes Simplex, Continues to        None
                     run all processes with no
                     interruption.

Restore CPU          Diagnose, if Passes, will         up to 10 second delay
                     "reintegrate" the CPU with        when copying the 64MB
                     the other 2.                      of memory from 2 other
                                                       CPUs

Pull 1 Disk          Simplex, No Data Lost             None

Restore Disk         Disk Goes Online, Administrator   Disk Contention will
                     must tell the machine to          rise on the disk
                     reattach the mirrors. (I suppose  during the re-mirror,
                     this is just in case it was a     however the delay was
                     new disk which hasn't been        no noticeable on all
                     formatted or partitioned)         but our busiest File
                                                       System.  No other user
                                                       impact.

Pull IOP             IO Boards (SCSI, Ethernet)        None
                     switch to Other IOP

Restore IOP          Administator issues a cfmove      None
                     command when the IOP is restored
                     to "move" SCSI/Ethernet back to
                     its original IO

Pull SCSI            2 mirrored drives on that         None
                     controller go offline.  Disks
                     and SCSI Controller are Simplex

Restore SCSI         Disks go Online, Admin issues     Disk Contention while
                     the command to reattach the       mirrors are reattached
                     mirrors.                          No Other Impact.

Pull Ethernet        Machine "switches" to other       None.  If a packet was
                     Controller.                       lost, it was 
                                                       retrasmitted.

Restore Ethernet     Admin can "switch" back or        None
                     continue with other controller.

Pull Main Power      1 of each Component will go       None
Supply               offline since 1 power supply
                     cannot supply all cards' power

Restore Main Power   Those cards above which require   None other than disk
Supply               Admin intervention will require   Contention Mentioned
                     same intervention.  We typed      above.
                     3 commands, although we could
                     have used "MIDAS(r)" which is
                     a menu driven Maintenance and
                     Diagnositics System.  It shows
                     a real-time view of component's
                     status and configuration.

Pull MSC Power       All 4 disks stayed online.  The   None
Supply               remaining power supply could
                     power all drives.

Restore MSC Power    Came back online, no Admin        None
Supply               intervention.

Pull Plug From       Power configuration in our        None until the machine
Wall                 setup kept the machine running    shutdown, then users
                     for 5 minutes then shutdown       were "kicked" off.
                     gracefully.  There is an option
                     to save the machine state instead
                     of shutting down which allows the
                     machine to resume where it left
                     off when power is restored.  We
                     figured a power failure (we have
                     UPS also) was a great time to
                     reboot.

Plug Power Back in   Returns to "PROM" mode, starts    Once rebooted, users
                     identifying cards, etc, then      could log in again.
                     will start to boot.  The machine
                     will wait for the batteries to
                     recharge before booting unless
                     the admin overrides OR the
                     max wait time is exceeded (as
                     setup in the power 
                     configuration)

Pull both "sides"    Machine Crashes (as expected)    Users get dumped off.
of SCSI, DISKs,      Note: When we pulled the         
2 of 3 CPUs          non-System disk (our 
and/or both          application disk), the 
IOP                  machine kept running but the
                     file systems on that disk
                     were invalidated (unavailable)

Restore both         Machine retured to PROM mode     Machine rebooted, users
"sides"              started booting, file system     logged back in.
                     checked all file systems
                     then started normally.  We
                     didn't lose any data in our
                     tests.

Do Jimmy Hendrix     Machine burned, Purple Haze      Users organized a coop,
Test.                played, the fire extinguishing   overthrew my government,
                     system truned on and I was       beheaded my mistress,
                     thrown out of the building :-)   and set me adrift. :-)


For the Sun workstations, during the crashes (we did 7 tests in all)
when the machine was available again, they "kept on truckin'" with 
their NFS requests (retrans).  The NFS on the Tandem is stateless as it
is on the SUN (of course, since SUN invented it in the first place), so the
SUNs didn't need to reboot.

According to Tandem, no single component failure will crash the machine 
nor will the user notice the machine react.  Alarms are generated on
the console about the changes in status (online/offline/errors, etc).  The
machine can be configured to call the vendor when it detects an error/fault.
In all of our tests, these held true.  The machine really impressed me.  I
really like it for what it does in our application.  I think that
our configuration above was about $300,000 US.  We have a lot of "stuff"
on it.  Our customer was just as impressed (they also tested the Stratus
Machine and liked this one better -- they didn't test the Sequoia and 
I haven't either).

Tandem says that it needs only an office environment (no special 
air conditioning).  We put it in our computer room anyhow which usually
runs 80-90 degree Fairenheit (our AC is on the blink).  We feel we gave
this a good test since the main office room was a lot cooler than the
computer room (go figure).  We also had the machine earthquake braced
(as required here in California) but we didn't have any quakes during
our trial to test with.

Those are the advantages.  The only disadvantages I could find were
that the machine we had was almost too small.  At 2 feet tall (the machine,
not me), I had to bend down to put tapes into the tape drives.  I'm not
tall by any stretch of the imagination, but I don't like leaning over to
feed my computers.  It made for a good seat when doing demos, however.

Also, for our application, the S3 had more than enough power, but I
think that is the top of the line Integrity (30 Mips).  It uses a Mips
R3000 Processor and VME Bus.  I don't know of Tandem's Plans for
the future of this particular machine, however, I have seen in a 
recent Uniforum(r) Newsletter the following:

HP/Sequoia will go with the PA-Risc in The future.  I don't remember what
they use right now (68040 but don't quote me).

IBM/Stratus is looking at the PA-Risc also.  I think thaty they threw out the
idea of the ALPHA Chip.  I believe that they are currently
Intel i860 RISC Based.  I know this platform runs IBM's VOS Operating
System, but I have heard that it runs or will run some dialect of UNIX(r) --
probably Ultrix.

Tandem, I think, is debating the MIPS R4000 processor vs the PA-RISC.  I 
also understand that they are working on their UNIX System V Release 4
hardened OS for this same platform (R3000 S3 Integrity).


Anyhow, the usual hub-bub about
"The ideas expressed in this message are those of John Sottile and in
 no way reflect the position of AT&T or any other person for that matter."


=============================
John Sottile                             "That Guy has to Oil his Brakes!"
AT&T Bell Labs                           #include <std/disclaimer/who-me?.h>
jjs@cblph.att.com