From: michael@cutler.com (Michael D. Shields)
Newsgroups: comp.databases.ingres
Subject: Re: Transaction system reliability
Date: 7 Oct 92 16:34:26 GMT
Article-I.D.: cutler.1992Oct7.093426.301
Organization: Cutler & Company, Inc., Medford OR (503) 770-9000

In article <9210061409.AA11415@valandil.YP.sun386i>, mail-to-news-gateway@cv3.cv.nrao.edu.ogi.edu writes:
> 
>  ...
> 
>   When you call iishutdown, it returns a status as
> to how it went.  If it returns zero, which is indicated
> as OK in the header of the iishutdown script, IT DOES
> NOT MEAN INGRES HAS SHUTDOWN SUCCESSFULLY!
> 
>   I repeat:  IT DOES NOT MEAN INGRES HAS SHUTDOWN SUCCESSFULLY!
> 
>   Ingres is still shutting down.  In MOST cases, you
> may not notice this, because under normal circumstances, 
> Ingres shuts down quite quickly.
>  
>   However, if something needs to be recovered, the dmfrcp
> process may still be running.
> 
>   It is very dangerous to shut down the operating while 
> the dmfrcp is still running.
> 
>   So, in any automatic system shutdown script, it is NOT
> enough to call iishutdown, and upon zero return status, 
> call the operating system shutdown.  You MUST make sure
> dmfrcp has completely shut down.

Back when IISHUTDOWN did not exist, I, like probably many of you, wrote my own
IISHUTDOWN tailored to my cluster environment.  My procedure shuts down as
gracefully as possible (SET SERVER SHUT instead of STOP SERVER, etc.).
Moreover, my procedure does not exit until all INGRES processes have
vanished.  If you're interested in this VMS proocedure, let me know.

>   One other neato implication Ingres' statement has:
> 
>     1) You should not allow your operating system to 
>        ever crash.  
> 
>     2) Your power supply must be perfect, and NEVER
>        fail, under any circumstances.
> 
>     3) Your hardware must never fault, in any way.
> 
>     4) You must never shut down the operating system
>        without shutting down Ingres first (and making
>        sure it is down, as stated above).
> 
>    If you cannot guarantee all of the above, Ingres will not 
> guarantee it can restart normally.

Now, now.  They may not guarantee, but the probability of a successful restart
is most likely upwards of 999,999,999 out of 1,000,000,000 events.

We had an interesting occurrence here the other day.  I SUSPENDed one of our
PSQ (power sucking query) threads.  Unfortunately, the thread was apparently
involved in a cluster deadlock resolution at the moment I suspended it.  The
result was every server hung throughout our cluster.  A call to technical
support indicated that the only resolution was a STOP/ID on the server
containing the suspended thread.  After a brief moment of prayer, I issued
the command and watched the other servers and the recovery process come to
life.  Then the worst.  We lost power as the recovery process was doing it's
job.  (A UPS is in next year's budget.)  When I got TS back on the phone,
their response was "Oh, you lost power, oh."  As the cluster rebooted,
I was pleased to see all of INGRES intact and all databases consistent.

>    To be fair, Ingres can and should not guarantee anything
> if disk corruption occurs.

For this case, and the once in 1,000,000,000 when the restart is
unsuccessful, use your checkpoints.  That's what they're there for.

                    DO YOUR CKPDBs.

>    Please call me if you can assure 1-4, as I would like to 
> find out how.

Get real.

  Michael

Michael D. Shields      |  Internet: michael@cutler.com
Vice President, IS      |  UUCP: ...hplabs!hp-pcd!orstcs!cutler!michael
Cutler & Company, Inc.  |  Voice: (503) 770-9000
Medford, OR  97504      |  FAX: (503) 779-0006
Newsgroups: comp.databases.ingres