Abstract:
When a fault-on-fault condition arises in a data processing system which follows a backup fault procedure in the fault handling process, control is passed to dedicated firmware. Fault flags are reset and information vital to maintaining operating system control is sent to a reserved memory (which can be written to in limited circumstances) under firmware control. Control is then transferred to an Intercept process resident in the reserved memory which attempts to build a stable environment for the operating system to dump the system memory. If possible, a dump is taken, and a normal operating system restart is carried out. If not possible, a message with the vital fault information is issued, and a full manual restart must be taken. Even in the latter case, the fault information is available to help in determining the cause of the fault-on-fault.

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This application is related to our co-pending patent applications entitled: “FAULT HANDLING IN A DATA PROCESSING SYSTEM UTILIZING A FAULT VECTOR POINTER TABLE”, application Ser. No. 09/742,457, filed Dec. 20, 2000, and assigned to the assignee hereof, now U.S. Pat. No. 6,697,959; and “FAULT VECTOR POINTER TABLE”, application Ser. No. 09/742,456, filed Dec. 20, 2000, and assigned to the assignee hereof, now U.S. Pat. No. 6,687,845. 
    
    
     FIELD OF THE INVENTION 
     The present invention generally relates to data processing system fault handling and more specifically to preserving the ability to obtain a valid dump printout for analysis during certain operations, most particularly after the occurrence of a fault-on-fault condition and also to increasing the chances that a useable dump can be obtained and a full system restart avoided after processing a fault-on-fault. 
     BACKGROUND OF THE INVENTION 
     In a typical data processing system, input and output completions are typically signaled by interrupts. This concept was extended to cover other external as well as internal events. Herein, a distinction will be made between responding to external events, herein termed “interrupts”, and responding to internal events, herein termed “exceptions” or “faults”. It should be noted that the distinction between interrupts and exceptions or faults is somewhat arbitrary, as some architectures do not make such a distinction. 
     An exception then is the happening of an internal event within a computer within a data processing system. Exception handling is the action taken by a computer processor to respond to the exception. Some typical exceptions are page faults, zero divide, supervisory call, illegal instruction, privileged instruction (when not in a mode allowing execution of such), security violations, timer or decrementer expiration, and traps. Other exceptions are within the ambit of this disclosure. 
     Typically, exception handling or exception processing involves diverting control or instruction flow from where the computer processor was executing prior to the exception to an exception handling routine. Typically again, there will be a different exception handling routine for each exception type and even subtype. The exception handling routines are typically a portion of the operating system controlling each computer processor in the data processing system. The exception handling routine for a given exception will typically be programmed to determine how to handle a particular exception type. For example, the task that attempts to execute a privileged instruction, performs a security violation, or a zero divide, will typically be aborted by the operating system, after providing for the possibility of dumping the job containing the task. On the other hand, in the case of a page fault, the operating system will typically suspend the task causing the page fault, initiate reading the requested page of memory from disk, and dispatch another task to execute. The task causing the page fault will be re-dispatched later after the missing page has been retrieved from disk. In the case of expiration of a timer, the executing task is placed on a dispatch queue, and another task is dispatched. 
     It should be noted here that the above mechanisms require that the exception handler save the current execution environment in the computer processor so that it can be returned to at some later time. Upon completion of exception processing for a given exception, control is returned to the saved environment, typically at either the instruction causing the exception (for example in responding to a page fault), or at the next instruction after that instruction (for example in responding to a supervisor request). Indeed, this mechanism is the fundamental method used by the dispatcher in a modern operating system to accomplish dispatching of tasks. Partly this is done through the fairly complete control over the information in the saved environment of a task that the operating system has. 
     Since exception handling is typically part of the operating system controlling a data processing system, and since exception handling routines typically require almost full control of the computer processor, including the ability to execute privileged instructions, and to read and write almost all memory, exception handling routines will typically be entered with the highest possible privilege level. Typically this means that exception handling will be entered in a pre-specified maximum security mode. 
     In order for a computer processor to respond to an exception, it must be aware of the location of the appropriate exception processing routine. In some data processing systems, such as GCOS® 8 from the assignee of this invention, the entry descriptor for a general exception or fault handling routine is retrieved from a specified location (octal 032) in memory and evaluated. The entry descriptor specifies the environment for the exception processing routine, including which segments are visible, the routine starting address, and what privileges to enable. It is treated by the computer processor almost like an ICLIMB subroutine call, laying down a Safe Store Stack Frame containing the saved environment. An OCLIMB instruction can be later executed to return control back to the location of the exception or fault. Within the fault handling routine (titled “Fault”), a determination is made as to the fault (or exception) code causing the exception. This then is used to invoke the appropriate exception processing routine for that type of fault, again with an “ICLIMB” instruction. 
     Other mechanisms are typically used in less secure data processing systems. For example, in the Intel X86 architecture, there is a fault or exception vector stored at a specified location in memory containing a number of exception handling routine addresses. When an exception occurs, control is transferred to the address at the specified location in the exception vector corresponding to that exception type. As noted above, the environment of the exception handling is automatically set to a pre-specified maximum security state. Most of the environmental saving and restoring required is done by general purpose instructions that store and later load processor registers. 
     Somewhat more sophisticated is the exception processing in a Motorola or IBM PowerPC® processor environment. Instead of having an exception (or fault) vector containing addresses of exception handling routines, the exception handling routine for each exception handling type begins execution in response to the occurrence of the exception being handled, at the first word in a block of memory at a specified location in memory. Each exception type has its own block of memory starting at its specified location in memory. The PowerPC architecture contains a couple of enhancements in sophistication over the X86 architecture discussed before. First, instead of one set of exception routine routines or exception vector, there are two. The selection of which of the two to utilize is determined by a static bit in a reserved status register in each computer processor. Typically, one set of exception routines are utilized at system startup. The bit is then toggled, and the other set of exception routines is then utilized thereafter. Second, instead of always initiating exception processing with the same high security environment, the PowerPC architecture specifies slightly different processing environments for the start of exception processing for different exception types. 
     Other data processing system architectures utilize similar mechanisms to the above. 
     There are problems with all of the above mechanisms. One problem with the GCOS 8 mechanism disclosed above is that it requires the equivalent to two ICLIMB instructions to enter the appropriate fault or exception handling routine, and two OCLIMB instructions to return. These are some of the most expensive instructions in the GCOS 8 processor instruction repertoire to execute in terms of computer instruction cycles, typically taking over 100 cycles each to execute. Thus, it would be preferable to be able to perform fault processing more efficiently, with the expenditure of fewer instruction cycles. 
     Both the X86 and PowerPC approaches suffer from being unable to automatically fine tune the processor environment to the exception type being processed. Thus, with the minor exceptions noted above for the PowerPC architecture, all exception handling in both architectures begins execution in the identical processor environment. This means that the same memory is visible to all fault handling routines, as well as most (PowerPC) or all (X86) of the same processor privileges are in effect. 
     One problem that is common to all three approaches or mechanisms is that in certain instances, the exception vector or exception handling routines are mistakenly overlaid by other data. This is compounded because these are typically in physical memory with low fixed addresses. In the X86 environment, given its minimal security, this overlaying happens frequently. However, even in the most secure operating system, such as GCOS 8, it still happens. One major cause of this is issuance of erroneous input/output (I/O) requests. 
     The problem that this causes is that exception processing will thereafter fail, when the processor is unable to either find the required exception processing routines, or if it can find such, it cannot execute them, as they no longer exist, having been overwritten. This sort of problem is often hard to diagnose since one of the functions that can result from exception processing is the generation of a dump of the processor and its memory. No exception processing typically means no dump. One advantage of the higher security GCOS 8 architecture is that overlaying of the entry descriptor for the fault handler is easily detected as it typically no longer is a valid entry descriptor. 
     When a computer processor causes an exception or fault while processing an exception or fault, it is termed here “fault-on-fault”. In the prior art, this typically ultimately resulted in halting the computer processor, if not explicitly, at least implicitly. In the above scenario, when either the exception vector, or the exception processing routines, are overlaid, even when exceptions are prioritized, the processor will ultimately end up attempting to process some exception while in the process of processing that very same exception. For example, if the exception handling routines have been overlaid, then the processor will (hopefully) recognize an illegal instruction exception while executing code in the overlaid area. If this in turn results in attempting to execute code in the overlaid area, recovery is impossible. 
     The GCOS 8 architecture does provide a partial solution to the “fault on fault” problem outlined above. When a program fault or exception is detected during fault processing, a second fault or exception handling routine is invoked, instead of the first one described above. It is entered by loading and evaluating a second entry descriptor located at another specified location in memory. However, this is not a complete solution since it sometimes happens that the same situation that resulted in the second fault (the “fault within fault”) also resulted in either the entry descriptor for the second fault handler being overlaid, or the code for the second fault handler itself being overlaid. 
     The fault handling procedures set forth in the above-identified related patent applications provide significant improvements in the art of fault handling in fault tolerant data processing systems. However, conditions remained in which it was impossible to obtain a valid dump to provide insight into a system failure, particularly those caused by software errors. The present invention serves to significantly enhance the chances that a valid dump can be obtained when a fault-on-fault condition occurs with the additional facility that the dump can be rendered automatic and can lead to an operating system restart rather than the need for a full system boot requiring direct operator intervention. 
    
    
     DESCRIPTION OF THE DRAWING 
     The features and advantages of the present invention will be more clearly understood from the following detailed description taken in conjunction with the accompanying FIGURES where like numerals refer to like and corresponding parts and in which: 
     FIG. 1 is a block diagram illustrating a General Purpose Computer, in which the present invention may be employed; 
     FIG. 2 is a block diagram of a more detailed view of a multiprocessor data processing system in accordance with the present invention; 
     FIG. 3 is a block diagram illustrating a processor (CPU) module as shown in FIG. 2; 
     FIG. 4 is a block diagram of a processor shown in FIG. 3; 
     FIG. 5 is a block diagram of an AX unit in the processor shown in FIG. 4; 
     FIG. 6 is a block diagram of the interrupt structure for each processor in FIG. 3, in accordance with the prior art; 
     FIG. 7 is a block diagram of the interrupt structure for each processor in FIG. 3; 
     FIG. 8 is a block diagram illustrating the data structures utilized in FIG. 7; 
     FIGS. 9 and 10 are diagrams that illustrate the format of two different types of descriptors in a GCOS 8 environment. 
     FIG. 11 is a process flow chart of the fault handling process employed in the systems described in the above-identified related patent applications and with an added exit under certain conditions to the process set forth in FIG. 12; and 
     FIG. 12 is a process flow chart particular to the present invention. 
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     A fault number is utilized by microcode fault handling to index into a fault array pointer table containing a plurality of pointers to entry descriptors describing fault handling routines. The pointer resulting from the indexing is utilized to retrieve an entry descriptor. The entry descriptor is verified and if valid, is utilized to setup the environment for the appropriate fault handling routine and to enter such. The fault array pointer table is located in a reserved memory that cannot be overwritten by I/O. During the boot process, the fault array pointer table entries, along with a fault-on-fault pointer are updated to point at entry descriptors stored in the reserved memory. Additionally, the fault-on-fault entry descriptor rebuilds the processor environment, if necessary, from information in reserved memory. 
     In the following description, numerous specific details are set forth such as specific word or byte lengths, etc. to provide a thorough understanding of the present invention. However, it will be obvious to those skilled in the art that the present invention may be practiced without such specific details. In other instances, circuits have been shown in block diagram form in order not to obscure the present invention in unnecessary detail. For the most part, details concerning timing considerations and the like have been omitted inasmuch as such details are not necessary to obtain a complete understanding of the present invention and are within the skills of persons of ordinary skill in the relevant art. 
     The term “bus” will be used to refer to a plurality of signals or conductors which may be used to transfer one or more various types of information, such as data, addresses, control, or status. The terms “assert” and “negate” will be used when referring to the rendering of a signal, status bit, or similar apparatus into its logically true or logically false state, respectively. If the logically true state is a logic level one, the logically false state will be a logic level zero. And if the logically true state is a logic level zero, the logically false state will be a logic level one. 
     FIG. 1 is a block diagram illustrating a General Purpose Computer  20 . The General Purpose Computer  20  has a Computer Processor  22 , and Memory  24 , connected by a Bus  26 . Memory  24  is a relatively high speed machine readable medium and includes Volatile Memories such as DRAM, and SRAM, and Non-Volatile Memories such as, ROM, FLASH, EPROM, EEPROM, and bubble memory. Also connected to the Bus are Secondary Storage  30 , External Storage  32 , output devices such as a monitor  34 , input devices such as a keyboard (with mouse)  36 , and printers  38 . Secondary Storage  30  includes machine-readable media such as hard disk drives, magnetic drum, and bubble memory. External Storage  32  includes machine-readable media such as floppy disks, removable hard drives, magnetic tape, CD-ROM, and even other computers, possibly connected via a communications line  28 . The distinction drawn here between Secondary Storage  30  and External Storage  32  is primarily for convenience in describing the invention. As such, it should be appreciated that there is substantial functional overlap between these elements. Computer software such test programs, operating systems, and user programs can be stored in a Computer Software Storage Medium, such as memory  24 , Secondary Storage  30 , and External Storage  32 . Executable versions of computer software  33 , can be read from a Non-Volatile Storage Medium such as External Storage  32 , Secondary Storage  30 , and Non-Volatile Memory and loaded for execution directly into Volatile Memory, executed directly out of Non-Volatile Memory, or stored on the Secondary Storage  30  prior to loading into Volatile Memory for execution. 
     FIG. 2 is a block diagram of a more detailed view of a multiprocessor data processing system, in accordance with the present invention. The multiprocessor data processing system  80  comprises a plurality of modules coupled together via an intra-module bus  82  controlled by a storage control unit  86 . In the preferred embodiment, each such module  84 ,  88 ,  90  is contained on a single board, with the boards connecting into a backplane. The backplane includes the intra-module bus  82 . In the representative data processing system  80  shown in FIG. 2, sixteen modules are shown. The system includes four (4) processor (“CPU”) modules  90 , four (4) Input/Output (“IOU”) modules  88 , and eight (8) memory (“MMU”) modules  84 . Each of the four Input/Output (“IOU”) modules  88  is shown coupled to secondary storage  30 . This is representative of the function of such IOU modules  88 . Each IOU module  88  will typically contain a plurality of IOU processors (not shown). Each of the eight memory modules  84  contains memory  24  and a memory controller (not shown). This memory  24  is typically Dynamic Random Access Memory (DRAM). Large quantities of such memory  24  are typically supported. Also shown in FIG. 2 is a Clock Management Unit  98 , which supplies a standard clock signal  99  to the remainder of the system  80 . As clock signals are ubiquitous in digital computer architectures, the clock signal  99  will not be shown further herein except where relevant. Note also that in the preferred embodiment, multiple Clock Management Units  98  are utilized to provide a redundant clock signal  99 . 
     Also b-directionally coupled to the intra-module bus  82  are a service processor (SP)  87  and reserved memory  85 . The service processor  87  is utilized to perform maintenance on the system  80 . It controls partitioning of processors  92 , IOUs  88 , and MMUs  84  into multiple system images, as well as determining which major components are available to which system at which time. Though not shown here, the SP  87  typically also contains nonvolatile storage to maintain static systems configuration information. It also typically contains a modem allowing remote support systems to be contacted automatically whenever an error is detected in the system  80 . 
     Reserved memory  85  is similar to the memory installed in the MMUs  84 , with the limitation that it is “write” protected except in special situations. The processors  92  utilize special instructions to write to reserved memory  85 . Additionally, reserved memory  85  cannot be written by Input or Output (I/O) operations. The SP  87  starts each processor  92  with a “Connect” command. Prior to this, it has initialized a processor specific area of the reserved memory  85  for that processor  92 . The processor  92  utilizes this processor specific area of reserved memory  85  to determine what channels and peripherals are configured as being connected to it. 
     FIG. 3 is a block diagram illustrating a processor (CPU) module  90  as shown in FIG.  2 . The CPU module  90  contains a plurality of processors (CPU)  92  and a cache memory system  94 . In the preferred embodiment, each processor (CPU) module  90  contains up to four (4) processors (CPU)  92 . The processors  92  and the cache memory system  94  are coupled together and communicate over an intra-processor bus  96 . 
     The cache memory system  94  is shared among the processors  92  on the CPU module  90  and maintains cache copies of data loaded into those processors  92 . The cache memory system  94  is considered here a Level  2  cache and is coupled to and communicates with the storage control system (SCU)  88  over the intra-module bus  82  in order to maintain cache coherency between Level  1  cache memories  94  on each of the processor modules  90 , as well as between cache memories  54 ,  56  in each of the processors  92 , and on the IOU modules  88 . The SCU  88  also maintains coherency between the various cache memories  94 ,  54 ,  56 , and the typically slower speed memory in the MMU modules  84 . In the preferred embodiment, a single block of memory will be owned by a single cache or memory at potentially each level in the memory hierarchy. Thus, a given memory block may be owned by one Level  1  cache  54 ,  56 , by one Level  2  cache  94 , and by one MMU  84 . 
     FIG. 4 is a block diagram of a processor  92  shown in FIG.  3 . The processor  92  communicates with the bus  96  utilizing a bus interface  78 . The bus interface is b-directionally coupled to a unified local cache  256 . Cache memories, such as this unified local cache  256 , are typically constructed as high speed Static Random Access Memories (SRAM). In the preferred embodiment, the local cache  256  is incorporated on the same integrated circuit as the remainder of the processor  92 . The local cache  256  is the primary block that interfaces with the bus interface  78 . Data and instructions are loaded via the bus  96  into the local cache  256 , and data is written back from the local cache  256  via the bus  96 . 
     The local cache  256  is b-directionally coupled to an AX module  260 . The AX unit  260  provides the bulk of the functionality of the processor  92 , including instruction decode. The AX unit  260  is b-directionally coupled to and controls execution of a floating point (FP) unit  268  and a decimal/numeric (DN) unit  262 . In the preferred embodiment, the floating point unit  268  performs both floating point operations, and fixed point multiplications and divisions. It is b-directionally coupled to the local cache  256 . The decimal/numeric (DN) unit  262  performs decimal and string operations. It is b-directionally coupled to the local cache  256 , allowing it to operate relatively autonomously from the AX unit  260 . Rather, once decimal or string operations are initiated in the DN unit  262 , the DN unit  262  is driven by operand availability in the local cache  256 . 
     B-directionally coupled to both the AX unit  260  and the local cache  256  is a Trace RAM cache  58  which is capable of caching the status of instruction or cache operation. The Trace RAM  58  is controlled by commands decoded and executed by the AX unit  260 . The Trace RAM  58  also selectively traces AX unit  260  statuses. The Trace RAM  58  receives and selectively traces cache state signals from the local cache  256 . When a Trace is complete, the Trace RAM  58  can be written out to the local cache  256 , and ultimately to slower memories. 
     FIG. 5 is a block diagram of an AX unit  260  in the processor  92  shown in FIG.  4 . The AX unit  260  comprises a Microprogram Control Section (MPS) unit  280 , an Auxiliary Operations Section (XOPS)  282 , a Basic Operations Section (BOPS)  284 , a Safe Store Buffer (SSB)  286 , an Address Preparation (AP) section  288 , and a NSA Virtual Segment Section  290 . The MPS  280  is b-directionally coupled to and receives instructions from the local cache  256 . The MPS  280  performs instruction decode and provides microprogram control of the processor  92 . The microprogram control utilizes a microengine executing microcode  281  stored in both dynamic and static memories in response to the execution of program instructions. The MPS  280  is b-directionally coupled to and controls operation of the Auxiliary Operations Section (XOPS)  282 , the Basic Operations Section (BOPS)  284 , the floating point (FP) unit  268 , the decimal/numeric (DN) unit  262 , the Address Preparation (AP) section  288 , and the NSA Virtual Segment Section  290 . The Basic Operations Section (BOPS)  284  is used to perform fixed point arithmetic, logical, and shift operations. The Auxiliary Operations Section (XOPS)  282  performs most other operations. The Address Preparation (AP) section  288  forms effective memory addresses utilizing virtual memory address translations. The NSA Virtual Segment Section  290  is b-directionally coupled to and operates in conjunction with the AP section  288 , in order to detect addressing violations. 
     The Safe Store Buffer (SSB)  286  stores the current status of the processor  92  environment, including user and segment registers, for the purpose of changing processor state. The SSB  286  is coupled to and receives signals from the BOPS  284 , the AP section  288 , the MPS  280 , and the NSA  290 . The SSB  286  is b-directionally coupled to the local cache  256 , allowing SSB  286  frames to be pushed out to cache  256  when entering a new processor environment, and pulled back from cache  256  when returning to an old processor environment. 
     FIG. 6 is a block diagram of the interrupt structure for each processor  92  in FIG. 3, in accordance with the prior art. Whenever a system fault occurs, an attempt by system microcode is made to invoke fault handling in the operating system through a special variant of a “Climb” instruction. A two word entry descriptor is retrieved from a specified location (032 Octal) in memory. This entry descriptor specifies the environment and starting address of the interrupt processing to be performed. The standard operating system fault processing code then utilizes a fault code supplied by the microcode to determine the exact actions that are required to be performed in response to the fault being handled. Should the processor  92  fail in its attempt to enter and execute fault handling code, a second two word entry descriptor is retrieved from a second specified location (040 Octal) in memory. This second entry descriptor specifies a “fault on fault” fault handler. 
     During execution of code  102 , a fault  104  is detected by a processor  92 . The fault  104  causes execution of microcode fault handler  110 . The microcode fault handler  110  causes a Safe Store Stack Frame to be laid down containing the environment of that processor  92  as the processor  92  executes a variant of an ICLIMB instruction  120  to enter a general fault routine  112  described by the entry descriptor  304  stored at the specified location (032) in memory. The general fault routine  112 , which is part of the operating system (OS) controlling the processor  92 , then determines what fault  104  occurred. The fault  104  is identified by a six bit fault number. Based on this six bit fault number, a specific fault handler  114  is selected and entered, again by use of an ICLIMB instruction  122 , again laying down a Safe Store Stack Frame. Upon completion of fault handling in the specific fault handler  114 , an OCLIMB instruction is executed  123  utilizing the second Safe Store Stack Frame to return to the general fault hander  112 , and it in turn causes execution of a second OCLIMB instruction  121  utilizing the first Safe Store Stack Frame to return  106  control to execution of the code  102  that had been interrupted by the fault  104 . 
     The prior art interrupt structure shown in FIG. 6 has a number of short comings. First, in order to enter fault handling for any faults, typically two different environment transfer (i.e. “ICLIMB”) instructions are executed, laying down two Safe Store Stack Frames. The first environment transfer is made automatically by the microcode and transfer is made into the specified general fault handling routine. Then this general fault handling routine determines which fault has occurred, and what routine needs to be called to handle that fault. This requires the second ICLIMB. After the specific fault handling routine has accomplished the appropriate fault handling, two more environment transfers (i.e. “OCLIMB”) instructions are executed to return to the code being executed when the original fault occurred. Thus four environment transfers are required to handle most faults. Environment transfers are typically quite expensive. 
     Second, it sometimes happens in even a system with the best security that memory areas are inadvertently overwritten. In systems with weak security, this can be done by user programs. In more secure systems, it can still be done by either privileged operating system functions, or Input/Output. This can cause serious problems when the area being overwritten contains the entry descriptors for the fault handling routines or the fault handling routines themselves. This is typically detected in the GCOS 8 system by the microcode when it attempts to utilize an entry descriptor retrieved from the overwritten area. If the “fault on fault” entry descriptor has also been overwritten, it becomes extremely difficult to dump the system in order to determine what caused the fault and what caused the inadvertent overwriting of system memory. 
     FIG. 7 is a block diagram of the interrupt structure for each processor  92  in FIG.  3 . During execution of code  102 , a processor  92  enters its microcode fault handler  111  upon detection of some fault  104 . The microcode fault handler  111  utilizes the six bit fault code to index into a 64-entry fault vector pointer table  130 . Each entry of the 64 entries fault vector pointer table  130  contains an address of a two word entry descriptor  304  which describes the specific fault handler  115  for the corresponding fault type. The appropriate entry descriptor  304  is retrieved and utilized to ICLIMB  126  to the specific fault handler  115 , laying down a Safe Store Stack Frame. Upon completion of the actions by the specific fault handler  115 , an OCLIMB  127  instruction is executed, utilizing the Safe Store Stack Frame to return  106  execution control to the code  102  originally being executed. 
     FIG. 8 is a block diagram illustrating the data structures utilized in FIG.  7 . The fault vector pointer table  130  contains sixty-four one-word entries and is located at a specified location (01500 Octal) in reserved memory  85 . Each entry  131  in the fault vector pointer table  130  contains the address of an entry descriptor  304  for a specific fault handler for the corresponding fault type. Each entry descriptor  304  points at a segment descriptor  302  in a linkage table  134 , with a corresponding segment descriptor  302  describing the segment containing the specific fault handler  115 . Each entry descriptor  304  also specifies the starting address for the specific fault handler  115  in the specified segment described by the corresponding segment descriptor  302 . Note that the entry descriptors  304  combined with the corresponding segment descriptors  302  specify the processor environment for each specific fault handler  115 . 
     Following the fault vector pointer table  130  in reserved memory  85  is a one word fault-on-fault entry descriptor address  138  of a two word entry descriptor  304  for the fault-on-fault fault handler. Upon system initialization, each of the entries in the fault vector pointer table  130  is initialized by the service processor  87  to a constant value (032 octal). This constant value (032 octal) is the address of a two word entry descriptor  304  of a general fault handler  112 . The fault-on-fault entry descriptor address  138  is similarly initialized to a constant value (040 octal). This second constant value (040) is the address of a two word entry descriptor  304  of a fault-on-fault handler  112 . Also stored in specific areas of reserved memory  85  by system initialization or “boot” software are the contents of specific areas of memory critical to rebuilding processor  92  environment upon detection of a fault-on-fault situation. Included in this saved information are the two word entry descriptors  304  for the general fault handler  112  and the fault-on-fault handler. 
     The reserved memory  85  contains both a processor specific portion for each processor  92  in the system  80  and a shared portion. In this shared portion of the reserved memory  85  is Intercept code. Within this Intercept code is an improved fault-on-fault hander. This improved fault-on-fault handler takes the information saved in the reserved memory  85  and rebuilds a minimal processor environment. This rebuilding includes loading environmental registers and initializing work space zero (WS 0 ) which contains critical operating system code and data. 
     Putting this information and code in reserved memory  85  has a number of advantages. Reserved memory  85  cannot be written inadvertently. It is never made part of any work space, nor ever framed by a segment descriptor  302 . In the preferred embodiment, it can be read via I/O, but not written. It is thus safe from being accidentally overwritten. Since the fault-on-fault handler, as well as the descriptors to it are stored in reserved memory  85 , neither the fault-on-fault handler, nor the descriptors  302 ,  304  needed to enter it, can be inadvertently overwritten. Thus, it is now possible to guarantee that memory dumps can be performed in a fault-on-fault situation where critical portions of the operating system have been overwritten. 
     As noted above, the service processor  87  initializes the fault vector pointer table  130  to a value ( 032 ) corresponding to the address of the entry descriptor  304  of a general fault hander  112  in the operating system. The operating system then rewrites some or all of these fault vector pointer table  130  entries  131  to point at entry descriptors  304  for specific fault handers  115 . This provides a transitional methodology. Initially, the system operates as shown in FIG.  6 . Then as fault handler code is modified over time, pointers to different specific fault handlers  115  can be written to the fault vector pointer table  130  as the code is modified to operate as shown in FIG.  7 . Some fault types are rare enough that it may not be economically worthwhile to modify the corresponding specific fault handers  115 . For example, in the exceedingly rare category are such faults as STUP (startup), LUF (lockup), and SDF (shutdown). Other fault types, being much more common, can be profitably migrated much quicker. For example, in the extremely common category are such faults as DVCF (divide check), OFL (overflow), MSG (missing segment), MWS (missing work space), MSCT (missing section), and MPF (missing page). 
     FIGS. 9 and 10 are diagrams that illustrate the format of two different types of descriptors in a GCOS 8 environment. Thirteen segment descriptor registers are supported in the GCOS 8 architecture, and they are: eight Segment Descriptor Registers (DR 0  through DR 7 ) for operand addressing; an Argument Stack Register (ASR); a Data Stack Descriptor Register (DSDR); an Instruction Segment Register (ISR); a Linkage Segment Register (LSR); and a Parameter Segment Register (PSR). In the GCOS 8 environment, segment descriptors are 72-bits in size and are used to describe a contiguous subset of a working space. 
     FIG. 9 is a diagram illustrating a standard Segment Descriptor. A Segment Descriptor defines a contiguous extent of virtual space. The Segment Descriptor  302  comprises two 36-bit words stored in two words of memory or in a single 72-bit register. The format of the Segment Descriptor is shown in table T-4: 
     
       
         
               
             
               
               
               
               
               
               
               
             
           
               
                 TABLE T-4 
               
             
             
               
                   
               
               
                 Standard Segment Descriptor 302 Format 
               
             
          
           
               
                 Ref 
                 W 
                   
                   
                   
                   
                   
               
               
                 # 
                 d 
                 St 
                 Ed 
                 B# 
                 Field Name 
                 Field Description 
               
               
                   
               
               
                 310 
                 0 
                 00 
                 19 
                 20 
                 Bound 
                 Segment upper bound 
               
               
                 312 
                 0 
                 20 
                 28 
                  9 
                 Flags 
                 Flags: 
               
               
                   
                   
                   
                   
                   
                   
                 20 - Read allowed 
               
               
                   
                   
                   
                   
                   
                   
                 21 - Write allowed 
               
               
                   
                   
                   
                   
                   
                   
                 22 - Store by STDn allowed 
               
               
                   
                   
                   
                   
                   
                   
                 23 - Cache use control 
               
               
                   
                   
                   
                   
                   
                   
                 24 - NS/ES mode 
               
               
                   
                   
                   
                   
                   
                   
                 25 - Execute allowed 
               
               
                   
                   
                   
                   
                   
                   
                 26 - Privilege required 
               
               
                   
                   
                   
                   
                   
                   
                 27 - Bound valid 
               
               
                   
                   
                   
                   
                   
                   
                 28 - Segment available 
               
               
                 314 
                 0 
                 29 
                 31 
                  3 
                 WSR 
                 Working Space Register 
               
               
                 316 
                 0 
                 32 
                 35 
                  4 
                 Type 
                 Segment Descriptor Type 
               
               
                   
                   
                   
                   
                   
                   
                 0 - frames operand space 
               
               
                   
                   
                   
                   
                   
                   
                 1 - frames descriptor space 
               
               
                   
                   
                   
                   
                   
                   
                 12 - extended descriptor 
               
               
                 318 
                 1 
                  0 
                 35 
                 36 
                 Base 
                 Segment Base Address 
               
               
                   
               
             
          
         
       
     
     The 3-bit Working Space Register (WSR)  314  field designates one of eight 9-bit working space registers. The contents of the selected WSR  314  are retrieved and used as the working space for the segment. The 20-bit bound field  324  contains the maximum valid byte address within the segment. The 36-bit base field  318  contains a virtual byte address that is relative to the start of the designated working space defined by the WSR  314 . Bits  0 : 33  are a 34-bit word address, and bits  34 : 35  identify a 9-bit byte within the word. 
     FIG. 10 is a diagram illustrating the format of an Entry Descriptor  304 . Entry Descriptors  304  are utilized by the ICLIMB instruction for domain transfer subroutine calls, as well as entry into Fault (or Exception) and Interrupt processing. The Entry Descriptor  304  is a Descriptor that defines the execution environment and starting address of a subroutine. The Entry Descriptor  304  comprises two 36-bit words stored in two words of memory or in a single 72-bit register. The format of an Entry Descriptor  304  is shown in table T-2: 
     
       
         
               
             
               
               
               
               
               
               
               
             
               
               
               
               
               
               
               
             
           
               
                 TABLE T-2 
               
             
             
               
                   
               
               
                 Entry Descriptor 304 Format 
               
             
          
           
               
                 Ref # 
                 Wd 
                 St 
                 Ed 
                 B # 
                 Field Name 
                 Field Description 
               
               
                   
               
             
          
           
               
                 320 
                 0 
                 00 
                 17 
                 18 
                 Entry 
                 Entry address relative 
               
               
                   
                   
                   
                   
                   
                 Loaction 
                 to base 
               
               
                   
                   
                   
                   
                   
                   
                 of new instruction 
               
               
                   
                   
                   
                   
                   
                   
                 segment 
               
               
                 324 
                 0 
                 18 
                 18 
                 1 
                 F 
                 Store permission bit 
               
               
                 322 
                 0 
                 19 
                 28 
                 10 
                 ISEG No. 
                 Number of descriptor 
               
               
                   
                   
                   
                   
                   
                   
                 to be loaded into the 
               
               
                   
                   
                   
                   
                   
                   
                 Instruction Segment 
               
               
                   
                   
                   
                   
                   
                   
                 Register (ISR) 
               
               
                 314 
                 0 
                 29 
                 31 
                 3 
                 WSR 
                 Working Space Register 
               
               
                 316 
                 0 
                 32 
                 35 
                 4 
                 Type 
                 Entry Descriptor Type 
               
               
                   
                   
                   
                   
                   
                   
                 8-16 word stack frame 
               
               
                   
                   
                   
                   
                   
                   
                 9-24 word stack frame 
               
               
                   
                   
                   
                   
                   
                   
                 11-64/80 word 
               
               
                   
                   
                   
                   
                   
                   
                 stack frame 
               
               
                 326 
                 1 
                 00 
                 10 
                 20 
                 LBound 
                 Linkage Segment upper 
               
               
                   
                   
                   
                   
                   
                   
                 bound (in descriptors) 
               
               
                 328 
                 1 
                 11 
                 23 
                 13 
                 Linkage Base 
                 Segment Base Address 
               
               
                 329 
                 1 
                 33 
                 35 
                 3 
                   
                 Zeroes 
               
               
                   
               
             
          
         
       
     
     An Entry Descriptor  304  describes a linkage section that defines a new domain, a segment containing instructions to be initially executed in the domain, and an offset relative to the origin of that segment to which control is transferred. 
     The 3-bit Working Space Register (WSR)  314  field designates one of eight 9-bit working space registers. The contents of the selected WSR  314  are retrieved and used as the working space for the Entry Descriptor  304 . The virtual starting address of a Linkage Section in the working space designated by the WSR field  314  is determined from the Linkage Base  328  address field. The Linkage Segment contains a number of Type=0 Segment Descriptors  302 . The number of Segment Descriptors  302  in the Linkage Segment is specified by the LBOUND field  326  in the Entry Descriptor. The ISEG number  322  in the Entry Descriptor  304  is utilized as an index to index into these Segment Descriptors  302 . The indexed Segment Descriptor  302  is then loaded into the Instruction Segment Register (ISR) in order to specify a new execution environment. The Entry Location  320  field in the Entry Descriptor  304  is then utilized to identify the starting address in the segment described by the ISR. 
     Thus, in accordance with the subject matter of the related patent applications identified above and also referring to FIG. 11, a fault may occur (step  500 A) during normal operation (step  500 ), and a routine fault handling procedure invoked (Step  501 ). If the firmware does not complete its preliminary handling of the fault before another fault occurs, a backup fault will be taken if available (Step  503 ) for the present fault. Else, the original fault is processed (step  501 ) and normal operation resumes (step  501 ). If there is no problem completing the handling of the backup fault (step  507 ) and if no dump is necessary (step  508 ), then the original fault is then processed ( 502 ), and normal operation resumes. If a dump is desired, then the dump (which is valid) is taken (step  509 ) after which a full manual restart will be taken (step  510 ). 
     However, as discussed above, no Safe Store Frame (SSF) is stored for the backup fault. If either the backup fault does not complete the fault firmware or the entry to the Operating System (OS) Fault Module (MFLT) does not get far enough into the code to issue the RBFF instruction to reset the backup fault flag, then a fault-on-fault (FOF) occurs (steps  503 ,  504 ). 
     In the earlier inventions disclosed in the related applications identified above, under certain conditions, the execute fault will not function because one or more of the hardware fault flags are still set. This results in the execute fault causing another FOF. Direct operator intervention is required to recover the system with no information about the problem which caused the FOF or a procedure to follow. Any dump taken is probably useless for analysis because the operating system has lost control of the status of the system; thus, complete manual reinitializion from the Service Processor must be undertaken (step  510 ). 
     This problem is addressed in the present invention in which, if there is no valid backup fault vector (step  503 ) or if there is a problem in processing a backup fault (step  504 ), fault handling is directed to the process flow shown in FIG. 12 via connector “B”. 
     Thus, referring to FIG. 12, firmware is provided to reset fault flags (step  550 ), get the processor back to a known basic state (step  551 ) and set Working Space Registers  0 _ 7  to  000  (step  552 ) to prepare for a return to the OS with a call to Intercept. During step  551 , internal registers are saved to reserved memory to preserve the minimum information needed to determine where in software the faults had occurred. These registers include: 
     
       
         
               
               
             
           
               
                   
               
             
             
               
                 IC &amp; I 
                 (Instruction Counter and Indicators) 
               
               
                 FAULT WORD 
                 (Of the Safe Store Frame) 
               
               
                 ISR BSE/BND 
                 (Instruction Segment Register, Base and Bounds) 
               
               
                 WS0_3 
                 (Working Space Registers) 
               
               
                 WS4_7 
                 (Working Space Registers) 
               
               
                   
               
             
          
         
       
     
     There are several registers and fixed locations in reserved memory which must be correct. These are: 
     
       
         
               
               
             
           
               
                   
               
             
             
               
                 ISR BSE/BND: 
                 (loaded to Reserved Memory base + 40000 = 77040000) 
               
               
                 IC: 
                 (loaded to the Intercept entry point for FOF Dump) 
               
               
                 WS0_7: 
                 (all Workspace Registers forced to 000 to use only WS0 
               
               
                   
                 page table) 
               
               
                 WS0: 
                 (only used by the OS; chances of page table corruption is 
               
               
                   
                 low). 
               
               
                 PDBR: 
                 (WS_0 Page Directory Base Register is stored in 
               
               
                   
                 RMS + xxxxxxx by the Service Processor during startup; 
               
               
                   
                 GCOS normally runs in WS1 which has a copy of WS0 
               
               
                   
                 page table. Intercept process requires 16k of PTW to 
               
               
                   
                 be correct.) 
               
               
                 PTDW: 
                 pointed to by the PDBR; should be re-loaded to 
               
               
                   
                 guarantee it is usable and pointing to W0 Page Table 
               
               
                   
                 Words) 
               
               
                   
               
             
          
         
       
     
     The Page Table Directory Word (PTDW) is loaded into the CPU (step  553 ). Since the state of the processor registers has not been saved, they are not available to be initialized to facilitate entry into the Intercept process. The process has been minimized to avoid use of any registers that have not been initialized. This maximizes the probability that the transfer to Intercept process (step  555 ) will function properly. 
     The Intercept process is capable of quickly determining the requirements to institute an operating system dump (step  556 ). It can verify PTWs, reload fault entry descriptors, reload hardware registers as needed and build a stable basic environment for the dump process. Even if an operating system dump is not possible, control by the operating system is maintained, and interactive messages can keep the operator aware of the system status. 
     If a valid operating system dump is possible (step  557 ), a dump of the system memory is taken automatically (step  560 ) after which it is only necessary to carry out a normal operating system restart (step  561 ), thus obviating the necessity for a full manual restart including preliminary initialization by the service processor, consequently not only saving time, but also avoiding customer aggravation. 
     If, however, a valid operating system environment is determined by Intercept process to be impossible (step  557 ), then a message to that effect is sent to the operator (step  564 ), operation is terminated (step  565 ), and a full manual restart must be undertaken (step  510 ). Intercept process has determined that no useful dump can be obtained because the information gathered in steps  551 - 556  is still not sufficient to carry out the dump. 
     However, the message sent to the operator in step  564  is important because, rather than experiencing a system hang-up with no information about the cause, this message contains valuable information (available because of the information gathered in steps  551 - 556  and stored in reserved memory) that can be used to analyze the fault event and take corrective action. 
     If a dump is possible as determined by Intercept process in step  557  (enough valid information is available to effect the dump), then the first Central Processing Unit that makes itself available (in the routine flow in which each CPU provides notice when it is ready for a new task) starts the Dump module. In addition, the CPU processing the Dump module will close a process gate preventing any other CPU from starting another dump (step  569 ). Thus, the selection of the CPU which executes the operating system dump (step  559 ) is made automatically by the first CPU finding the open gate which it immediately closes. Accordingly, all other CPUs are locked out of execution (step  563 ) until Normal System Restart (operating system restart) resumes multiprocessor operation. 
     In order to provide for the unlikely event that the single CPU processing the Dump module is, itself, not operating properly, the process gate is opened periodically (step  562 ) to allow another CPU to offer its availability to take over processing the Dump module. 
     Those skilled in the art will recognize that modifications and variations can be made without departing from the spirit of the invention. Therefore, it is intended that this invention encompass all such variations and modifications as fall within the scope of the appended claims. 
     Claim elements and steps herein have been numbered and/or lettered solely as an aid in readability and understanding. As such, the numbering and/or lettering in itself is not intended to and should not be taken to indicate the ordering of elements and/or steps in the claims.