Free resource error/event log for autonomic data processing system

System, method, and computer program product for monitoring resources capable of being assigned to different partitions of a data processing system from a free resources pool of the data processing system. A free resources event log is associated with the free resources pool for maintaining a log of events for each resource that is in the free resources pool. When a resource is assigned to a partition from the free resources pool, a log of events for the assigned resource is transferred from the free resources event log to an event log associated with the partition.

TECHNICAL FIELD

The present invention relates generally to the data processing field, and, more particularly, to a system, method and computer program product for monitoring resources capable of being assigned to different partitions of a data processing system from a free resources pool of the data processing system.

DESCRIPTION OF RELATED ART

Increasingly, large symmetric multi-processor data processing systems are not being used as single large data processing systems; but are being partitioned and used as a plurality of smaller systems. These systems are commonly referred to as logical partitioned (LPAR) data processing systems. A logical partitioned functionality within a data processing system allows multiple copies of a single operating system or multiple heterogeneous operating systems to be simultaneously run on a single data processing system platform. A partition, within which an operating system runs, is assigned a non-overlapping subset of platform resources. These platform resources may include one or more architecturally distinct processors with their interrupt management area, regions of system memory, and input/output (I/O) adapter bus slots. The partition's resources are represented by the platform's firmware to the operating system.

Each operating system running within a platform is protected from other operating systems such that software errors in one logical partition cannot affect the correct operations of any of the other partitions. This protection is provided by allocating a disjointed set of platform resources to be directly managed by each operating system and by providing mechanisms for insuring that the various operating systems cannot control any resources that have not been allocated to that system. Furthermore, software errors in the control of an operating system's allocated resources are prevented from affecting the resources of any other operating system. Thus, each operating system directly controls a distinct set of allocatable resources within the platform.

With respect to hardware resources in a logical partitioned data processing system, these resources are disjointly shared among various partitions. Hardware resources may include, for example, input/output (I/O) adapters, memory DIMMs, non-volatile random access memory (NVRAM), and hard disk drives.

In an autonomic data processing system, for example, a Dynamic Logical Partitioned (DLPAR) data processing system or a Dynamic Capacity Upgrade on Demand (DCUoD) data processing system, allocatable resources are often moved between a free resources pool and logical partitions in order to meet the dynamic workload requirements of a partition. An important aspect of autonomic computing is the ability of a server to “heal” itself. One technique for implementing self-healing is for the data processing system to replace a resource that is predicted to fail with a similar resource from a pool of available resources or by a capacity upgrade on demand resource. Failure prediction is typically achieved by analyzing logged errors associated with each resource.

A problem that exists in implementing this self-healing aspect of a data processing system is the inability of the system to monitor pertinent errors logged against a resource as the resource moves between a partition and the free resources pool, i.e., as the resource moves between a state in which the resource is assigned to a partition, and a state in which the resource is not assigned to a partition but is available for assignment as needed. In particular, errors that may have been logged against a resource while in a first partition are lost when the resource is subsequently moved from that partition to the free resources pool, and then later assigned to a second partition. If errors pertinent to predicting a failure of the resource were logged in the first partition while the resource was assigned to that partition, the resource can unexpectedly fail while assigned to the second partition since the errors logged against the resource while assigned to the first partition are unable to be considered for Predictive Failure Analysis (PFA) in the second partition.

It would, accordingly, be advantageous to accurately monitor resources capable of being assigned to different partitions of a data processing system from a free resources pool of the data processing system.

SUMMARY OF THE INVENTION

The present invention provides a system, method, and computer program product for monitoring resources capable of being assigned to different partitions of a data processing system from a free resources pool of the data processing system. A free resources event log is associated with the free resources pool for maintaining a log of events for each resource that is in the free resources pool. When a resource is assigned to a partition from the free resources pool, the log of events for the assigned resource is transferred from the free resources event log to an event log associated with the partition.

The invention permits resources capable of being assigned to different partitions of a data processing system from a free resources pool of the data processing system to be accurately monitored for Predictive Failure Analysis and for other purposes.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

With reference now to the figures,FIG. 1depicts a block diagram of a data processing system in which the present invention may be implemented. Data processing system100may be a symmetric multiprocessor (SMP) system including a plurality of processors101,102,103, and104connected to system bus106. Also connected to system bus106is memory controller/cache108, which provides an interface to a plurality of local memories160-163. I/O bus bridge110is connected to system bus106and provides an interface to I/O bus112. Memory controller/cache108and I/O bus bridge110may be integrated as depicted.

Data processing system100is a logical partitioned (LPAR) data processing system. Thus, data processing system100may have multiple heterogeneous operating systems (or multiple instances of a single operating system) running simultaneously. Each of these multiple operating systems may have any number of software programs executing within it. Data processing system100is logically partitioned such that different PCI I/O adapters120-121,128-129, and136, graphics adapter148, and hard disk adapter149may be assigned to different logical partitions. In this case, graphics adapter148provides a connection for a display device (not shown), while hard disk adapter149provides a connection to control hard disk150.

Thus, for example, suppose data processing system100is divided into three logical partitions, P1, P2, and P3. Each of PCI I/O adapters120-121,128-129,136, graphics adapter148, hard disk adapter149, each of host processors101-104, and memory from local memories160-163is assigned to each of the three partitions. In these examples, memories160-163may take the form of dual in-line memory modules (DIMMs). DIMMs are not normally assigned on a per DIMM basis to partitions. Instead, a partition will get a portion of the overall memory seen by the platform. For example, processor101, some portion of memory from local memories160-163, and I/O adapters120,128, and129may be assigned to logical partition P1; processors102-103, some portion of memory from local memories160-163, and PCI I/O adapters121and136may be assigned to partition P2; and processor104, some portion of memory from local memories160-163, graphics adapter148and hard disk adapter149may be assigned to logical partition P3.

Each operating system executing within data processing system100is assigned to a different logical partition. Thus, each operating system executing within data processing system100may access only those I/O units that are within its logical partition. Thus, for example, one instance of the Advanced Interactive Executive (AIX) operating system may be executing within partition P1, a second instance (image) of the AIX operating system may be executing within partition P2, and a Linux or OS/400 operating system may be operating within logical partition P3.

Peripheral component interconnect (PCI) host bridge114connected to I/O bus112provides an interface to PCI local bus115. A number of PCI input/output adapters120-121may be connected to PCI bus115through PCI-to-PCI bridge116, PCI bus118, PCI bus119, I/O slot170, and I/O slot171. PCI-to-PCI bridge116provides an interface to PCI bus118and PCI bus119. PCI I/O adapters120and121are placed into I/O slots170and171, respectively. Typical PCI bus implementations will support between four and eight I/O adapters (i.e. expansion slots for add-in connectors). Each PCI I/O adapter120-121provides an interface between data processing system100and input/output devices such as, for example, other network computers, which are clients to data processing system100.

An additional PCI host bridge122provides an interface for an additional PCI bus123. PCI bus123is connected to a plurality of PCI I/O adapters128-129. PCI I/O adapters128-129may be connected to PCI bus123through PCI-to-PCI bridge124, PCI bus126, PCI bus127, I/O slot172, and I/O slot173. PCI-to-PCI bridge124provides an interface to PCI bus126and PCI bus127. PCI I/O adapters128and129are placed into I/O slots172and173, respectively. In this manner, additional I/O devices, such as, for example, modems or network adapters may be supported through each of PCI I/O adapters128-129. In this manner, data processing system100allows connections to multiple network computers.

A memory mapped graphics adapter148inserted into I/O slot174may be connected to I/O bus112through PCI bus144, PCI-to-PCI bridge142, PCI bus141and PCI host bridge140. Hard disk adapter149may be placed into I/O slot175, which is connected to PCI bus145. In turn, this bus is connected to PCI-to-PCI bridge142, which is connected to PCI host bridge140by PCI bus141.

A PCI host bridge130provides an interface for a PCI bus131to connect to I/O bus112. PCI I/O adapter136is connected to I/O slot176, which is connected to PCI-to-PCI bridge132by PCI bus133. PCI-to-PCI bridge132is connected to PCI bus131. This PCI bus also connects PCI host bridge130to the service processor mailbox interface and ISA bus access pass-through logic194and PCI-to-PCI bridge132. Service processor mailbox interface and ISA bus access pass-through logic194forwards PCI accesses destined to the PCI/ISA bridge193. NVRAM storage192is connected to the ISA bus196. Service processor135is coupled to service processor mailbox interface and ISA bus access pass-through logic194through its local PCI bus195. Service processor135is also connected to processors101-104via a plurality of JTAG/I2C busses134. JTAG/I2C busses134are a combination of JTAG/scan busses (see IEEE 1149.1) and Phillips I2C busses. However, alternatively, JTAG/I2C busses134may be replaced by only Phillips I2C busses or only JTAG/scan busses. All SP-ATTN signals of the host processors101,102,103, and104are connected together to an interrupt input signal of the service processor. The service processor135has its own local memory191, and has access to the hardware OP-panel190.

When data processing system100is initially powered up, service processor135uses the JTAG/I2C busses134to interrogate the system (host) processors101-104, memory controller/cache108, and I/O bridge110. At completion of this step, service processor135has an inventory and topology understanding of data processing system100. Service processor135also executes Built-In-Self-Tests (BISTs), Basic Assurance Tests (BATs), and memory tests on all elements found by interrogating the host processors101-104, memory controller/cache108, and I/O bridge110. Any error information for failures detected during the BISTs, BATs, and memory tests are gathered and reported by service processor135.

If a meaningful/valid configuration of system resources is still possible after taking out the elements found to be faulty during the BISTs, BATs, and memory tests, then data processing system100is allowed to proceed to load executable code into local (host) memories160-163. Service processor135then releases host processors101-104for execution of the code loaded into local memory160-163. While host processors101-104are executing code from respective operating systems within data processing system100, service processor135enters a mode of monitoring and reporting errors. The type of items monitored by service processor135include, for example, the cooling fan speed and operation, thermal sensors, power supply regulators, and recoverable and non-recoverable errors reported by processors101-104, local memories160-163, and I/O bridge110.

Service processor135is responsible for saving and reporting error information related to all the monitored items in data processing system100. Service processor135also takes action based on the type of errors and defined thresholds. For example, service processor135may take note of excessive recoverable errors on a processor's cache memory and decide that this is predictive of a hard failure. Based on this determination, service processor135may mark that resource for deconfiguration during the current running session and future Initial Program Loads (IPLs). IPLs are also sometimes referred to as a “boot” or “bootstrap”.

With reference now toFIG. 2, a block diagram of an exemplary logical partitioned platform is depicted in which the present invention may be implemented. The hardware in logical partitioned platform200may be implemented as, for example, data processing system100inFIG. 1. Logical partitioned platform200includes partitioned hardware230, operating systems202,204,206,208, and partition management firmware210. Operating systems202,204,206, and208may be multiple copies of a single operating system or multiple heterogeneous operating systems simultaneously run on logical partitioned platform200. These operating systems may be implemented using an OS/400 operating system available from International Business Machine Corporation of Armonk, N.Y., which is designed to interface with a partition management firmware, such as Hypervisor. OS/400 is used only as an example in these illustrative embodiments. Of course, other types of operating systems, such as AIX and Linux, may also be used depending on the particular implementation. Operating systems202,204,206, and208are located in partitions203,205,207, and209. Hypervisor software is an example of software that may be used to implement partition management firmware210and is available from International Business Machines Corporation. Firmware is “software” stored in a memory chip that holds its content without electrical power, such as, for example, read-only memory (ROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), and nonvolatile random access memory (nonvolatile RAM).

Additionally, these partitions also include partition firmware211,213,215, and217. Partition firmware211,213,215, and217may be implemented using initial boot strap code, IEEE-1275 Standard Open Firmware, and runtime abstraction software (RTAS), which is available from International Business Machines Corporation. When partitions203,205,207, and209are instantiated, a copy of boot strap code is loaded onto partitions203,205,207, and209by platform firmware210. Thereafter, control is transferred to the boot strap code with the boot strap code then loading the open firmware and RTAS. The processors associated or assigned to the partitions are then dispatched to the partition's memory to execute the partition firmware.

Partitioned hardware230includes a plurality of processors232-238, a plurality of system memory units240-246, a plurality of input/output (I/O) adapters248-262, and a storage unit270. Each of the processors232-238, memory units240-246, NVRAM storage298, and I/O adapters248-262may be assigned to one of multiple partitions within logical partitioned platform200, each of which corresponds to one of operating systems202,204,206, and208.

Partition management firmware210performs a number of functions and services for partitions203,205,207, and209to create and enforce the partitioning of logical partitioned platform200. Partition management firmware210is a firmware implemented virtual machine identical to the underlying hardware. Thus, partition management firmware210allows the simultaneous execution of independent OS images202,204,206, and208by virtualizing all the hardware resources of logical partitioned platform200.

Service processor290may be used to provide various services, such as processing of platform errors in the partitions. These services also may act as a service agent to report errors back to a vendor, such as International Business Machines Corporation. Operations of the different partitions may be controlled through a hardware management console, such as hardware management console280. Hardware management console280is a separate data processing system from which a system administrator may perform various functions including reallocation of resources to different partitions.

FIGS. 3A,3B and3C are diagrams that schematically illustrate a known procedure for monitoring resources capable of being assigned to different partitions of a data processing system from a free resources pool of the data processing system in order to assist in explaining the present invention.FIG. 3Aillustrates State1of data processing system300having a plurality of logical partitions including partitions302and304, and a free resources pool306. Operating system312is running in partition302and operating system314is running in partition304.

Three resources320,322and324are assigned to partition302; three resources330,332and334are assigned to partition304; and three resources340,342and344are not currently assigned to any partition and are maintained in free resources pool306. Resources340,342and344in free resources pool306are available for assignment to any of the plurality of partitions in data processing system300; including partitions302and304, as the need arises. Resources320-324,330-334and340-344may, for example, be I/O adapters or devices that are connected to system300through I/O adapters.

Event log352is associated with partition302and event log354is associated with partition304. Event logs352and354maintain a log of events associated with each of the resources assigned to partitions302and304, respectively. The events may be the result of some error condition encountered by a resource, or some informational type of message associated with a resource. Events in an event log are commonly used to predict a failure of a resource based on an analysis of the event history of the resource.

As illustrated inFIG. 3A, event log352associated with partition302includes a log of events for each of resources320,322and324assigned to partition302. Event log354associated with partition304includes a log of events for each of resources330,332and334assigned to partition304.

FIG. 3Billustrates State2of data processing system300. In State2, resource332has been moved from partition304to free resources pool306, for example, as a result of having satisfied a temporary need for the resource in partition304. As shown inFIG. 3B, the events for resource332in event log354associated with partition304remain in event log354after resource332has been moved to free resources pool306.

FIG. 3Cillustrates State3of data processing system300. In State3, resource332has now been assigned to partition302from free resources pool306, for example, due to a need to satisfy increased I/O requirements of partition302. Note that events for resource332that occurred while resource332was assigned to partition304remain in event log354associated with partition304.

FIGS. 4A,4B and4C are diagrams that schematically illustrate a procedure for monitoring resources capable of being assigned to different partitions of a data processing system from a free resources pool of the data processing system in accordance with a preferred embodiment of the present invention.FIG. 4Aillustrates State1of data processing system400having a plurality of logical partitions including partitions402and404, and a free resources pool406. Operating system412is running in partition402and operating system414is running in partition404.

Three resources420,422and424are assigned to partition402; three resources430,432and434are assigned to partition404; and three resources440,442and444are not currently assigned to any partition and are maintained in free resources pool406. Resources440,442and444in free resources pool are available for assignment to any of the plurality of partitions in data processing system400; including partitions402and404, as the need arises.

Data processing system400illustrated inFIGS. 4A-4Cdiffers from data processing system300illustrated inFIGS. 3A-3Cin that data processing system400includes a Free Resources Event Log (FREL)456that is associated with free resources pool406. FREL456includes a log of events for resources in free resources pool406. In particular, events logged in FREL456are the result of events that occurred with respect to resources440,442and444when the resources were assigned to a partition of data processing system400such as one of partitions402and404.

In State1of data processing system400illustrated inFIG. 4A, FREL456includes a log of events for resources440,442and444in free resources pool406.

FIG. 4Billustrates State2of data processing system400. In State2, resource432has been moved from partition404to free resources pool406, for example, as a result of having satisfied a temporary need for the resource in partition404. As shown inFIG. 4B, the log of events for resource432in event log454associated with partition404has been moved from event log454to FREL456along with the move of resource432to free resources pool406. Thus, in State2, FREL456also includes a log of events for resource432as well as a log of events for resources440,442and444that remain in FREL456.

FIG. 4Cillustrates State3of data processing system400. In State3, resource432has now been assigned to partition402from free resources pool406. As shown inFIG. 4C, events in FREL456associated with resource432has also been moved from FREL456to event log452associated with partition402along with the move of resource432from free resources pool406to partition402.

In system300illustrated inFIGS. 3A-3C, operating system312in partition302has no visibility to the event history of resource332while the resource was assigned to partition304or to any other partition of data processing system300. This lack of information can prevent an application from making an accurate failure prediction with respect to resource332, and can result in resource332failing without warning while assigned to partition302.

In system400illustrated inFIGS. 4A-4C, on the other hand, operating system412has visibility to the complete event history of resource432and is in a position to more accurately predict a failure of resource432based on its previous event history while assigned to another partition of data processing system400, and on any new events that may occur during the current assignment of resource432to partition402.

FIG. 5is a flowchart that illustrates a method for monitoring a resource transferred from a partition to a free resources pool of a data processing system in accordance with a preferred embodiment of the present invention. The method is generally designated by reference number500, and begins by determining if a resource should be moved from a partition to the free resources pool (step502). If No, the method returns. If Yes, a determination is made if there are events for the resource in an event log associated with the partition (step504). If No, the resource is transferred to the free resources pool and the method ends. If Yes, any events associated with the resource that are in the event log of the partition are copied to a FREL in the free resources pool (step506) and the method ends. The events may also be deleted from the partition's event log (step506), or, alternatively, the event log of the partition may retain the event record of the moved resource. In such alternative embodiment, if the resource is later reassigned to the partition from the free resources pool, any events for the resource in the FREL that are duplicates of errors already recorded in the partition's event log are not transferred. In this alternative embodiment the FREL also maintains a record of the particular partition in which an event occurred with respect to a resource, as well as the number of events for that resource.

FIG. 6is a flowchart that illustrates a method for monitoring a resource assigned to a partition from a free resources pool of a data processing system in accordance with a preferred embodiment of the invention. The method is generally designated by reference number600and begins by determining if a resource should be moved from the free resources pool to a partition (step602). If No, the method returns. If Yes, a determination is made if a FREL associated with the free resources pool has any events logged for the resource (step604). If No, the resource is assigned to the partition and the method ends. If Yes, the events for the resource in the FREL are copied to the event log of the partition, the log of events of the resource in the FREL is deleted (step606), and the method ends.

With the present invention, an operating system associated with a partition in a data processing system in which resources are assigned to a partition from a free resources pool has visibility to all events associated with the resources currently assigned to the partition. Accordingly, a more accurate determination of when a particular resource should be replaced can be made based on a system-wide view of all events logged against the resource. This will reduce the occurrences of a resource failure while the resource is assigned to a partition.

In addition, with the present invention, an application is better able to determine which of identical resources should be assigned to a partition that requires additional resources to service a critical application. For example, if a resource has a predictive failure threshold of ten errors of a certain type and eight of those errors have already occurred for that resource as shown by the record for that resource in the FREL, the application may decide to use another, identical resource that has fewer or no errors logged against it in order to mitigate the risk of a failure or of having to perform deferred maintenance.

In general, the present invention provides a method, apparatus and system for monitoring resources capable of being assigned to different partitions of a data processing system from a free resources pool of the data processing system. The invention permits errors or other events associated with resources to be accurately monitored as the resources are assigned to different partitions from the free resources pool during operation of the data processing system.