System administration method and apparatus

A system, device, and method are provided for managing one or more computer resource systems. More specifically, the present invention provides for the automated maintenance of a computer resource and gradual change of management policies based upon a state of the computer resource. Moreover, the present invention is capable of generating commands related to changing the configuration of the computer resource and maintaining a log of generated commands for later reference by a system administrator. Further, the present invention includes one or more human user written policies, in some computer understandable format, to manage the system and accomplish specific goals based on the system's dynamic nature or behavior.

FIELD

The present invention is directed to system administration. In particular, the present invention describes methods and apparatuses for automatically controlling parameters of a computer resource and updating control thresholds of the same.

BACKGROUND

The need to process and store digital files, documents, pictures, images and other data continues to increase rapidly. Accordingly, computer resources continue to be developed and refined. In connection with the electronic storage of data, various data storage systems have been devised for the rapid and secure storage of large amounts of data. Such systems may include one or a plurality of storage devices that are used in a coordinated fashion. Devices can also provide improved data access and/or storage times. Whether implemented using one or a plurality of storage devices, the storage provided by a data storage system can be treated as one or more storage volumes.

As computer resources have evolved, increasing amounts of data have been made available to system administrators. For example, data storage products and other computer resource systems have been created that provide more information in the form of performance metrics relating to system performance that are stored in a performance log. In some instances, a performance log may contain hundreds of entries for a large number of performance metrics related to the system.

The system administrator reads the performance metrics of the system and makes decisions as to how storage space, storage controllers, data channels, processors, and other storage assets should be allocated. Typically, the system administrator is expected to analyze most or all of the performance metrics provided to him/her while monitoring the system performance. Thus, as the number of performance metrics provided by a system increases, the job of the administrator increases in difficulty. Because most computer resource systems require a high level of quality control, experienced system administrators are typically hired to manage those systems. As the amount of performance data available to administrators has increased, the number of administrators required to review the data has increased. This in turn increases the cost of maintaining computer resource systems. One way to decreases costs would be to hire a less experienced person to manage the system. Unfortunately, this may lead to improper decisions and may sacrifice system performance and data. One goal of computer resource system developers is to lower costs for their products. However, if a low cost system still requires high priced personnel to manage the system, then the cost savings of the system are offset, thus decreasing the cost effectiveness of the system.

There have been some attempts to reduce the level of expertise required of the system administrator. Specifically, automated expert systems have been deployed in some data storage systems. These expert systems automatically monitor some or all of the performance metrics and compare them with predetermined metric thresholds when the performance metric is received. If the received performance metric meets or exceeds a corresponding metric threshold, then the expert system can notify the system administrator that their attention is required. Such expert systems can operate rather effectively and efficiently in a particular setting as long as the behavior of the system or system environment does not change. However, such expert systems can only be effectively used with systems in which they are an “expert”. In other words, the expert system must be programmed to deal with certain situations that occur within a particular system under certain conditions. If the behavior of the system does change, then the expert system is no longer an expert. Rather, the thresholds programmed in the expert system along with other algorithms provided therein may become useless when trying to manage the system having new behaviors.

SUMMARY

The present invention is directed to solving these and other problems and disadvantages of the prior art. In accordance with embodiments of the present invention, a computer resource management system is provided. More specifically, the computer resource management system is equipped with a management application. In one embodiment, the management application comprises an inference engine that is capable of receiving various data inputs related to the operation of the computer resource. The inputs received at the management system are mapped against a set of policies that define how the computer resource is to be managed. Based on the mapping, the inference engine can then make asynchronous decisions regarding how the computer resource should be managed.

In accordance with embodiments of the present invention, a decision can be made to change the configuration of the computer resource. A configuration change order can be issued by the management system, which in turn can be executed by the computer resource and/or a system administrator. The configuration change may require the redistribution of computer resource assets within the computer resource and between other computer resources.

In accordance with further embodiments of the present invention, a decision can be made to change one or more policies in the set of policies that define how the computer resource is to be managed. A policy change can be instituted in response to a change in activity of a particular computer resource. For example, if an initial set of policies was based on a particular estimated amount of activity and the activity of the computer resource varies from the estimated activity, then one or more policies in the set of policies can be updated such that the computer resource is properly managed based on the actual level of activity and not the estimated level of activity.

In accordance with still further embodiments of the present invention, when a configuration and/or policy change is initiated, feedback may be provided to the computer resource management system. The feedback can show what type of change was made and how it affects the computer resource. The feedback also provides the computer resource management system a mechanism for staying updated on the current state of the computer resource. In one embodiment, the computer resource management system uses the feedback as input for making subsequent management decisions.

Additional features and advantages of embodiments of the present invention will become more readily apparent from the following description, particularly when taken together with the accompanying drawings.

DETAILED DESCRIPTION

FIG. 1Ais a block diagram depicting an electronic data system100in accordance with embodiments of the present invention comprising a computer resource102. The computer resource102generally includes intelligence or functions for managing or controlling computer resource assets within the computer resource102. In one embodiment, the computer resource102may comprise a storage system104such as a RAID storage system or the like for storing large amounts of data.

The electronic data system100may also include a host processor, computer or computer system108. Additionally, the electronic data system100can include an external management workstation110comprising a remote storage monitor113. The host108can be connected to the storage system104through an in-band network116and/or an out-of-band network120. The networks116,120generally provide or support communication capabilities between the host108, the storage system104, and, if included, the external management workstation110.

The in-band network116generally functions to transport data between the storage system104and host108, and can be any data pipe capable of supporting storage system104data transfers. Accordingly, examples of in-band networks116include Fibre Channel (FC), iSCSI, parallel SCSI, Ethernet, ESCON, or FICON connections or networks, which may typically be characterized by an ability to transfer relatively large amounts of data at medium to high bandwidths.

The out-of-band network120generally functions to support the transfer of communications events and/or messages between the external management workstation110, the storage system104, and the host108. Examples of an out-of-band communication network120include a local area network (LAN) or other transmission control protocol/Internet protocol (TCP/IP) network. In general, the out-of-band communication network120is characterized by an ability to interconnect disparate nodes or other devices through uniform user interfaces, such as a web browser without impact to the data pipe feeding system104.

In one embodiment, the remote storage monitor113application or software utility residing on and executed by the external management workstation110receives event and other performance data from the computer resource102such as the storage system104and/or host108. The external management workstation110employs the remote storage monitor113to analyze the event and performance data to automatically manage the operation of the computer resource102.

In another embodiment, a storage monitor112comprises an application or software utility executed by a controller111that resides in the storage system104and manages the operation of the storage system104. The storage monitor112can receive data or commands from the storage system104, the host108and/or the external management workstation110notifying it of certain conditions associated with the electronic data system100. Based on the data received, the storage monitor112can adjust the configuration of computer resources employed by the storage system104and/or policies governing the operation of the storage system104. In accordance with still other embodiments of the present invention, functions related to automatic management of the storage system104may be distributed between one or more storage monitors113running on one or more workstations110and/or one or more storage monitors112running on one or more storage systems104. The storage monitors112and/or113may further provide a user interface for controlling aspects of the operation of the storage system104.

The storage monitors112and/or113are typically in communication with a resource asset pool106, which is a logical representation of the resource assets not currently being employed by the storage system104but otherwise available for use by the storage system104. Some of the computer resource assets listed in the resource asset pool106may be physically associated with the storage system104whereas other computer resource assets may be external to the storage system104and are associated with another computer resource102. Computer resource assets listed in the resource asset pool106may include spare disk drives or other storage devices, spare controllers, spare processors, spare data channels, and the like.

FIG. 1Bis a block diagram depicting a data system100in accordance with embodiments of the present invention incorporating a computer resource102such as a simple storage system105. In contrast to the storage system104described above in relation toFIG. 1A, the simple storage system105does not include a controller or intelligence for controlling computer resource assets internally. Accordingly, the configuration of computer resource assets within the simple storage system105is controlled and managed by the external management workstation110or through the host108. Thus, the simple storage system105is only connected to the host108through the in-band network116. The external management workstation110may communicate with the host108via the out-of-band network120. The host108can then forward commands received from the management workstation110to the simple storage system105through the in-band network116. In accordance with still other embodiments of the present invention, the functions of the host108and the external management workstation110can be performed by a single device.

The host108may also comprise a host system monitor114for controlling the operation of the simple storage system105. Accordingly, a host108may also include a host system monitor114for controlling a storage system104whether the storage system includes a storage monitor112or not. The host system monitor114can be used to control the operation of the computer resource assets within the simple storage system105. Alternatively, the host system monitor114may receive commands and/or other data from a remote storage monitor113to control the computer resource assets of the simple storage system105.

Generally speaking, the host system monitor114operates in a similar fashion to the storage monitor112described above in that the host system monitor114receives performance and other data related to the simple storage system105and then controls the operation of the simple storage system105based on the received data. The data may be received from the simple storage system105and/or may be received through the operation of the host108. As can be appreciated, the types of data received from the simple storage system105may differ from data received through the host108.

Certain embodiments of the present invention do not employ both the host system monitor114and the remote storage monitor113to manage the simple storage system105. For example, the host system monitor114may be the sole monitoring application required to manage the simple storage system105. In such embodiments, an out-of-band network120may not be necessary to manage the simple storage system105. Alternatively, the remote storage monitor113may be the only monitoring application that manages the simple storage system105.

FIG. 1Cis a block diagram depicting a data system100in accordance with embodiments of the present invention incorporating a plurality of hosts108a-N connected to a computer resource102, such as a storage system104, via a Storage Area Network (SAN), where N is greater than or equal to one. The SAN represents one type of in-band network116that can be used to connect multiple hosts108to a storage system104. Each host108can communicate through the in-band network116with the storage system104. Additionally, the storage monitor112and/or the remote storage monitor113may be capable of communicating and of receiving performance data from the plurality of hosts108either through the in-band network116and/or the out-of-band network120.

In accordance with embodiments of the present invention, only one storage monitor112or113is used to manage the storage system104. For example, the storage monitor112running on the storage system104controller111can comprise the application managing the storage system104. On the other hand, the remote storage monitor113may comprise the application monitoring and managing the storage system104.

FIG. 1Dis a block diagram depicting a data system100in accordance with embodiments of the present invention incorporating a plurality of hosts108a-N connected to a plurality of storage systems104a-M, where M is greater than or equal to one. The hosts108a-N are generally connected to the storage systems104a-M through in-band network116. The configuration of the in-band network116is similar to the SAN described above in that any of the hosts108a-N can communicate with any storage system104a-M through the in-band network116. Moreover, the hosts108a-N can communicate with the storage systems104a-M via the out-of-band network120if desired.

FIG. 2Aillustrates components that may be included in a storage system104in accordance with embodiments of the present invention. In general, the storage system104includes a number of storage devices204. Examples of storage devices204include hard disk drives, such as serial advanced technology attachment (SATA), small computer system interface (SCSI), serial attached SCSI (SAS), Fibre Channel (FC), or parallel advanced technology attached (ATA) hard disk drives. Other examples of storage devices204include magnetic tape storage devices, optical storage devices or solid-state disk devices. Furthermore, although three storage devices204a-care illustrated, it should be appreciated that embodiments of the present invention are not limited to any particular number of storage devices, and that a lesser or greater number of storage devices204may be provided as part of a storage system104.

A storage system104in accordance with embodiments of the present invention may be provided with a first controller208afor controlling the flow of data to the storage devices204. In addition, other embodiments may include one or more additional controllers, such as a second controller208b. The storage system104may be operated by a single controller208ain a non-redundant mode. As can be appreciated by one of skill in the art, the provision of two controllers208a,208bpermits data to be mirrored between the controllers208a,208b, providing improved performance and data availability over a single controller embodiment.

The controllers208receive instructions and/or data from external devices through the in-band network116and/or the out-of-band network120. The remote storage monitor113running on the external management workstation110and/or the storage monitor112can determine which controllers208should be assigned to which storage devices204. Accordingly, one controller may be assigned to send data to a first subset of the storage devices204while another controller may be assigned a second subset of storage devices204to which it sends data. Moreover, the storage system104is not required to employ all of the storage devices204controllers208or other associated assets at once. Instead, some computer resource assets may remain in an idle state and thus are a part of the resource asset pool106. Computer resource assets in the resource asset pool106can be made available to the storage system104upon request.

One or more device channels117are generally provided for each controller208. The device channels117generally interconnect the corresponding controller208to the storage devices204. Furthermore, while illustrated as a number of dedicated channels117, it can be appreciated that a single shared bus or channel117can be shared between both controllers208. The channels117may further provide the controllers208with the capability to connect to additional computer resources102like a simple storage system105or other devices.

Although some storage systems104are generally implemented as a complete RAID system, it should be appreciated that the storage system104can comprise one or more storage volumes implemented in various other ways. For example, the storage system104may comprise a hard disk drive or other storage device204connected or associated with a server or a general-purpose computer.

Referring toFIG. 2B, a simple storage system105will be described in accordance with embodiments of the present invention. The simple storage system105may comprise a Just a Bunch of Disks (JBOD) system or a Switched Bunch of Disks (SBOD) system. The simple storage system105may further comprise a bus or network interface212for connecting to either the in-band network116or a storage system104. The simple storage system105does not generally comprise a controller or the like. Rather, data is sent to the simple storage system105with a destination address associated therewith. The data is already directed toward a predetermined storage device204. Thus, no controller is necessary to route the data to any location. As can be seen, additional computer resources102including storage systems104,105may be linked together in known configurations such that a storage system104has more computer resource assets available if they become needed.

FIG. 3is a block diagram depicting a controller208such as may be included in a storage system104in accordance with embodiments of the present invention. In general, the controller208includes a processor subsystem304capable of executing instructions for performing, implementing and or controlling various controller208functions. Such instructions may include instructions for monitoring and managing the operation of a storage system104. Furthermore, such instructions may be stored as software and/or firmware. As can be appreciated by one of skill in the art, operations concerning the monitoring and management of the storage system104may be performed using one or more hardwired and or programmable logic circuits provided as part of the processor subsystem304. Accordingly, the processor subsystem304may be implemented as a number of discrete components, such as one or more programmable processors in combination with one or more logic circuits. Processor subsystem304may also include or be implemented as one or more integrated devices or processors. For example, a processor subsystem may comprise a complex programmable logic device (CPLD).

The processor304is operable to execute instructions and perform tasks associated with the storage monitor112function and in connection with the resource asset pool106function. The storage monitor112may include an inference engine308, or means for making asynchronous decisions, that is capable of making fuzzy logic decisions based on incomplete and/or erratic data. The inference engine308is responsible for analyzing performance and other types of data and mapping them to system management policies. Based on the mapping of the data to the policies the inference engine308can make asynchronous management decisions for computer resources102such as the storage system104,105. In contrast to expert systems, the inference engine308does not necessarily rely upon rigid, predetermined thresholds to make a management decision. Rather, the inference engine308is capable of making management decisions based on loosely defined criterion.

The controller208may include a number of communication interfaces such as one or more host interfaces312, one or more device interfaces316, and one or more management interfaces320. Each of these interfaces represents a potential means for receiving data related to operation of a computer resource102. The host interface312is designed to interconnect the controller208with the host108via the in-band network116. Examples of the host interface312include, without limitation, Fibre channel ports, iSCSI ports, parallel SCSI ports, and so on depending upon the configuration of the in-band network116. The device interface316may be provided for operably interconnecting the controller208to the remainder of the data storage system104, for example the storage devices through channel117. Furthermore, the management interface320may be configured to facilitate communication between the controller208and one or both of a storage monitor112running on another storage system104and a remote storage monitor113running on a management workstation110through the out-of-band network120.

FIG. 4depicts functional aspects of an exemplary storage monitor112,113capable of performing management application functions and data flows in accordance with embodiments of the present invention. As previously noted, the storage monitor112,113may be implemented in a storage system104, a host108, and/or an external management workstation110. The storage monitor112,113generally comprises an inference engine308, a configuration data store400for maintaining configuration data401and resource asset pool106data, a statistics data input402, an event input403, and a policy store404. The statistics data input402and event input403, which typically function as a means for receiving data, are generally considered system operational data since they relate to the operation of the computer resource102, for instance the storage system104,105. The configuration data store400and policy store404may comprise data sets stored in memory or other data storage.

The inference engine308receives data from the configuration data store400, the statistics data input402, and the event input403and analyzes the data from each of the inputs to determine a state of the computer resource102. The state of the computer resource102, in one embodiment, is a determined status of the system based on available operational information and configuration data401. Additionally, a projected future state of the system may be determined based upon the configuration data401and the resource asset pool106data. Examples of states that may be identified by the inference engine308include system stable, system unstable, system nearly full, system nearing failure, system failed, system behind schedule, storage device nearing failure, storage device failed, controller near failure, controller failed, or any other known state related the computer resource102or assets associated with the computer resource.

In determining a system state, each of the inputs may be assigned a different weight. Thereafter, the inference engine308maps the weighted inputs to one or more policies received from the policy data store404to determine if any actions should be taken to control the computer resource102. Generally, the inference engine308can make asynchronous decisions based on real-time inputs because the inference engine308does not rely on hard thresholds. Rather, the inference engine308can receive an input and wait to see if subsequent inputs drive the computer resource102status to a point where an action is warranted.

If the inference engine308determines that an action should be taken, then the inference engine308formulates the action consistent with the policies from the policy data store404and sends a message to the computer resource102requesting the implementation of the action. The message is transmitted through a message output413to the computer resource102. Additionally, the message is provided as feedback to the event input403thus making the inference engine308aware of the actions it has just requested. In other words, messages are provided as feedback to the inference engine308so that the inputs to the inference engine308are current.

In one embodiment, the message may generally include executable instructions for storage system controllers208to perform. Alternatively, the message may include a request that actuates one or more switches connecting more computer resource assets to the computer resource102. Another type of message that can be generated is a message to a system administrator requesting the execution of certain actions or identifying certain computer resource102issues. Messages may be sent via any suitable message transmission protocol including, without limitation, short message service (SMS), TCP/IP protocols, and SCSI protocols.

If the action corresponds to a configuration change or some other change related to the computer resource102, then the action is sent to the computer resource102in the form of a configuration change order. When the inference engine308issues a configuration change order, the order is also provided as feedback411to the configuration data store400. This update of the configuration data store400allows the inference engine308to use updated configuration data401when determining whether subsequent actions are warranted. As can be appreciated by one of skill in the art, an update to the configuration data store400may include transferring descriptors of computer resource assets between the configuration data401and the resource asset pool106data. If computer resource assets were added to the computer resource102from the resource asset pool106, then the listing of those transferred resource assets are removed from the resource asset pool106data and are added to the configuration data401. Configuration changes to the computer resource102may be carried out through a suitable interface, such as SMI-S, SNMP or proprietary APIs.

A further type of message that can be generated by the inference engine308is a policy modification or update message414. As changes are implemented in the computer resource102or as changes to the environment about the computer resource102occur, it may be necessary to update policies in the policy data store404in connection with those changes. A policy update414is issued by the inference engine308and received at the policy data store404. The creation of this type of message may also automatically generate a policy change notification message that can be sent to a system administrator notifying them of the change to the policy data store404. As can be appreciated, since the inference engine308can alter its policy data store404as changes are implemented, the management of the computer resource102becomes dynamic and responsive to changes when they are made.

When the inference engine308makes a decision regarding the management of the computer resource102and issues a configuration change order411, a policy modification message414, and/or any other type of message, it may also send a copy of the message to an action log412. A system administrator can later reference details about the action through the action log412. The action log412provides a single point where the performance of the inference engine308and any computer resource102managed thereby can be audited and analyzed.

Initially at least, the policy data in the policy data store404may comprise a set of default heuristics regarding the operation of its computer resource102being managed. These policies or heuristics are generally described in terms using conceptual event and statistical data. For example, the storage volume is “almost full,” the job is “behind schedule,” the CPIJ is “about to fail” and so forth. In addition, the provided heuristics are appropriately weighted to best describe the goals of the computer resource102user community. For example, reliability may be the highest priority for the user community, while performance may be secondary. These policies may be modified, substantially or in part, deleted, or added to as needed to meet the intended goals of the user community. In addition to changes to policy effected by the operation of the system monitor112,113, policy changes may be entered directly by a system administrator.

The inference engine308evaluates the streams of information against the provided (or effective) heuristics or policies. This can include categorizing the information into a weighted priority scheme to best meet the policies of the computer resource102. Moreover, even though streams of information may be complete, incomplete or noisy, such combinations will not prevent the inference engine308from taking timely or appropriate action. If necessary, the inference engine308will perform an extrapolation of the available data to make the best possible decision based on the provided policies. Accordingly, actions taken by the inference engine308(or more generally the storage monitor112,113) will be based on the effective policies (either as supplied or as modified), the configuration of the target system, the available resources of the target system, and the relevant statistical and event data. Action may therefore be taken based on vague, incomplete or noisy data from or about the computer resource102by evaluating the available data using intelligent (fuzzy) logic to best interpret the data against the policies.

With reference toFIG. 5, a process for managing a computer resource102is illustrated in accordance with embodiments of the present invention. Initially, at step504, data is received by the inference engine308. The received input data may include configuration data401and/or resource asset pool106data from the configuration data store400, statistical data from the statistics data input402, and event data from the event input403. In some instances the received input data may be incomplete or otherwise unusable due to noise or interference encountered during transmission to the inference engine308. Additional data that may be received by the inference engine308includes policy data from the policy data store404. The inference engine308then maps the input data to the policy data (step508). The policy data used in accordance with embodiments of the present invention may use descriptive language to identify conditions that are one of specific and vague in evaluation. As used herein, “vague” is understood to refer to an inexact value as opposed to a definitive threshold. As an example, a policy may be worded, “When the volume is nearly full—incrementally expand the volume to support the goals.” The use of imprecise values (i.e., nearly full and incrementally expand) liberates the inference engine308to evaluate, for example, a rate of consumption, which may be faster than the time it takes to institute any remedial action, possibly resulting in a loss of availability or significant reduction in performance. The inference engine308may then choose to implement an action at 70%, 76%, 82%, or possibly 97% of volume capacity, expanding the volume incrementally (e.g., by percentage of use) as necessary. The inference engine308may further attempt to ensure fulfillment of the policies without loss of availability and minimal reduction in performance. It is this imprecise definition used in accordance with policy data that allows the inference engine308to manage a computer resource dynamically and effectively.

In the event that the received input data was incomplete or unusable, then the inference engine308may only map the usable input data to the policy data. Alternatively, the inference engine308may guess or interpolate values for the unusable input data and map the interpolated data to the policy data. In accordance with some embodiments, raw input data may be mapped to the policy data. In other embodiments, some or all of the input data may be weighted before it is mapped to the policy data as noted above.

After the input data has been mapped to the policy data, the inference engine determines whether an action is necessitated based on the state of the computer resource102and the policy data governing the management of the computer resource102(step512). As noted above, the decision made by the inference engine308may be an asynchronous decision meaning that it is not made at the instant when a particular threshold is met or exceeded. In some storage systems input data may be volatile or erratic and for this reason the inference engine308does not rely on hard thresholds. The inference engine308can note that some thresholds are being exceeded but based on the policy data may wait to make a decision to implement an action. For example, if the input data continues to exceed a threshold for a certain amount of time then the inference engine308may decide to implement an action consistent with the policy data.

If an action has been deemed necessary by the inference engine308, then the requisite action is identified by the inference engine308(step516). An action may include the addition, removal, transfer, reconfiguration, or other adjustment to one or more computer resource assets currently employed by the computer resource102. After the action has been identified, the computer resource asset(s) that will be affected by the action is identified (step520). The identified computer resource asset(s) is then redeployed according to the determined action (step524). The computer resource asset(s) may be added from, or returned to, the resource asset pool106and a descriptor may be updated in the resource asset pool106data to reflect the same. If an additional computer resource asset is required and no such assets are available in the resource asset pool106they may be ordered from another source, such as an outside vendor. Alternatively, the inference engine308may scan lower priority computer resources102for suitable computer resource assets. If such a resource asset is found in a lower priority computer resource102then computer resource assets may be disconnected therefrom and reassigned to the higher priority computer resource102. In another embodiment, an identified action may include sending a message to prepare for the addition or removal of a computer resource asset. In still other embodiments, an action might include the adjustment to the frequency with which backups are performed and/or snapshots are taken in connection with a computer resource102comprising a storage system104,105.

After the computer resource asset has been redeployed, the action is recorded in the action log412(step528). The recordation in the action log412may include an indication that the configuration data401and resource asset pool106data have been changed in the configuration data store400. As previously noted, this allows the system administrator to stay apprised of the computer resource102management decisions made by the inference engine308. Of course, if no actions were deemed necessary by the inference engine308in step512, then steps516-528can be skipped.

In addition to action decisions, the inference engine308may determine whether a change in policy data is necessary (step532). The issuance of a configuration change order411or other type of action request may warrant a change in policy data. Alternatively, a change in the environment or usage of the computer resource102may warrant a policy change without necessarily requiring a configuration change. Examples of changes to the usage of the computer resource102that may result in a policy change include an increase in rate of data storage capacity consumption, a decrease in rate of data storage capacity consumption, an increase in the number of tasks being processed per unit time, a decrease in the number of tasks being processed per unit time, and so forth. Similar to the action decision made in step512by the inference engine308, the policy decision made in step532can be an asynchronous decision. In making asynchronous policy decisions the inference engine308gives itself time to further analyze the state of the computer resource102or to receive additional data before a policy decision is made.

If the decision has been made to change a policy, then the inference engine identifies which policy or policies should be changed (step536). The identified policy or policies are then changed according to the decision of the inference engine308(step540). In one example, the relative importance of various policies in the policy data store404can be changed if more or less activity has been observed in certain computer resources102. As a further example, an initial policy might be stated as, create a 5 TB Online Transaction Processing (OLTP) database that will handle 1200 Transactions per second (TPS) and grow at the rate of 1 TB per year. If the number of transactions per month is significantly higher than was expected when the initial policy was instituted, then the policy may need to be changed to have the system grow at a rate of 1.5 TB per year. Alternatively, the policy may be changed if more users are requiring data storage at the same time. In this instance, the policy may be changed to require the database to handle 1500 TPS instead of the original 1200 TPS. As can be appreciated, other aspects of policies may be modified or updated based upon certain conditions of the computer resource102. Once a policy has been changed, the change is recorded in the action log412(step544). The system administrator can then be notified of the change to the policy as well as any other actions that have been taken by the inference engine308(step548).

If the inference engine308determines that no policy decision is necessary, then a system administrator can be notified of any actions or inactions of the inference engine308(step548). A system administrator may be notified through a message sent directly to him/her or the system administrator may be notified through a request to view the action log412. After the system administrator has been notified of the change, if any, then the process continues to step552where it is determined if the storage monitor112,113is deactivated. If the storage monitor112,113is not deactivated (i.e., it is still monitoring the computer resource102), then the process may return to step504to receive inputs and determine if an action is necessary based on new policies or existing policies if there was no policy change. However, if the storage monitor112,113is deactivated, then the process ends (step556).

Referring now toFIG. 6, a process for managing a computer resource102such as may be implemented by a storage monitor112,113in accordance with other embodiments of the present invention is illustrated. The method starts in step604then proceeds to step608where information regarding computer resource assets is collected. In this particular step the inference engine308may scan for all computer resource assets associated with the computer resource102under its management. The computer resource assets discovered during this scanning step can then be compared to a list of known resources in the configuration data store400. If one of the resources is not in the list of known resources in the configuration data store400, then the attributes for that resource are discovered by the inference engine308using known query and response methods. Then the information regarding the computer resource assets is stored or otherwise maintained in the configuration data store400(step612). Those computer resource assets that are actively associated with the computer resource102may have their descriptors stored in the configuration data401whereas those computer resource assets not actively associated with the computer resource102may have their descriptors stored in the resource asset pool106data portion of the configuration data store400. Once the configuration data store400has been updated properly, any policy changes that are made in response to the determination of computer resource assets and configuration data may be recorded in the policy data store404(step616). The inference engine308then evaluates and uses the most recent policies in the policy data store404and compares them with input data from the computer resource102(step620).

The inference engine308continues to receive input data from the computer resource102and maps that to the policy data. The input data may include event data as well as statistical data. An event may occur that drives the inference engine308to determine that a configuration change is warranted in order to comply with or satisfy the current policies. Such events and subsequent asynchronous decisions are illustrated as broken lines connecting step620and step624. Examples of an event that may lead to an asynchronous decision by the inference engine308includes, without limitation, event data indicating that a job is not in compliance with policies or statistics data indicating that the computer resource102performance is changing or that a computer resource asset has failed. If such an event occurs, then the inference engine308determines if any computer resource assets are readily available in the resource asset pool106for use by the computer resource102that can be activated in order to satisfy the current policies (step624). In the event that computer resource assets are available in the resource asset pool106that can be applied so that the computer resource102will satisfy (or better satisfy) the current policies, then those resource assets are allocated to the computer resource102(step628). Thereafter, the changes in the computer resource102are logged in the action log412(step632). Any configuration changes are also recorded in the configuration data store400(step612). The change to the configuration data store400may include moving descriptors of the allocated computer resource assets from the resource asset pool106data to the configuration data401.

If there are no computer resource assets readily available for the computer resource's102use, then the inference engine308analyzes existing resource assets that are currently being used by other computer resources102to determine if existing computer resource assets can be adjusted (step636). Specifically, the inference engine308may scan all computer resources102that have a lower priority than the computer resource102requiring an additional computer resource asset. If it is determined that existing computer resource assets can be adjusted, then those computer resource assets from the lower priority computer resource102can be reallocated to the higher priority computer resource102(step640). In this step, the computer resource assets that are being reallocated are disconnected from the lower priority computer resource102and connected to the higher priority computer resource102. Thereafter, the changes to the configuration of all affected computer resources102are logged in the action log412(step632). Then the configuration data store400is updated to reflect the same changes (step612). However, if there are no computer resource assets that can be reallocated, the inference engine308continues monitoring the computer resource102performance and may notify a system administrator that additional computer resource assets are required from an outside source (step644). As a part of this step, the inference engine308may update the policy data store404to show that the computer resource102is malfunctioning or operating in a near failure or non-fault tolerant state. This may result in alteration to the policy data. For example, policies may be altered so that backups and/or snapshots are taken more frequently.

With reference toFIG. 7, an exemplary storage monitor112,113process of expanding capacity in a computer resource102will be described in accordance with embodiments of the present invention. The method starts in step704then proceeds to step708where event triggers are initialized. An event trigger may be associated with policy data used to make asynchronous decisions regarding the management of the computer resource102. Once the event triggers are initialized, meaning that they are set to a default value where no action is to be taken, the inference engine308monitors the computer resource102and waits until the occurrence of an event that changes the value of the event trigger. The broken arrows depicted connecting step708and712represent an asynchronous decision that one or more event triggers should have their default value changed. Examples of such events include the inference engine308determining that the computer resource102is in an almost full state or that the rate of tasks being processed by the computer resource102has changed. Other examples of events that may be received from the event input403include job start notifications, job completion notifications, failure notifications, system change notifications, abnormal storage condition notifications, and so forth. If one of these or a similar event occurs, then the inference engine308determines if an increase in capacity of the computer resource102is required (step712). The inference engine may wait and continue monitoring input data for a certain amount of time after the event trigger has changed values to ensure that an expansion is required. If after analyzing further input data the inference engine308determines that no increase is required, then the values of the event triggers are reset to their default value and the inference engine308continues monitoring the computer resource102(step716). On the other hand, if the inference engine308determines that an increase in capacity is required, then the inference engine308determines the amount of computer resource assets needed to make the determined expansion (step720). The inference engine308uses the configuration data401and/or resource asset pool106data from the configuration data store400to determine what amount of computer resource assets are needed and what computer resource assets are available (step724). The inference engine308may determine that additional storage devices204or controllers208are needed to make the expansion. Alternatively, the inference engine308may determine that other computer resource assets are needed, such as additional processors or servers to make the desired expansion.

After the inference engine308has determined the amount of computer resource assets needed to accommodate the expansion, it calculates the amount of time needed to make the expansion (step728). The amount of time needed can be determined by using statistical data from the statistics data input402(step732). The statistical data may help determine how much time the computer resource102has before it completely fails or otherwise becomes unable to meet the policies contained in the policy data store404. Examples of statistical data include, but are not limited to computer resource102loading statistics, deviations in operation performance, number of jobs currently in queue, job receipt rate, job completion rate, and the like. Moreover, the statistical data may comprise a mathematical indicator of computer resource102performance parameters and may be an absolute value or a relative deviation from desired system performance. The inference engine308may also estimate the amount of time needed to add the required computer resource assets based on their availability or the lag time inherent in ordering such a computer resource asset.

Once the amount of time needed to expand is determined, the inference engine308determines if the expansion can be made slowly or whether the expansion needs to be made relatively quickly (step736). This determination can be based upon current policy data received from the policy data store404(step740). For example, if the subject computer resource102requiring expansion is relatively important to business operations and thus has a high priority, then the inference engine308may determine that a fast change is required. However, if the subject computer resource102has a relatively low priority, then the inference engine308may determine that a slow change will suffice. If a fast change is required, then the inference engine308sends a request to expand the computer resource102by adding computer resource assets thereto (step744). These resource assets may be retrieved from the resource asset pool106or may be borrowed from a lower priority computer resource102. Alternatively, if a slow change can be accommodated, then the inference engine308may schedule an expansion of the computer resource102by ordering computer resource assets from an alternative source and may notify a system administrator that computer resource assets will arrive and should be added to the computer resource102by a certain date.

After the expansion has either been implemented or scheduled, then the corresponding policies are updated in the policy data store404(step752). As can be appreciated, policy data for the computer resource102receiving the expansion may not be the only policy data that is updated. If computer resource assets were borrowed from a lower priority computer resource102, then the policy data associated with that computer resource102can be updated as well. Furthermore, the statistic input is updated to reflect the expansion of the computer resource102(step756). In addition to updating statistical data, the inference engine308may update the configuration data store400to reflect the configuration changes (step760). As noted above, descriptors of new computer resource assets may be added to either the configuration data401or the resource asset pool106data depending upon where the computer resource asset is added. Moreover, descriptors for computer resource assets moved from the resource asset pool106to an active state in the computer resource102may be moved from the resource asset pool106data to the configuration data401. After all of the requisite data and data stores have been updated, the method ends (step764). As can be appreciated by one skilled in the art, when the method ends, it may cycle back to the start and continue monitoring/managing the computer resource102.

A particular example of updating policies in response to the discovery (or alternatively the reading) of computer resource configuration data, expressed in pseudo-code, is as follows:

{Scan for all physical devicesCompare discovered devices to known devices in the systemIf device is new{Get its attributesPrepare it for useAdd it to the Resource Pool}Else if known device is discovered to be malfunctioning or missingIf known device was in use{Isolate it until maintenance can determine actionMove device from Configuration DB to Resource PoolUpdate Resource Pool descriptor showing device as blacklistedLocate a suitable replacement from the Resource PoolIf replacement found{Prepare it for useRemove it from Resource PoolConfigure it into the target vdiskStart rebuild, if neededWhen finished, add it to the Configuration DB}Else{Evaluate usage of resources in Configuration DBIf a lower priority vdisk has a suitable replacement{Disconnect that device from the lower priority vdiskConnect that device to target vdiskUpdate policies to show disabled lower priority taskUpdate configuration DB to show new disk in target}Else{Update policies to show disabled target vdiskGenerate event(s) to reflect conditionUpdate logs}}}}

Although the above description generally includes examples of storage systems104,105as the computer resource102, the present invention is not limited to the management of storage systems104,105. Instead, novel aspects of the present invention may be deployed in the automatic management of any computer resource102including, without limitation, storage systems104,105, a collection of servers, a processor bank, and/or other collections of computer enabled devices or facilities.