Method and system for throttling log messages for multiple entities

A software module capable of simultaneously supporting multiple services provides log message throttling for each service with a separate “per service” log message buffer. When the software module is a device driver, for example, each device controlled by the device driver is allocated a message buffer to store descriptive log messages. Upon generation of a warning log message, descriptive log messages in the message buffer are flushed to a log file for review by an administrator. Furthermore, log message throttling may be implemented by only flushing the message buffer upon certain occurrences of warning log messages, such as in accordance with an exponential back-off algorithm.

BACKGROUND OF THE INVENTION

Computer administrators typically review error log files stored on a disk drive or other permanent storage media (e.g., flash drive) to investigate problems relating to the configuration and operation of devices in a computer system. These error log files contain text messages (or “log messages”) that provide warnings and/or errors (referred to herein as “warning log messages”) and descriptions and/or other information (referred to herein as “descriptive log messages”) that enable the computer administrator to better understand why a warning log message may have occurred. Log messages in an error log file pertaining to a particular device are added to the log file as a result of log requests made by a device driver communicating with the device. A device driver, as part of its typical operation, will continually request descriptive log messages to be written to the error log file. Such descriptive log messages relate to a state of the device at the time the request was made. Additionally, to the extent that an error occurs during the device driver's communication with the device, the device driver will also request that a warning log message be written to the error log file.

Continual writing of log messages to an error log file requires continual access to the disk drive through repeated disk I/O (Input/Output) operations. To reduce such repeated disk I/O operations, a device driver can allocate a message buffer (typically, as a circular buffer) in RAM (Random Access Memory) of the computer system to preliminarily store descriptive log messages. When the device driver generates a warning log message, the device driver then flushes the descriptive log messages stored in the message buffer to the error log file. In this manner, descriptive log messages that arise during a time distant from the occurrence of a warning log message are simply overwritten in the circular buffer by other descriptive log messages and are therefore never written to the error log file. This result is desirable because such overwritten descriptive log messages are not relevant to resolving a warning log message since they relate to states of the computer system during times that are distant from the occurrence of the warning log message.

Another technique to minimize disk I/O operations is to “throttle” warning log messages. In many instances a device driver repetitively generates a great many warning log messages relating to the same error or device driver state because, for example, the device driver continually tries to unsuccessfully recover from the error or because the device driver repetitively encounters the same issue generating the warning (e.g., the device has become non-responsive or is otherwise not performing as expected). Because the same issue is repeatedly encountered, the same descriptive log messages that are relevant to the issue are continually generated and inserted into the circular buffer. During a first occurrence of the warning log message, these relevant descriptive log messages are flushed from the circular buffer into the error log file. As such, the occurrence of the repeated warning log messages causes the device driver to repeatedly flush the same relevant descriptive log messages from the message buffer to the error log file, thereby unnecessarily consuming disk I/O resources.

A device driver can “throttle” back warning log messages (i.e., flush the circular buffer only on certain occurrences of the repeated warning log message) utilizing a heuristic, such as an exponential back-off algorithm, to reduce its consumption of disk I/O resources. In one example of log throttling using exponential back-off, a device driver maintains a count, referred to herein as a “throttle count,” and flushes the circular buffer each time a particular warning log message occurs until the throttle count is reached. Subsequent occurrences of the warning log message are exponentially ignored such that only the subsequent 2Noccurrences (i.e., 2nd, 4th, 8th, 16th, etc.) of the warning log message cause a flush of the message buffer. To properly keep track of different warning log messages that may be simultaneously occurring, the device driver needs to maintain a count for each particular warning log message instance. The device driver therefore needs to allocate a static local variable for each particular warning log message which can significantly increase the memory requirements of the device driver.

With the prevalence of standardized physical interfaces, such as USB (Universal Serial Bus), that support multiple types of devices, current device drivers have a corresponding capability to simultaneously support multiple devices that utilize such standardized physical interfaces. For example, a particular USB device driver may support multiple device types within a particular USB “class,” such as a mass storage device class or an HID (human interface device) class. A mass storage USB device driver may support a computer system's communication with external hard drives, portable flash drives, digital cameras, portable media players and mobile phones. Similarly, an HID USB device driver may support the computer system's communication with a keyboard and a mouse, as well as joysticks, trackballs and other input devices. When such a “device class” driver simultaneously supports different connected devices, log messages are generated and stored in a circular buffer that is shared by all of the devices. As such, when the device driver generates a warning log message relating to one device, the log messages that are flushed from the message buffer to the error log file may relate to multiple devices, thereby complicating the administrator's task of isolating log messages relating to the device causing the warning log message. Furthermore, because the circular buffer has a limited size, descriptive log messages that are relevant to debugging a warning log message for one device may be inadvertently overwritten by descriptive log messages of other devices prior to being flushed to the error log file. This is particularly likely to be the case if one device has malfunctioned before another, thereby filling up the circular buffer and the error log file with log messages that have no relevance to a second device's failure. Additionally, a device driver that supports multiple devices and implements the foregoing log message throttling techniques may discard all of the warning messages related to the failure of a second failing device because throttling has occurred in response to the failure of a first device.

SUMMARY OF THE INVENTION

One or more embodiments of the present invention provide methods and systems for logging messages generated by a device driver that simultaneously supports multiple devices. Each device supported by the device driver is allocated its own message buffer to store log messages relating to such devices operations. Such “per-device” message buffers eliminate the foregoing discussed issues relating to loss of relevant log messages, difficulty in analyzing the log file and maintenance of “per-warning log message” throttle counts for each device.

In one such method, a device driver allocates a plurality of message buffers equal to the plurality of devices that it is simultaneously supporting, wherein each message buffer in the plurality of message buffers corresponds to one of the devices in the plurality of devices and is allocated to store log messages for the corresponding device. When the device driver generates a warning log message in response to interacting with one of the devices, it flushes log messages in the corresponding message buffer of the device to a log file upon generating the warning log message in accordance with a log message throttling heuristic. In this manner, log messages stored in a message buffer for one device are not overwritten by log messages relating to other devices. The error log file is more easily reviewable because log messages relating to a device are clustered together.

Additionally, memory space utilized by the device driver to maintain throttle counts can be reduced by allocating a throttle count for each device rather than allocating a throttle count for each warning log message of each device. A device driver's use of memory space is reduced and exponential back-off throttling of log messages due to a first failing device does not affect a second failing device.

DETAILED DESCRIPTION

FIG. 1depicts a block diagram of a computer system in which one or more embodiments of the invention may be utilized. Computer system100comprises a processor105, a system memory (e.g., RAM)110, and an internal hard drive115. These components of computer system100communicate with each other through an internal bus system120. System memory110supports the execution of an operating system125that enables a variety of applications130to run in computer system100. In order to communicate with peripheral I/O devices, operating system125includes device drivers that serve as a communication intermediary between the peripheral I/O devices and operating system125. For example, operating system125includes a USB mass storage device driver135and a USB HID driver140that enable operating system125to interact with the various peripheral USB devices. As shown, USB mass storage device driver135simultaneously supports USB CD-ROM/DVD-ROM drive145, USB flash drive150and USB digital camera155, and USB HID driver simultaneously supports USB keyboard160and USB mouse165. Operating system125further includes a system error handler component170that receives requests from USB mass storage device driver135and USB HID driver140to handle log messages generated by such drivers. System error handler component170, in turn, interacts with internal hard drive115(via a device driver for the internal hard drive, not shown) in order to write log messages received from such device drivers to a system error log file175stored in internal hard drive115(as indicated by the dotted line inFIG. 1). It should be recognized that the component architecture shown inFIG. 1is a simplified representation of an actual computer system and one or more embodiments of the invention may be used in many other computer system architectures other than computer system100. For example, while the USB standard is shown inFIG. 1as the supported interface between the depicted device drivers of operating system125and the depicted peripheral devices, it should be recognized that various other standards may also be utilized, including, for example, a SCSI (Small Computer System Interface) standard, a PCI (Peripheral Component Interconnect) standard, and the like.

FIG. 2is a schematic diagram of a device driver. As depicted, USB mass storage device driver135allocates a separate log message buffer for each device that it simultaneously supports. Message buffer200has been allocated for USB CD-ROM/DVD-ROM drive145. Message buffer205has been allocated for USB flash drive150. Message buffer210has been allocated for USB digital camera155. In one embodiment, each message buffer is allocated during an initialization process when the corresponding USB device connects to the computer system (i.e., through a USB port).

Each of message buffer200,205and210may be implemented as a circular buffer shown inFIG. 3. Circular buffer300includes a vector of entries. Each entry, such as entry305, accommodates a text string of a specified length to accommodate a log message. Circular buffer300also utilizes a start index310and a current index315such that the current log messages in circular buffer300are flushed (e.g., upon generation of a warning log message) to the error log file in the appropriate order in which they were generated during operation of the device driver. Prior to inserting another log message into circular buffer300, current index315is incremented to the next available entry in the buffer. If incrementing current index315causes current index315to refer to the same entry as start index310, then start index310is also incremented. As depicted inFIG. 3, the shaded entries of circular buffer300are filled with current log messages while the unshaded entries are empty (or otherwise contain previously flushed log messages). It should be recognized that any other data structures may be utilized to implement a log message buffer, including, for example, linked lists, arrays, collections and the like.

FIG. 4depicts a flow chart of throttling log messages on a per-device basis, in accordance with one or more embodiments of the present invention. In step400, the device driver communicates with one of the devices that it supports. In step405, depending upon a state of the device, the device driver generates a log message. If, in step410, the log message is a descriptive log message rather than a warning log message, step415is executed. In step415, if the log message buffer has already been allocated for the supported device, the generated log message is inserted into the device's allocated log message buffer in step420. If the log message buffer has not been allocated, it is allocated in step425prior to the insertion step of420.

Returning to step410, if the log message is a warning log message, step450is executed. In step450, the device driver determines whether it is already throttling back warning log messages for the device. If warning log messages are not being throttled, the throttle count for the device is incremented in step455. In step460, if the throttle count has reached a specified limit for the device, in step465, the device driver sets warning log messages for the device to be throttled back using, for example, an exponential back-off algorithm and then proceeds to write the warning log message to the error log file in step435, and, if the log message buffer is not empty as determined in step440, flush the log message buffer to the error log file in step445. If the throttle count has not reached the specified limit, then the device driver simply proceeds to write the warning log message to the error log file in step435, and, if the log message buffer is not empty as determined in step440, flush the log message buffer to the error log file in step445, without setting the device to be throttled.

Returning to step450, if the warning log message are already being throttled back (e.g., from a previous setting of the throttling in step465), then the device driver assesses whether the warning log message should be throttled back (e.g., ignored), or whether the warning log message should cause the log message buffer to be flushed. For example, in accordance with one embodiment of an exponential back-off algorithm, if the device driver determines that the warning log message is a 2Noccurrence of a warning log message for the device, then the warning log message is written to the error log file in step435, and the log message buffer is flushed to the error log file in step445if it is not empty as determined in step440. If, however, the warning log message is not a 2Noccurrence, then the warning log message is ignored by the device driver.

As illustrated inFIG. 4, step410does not distinguish whether a warning log message is a repeat of a previously generated warning log message. As such, an embodiment of the invention that implements a flow similar toFIG. 4maintains a throttle count for each device rather for each warning log message of each device. Such an embodiment reduces the amount of memory that the device driver utilizes to store throttle counts (i.e., one throttle count per device).

FIG. 5depicts an alternative flow chart of throttling log messages on a per-device basis, in accordance with one or more embodiments of the present invention. In contrast to an embodiment implementing the flow ofFIG. 4, an embodiment implementing the flow ofFIG. 5maintains a throttle count for each warning log message of each device. The flow ofFIG. 5is similar to the flow ofFIG. 4except that a new step430is executed by the device driver if the generated log message is a warning log message as determined by step410. In step430, if the warning log message is a repeat of a previous warning log message, the flow continues to step450in a fashion similar to that ofFIG. 4. However, if the warning log message is not a repeat of a previous warning log message, then in new step432, the device driver creates a new throttle count for the warning log message.

It should be recognized that implementing “per-device” log message throttling as disclosed herein, wherein each device supported by a device driver is allocated its own message buffer, eliminates the possibility that descriptive log messages stored in a single message buffer that are relevant to resolving a warning log message for one device are overwritten by descriptive log messages relating to other devices prior to being flushed to the error log file. Additionally, the log messages are easier to interpret by an administrator when reviewing the error log file because clusters of log messages in the error log file relate to a single device. Furthermore, in embodiments that implement a throttling flow similar toFIG. 4, the number of throttle counts that are maintained by the device driver is also reduced to the number of supported devices (as opposed to maintaining separate throttle counts for each different warning log message of each different device).

In addition, because each device supported by a device driver is allocated a separate log message buffer, different limits on throttle counts (as checked in step460ofFIG. 4) can be specified for different devices. Specifying different throttle count limits for each supported device provides an administrator a finer granularity of control, enabling certain devices to be more aggressively throttled (i.e., setting a smaller limit on its throttle count), for example, depending upon the criticality of the device to the operation of the computer system.

It should be recognized that various modifications and changes may be made to the specific embodiments described herein without departing from the broader spirit and scope of the invention as set forth in the appended claims. For example, while the foregoing embodiments have described error log message throttling with respect to device drivers communicating with physical peripheral devices, it should be recognized that any type of log messaging system where a management module or component of a computer system simultaneously supports multiple services may utilize the techniques disclosed herein. Rather having the management module or component allocate a single message buffer to log all messages relating to the different supported services, the management module or component allocates a message buffer for each supported service. For example, the per-device log messaging throttling techniques herein may be utilized in a device driver of a guest operating system of a virtual machine that interacts with virtual devices in an emulation layer of a hypervisor. In addition to device drivers, error log files may be similarly utilized by various system software modules of a computer system. System software modules typically add log messages to error log files by calling into another dedicated software module, such as an error handler, that prints log messages on behalf of all other system software modules. These system software modules support the execution of software applications and may result in a stream of descriptive log messages as memory is allocated for processes, file I/O occurs, etc. and, in the event of errors, warning log messages will also be generated.

One example of a system software module is a hypervisor or virtual machine monitor (VMM) that is capable of supporting multiple virtual machines. Each virtual machine supported by the hypervisor is a distinct source of descriptive and warning log messages. Therefore, it is useful to segregate the descriptive log messages according to the virtual machines with which they are associated. A hypervisor of a virtualized computer system may thus utilize “per-virtual machine” log throttling using the techniques disclosed herein to manage the logging of messages across simultaneously running virtual machines. For example, the hypervisor could maintain separate circular buffers for descriptive log messages of each supported virtual machine and, when a warning log message occurs, flush only the circular buffers associated with that virtual machine, as well as the warning log message to the error log file.

In one environment utilizing virtual machines, a computing infrastructure service provider (e.g., a “cloud computing” provider) may offer an online service for customers to purchase and access computer power on a “per-virtual machine” basis. The techniques disclosed herein may be utilized to provide varying levels of service for message logging relating to such purchased virtual machines by specifying the limits on throttle count (i.e., a higher specified limit, n, for more aggressive log messaging and less log message throttling may correspond to a higher level of service). As such, customers can purchase virtual machine computing power based upon a degree of ease of diagnosis in the event that problems occur. Alternatively, the logging techniques herein could also enable the service provider to easily run workloads expected to be problematic with additional diagnostics turned on without impacting other workloads running on the same computer system (i.e., supported by the same hypervisor). It should further be recognized that alternative embodiments may utilize different log files or utilize different operating system components to allocate per-device log message buffers. For example, in one alternative embodiment, the device driver may utilize a separate error log file for each device that it supports rather than communicating with a system error handler to write to a more global system error log file. Similarly, the system error handler of the operating system (or any other operating system component) may allocate the per-device log message buffers for each of a device driver's supported devices rather than having the device driver allocate the buffers. In an alternative embodiment, the size of the message buffers for each device may be configurable during run-time. Alternative embodiments may provide the ability to permanently throttle a device, for example, if the occurrences of warning log messages are exceedingly high. Alternative embodiments may provide a further level of granularity than the flow depicted inFIG. 4, for example, enabling the device driver to maintain throttle counts for different types of warning log messages for each device (i.e., in exchange for savings in reducing memory usage for storing local variables for such throttle counts).