Patent Description:
<CIT> discloses methods for automatically generating help desk tickets in response to detecting performance and/or availability issues that occur throughout multiple layers of a networked computing environment.

The invention is defined by the independent claims <NUM> and <NUM>.

Entities represented in the figures may be indicative of one or more entities and thus reference may be made interchangeably to single or plural forms of the entities in the discussion.

A modularized collaborative performance issue diagnostic system is discussed herein. A collaborative diagnostic system monitors events in a system and identifies a causality chain from a detected performance issue to the root cause of that performance issue. The collaborative diagnostic system includes multiple issue detectors, multiple analysis core modules, and multiple scenario modules.

Each issue detector is a module or component that includes logic to detect known behaviors in the system, such as performance issues in the system, bad behaviors that can lead to performance issues in the system, and so forth. Each analysis core module includes logic to analyze and correlate low level system behavior within a conceptual area. Each conceptual area refers to a class or type of behavior, such as I/O behavior or operations, CPU behavior or operations, power management behavior or operations, and so forth. Within each analysis core module are one or more diagnostic modules (also referred to as helper modules) that are specific to that analysis core module to help determine what is happening in the system. Each scenario module includes logic to take an appropriate responsive action in response to the root cause of a performance issue being determined.

The various diagnostic modules and analysis core modules work together to identify the causality chain and root cause of a performance issue. The diagnostic modules and analysis core modules include query interfaces allowing them to collaborate with one another to track down the root cause of a performance issue and generate a causality chain.

The techniques discussed herein provide a scalable, expandable collaborative diagnostic system allowing root causes of various different performance issues to be determined. Additional issue detectors, analysis core modules, diagnostic modules, and/or scenario modules can be added into the collaborative diagnostic system as desired for newly discovered information regarding the system, such as newly discovered or potential problems in a system, newly implemented system behaviors, newly created scenarios and so forth. Furthermore, the techniques discussed herein avoid the need for the developer to know ahead of time what the causality chain is for a particular performance issue. Rather, the individual analysis core modules and diagnostic modules can operate to identify the causality chain in response to detection of the performance issue.

<FIG> illustrates an example system <NUM> implementing the modularized collaborative performance issue diagnostic system in accordance with one or more embodiments. The system <NUM> can be implemented by one or more computing devices. A computing device implementing the system <NUM> can be a variety of different types of devices, such as a physical device or a virtual device. For example, a computing device implementing the system <NUM> can be a physical device such as a desktop computer, a server computer, a laptop or netbook computer, a mobile device (e.g., a tablet or phablet device, a cellular or other wireless phone (e.g., a smartphone), a notepad computer, a mobile station), a wearable device (e.g., eyeglasses, head-mounted display, watch, bracelet, augmented reality (AR) devices, virtual reality (VR) devices), an entertainment device (e.g., an entertainment appliance, a set-top box communicatively coupled to a display device, a game console), Internet of Things (IoT) devices (e.g., objects or things with software, firmware, and/or hardware to allow communication with other devices), a television or other display device, an automotive computer, and so forth. A computing device implementing the system <NUM> can also be a virtual device, such as a virtual machine running on a physical device. A virtual machine can be run on any of a variety of different types of physical devices (e.g., any of the various types listed above). Thus, computing devices implementing the system <NUM> may range from full resource devices with substantial memory and processor resources (e.g., personal computers, game consoles) to low-resource devices with limited memory and/or processing resources (e.g., traditional set-top boxes, hand-held game consoles).

In one or more embodiments, the system <NUM> is implemented as a service that is run on one or more computing devices. Such a service can be, for example, a service hosted in the cloud. In such embodiments, the techniques discussed herein are used to perform diagnostics for the service rather than for a specific computing device.

The system <NUM> includes a user input module <NUM> and an output module <NUM>. The user input module <NUM> receives user inputs from a user of the system <NUM>. User inputs can be provided in a variety of different manners, such as by pressing one or more keys of a keypad or keyboard of the system <NUM>, pressing one or more keys of a controller (e.g., remote control device, mouse, track pad, etc.) of the system <NUM>, pressing a particular portion of a touchpad or touchscreen of the system <NUM>, making a particular gesture on a touchpad or touchscreen of the system <NUM>, and/or making a particular gesture on a controller (e.g., remote control device, mouse, track pad, etc.) of the system <NUM>. User inputs can also be provided via other physical feedback input to the system <NUM>, such as tapping any portion of a computing device implementing the system <NUM>, an action that can be recognized by a motion detection or other component of the system <NUM> (such as shaking a computing device implementing the system <NUM>, rotating a computing device implementing the system <NUM>, bending or flexing a computing device implementing the system <NUM>, etc.), and so forth. User inputs can also be provided in other manners, such as via voice or other audible inputs to a microphone, via motions of hands or other body parts observed by an image capture device, and so forth.

The output module <NUM> generates, manages, and/or outputs content for display, playback, and/or other presentation. This content can be created by the output module <NUM> or obtained from other modules of the system <NUM>. This content can be, for example, a display or playback portion of a user interface (UI). The content can be displayed or otherwise played back by components of a computing device implementing the system <NUM> (e.g., speakers, interactive display devices, etc.). Alternatively, the output module <NUM> can generate one or more signals that are output to other devices or components (e.g., speakers, display devices, etc.) that are separate from a computing device implementing the system <NUM>.

The system <NUM> also includes an operating system <NUM> and one or more applications <NUM>. The operating system <NUM> includes one or more programs or modules that manage various aspects of the system <NUM>, allow applications <NUM> to run, and so forth. Although the user input module <NUM> and the output module <NUM> are illustrated separately from the operating system <NUM>, it should be noted that the operating system <NUM> can optionally include the user input module <NUM> and the output module <NUM>.

The applications <NUM> are various different programs that can be run by the operating system <NUM>. The applications can be any of a variety of different application programs, such as utility programs, educational programs, entertainment programs, productivity programs, and so forth.

The system <NUM> also includes an event tracing module <NUM>. The applications <NUM> as well as the programs of the operating system <NUM> are run as one or more processes on the system <NUM>. These processes perform various tasks, which are logged by the event tracing module <NUM> in an event log <NUM>. The event log <NUM> can be any of a variety of different storage devices, such as a magnetic disk, optical disc, Flash memory, random access memory (RAM), and so forth.

The event tracing module <NUM> logs various different data (also referred to as metadata) regarding events that occur in the system <NUM>. These events refer to various calls made by a process in the system <NUM> or other events in the system <NUM> (e.g., changes in whether a desktop is displayed, changes in which window is displayed, changes in power state (e.g., whether the system <NUM> is operating in a battery save mode), and so forth). For example, the event tracing module <NUM> can log data regarding inter-process calls in the system <NUM>, file I/O access by a process in the system <NUM>, calls by a process to the operating system <NUM>, network access by a process in the system <NUM>, and so forth. Various data regarding the calls or events can be logged, such as a timestamp (e.g., date and time) of when the call was made or the event occurred, an identifier of the process that made the call, what the response was (if any) that the process received to the call, what the target of the call was (e.g., an identifier of another process, an identifier of a file, an identifier of a network address (e.g., an Internet protocol (IP) address), and so forth). Although illustrated as separate from the operating system <NUM>, the event tracing module <NUM> can optionally be part of the operating system <NUM>.

The event tracing module <NUM> can be implemented in a variety of different manners. For example, the event tracing module <NUM> can be the Event Tracing for Windows (ETW) service or an analogous service. By way of another example, the event tracing module <NUM> can be a module or system that provides performance counters' values, telemetry data, and so forth.

The system <NUM> also includes a collaborative diagnostic system <NUM>. The collaborative diagnostic system <NUM> identifies, based at least in part on the information stored in the event log <NUM>, performance issues within the system and a causality chain from an identified performance issue to the root cause of that performance issue. The collaborative diagnostic system <NUM> includes multiple issue detectors <NUM>, multiple analysis core modules <NUM>, and multiple scenario modules <NUM>.

Each issue detector <NUM> is a module or component that includes logic to detect known behaviors in the system <NUM>, such as performance issues in the system <NUM>, bad behaviors that can lead to performance issues in the system <NUM>, and so forth. Various different behaviors or performance issues can be detected, such as video glitches, audio glitches, high energy consumption, a long duration in performing some operation, and so forth. As used herein, a performance issue refers to a behavior in the system <NUM> that leads to, or may lead to, an undesirable result. An undesirable result is a result that a creator of an issue detector <NUM> (e.g., a user or developer of the system <NUM>) deems undesirable. Each issue detector <NUM> can be implemented in a variety of different manners, but analyzes the information in the event log <NUM> and determines whether one or more of various rules, criteria, parameters, etc. are satisfied. If the appropriate rules, criteria, parameters, etc. are satisfied, then the issue detector determines that the a particular behavior or performance issue has been detected. For example, an issue detector <NUM> may detect video glitches and determine that a video glitch has occurred if a video data buffer has not been refreshed within a threshold amount of time (e.g., <NUM> milliseconds). A behavior that is considered undesirable by a user or a designer of the system <NUM> (e.g., video glitches, high energy consumption) is also referred to as a bad behavior. Bad behaviors can be specified by a user or designer of the system <NUM> by specifying an issue detector <NUM> that detects those bad behaviors.

In one or more embodiments, the issue detector <NUM> is hard-coded logic. Alternatively, the issue detector <NUM> can be implemented as a scalable behavior detection system that includes one or more behavior definitions that each identify one or more basic operations the system <NUM> (such as data storage device I/O, network I/O, memory accesses, and so forth). If the basic operations specified by a behavior definition are performed in the system <NUM> as specified in the behavior definition, then the issue detector determines that the particular behavior has been detected. Such a behavior definition is optionally scalable, allowing the behavior definition to be modified over time and during operation of the system <NUM>.

At least one issue detector <NUM> is provided by a vendor of designer of the operating system <NUM> and/or system <NUM>. Additionally or alternatively, at least one issue detector <NUM> can be provided by another entity or vendor. For example, a developer of an application <NUM> can provide an issue detector <NUM> to allow performance issues in that application <NUM> (e.g., the application <NUM> takes too long to launch) to be detected.

Each analysis core module <NUM> includes logic to analyze and correlate low level system behavior within a conceptual area. Each conceptual area refers to a class or type of behavior, such as I/O behavior or operations, CPU behavior or operations, power management behavior or operations, and so forth. Different analysis core modules <NUM> are included for different conceptual areas. For example, one analysis core module <NUM> can be a central processing unit (CPU) analysis module that analyzes CPU starvation, CPU contention, as well as deferred procedure call (DPC) and interrupt related activity. By way of another example, another analysis core module <NUM> can be an I/O analysis module that analyzes and correlates I/O delays, I/O contention, I/O starvation, and so forth.

Within each analysis core module <NUM> are one or more diagnostic modules (also referred to as helper modules) that are specific to that analysis core module <NUM> to help determine what is happening in the system <NUM>. Each analysis core module <NUM> is responsible for managing the diagnostic modules included in that analysis core module <NUM>. These diagnostic modules perform behavior-specific analysis ranging from low level behaviors (e.g., resource starvation or contention) to higher level behaviors (e.g., fragmented access patterns and special activities such as prefetching I/O and page faults). One or more analysis core modules <NUM>, employing the various diagnostic modules, determine the root cause of the performance issue detected by the issue detector <NUM>.

Each scenario module <NUM> includes logic to take an appropriate responsive action in response to the root cause of a performance issue being determined. Different scenario modules <NUM> are associated with different performance issues and root causes. For example, a scenario module <NUM> can be associated with a scenario where process foo. exe is executing having a specific function in the executing call stack. Another scenario module <NUM> can be associated with a scenario where process bar. exe accesses files f1 and f2.

Various different responsive actions can be taken by a scenario module <NUM>. Examples of such responsive actions include displaying a notification on a display device of the system <NUM> for viewing by a user of the system <NUM>, recording the occurrence of the detected performance issue and root cause (and optionally causality chain) in a file or other record, notifying an administrator or designer of the service <NUM> that the performance issue occurred (and optionally the root cause and causality chain), and so forth.

In one or more embodiments, the scenario modules <NUM> can also aggregate information, providing a summary or aggregated view of the causality chain and/or root cause. For example, rather than displaying or otherwise presenting the entire causality chain to the user, a single element of the causality chain or the root cause can be displayed (e.g., an indication that there was a delay because the operating system was installing or downloading an update). This provides a more user-friendly indication of the root cause and/or causality chain to a user of the system <NUM>.

Generally, during operation of the system <NUM>, each issue detector <NUM> analyzes the information in the event log <NUM> and determines based on this analysis whether a performance issue has occurred. In response to an issue detector <NUM> determining that a performance issue has occurred, the issue detector <NUM> sends a communication to one or more of the analysis core modules <NUM> querying what operations were being performed by a thread being executed at the time the performance issue was detected. An analysis core module <NUM> (e.g., a CPU scheduling module) that is associated with (also referred to herein as owns) a CPU scheduler receives the query (including an indication of the time at which the performance issue was detected) from the issue detector <NUM> and proceed to determine what operations performed by a thread being executed at the time the performance issue was detected. The CPU scheduler manages execution of different processes and process threads on the CPU, and thus is aware of which threads were scheduled to execute at which times on the CPU.

Different analysis core modules <NUM> are associated with different conceptual areas, and the analysis core modules <NUM> communicate with one another to track down (e.g., identify) the root cause of the performance issue. This tracking down of the root cause of the performance issue results in a causality chain from an identified performance issue to the root cause of that performance issue. The causality chain is a path from the performance issue to the root cause, with each node or element in the path indicating an issue and what caused the issue (e.g., the performance issue was due to a slow I/O operation, the slow I/O operation was due to a memory page needing to be paged into memory, the memory page needed to be paged in because there was memory pressure in the system, and so forth). It should be noted that the causality chain can have multiple branches at each node (e.g., the slow I/O operation was due to both a memory page needing to be paged into memory and I/O contention from multiple processes). Thus, the causality chain may take the form of a tree or other graph.

Each of the analysis core modules <NUM> has a query interface that allows an issue detector <NUM> to inquire into the various analysis core modules <NUM> for the cause of an observed behavior or presence of a certain activity. An issue detector <NUM> communicates or sends a query as to why the observed behavior or certain activity occurred detected by the issue detector <NUM> occurred. Examples of such queries include why was a synchronization deadline missed, why did a particular operation take too long to complete, why is more than a threshold amount (e.g., <NUM>% of capacity) of energy being consumed in the system <NUM>, and so forth. In one or more embodiments, the issue detectors <NUM> communicate the query to a particular analysis core module directly (e.g., the CPU scheduling module), which operates as a central orchestrator for identifying the root cause and invokes the appropriate analysis core modules <NUM> when needed. In such embodiments, this central orchestrator includes logic to identify which of the analysis core modules <NUM> is the appropriate analysis core module <NUM> to respond to the query. Various different rules, criteria, parameters, and so forth can be used by such a central orchestrator to determine which of the analysis core modules <NUM> is the appropriate analysis core module <NUM> to respond to the query. For example, such a central orchestrator can determine that an I/O core module is the appropriate analysis core module <NUM> to respond to a query regarding why a long I/O occurred.

Additionally or alternatively, an issue detector <NUM> can include logic that allows the issue detector <NUM> to know which analysis core module <NUM> to send the query to. For example, the issue detector <NUM> may be a video glitch detector that includes logic to determine that the video glitch was caused by a desktop window manager (dwm. exe) process not being able to provide an image bitmap in time, and that this happened because the dwm. exe process was waiting on storage I/O. Having that knowledge, the video glitch detector interacts with one of the analysis core modules <NUM> that is responsible for diagnosing storage I/O related issues.

Additionally or alternatively, the issue detectors <NUM> can broadcast the query to all analysis core modules <NUM>, and the appropriate analysis core module <NUM> responds. In such embodiments, each analysis core module <NUM> includes logic to determine whether it is the appropriate analysis core module <NUM> to respond to the query. Various different rules, criteria, parameters, and so forth can be used by each analysis core module <NUM> to determine whether it is the appropriate analysis core module <NUM> to respond to the query. For example, an I/O core module can determine that it is the appropriate analysis core module <NUM> to respond to a query regarding why a long I/O occurred.

Similarly, the query interfaces of the analysis core modules <NUM> allow one or more other analysis core modules <NUM> to inquire into the various analysis core modules <NUM> for the cause of an observed behavior or presence of a certain activity. An analysis core module <NUM> communicates or sends a query as to why the observed behavior or certain activity occurred being analyzed by or identified by the analysis core module <NUM> occurred. Examples of such queries include what was the reason behind a paging I/O, why was a prefetch I/O being performed, and so forth. In one or more embodiments, the analysis core modules <NUM> communicate the query to a particular analysis core module directly (e.g., the CPU scheduling module), which operates as a central orchestrator for identifying the root cause and invokes the appropriate analysis core modules <NUM> when needed. In such embodiments, this central orchestrator includes logic to identify which of the analysis core modules <NUM> is the appropriate analysis core module <NUM> to respond to the query, analogous to determining which of the analysis core modules <NUM> is the appropriate analysis core module to respond to a query from an issue detector <NUM>. Additionally or alternatively, the issue detectors <NUM> can broadcast the query to all analysis core modules <NUM>, and the appropriate analysis core module <NUM> responds. In such embodiments, each analysis core module <NUM> includes logic to determine whether it is the appropriate analysis core module <NUM> to respond to the query, analogous to determining which of the analysis core modules <NUM> is the appropriate analysis core module to respond to a query from an issue detector <NUM>.

It should be noted that a single query interface can be implemented for (included in) an analysis core module <NUM>, in which case the single query interface includes logic to route the query to an appropriate diagnostic module within the analysis core module <NUM>. Additionally or alternatively each individual diagnostic module within an analysis core module can include a query interface, and each such query interface operates as a query interface for the analysis core module <NUM>.

The issue detectors <NUM>, analysis core modules <NUM>, diagnostic modules, and scenario modules <NUM> communicate via these query interfaces. These query interfaces support queries in the general format of what caused or is responsible for something happening, such as "what caused operation P or behavior B?", "what scenario module is responsible for process P executing code C?", "who is contending with operations O1 and O2?", and so forth. Note that such a query interface framework is easily extensible to newly discovered problems, newly implemented system behaviors, newly created scenarios, and so forth.

The communications between the issue detectors <NUM> and the analysis core modules <NUM>, as well as among the analysis core modules <NUM>, can be implemented using any of a variety of different public and/or proprietary techniques. For example, a particular analysis core module (e.g., the CPU scheduling module) can be communicated with directly using any of a variety of inter-process communication protocols. By way of another example, each analysis core module <NUM> can subscribe to events or messages on a particular communication channel, and any query can be submitted as an event or message on that particular communication channel.

<FIG> illustrates an example set of issue detectors <NUM> in accordance with one or more embodiments. The issue detectors <NUM> illustrated in <FIG> include a UI delay detector <NUM>, an audio glitch detector <NUM>, a video glitch detector <NUM>, a high energy consumption detector <NUM>, a long duration scenario <NUM> detector <NUM>, a long duration scenario <NUM> detector <NUM>, and an application A bad behavior detector <NUM>. It is to be appreciated that the example issue detectors <NUM> shown in <FIG> are only examples. Not all of the issue detectors <NUM> shown in <FIG> need be included in a system <NUM> of <FIG> and/or additional issue detectors <NUM> can be included in a system <NUM>.

The UI delay detector <NUM> includes logic to detect known behaviors (e.g., performance issues or bad behaviors) related to the UI displayed by the system <NUM>. The UI delay detector <NUM> analyzes various information in the event log <NUM> to identify situations in which the UI displayed by the system <NUM> (e.g., by the output module <NUM>) is delayed (e.g., an initial UI display when starting or rebooting the system <NUM> takes longer than a threshold amount of time (e.g., <NUM> seconds), changes to a UI display take longer than a threshold amount of time (e.g., <NUM> milliseconds), and so forth).

The audio glitch detector <NUM> includes logic to detect known behaviors (e.g., performance issues or bad behaviors) related to audio playback in the system <NUM>. The audio glitch detector <NUM> analyzes various information in the event log <NUM> to identify situations in which there is a glitch in audio playback in the system <NUM> (e.g., an audio data buffer has not been refreshed within a threshold amount of time (e.g., <NUM> milliseconds)).

The video glitch detector <NUM> includes logic to detect known behaviors (e.g., performance issues or bad behaviors) related to video playback in the system <NUM>. The video glitch detector <NUM> analyzes various information in the event log <NUM> to identify situations in which there is a glitch in video playback in the system <NUM> (e.g., a video data buffer has not been refreshed within a threshold amount of time (e.g., <NUM> milliseconds)).

The high energy consumption detector <NUM> includes logic to detect known behaviors (e.g., performance issues or bad behaviors) related to energy consumption in the system <NUM>. The high energy consumption detector <NUM> analyzes various information in the event log <NUM> to identify situations in which energy consumption in the system <NUM> is deemed to be too high (e.g., the energy consumption has exceeded a threshold amount (e.g., <NUM>% of a maximum amount of energy available from a battery of the system <NUM>), energy is being consumed at greater than a threshold rate (e.g., at a rate of <NUM> watts per hour), and so forth).

The long duration scenario <NUM> detector <NUM> includes logic to detect known behaviors (e.g., performance issues or bad behaviors) related to delays in the system <NUM> of <FIG>. The long duration scenario <NUM> detector <NUM> analyzes various information in the event log <NUM> to identify situations in which a particular operation or action is taking too long (e.g., greater than a threshold amount of time, such as <NUM> seconds).

The long duration scenario <NUM> detector <NUM> includes logic to detect known behaviors (e.g., performance issues or bad behaviors) related to delays in the system <NUM> of <FIG>. The long duration scenario <NUM> detector <NUM> analyzes various information in the event log <NUM> to identify situations in which a particular operation or action is taking too long (e.g., greater than a threshold amount of time, such as <NUM> milliseconds). The long duration scenario <NUM> detector <NUM> is similar to the long duration scenario <NUM> detector <NUM>, but identifies different delay situations (e.g., delays for different operations or actions, delays for different threshold amounts of time, combinations thereof, and so forth).

The application A bad behavior detector <NUM> includes logic to detect known behaviors (e.g., performance issues or bad behaviors) related to a particular application <NUM> (referred to as application A) in the system <NUM> of <FIG>. The application A bad behavior detector <NUM> analyzes various information in the event log <NUM> to identify situations in which the known behavior occurs in the application. The application A bad behavior detector <NUM> can be provided or specified by, for example, a developer of application A to allow the collaborative diagnostic system <NUM> of <FIG> to identify the occurrence of particular behaviors in the application A. Any of various different behaviors can be detected, such as delays in particular operations occurring, accesses to particular resources (e.g., networks, storage devices) exceeding a threshold rate (e.g., more than <NUM> accesses per minute) or duration (e.g., more than <NUM> seconds), and so forth.

<FIG> and <FIG> illustrate an example set of core modules <NUM> in accordance with one or more embodiments. The core modules <NUM> illustrated in <FIG> and <FIG> include a CPU scheduling core module <NUM>(<NUM>), a settings store core module <NUM>(<NUM>), a CPU core module <NUM>(<NUM>), a power management core module <NUM>(<NUM>), an I/O core module <NUM>(<NUM>), and a memory core module <NUM>(<NUM>). It is to be appreciated that the example core modules <NUM> shown in <FIG> and <FIG> are only examples. Not all of the core modules <NUM> shown in <FIG> and <FIG> need be included in a system <NUM> of <FIG> and/or additional core modules <NUM> can be included in a system <NUM>.

The CPU scheduling core module <NUM>(<NUM>) includes logic to analyze and correlate low level system behavior within the conceptual area of CPU scheduling. The CPU scheduling core module <NUM>(<NUM>) includes a root cause diagnostic module <NUM> that maintains a timeline of the activities of threads of processes in the system <NUM>. The root cause diagnostic module <NUM> knows, based on information in the event log <NUM>, what class or type of behavior (e.g., network I/O, storage device I/O, CPU execution, etc.) at any given time. The root cause diagnostic module <NUM> can then communicate the query to the appropriate one of the other analysis core modules <NUM> based on the query received from the issue detector <NUM>.

The settings store core module <NUM>(<NUM>) includes logic to analyze and correlate low level system behavior within the conceptual area of operating system settings (e.g., an operating system registry). The settings store core module <NUM>(<NUM>) includes a saving a backup diagnostic module <NUM>, a flushing keys diagnostic module <NUM>, and a mounting a hive diagnostic module <NUM>. The saving a backup diagnostic module includes logic to determine when a backup of the operating system settings is being saved. The flushing keys diagnostic module <NUM> includes logic to determine when keys or other operating system settings are being written to (flushed to) to disk or another storage device. The mounting a hive diagnostic module <NUM> includes logic to determine when keys or other operating system settings are being read from (mounted) disk or another storage device.

The CPU core module <NUM>(<NUM>) includes logic to analyze and correlate low level system behavior within the conceptual area of CPU operation. The CPU core module <NUM>(<NUM>) includes a CPU contention diagnostic module <NUM>, a CPU starvation diagnostic module <NUM>, and a disk encryption diagnostic module <NUM>. The CPU contention diagnostic module <NUM> includes logic to determine when there is high contention for the CPU (e.g., situations where a virtual machine wait too long (e.g., greater than a threshold amount of time, such as <NUM> second) to be executed on the CPU). The CPU starvation diagnostic module <NUM> includes logic to determine when one or more threads are being starved (e.g., situations where threads wait too long (e.g., greater than a threshold amount of time, such as <NUM> milliseconds) to be scheduled for execution on the CPU). The disk encryption diagnostic module <NUM> includes logic to determine when the CPU is encrypting a portion of the disk or other storage device.

The power management core module <NUM>(<NUM>) includes logic to analyze and correlate low level system behavior within the conceptual area of power management. The power management core module <NUM>(<NUM>) includes an idle states diagnostic module <NUM>, a connected standby diagnostic module <NUM>, and a CPU frequency throttling diagnostic module <NUM>. The idle states diagnostic module <NUM> includes logic to determine when the CPU is in an idle state (e.g., not running any applications <NUM> or operating system <NUM> programs). The connected standby diagnostic module <NUM> includes logic to determine when the system is in a connected standby state (e.g., the system is running in a low power mode but is still able to receive communications from other devices or systems). The CPU frequency throttling diagnostic module <NUM> includes logic to determine when the CPU frequency has been reduced (e.g., in order to conserve power).

The I/O core module <NUM>(<NUM>) includes logic to analyze and correlate low level system behavior within the conceptual area of I/O. The I/O core module <NUM>(<NUM>) includes an I/O delay diagnostic module <NUM>, an I/O contention diagnostic module <NUM>, a page fault I/O diagnostic module <NUM>, a prefetch I/O diagnostic module <NUM>, and a fragmented I/O diagnostic module <NUM>. The I/O delay diagnostic module <NUM> includes logic to determine when an I/O delay occurred (e.g., an I/O took longer than a threshold amount of time, such as <NUM> second). The I/O contention diagnostic module <NUM> includes logic to determine when there are two sources of contending I/O activity (e.g., two streams of write requests each from a different process). The page fault I/O diagnostic module includes logic to determine when I/O data was read from disk or other storage device not long ago (e.g., less that a threshold amount of time, such as <NUM> seconds) and should be cached in memory unless it was evicted due to memory pressure. The prefetch I/O diagnostic module <NUM> includes logic to determine when I/O data was read from disk or other storage device as part of a prefetch operation not long ago (e.g., less that a threshold amount of time, such as <NUM> seconds) and should be cached in memory. The fragmented I/O diagnostic module <NUM> includes logic to determine when I/O data is fragmented or otherwise scattered across the disk or other storage device.

The memory core module <NUM>(<NUM>) includes logic to analyze and correlate low level system behavior within the conceptual area of memory (e.g., random access memory (RAM)). The memory core module <NUM>(<NUM>) includes a binaries paged out diagnostic module <NUM>, a memory leaks diagnostic module <NUM>, and a memory pressure diagnostic module <NUM>. The binaries paged out diagnostic module <NUM> includes logic to determine when a memory page including a binary to be executed had been paged out and needed to be paged back in in order to be executed. The memory leaks diagnostic module <NUM> includes logic to determine when allocated memory is old (e.g., the memory was allocated more than a threshold amount of time ago, such as <NUM> hours). The memory pressure diagnostic module <NUM> includes logic to determine why memory pressure existed in the system.

<FIG> illustrates an example set of scenario modules <NUM> in accordance with one or more embodiments. The scenario modules <NUM> illustrated in <FIG> include an operating system update installing module <NUM>, a disk defragmentation module <NUM>, a network download module <NUM>, a loading pictures by camera software module <NUM>, a known activity <NUM> module <NUM>, a known activity <NUM> module <NUM>, and an application A activity module <NUM>. It is to be appreciated that the example scenario modules <NUM> shown in <FIG> are only examples. Not all of the scenario modules <NUM> shown in <FIG> need be included in a system <NUM> of <FIG> and/or additional scenario modules <NUM> can be included in a system <NUM>.

The operating system update installing module <NUM> includes logic to take an appropriate responsive action in response to a performance issue having a root cause that was an operating system update being installed in the system <NUM>. The disk defragmentation module <NUM> includes logic to take an appropriate responsive action in response to a performance issue having a root cause that was a disk (or other storage device) defragmentation occurring in the system <NUM>.

The network download module <NUM> includes logic to take an appropriate responsive action in response to a performance issue having a root cause that was a download over a network occurring. The loading pictures by camera software module <NUM> includes logic to take an appropriate responsive action in response to a performance issue having a root cause that was data (e.g., pictures, video, or other recordings) being downloaded by a program from a camera or other image capture device.

The known activity <NUM> module <NUM> includes logic to take an appropriate responsive action in response to a performance issue having a root cause that was some other known activity, referred to as activity <NUM>. This known activity can be any of a variety of different activities, such as data I/O or network accesses, power modes or settings, CPU state or workload, and so forth.

The known activity <NUM> module <NUM> includes logic to take an appropriate responsive action in response to a performance issue having a root cause that was some other known activity, referred to as activity <NUM>. This known activity can be any of a variety of different activities, analogous to the activity associated with known activity <NUM> module <NUM>, although the modules <NUM> and <NUM> are associated with different known activities.

The application A activity module <NUM> includes logic to take an appropriate responsive action in response to a performance issue having a root cause that was due to some activity in a particular application <NUM> (referred to as application A). This activity can be any of a variety of different activities, including activities specific to the operation of application A. The application A activity module <NUM> can be provided or specified by, for example, a developer of application A.

Returning to <FIG>, the collaborative diagnostic system <NUM> allows a root cause for a performance issue to be identified as well as a causality chain to the root cause to be identified. An issue detector <NUM> communicates a query that is analyzed by one or more analysis core modules <NUM>. Various diagnostic modules within the analysis core modules <NUM> can perform various analysis and various determinations, and each can query one or more other analysis core modules <NUM> and/or diagnostic modules. In one or more embodiments, each query includes an indication of any previous information received by the analysis core module <NUM> and/or diagnostic module regarding the performance issue. This information is passed through to a scenario module <NUM>, which can use the information to construct a causality chain from the performance issue to the root cause. Additionally or alternatively, in situations in which a central orchestrator invokes the appropriate analysis core modules and/or diagnostic modules, the central orchestrator can maintain the information associated with each query rather than passing the information through to each analysis core module and/or diagnostic module.

<FIG> illustrates an example <NUM> of using the collaborative diagnostic system to identify the root cause for a performance issue as well as a causality chain to the root cause in accordance with one or more embodiments. The example <NUM> is discussed with further reference to the issue detectors discussed in <FIG>, the analysis core modules discussed in <FIG> and <FIG>, and the scenario modules discussed in <FIG>. The example <NUM> is discussed with reference to the issue detectors and diagnostic modules issuing queries to other diagnostic modules and the appropriate diagnostic modules responding to the queries. Alternatively, a central orchestrator could be used as discussed above.

In the example <NUM>, a video glitch is observed by the video glitch detector <NUM>. The video glitch detector analyzes a sequence of events from the event log <NUM> related to video glitches since it knows about the conditions which are to be met in order for a video glitch to occur. In this example, the desktop window manager couldn't push out a frame of video because of a long storage I/O it had to issue. Having that knowledge, the video glitch detector <NUM> passes the analysis on to the root cause diagnostic module <NUM> with a query of "desktop window manager couldn't meet the video synchronization (v-synch) deadline due to a long I/O - what happened?".

The root cause diagnostic module <NUM> determines that execution of the thread that implements the desktop window manager was late. The root cause diagnostic module <NUM> passes the analysis on to the I/O delay diagnostic module <NUM> with a query of "who owned the desktop window manager thread and why was the thread late?".

The I/O delay diagnostic module <NUM> (inside the I/O core module <NUM>(<NUM>)) examines the long I/O for the thread. There are two fundamental queries that the I/O delay diagnostic module <NUM> makes - "why did the I/O happen in the first place?" and "why was that I/O slow?" The page fault I/O diagnostic module <NUM> receives the query and determines that the I/O is a page fault, and proceeds to determine why the page fault happened. The I/O contention diagnostic module <NUM> also receives the query and determines that the I/O was slow due to disk contention.

The I/O contention diagnostic module <NUM> analyzes I/O contention and discovers that that there are two other sources of the contending I/O activity: a stream of write requests by a thread of a trustedinstaller. exe process and another stream of write requests by a thread of a p2pclient. exe process. The I/O contention diagnostic module <NUM> then examines a list of known behaviors/scenarios for mention of I/O by these two processes and indeed finds two scenario modules, the operating system update installing module <NUM> and the network download module <NUM>, which "own" (are associated with) both of these behaviors. The I/O contention diagnostic module <NUM> passes the contending I/O information to those two scenario modules, which each then create a final root-cause description for the issue.

The page fault I/O diagnostic module <NUM> passes pass the issue information to the binaries paged out diagnostic module <NUM>, which recognizes that the I/O in question belongs to the dwm. exe process, was already read from disk not so long ago (e.g., less than a threshold amount of time ago, such as <NUM> minutes), and should be cached in memory unless it was evicted due to memory pressure. The page fault I/O diagnostic module <NUM> passes the issue information to the binaries paged out diagnostic module <NUM>, which determines that the a memory page including binary of the dwm. exe process to be executed had been paged out and needed to be paged back in in order to be executed.

The binaries paged out diagnostic module <NUM> passes the issue information to the memory pressure diagnostic module <NUM>, which determines why memory pressure existed on the system and caused the memory page to be paged out. The memory pressure diagnostic module <NUM> determines that the system has a 1GB (gigabyte) outstanding memory allocation by a process camerasoftware. exe and inquires the memory leaks diagnostic module <NUM> whether these are memory leaks. The memory leaks diagnostic module <NUM> confirms that these allocations are several hours old (e.g., allocations made at least a threshold amount of time ago, such as <NUM> hours), and examines a list of known behaviors/scenarios for mention of large memory allocation activity by the camerasoftware. exe process and finds a scenario module (the loading pictures by camera software module <NUM>) that describes building an image database scenario by the camerasoftware. exe process. The memory leaks diagnostic module <NUM> then passes the allocation information to the loading pictures by camera software module <NUM>, which then creates the final root-cause descriptions for the issue.

As a result of this analysis, three scenario-tied root causes were found for the video glitch: a download by a p2p client, an operating system update service (trustedinstaller. exe) installing updates, and the memory pressure caused by a photography software suite. Each of the scenario modules (the operating system update installing module <NUM>, the network download module <NUM>, and the loading pictures by camera software module <NUM>) can take a responsive action that incorporates the root cause and causality chain. For example, the operating system update installing module <NUM> can include an indication that the root cause of the video glitch is an operating system update service installing updates, and that the causality chain is: the I/O was slow, which was caused by two sources of contending I/O activity. By way of another example, the loading pictures by camera software module <NUM> can include an indication that the root cause of the video glitch is the memory pressure caused by a photography software suite, and that the causality chain is: the I/O was slow, which was caused by a page fault, which was for a multiple that was already read not long ago and should be in cached memory but was paged out and needed to be paged back in due to memory pressure, and that the memory pressure was caused by memory leaks due to memory allocations made to the photography software suite.

The scenario modules (the operating system update installing module <NUM>, the network download module <NUM>, and the loading pictures by camera software module <NUM>) can optionally communicate with one another to determine an appropriate fix or responsive action to take, which would be a fix to the photography software suite to reduce or eliminate the memory leaks. Various different rules or criteria can be used to determine which scenario module includes the appropriate responsive action. For example, the scenario modules may be ranked with the responsive action for the highest ranking scenario module being used (e.g., windows update installing is something that is appropriate for the system and wouldn't typically need a fix, and thus the fix would be for the photography software suite).

Returning to <FIG>, it should be noted that situations can arise in which no root cause is identified for a particular performance issue. Such situations are resolved by having additional cycles or iterations through the collaborative diagnostic system <NUM>. These additional cycles or iterations can return back to the same issue detector <NUM> as detected the performance issue. The issue detector <NUM> can go back to a previous time (e.g., <NUM> milliseconds before the last performance issue was detected), to a previous event (e.g., the last time, prior to the performance issue being detected, that the v-synch deadline), and so forth. For example, assume that the issue detector <NUM> is a video glitch detector and that the issue detector detects a video glitch that because the v-synch deadline was missed due to a performance problem <NUM> display frames (<NUM> v-synch deadlines) ago, but the performance problem has since been resolved and thus no root cause for the video glitch is identified. The issue detector <NUM> is notified (e.g., by a scenario module <NUM>) of this inability to identify a root cause, and in response can identify the previous v-synch deadline and submit a query to the analysis core modules <NUM> regarding why the previous v-synch deadline was missed, which may or may not identify a root cause. This process can be repeated until a query is submitted to the analysis core module <NUM> regarding why the v-synch deadline <NUM> display frames ago was missed, which will identify a root cause for the v-synch deadline being missed.

It should also be noted that, as discussed above, the event tracing module <NUM> logs various different data regarding events that occur in the system <NUM> in the event log <NUM>. In one or more embodiments, the logging of data regarding events can be turned on and off at different times for different applications or programs in order to reduce the amount of time and storage space expended to record the data. For example, event tracing module <NUM> may be able to log data regarding different events for a particular application <NUM>, although by default turns off logging of data for events regarding that particular application <NUM>. If a situation arises later indicating that logging of data for events regarding that particular application <NUM> should be turned on, then the event tracing module <NUM> begins logging data for events regarding that particular application.

By way of example, assume that a video glitch performance issue was detected, and the root cause ended up being an issue with a thread running as part of a Web browser application <NUM>. However, logging of data for events regarding the Web browser application <NUM> was not previously being performed by the event tracing module <NUM>. A scenario module <NUM> can take the responsive action of having the event tracing module <NUM> begin logging data for events regarding the Web browser application <NUM>. Accordingly, if a video glitch performance issue is again detected and is due to an issue with a thread running as part of the Web browser application <NUM>, one or more analysis core modules <NUM> can further analyze the performance issue and determine the root cause of the performance issue in the thread of the Web browser application <NUM>. Thus, the collaborative diagnostic system <NUM> can self-identify additional diagnostics that are needed, and turn on event logging to obtain data for those additional diagnostics in the future.

<FIG> is a flowchart illustrating an example process <NUM> for implementing the modularized performance issue diagnostic system in accordance with one or more embodiments. Process <NUM> is carried out by a system, such as collaborative diagnostic system <NUM> of <FIG>, and can be implemented in software, firmware, hardware, or combinations thereof. Process <NUM> is shown as a set of acts and is not limited to the order shown for performing the operations of the various acts. Process <NUM> is an example process for implementing the modularized performance issue diagnostic system; additional discussions of implementing the modularized performance issue diagnostic system are included herein with reference to different figures.

In process <NUM>, one or more performance issues in a system are detected (act <NUM>). This system refers to a system (e.g., a service hosted in the cloud, a computing device, etc.) that includes the system implementing process <NUM> (e.g., that includes the collaborative diagnostic system <NUM> of <FIG>). The one or more performance issues are detected by one or more issue detectors as discussed above.

One or more analysis core modules are used to analyze system behavior to determine root causes for the detected one or more performance issues (act <NUM>). Each of the analysis core modules includes diagnostic modules that are specific to that analysis core module to help determine what is happening in the system. The diagnostic modules collaborate with one another, submitting queries that are responded to by the appropriate diagnostic module as discussed above.

A responsive action is taken in response to determining the root cause of the detected performance issue (act <NUM>). The responsive action is taken by one or more scenario modules that are associated with the root cause, as discussed above.

Although particular functionality is discussed herein with reference to particular modules, it should be noted that the functionality of individual modules discussed herein can be separated into multiple modules, and/or at least some functionality of multiple modules can be combined into a single module. Additionally, a particular module discussed herein as performing an action includes that particular module itself performing the action, or alternatively that particular module invoking or otherwise accessing another component or module that performs the action (or performs the action in conjunction with that particular module). Thus, a particular module performing an action includes that particular module itself performing the action and/or another module invoked or otherwise accessed by that particular module performing the action.

<FIG> illustrates an example system generally at <NUM> that includes an example computing device <NUM> that is representative of one or more systems and/or devices that may implement the various techniques described herein. The computing device <NUM> may be, for example, a server of a service provider, a device associated with a client (e.g., a client device), an on-chip system, and/or any other suitable computing device or computing system.

The example computing device <NUM> as illustrated includes a processing system <NUM>, one or more computer-readable media <NUM>, and one or more I/O Interfaces <NUM> that are communicatively coupled, one to another. Although not shown, the computing device <NUM> may further include a system bus or other data and command transfer system that couples the various components, one to another. A system bus can include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and/or a processor or local bus that utilizes any of a variety of bus architectures. A variety of other examples are also contemplated, such as control and data lines.

The processing system <NUM> is representative of functionality to perform one or more operations using hardware. Accordingly, the processing system <NUM> is illustrated as including hardware elements <NUM> that may be configured as processors, functional blocks, and so forth. This may include implementation in hardware as an application specific integrated circuit or other logic device formed using one or more semiconductors. The hardware elements <NUM> are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, processors may be comprised of semiconductor(s) and/or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor-executable instructions may be electronically-executable instructions.

The computer-readable media <NUM> is illustrated as including memory/storage <NUM>. The memory/storage <NUM> represents memory/storage capacity associated with one or more computer-readable media. The memory/storage <NUM> may include volatile media (such as random access memory (RAM)) and/or nonvolatile media (such as read only memory (ROM), Resistive RAM (ReRAM), Flash memory, optical disks, magnetic disks, and so forth). The memory/storage <NUM> may include fixed media (e.g., RAM, ROM, a fixed hard drive, and so on) as well as removable media (e.g., Flash memory, a removable hard drive, an optical disc, and so forth). The computer-readable media <NUM> may be configured in a variety of other ways as further described below.

The one or more input/output interface(s) <NUM> are representative of functionality to allow a user to enter commands and information to computing device <NUM>, and also allow information to be presented to the user and/or other components or devices using various input/output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone (e.g., for voice inputs), a scanner, touch functionality (e.g., capacitive or other sensors that are configured to detect physical touch), a camera (e.g., which may employ visible or non-visible wavelengths such as infrared frequencies to detect movement that does not involve touch as gestures), and so forth. Examples of output devices include a display device (e.g., a monitor or projector), speakers, a printer, a network card, tactile-response device, and so forth. Thus, the computing device <NUM> may be configured in a variety of ways as further described below to support user interaction.

The computing device <NUM> also includes a collaborative diagnostic system <NUM>. The collaborative diagnostic system <NUM> provides various functionality to identify performance issues within the system and a causality chain from an identified performance issue to the root cause of that performance issue, as discussed above. The collaborative diagnostic system <NUM> can implement, for example, the collaborative diagnostic system <NUM> of <FIG>.

Various techniques may be described herein in the general context of software, hardware elements, or program modules. Generally, such modules include routines, programs, objects, elements, components, data structures, and so forth that perform particular tasks or implement particular abstract data types. The terms "module," "functionality," and "component" as used herein generally represent software, firmware, hardware, or a combination thereof. The features of the techniques described herein are platform-independent, meaning that the techniques may be implemented on a variety of computing platforms having a variety of processors.

"Computer-readable storage media" refers to media and/or devices that enable persistent storage of information and/or storage that is tangible, in contrast to mere signal transmission, carrier waves, or signals per se. Thus, computer-readable storage media refers to non-signal bearing media. The computer-readable storage media includes hardware such as volatile and non-volatile, removable and non-removable media and/or storage devices implemented in a method or technology suitable for storage of information such as computer readable instructions, data structures, program modules, logic elements/circuits, or other data. Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, hard disks, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other storage device, tangible media, or article of manufacture suitable to store the desired information and which may be accessed by a computer.

"Computer-readable signal media" refers to a signal-bearing medium that is configured to transmit instructions to the hardware of the computing device <NUM>, such as via a network. Signal media typically may embody computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as carrier waves, data signals, or other transport mechanism. Signal media also include any information delivery media. By way of example, and not limitation, communication media include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

As previously described, the hardware elements <NUM> and computer-readable media <NUM> are representative of instructions, modules, programmable device logic and/or fixed device logic implemented in a hardware form that may be employed in some embodiments to implement at least some aspects of the techniques described herein. Hardware elements may include components of an integrated circuit or on-chip system, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), and other implementations in silicon or other hardware devices. In this context, a hardware element may operate as a processing device that performs program tasks defined by instructions, modules, and/or logic embodied by the hardware element as well as a hardware device utilized to store instructions for execution, e.g., the computer-readable storage media described previously.

Combinations of the foregoing may also be employed to implement various techniques and modules described herein. Accordingly, software, hardware, or program modules and other program modules may be implemented as one or more instructions and/or logic embodied on some form of computer-readable storage media and/or by one or more hardware elements <NUM>. The computing device <NUM> may be configured to implement particular instructions and/or functions corresponding to the software and/or hardware modules. Accordingly, implementation of modules as a module that is executable by the computing device <NUM> as software may be achieved at least partially in hardware, e.g., through use of computer-readable storage media and/or hardware elements <NUM> of the processing system. The instructions and/or functions may be executable/operable by one or more articles of manufacture (for example, one or more computing devices <NUM> and/or processing systems <NUM>) to implement techniques, modules, and examples described herein.

As further illustrated in <FIG>, the example system <NUM> enables ubiquitous environments for a seamless user experience when running applications on a personal computer (PC), a television device, and/or a mobile device. Services and applications run substantially similar in all three environments for a common user experience when transitioning from one device to the next while utilizing an application, playing a video game, watching a video, and so on.

In the example system <NUM>, multiple devices are interconnected through a central computing device. The central computing device may be local to the multiple devices or may be located remotely from the multiple devices. In one or more embodiments, the central computing device may be a cloud of one or more server computers that are connected to the multiple devices through a network, the Internet, or other data communication link.

In one or more embodiments, this interconnection architecture enables functionality to be delivered across multiple devices to provide a common and seamless experience to a user of the multiple devices. Each of the multiple devices may have different physical requirements and capabilities, and the central computing device uses a platform to enable the delivery of an experience to the device that is both tailored to the device and yet common to all devices. In one or more embodiments, a class of target devices is created and experiences are tailored to the generic class of devices. A class of devices may be defined by physical features, types of usage, or other common characteristics of the devices.

In various implementations, the computing device <NUM> may assume a variety of different configurations, such as for computer <NUM>, mobile <NUM>, and television <NUM> uses. Each of these configurations includes devices that may have generally different constructs and capabilities, and thus the computing device <NUM> may be configured according to one or more of the different device classes. For instance, the computing device <NUM> may be implemented as the computer <NUM> class of a device that includes a personal computer, desktop computer, a multi-screen computer, laptop computer, netbook, and so on.

The computing device <NUM> may also be implemented as the mobile <NUM> class of device that includes mobile devices, such as a mobile phone, portable music player, portable gaming device, a tablet computer, a multi-screen computer, and so on. The computing device <NUM> may also be implemented as the television <NUM> class of device that includes devices having or connected to generally larger screens in casual viewing environments. These devices include televisions, set-top boxes, gaming consoles, and so on.

The techniques described herein may be supported by these various configurations of the computing device <NUM> and are not limited to the specific examples of the techniques described herein. This functionality may also optionally be implemented all or in part through use of a distributed system, such as over a "cloud" <NUM> via a platform <NUM> as described below.

The platform <NUM> may abstract resources and functions to connect the computing device <NUM> with other computing devices. The platform <NUM> may also serve to abstract scaling of resources to provide a corresponding level of scale to encountered demand for the resources <NUM> that are implemented via the platform <NUM>. Accordingly, in an interconnected device embodiment, implementation of functionality described herein may be distributed throughout the system <NUM>. For example, the functionality may be implemented in part on the computing device <NUM> as well as via the platform <NUM> that abstracts the functionality of the cloud <NUM>.

In the discussions herein, various different embodiments are described. It is to be appreciated and understood that each embodiment described herein can be used on its own or in connection with one or more other embodiments described herein.

Claim 1:
A method comprising:
detecting, by at least one of multiple issue detectors in a system (<NUM>), one or more performance issues in the computing system (<NUM>);
analyzing, by the at least one issue detector, a sequence of events from an event log (<NUM>) relating to the detected one or more performance issues to determine a first system behavior;
issuing, by the at least one issue detector, a first query relating to the first system behavior;
receiving, by one or more analysis core modules in the system, the first query;
determining, by the one or more analysis core modules, a second system behavior based on the first query;
issuing, by the one or more analysis core modules, a second query relating to the second system behavior;
repeating the steps of receiving, determining, and the issuing by another one or more analysis core modules for the second and further queries until the root cause for the detected performance issue is identified; and
performing, in the computer system (<NUM>), a responsive action in response to the root cause for the detected performance issue being determined by the multiple analysis modules.