Patent Description:
Often times, traces are used to identify and fix common performance bottlenecks in computing hardware, firmware, and software. For example, developers may identify log events that represent execution delays, and that are frequent or common across a plurality of traces. Thus, developers they can focus developer time on fixing these common performance issues. <NPL>, proposes a new trace-based approach consisting of two steps: impact analysis and causality analysis. The impact analysis measures performance impacts on a component basis, and the causality analysis discovers patterns of runtime behaviors that are likely to cause the measured impacts. The discovered patterns can help performance analysts quickly identify root causes of perceived performance problems. <CIT> relates to techniques for identifying a root cause of a wait in a computer system. Given the identity of a thread of interest and time window, a longest wait period for the thread of interest within the time window is identified. The longest wait period is used as a starting node to generate a ready tree by walking backwards through the data in a system trace to construct a tree of readying events that ready threads for running on a processor. A potentially anomalous chain of events is automatically identified and highlighted in the ready tree. <CIT> relates to an application programming interface (API). The API that leverages operating system instrumentation to provide a chain of threads and processes may alleviate some debugging complications. Specifically, the chain may start with the first thread in the process that experienced the original failure and end with the last thread upon which the first thread directly or indirectly depends. The API may aid debugging efforts by classifying all threads related or dependent upon an original failed thread into specific categories of failures, requesting further information from the originating OS concerning specific failed threads, and using that information to debug the failed application or process more thoroughly.

This object is solved by the subject matter of the independent claim.

While identifying and fixing common computing performance issues is important, in many situations it can also be desirable to identify and fix rarely-occurring computing performance issues. As one example, cloud computing providers often guarantee strict uptime and performance service-level agreements (SLAs), such as "five nines" (i.e., <NUM>%) or "six nines" (i.e., <NUM>%) uptime guarantees. Under these SLAs, the cloud computing provider can have only brief amounts of downtime (i.e., <NUM> seconds under six nines, or <NUM> minutes under five nines) each year without breaching the SLA. Under these constraints, cloud computing providers endeavor to highly optimize any system maintenance (e.g., a hardware and/or operating system reboot) that causes downtime. This means identifying and fixing not only common performance bottlenecks, but also identifying and fixing rare performance bottlenecks. For example, even assuming that all common system boot performance issues have been addressed, when the cloud computing provider hosts and maintains tens to hundreds of thousands of servers, the existence of even one rare system boot performance issue has the potential to break an unacceptable number of SLAs when the cloud computing provider performs system maintenance. Using conventional techniques, identifying traces that capture these rare issues (and, thus, identifying those rare issues) is exceedingly difficult and time-consuming.

At least some embodiments described herein perform an automated wait chain-based analysis of trace data in order to identify traces that contain relatively larger durations of unknown wait events. In particular, the embodiments herein identify a computing scenario, such as system initialization/boot, that has a definable beginning and ending point, as well as statistically independent phases of this scenario (i.e., in which the duration of one phase is an independent variable to the duration of another phase). For each phase, the embodiments herein perform a critical path analysis of different traces of that phase (e.g., each trace corresponding to a different execution of the phase across one or more computers). An output of the critical path analysis of a trace is a wait chain, which identifies a chain of wait (i.e., blocking) operations (e.g., thread sleeps, I/O blocks, CPU blocks, etc.) between the beginning of the phase and the ending of the phase. Based on a collection of signatures that match known wait chain patterns (i.e., known/identified performance issues), these embodiments determine a signature coverage over the identified wait chains. Wait chains with a higher degree of signature coverage are considered to have known and/or common performance issues, while wait chains with a lesser degree of signature coverage are considered to have unknown and/or rare performance issues. Thus, the embodiments herein are usable to identify-and trigger an analysis of-traces having wait chains with lesser degrees of signature coverage for further analysis to identify/fix rare performance issues.

In embodiments, signature coverage is also utilized for additional analysis, such as to determine how widespread a newly-identified performance issue is, to validate if a newly-identified performance issue has actually been fixed and properly deployed, or to perform a "worst case scenario" to determine readiness to meet determined SLA goals.

Embodiments are directed to methods, systems, and computer program products that identify a trace based on wait chain coverage analysis. In one or more embodiments, a computer system identifies a computing scenario having a scenario beginning and a scenario ending. The scenario is covered by a plurality of traces corresponding to execution of a plurality of instances of the scenario. The computer system identifies a plurality of scenario phases between the scenario beginning and the scenario ending. Each phase has a corresponding phase beginning and a corresponding phase ending, and is covered by one or more corresponding traces of the plurality of traces. The one or more corresponding traces are usable to identify one or more wait operations that occurred during a prior execution of a prior instance of the phase in connection with execution of a prior instance of the scenario. For each prior instance of each phase, the computer system identifies, based at least on the one or more corresponding traces, a corresponding wait chain comprising a series of wait operations between the corresponding phase beginning and the corresponding phase ending. The computer system identifies one or more signatures, each configured to match a subset of wait operations in one or more wait chains, and calculates one or more coverages, and calculates a signature coverage that characterizes one or more portions of the identified corresponding wait chains that have one or more wait operations that are matched by the one or more signatures. The computer system triggers an analysis of one or more of the plurality of traces as having unknown wait states based on the identified one or more traces corresponding to a larger amount of uncovered wait chain portions than one or more others of the plurality of traces.

To the accomplishment of the foregoing, <FIG> illustrates an example computer architecture <NUM> that facilitates identifying a trace based on wait chain coverage analysis. As depicted, the computer architecture <NUM> comprises or utilizes a computer system <NUM>, which can be special-purpose or general-purpose, and which includes computer hardware, such as, for example, a processor <NUM> (i.e., or plurality of processors), system memory <NUM>, I/O devices <NUM>, and durable storage <NUM>, which are communicatively coupled using a communications bus <NUM> (or a plurality of communications buses). As indicated by an ellipsis <NUM>, the computer system <NUM> can include additional hardware components, as appropriate.

Embodiments within the scope of the present invention can include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions and/or data structures are computer storage media. Computer-readable media that carry computer-executable instructions and/or data structures are transmission media. Thus, by way of example, and not limitation, embodiments of the invention can comprise at least two distinctly different kinds of computer-readable media: computer storage media and transmission media.

Computer storage media are physical storage media (e.g., system memory <NUM> and/or durable storage <NUM>) that store computer-executable instructions and/or data structures. Physical storage media include computer hardware, such as RAM, ROM, EEPROM, solid state drives ("SSDs"), flash memory, phase-change memory ("PCM"), optical disk storage, magnetic disk storage or other magnetic storage devices, or any other hardware storage device(s) which can be used to store program code in the form of computer-executable instructions or data structures, which can be accessed and executed by a general-purpose or special-purpose computer system to implement the disclosed functionality of the invention.

Further, upon reaching various computer system components, program code in the form of computer-executable instructions or data structures can be transferred automatically from transmission media to computer storage media (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., I/O devices <NUM>), and then eventually transferred to computer system RAM (e.g., system memory <NUM>) and/or to less volatile computer storage media (e.g., durable storage <NUM>) at the computer system.

Computer-executable instructions may be, for example, machine code instructions (e.g., binaries), intermediate format instructions such as assembly language, or even source code.

Those skilled in the art will appreciate that the invention may be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, and the like. As such, in a distributed system environment, a computer system may include a plurality of constituent computer systems.

A cloud computing model can be composed of various characteristics, such as on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, and so forth. A cloud computing model may also come in the form of various service models such as, for example, Software as a Service (" SaaS"), Platform as a Service ("PaaS"), and Infrastructure as a Service ("IaaS"). The cloud computing model may also be deployed using different deployment models such as private cloud, community cloud, public cloud, hybrid cloud, and so forth.

Some embodiments, such as a cloud computing environment, may comprise a system that includes one or more hosts that are each capable of running one or more virtual machines. During operation, virtual machines emulate an operational computing system, supporting an operating system and perhaps one or more other applications as well. In some embodiments, each host includes a hypervisor that emulates virtual resources for the virtual machines using physical resources that are abstracted from view of the virtual machines. The hypervisor also provides proper isolation between the virtual machines. Thus, from the perspective of any given virtual machine, the hypervisor provides the illusion that the virtual machine is interfacing with a physical resource, even though the virtual machine only interfaces with the appearance (e.g., a virtual resource) of a physical resource. Examples of physical resources including processing capacity, memory, disk space, network bandwidth, media drives, and so forth.

As illustrated, the durable storage <NUM> stores computer-executable instructions and/or data structures representing executable software components; correspondingly, during execution of this software at the processor <NUM> (or processors), one or more portions of these computer-executable instructions and/or data structures are loaded into system memory <NUM>. For example, the durable storage <NUM> is shown as potentially storing computer-executable instructions and/or data structures corresponding to a wait chain analysis component <NUM>, a trace generation component <NUM>, and executables <NUM>. The durable storage <NUM> is also illustrated as storing data, including signatures <NUM> and traces <NUM>.

If included, the trace generation component <NUM> generates one or more of the traces <NUM> based on execution of executables <NUM> at the processor <NUM> and/or based on operation of hardware of computer system <NUM> and/or its associated firmware. While <FIG> illustrates only a single trace generation component <NUM>, in embodiments the computer system <NUM> includes a plurality of trace generation components, such as a different trace generation component for each type of trace in traces <NUM>. The particular form, and information fidelity, of each of the traces <NUM> may vary by implementation of its corresponding trace generation component. In some embodiments, one or more of the traces <NUM> comprise events generated by an operating system kernel, such as Event Tracing for Windows (ETW) events, DTrace events, and the like.

Regardless of the particular the type(s) of the traces <NUM>, in embodiments each of the traces <NUM> comprises a record of, or are usable to identify (e.g., via emulation-based deterministic code replay), information about executing threads-including wait events/operations that occurred during execution of those threads. For example, in some embodiments the traces <NUM> include a record of, or are usable to identify, one or more of a first call stack of a first thread that initiated a given wait operation, a second call stack of a second thread that terminated the wait operation, a type of the wait operation (e.g., timer, disk I/O, network I/O, CPU busy, etc.), or context information for a thread involved in the wait operation (e.g., an identity of a process to which the thread belongs, command line parameters used to initiate the process, etc.).

In embodiments, the computer system <NUM> additionally, or alternatively, receives at least one of the traces <NUM> from another computer system (e.g., using one or more of the I/O devices <NUM>). For example, <FIG> illustrates an example computing environment <NUM> in which computer system <NUM> of <FIG> is connected to one or more of computer systems <NUM> (i.e., computer system 202a to computer system 202n) over a network <NUM> (or a plurality of networks). As shown in <FIG>, each of the computer systems <NUM> includes a trace generation component <NUM> (or a plurality of trace generators for different types of traces) and stores a copy of executables <NUM>. As such, the computer system <NUM> is enabled to receive, over the network <NUM>, one or more of the traces <NUM> from one or more of these computer systems <NUM>. In one example, the computing environment <NUM> is a cloud data center, the computer systems <NUM> are virtualization hosts, and the computer system <NUM> receives the traces <NUM> from these virtualization hosts for analysis by the wait chain analysis component <NUM>.

Each of the signatures <NUM> is configured to match subsets of one or more wait chains that the wait chain analysis component <NUM> has produced from the traces <NUM>. For example, each of the signatures <NUM> could match a single wait event/operation (e.g., thread sleeps, I/O blocks, CPU blocks, etc.) within a given wait chain, or a sequence of two or more wait operations in the wait chain, if appropriate matching wait operations are present in the wait chain. In embodiments, each of the signatures <NUM> comprises one or more regular expressions and/or one or more conditional statements that are configured to match one or more attributes of one or more wait operations.

In general, the wait chain analysis component <NUM> provides functionality for performing an automated wait chain-based analysis of trace data in order to identify one or more of the traces <NUM> that contain relatively larger portions of unknown wait events than one or more others of traces <NUM>. According to the invention, unknown wait events are those operations that have not matched to one or more of the signatures <NUM>. With this identification, the wait chain analysis component <NUM> may perform additional analysis, such as to determine how widespread a newly-identified performance issue is, to validate if a newly-identified performance issue has actually been fixed and properly deployed, or to perform a "worst case scenario" to determine readiness to meet determined SLA goals.

To demonstrate embodiments for how the wait chain analysis component <NUM> accomplishes the foregoing, <FIG> illustrates more particular detail of the wait chain analysis component <NUM> of <FIG>. As depicted, the wait chain analysis component <NUM> includes a variety of components, including a scenario identification component <NUM>, a phase identification component <NUM>, a critical path extraction component <NUM>, a signature identification component <NUM>, a coverage calculation component <NUM>, and an analysis component <NUM>, that represent various functions that the wait chain analysis component <NUM> implements in accordance with various embodiments described herein. It will be appreciated that the depicted components-including their identity, sub-components, and arrangement-are presented merely as an aid in describing a particular embodiment of the wait chain analysis component <NUM> described herein, and that these components are nonlimiting to how software and/or hardware might implement various embodiments of the wait chain analysis component <NUM> described herein, or of the particular functionality thereof.

In embodiments, the scenario identification component <NUM> identifies a computing scenario having a defined start and a defined end, and one or more instances of which areat least partially-covered (i.e., logged/recorded) by the traces <NUM>. A computing scenario can comprise any series of operations performed by computer system <NUM> and/or computer systems <NUM>, such as system boot/initialization, processing a database transaction, performing a computational task, communicating with another computer system, etc. Taking the example of the computing scenario comprising a system boot/initialization, in embodiments the scenario begins with a hardware power on/reset (or a software reset), and ends when an operating system reaches a certain initialization state.

To illustrate, <FIG> illustrates an example 300a of critical path extraction and wait chain analysis of a computing scenario, and depicts a plurality of instances of a computing scenario <NUM>, execution of which is recorded in traces <NUM>. As shown, at least one of these instances begins at timestamp zero and ends at timestamp forty-two. In an example, for this instance, timestamp zero corresponds to a timestamp at which a hardware power on/reset (or a software reset) occurred, and timestamp forty-two corresponds to a timestamp at which an operating system reached a fully initialized state. Each instance of computing scenario <NUM> has the same defined beginning and ending events, although the particular duration of each instance may vary due to different initial environmental conditions, different external inputs, etc..

In embodiments, the phase identification component <NUM> identifies one or more phases of the computing scenario. As used herein, a phase is a subset of a scenario for which one or more of traces <NUM> exists, and which is independent of the other phases. For example, returning to the example of the computing scenario comprising a system boot/initialization, in embodiments a first scenario phase corresponds to a low-level hardware and BIOS/EFI initialization, a second phase corresponds a boot loader initialization, a third phase corresponds to initialization of hardware by an operating system, a fourth phase corresponds to loading of system services, etc. Notably, each of these phases is independent of one another. For example, if the low-level hardware and BIOS/EFI initialization of the first phase takes longer than usual due to a memory upgrade, beginning of the second phase of boot loader initialization will be delayed but the actual duration of the boot loader initialization is unaffected by the longer than usual first phase. In embodiments, operation of the phase identification component <NUM> is optional, in which case the entire computing scenario is considered to be single phase.

In embodiments, identifying different phases of a scenario is useful for segmenting and/or focusing further analysis by the wait chain analysis component <NUM>. For example, the software and/or hardware involved in different phases may be the responsibility of different development teams; may have different fidelities, qualities, or quantities, of trace data available; may be in different developmental stages (e.g., mature, immature, etc.), may have different overall stability, etc. Given these considerations, it may make sense to devote more computing resources to analysis of some phase(s) of a scenario than to other phases (e.g., in terms of gathering trace data, in terms of operation of the wait chain analysis component <NUM>, in terms of development team availability and responsiveness, etc.).

In <FIG>, example 300a shows a plurality of phases <NUM> (i.e., phase 302a to phase 302e) of computing scenario <NUM>. In some embodiments, each instance of the computing scenario <NUM> has the same phases, though the particular duration of each phase may vary from scenario instance to scenario instance. In embodiments, there may be gaps in trace coverage for one or more scenario instances. For example, as highlighted by diagonal shading, there is a gap between phase 302a and phase 302b (i.e., corresponding to timestamp six to timestamp nine) in at least one instance of computing scenario <NUM>. For example, perhaps no trace data is generated during boot loader initialization. Depending on coverage by traces <NUM>, some instances of computing scenario <NUM> may have gaps that others do not. Thus, in some embodiments, different instances of the computing scenario <NUM> may have different subsets of a set of phases. However, for purposes of the example 300a, it is assumed that each instance of computing scenario <NUM> has a gap between phase 302a and phase 302b.

In embodiments, for each trace of each phase contained in traces <NUM>, the critical path extraction component <NUM> performs a critical path extraction on the trace to create a wait chain. In one embodiment, beginning at the end of the phase, the critical path extraction component <NUM> analyzes a given trace of the phase to identify the last wait operation that occurred before the phase ended. Next, the critical path extraction component <NUM> further analyzes the trace to identify the last wait operation that occurred before the identified wait operation. The critical path extraction component <NUM> continues this analysis until it reaches the beginning of the trace/phase. As a result of the critical path extraction, the critical path extraction component <NUM> identifies a sequential chain of wait operations from the trace, which wait operations proceed between the beginning of the phase to the ending of the phase.

In <FIG>, example 300a shows a plurality of wait chain sets <NUM> (i.e., wait chain set 303a to wait chain set 303e) which-as indicated by downward arrows-each corresponds to one phase in phases <NUM>. Each of wait chain sets <NUM> corresponds to a different trace of a corresponding phase in phases <NUM>. Thus, for example, for at least one trace of phase 302a (i.e., contained in traces <NUM>), the critical path extraction component <NUM> has extracted a wait chain comprising wait operations A, B, and C; for at least one trace of phase 302b, the critical path extraction component <NUM> has extracted a wait chain comprising wait operations D, E, and F; and so on to at least one trace of phase 302e and a wait chain comprising wait operations M, N, and O. Notably, in embodiments, different wait chains in a given one of wait chain sets <NUM> can include a different number and/or identity of wait operations, depending on the data contained in the different traces of the corresponding phase.

In embodiments, the critical path extraction component <NUM> also identifies, for each identified wait operation, one or more attributes of the wait operation. In embodiments, these attributes include one or more of a first call stack of a first thread that initiated a given wait operation, a second call stack of a second thread that terminated the wait operation, a type of the wait operation (e.g., timer, disk I/O, network I/O, CPU busy, etc.), or context information for a thread involved in the wait operation (e.g., an identity of a process to which the thread belongs, command line parameters used to initiate the process, etc.).

In embodiments, the signature identification component <NUM> identifies one or more signatures (i.e., signatures <NUM>), each of which is configured to match to a subset of one or more wait operations in a wait chain identified by the critical path extraction component <NUM>. As mentioned, in embodiments, each of the signatures <NUM> comprises one or more regular expressions and/or one or more conditional statements that are configured to match one or more attributes of one or more wait operations-such as the attributes that were identified by the critical path extraction component <NUM>.

In embodiments, the coverage calculation component <NUM> calculates one or more "coverages," including at least a signature coverage. In embodiments, the coverage calculation component <NUM> calculates a signature coverage for each wait chain based on determining how much of each wait chain has wait operations to which at least one of signatures <NUM> match. As will be appreciated, wait chains having a larger portion of their wait operations to which the signatures <NUM> match have a larger portion of "known" wait operations than wait chains having a lesser portion of their wait operations to which the signatures <NUM> match.

Turning to <FIG>, in example 300b shows the wait chains <NUM> of example 300a now show subsets of wait operations in bold, indicating that the coverage calculation component <NUM> has matched one or more signatures to those wait operations (either individually or as a group/sequence). Here, the wait operations in the illustrated wait chain within wait chain set 303a are about <NUM>% "covered" by signatures <NUM>, the wait operations in the illustrated wait chain within wait chain set 303b are about <NUM>% "covered" by signatures <NUM>, the wait operations in the illustrated wait chain within wait chain set 303c are <NUM>% "covered" by signatures <NUM>, the wait operations in the illustrated wait chain within wait chain set 303d are <NUM>% "covered" by signatures <NUM>, and the wait operations in the illustrated wait chain within wait chain set 303e are <NUM>% "covered" by signatures <NUM>.

In embodiments, the coverage calculation component <NUM> also calculates at least one of a phase coverage that characterizes how much of a computing scenario is covered by identified phases, or a wait coverage that characterizes how much of the identified phases have identified corresponding wait chains. For example, in examples 300a/300b, the coverage calculation component <NUM> might determine that computing scenario <NUM> has about <NUM>% phase coverage (i.e., accounting for the phase gap between timestamp six and timestamp nine) and that computing scenario <NUM> has <NUM>% wait coverage (because each phase has a corresponding wait chain set). In embodiments, these additional coverages are used to drive further gathering of traces <NUM> in order to increase an amount of phase coverage and/or wait coverage (which, in turn, can lead to increased signature coverage).

In embodiments, the analysis component <NUM> uses at least the coverage calculations by the coverage calculation component <NUM> to classify and/or direct further analysis of the traces <NUM>. In one embodiment, the analysis component <NUM> classifies the traces into at least two categories: those corresponding to wait chains having a relatively greater amount of signature coverage, and those corresponding to wait chains having a relatively lesser amount of signature coverage. Since signatures are used to match "known" wait operations (or sequences thereof) within wait chains, then in embodiments this classification is used by the analysis component <NUM> to identify one or more of the traces <NUM> that have a relatively lesser amount of signature coverage than one or more others of the traces <NUM> and, thus, a greater amount of unknown wait operations. Thus, in these embodiments, an output of the analysis component <NUM> is an identity of one or more of the traces <NUM> having a greater number, percentage, etc. of unknown wait states than others of the traces <NUM>. In embodiments, having a greater number, percentage, etc. of unknown wait states is interpreted as meaning that these trace(s) may have captured an unknown-and potentially rare-performance issue. Thus, in these embodiments, the output by the analysis component <NUM> triggers further analysis of these identified trace(s) (e.g., by an automated computer analysis, by a human developer, etc.). As will be appreciated, these embodiments can be extremely useful for focusing resources-whether they be computer hardware or human-to analysis of a potentially small subset of the traces <NUM> that have captured delays (wait operations) that are not from known issues.

In some embodiments, after having triggered the further analysis of the identified trace(s), the analysis component <NUM> re-runs one or more of the critical path extraction component <NUM>, the signature identification component <NUM>, or the coverage calculation component <NUM> in view of one or more new signatures that are added to signatures <NUM> as a result of the triggered analysis of the identified trace(s). In some embodiments, the analysis component <NUM> analyzes coverage of these new signature(s) against the existing corpus of traces <NUM> to determine how widespread a newly-identified performance issue is in these traces <NUM>, which is helpful for providing a data-driven decision for triage (e.g., to prioritize fixing/mitigating a bug). In other embodiments, the analysis component <NUM> analyzes coverage of these new signature(s) against newly generated traces <NUM> to validate if a newly-identified performance issue has actually been fixed and properly deployed, identify a later regression of the performance issue, etc..

In some embodiments, the analysis component <NUM> uses the data generated by the coverage calculation component <NUM> to perform a "worst case scenario" analysis that determines if one or more scenario goals would be met using only uncovered wait chain durations. For example, a scenario may be associated with an SLA, such as five nines, six nines, etc. uptime guarantees, thereby driving scenario or phase goals for a maximum amount of time that the scenario/phase can take to execute. In embodiments, the analysis component <NUM> identifies which portion(s) of the traces <NUM> correspond to wait chain sections that are covered by the signatures <NUM> (i.e., covered duration), and which portion(s) of the traces <NUM> correspond to wait chain section that are not covered by the signatures <NUM> (i.e., uncovered duration). The analysis component <NUM> then considers only the uncovered durations of these traces (i.e., assuming that all known performance issues matched by the signatures <NUM> are, or will, be addressed), and determines if the uncovered duration would be sufficient to meet one or more performance goals.

A more particular description of operation of the wait chain analysis component <NUM> is now given in connection with <FIG>, which illustrates a flow chart of an example method <NUM> for identifying a trace based on wait chain coverage analysis. Method <NUM> will be described with respect to the components and data of computer architecture <NUM>, with particular focus on the wait chain analysis component <NUM> of <FIG>, as well as with respect to the examples 300a/300b of critical path extraction and wait chain analysis of a computing scenario as illustrated in <FIG> and <FIG>. Although the acts of method <NUM> may be discussed in a certain order or may be illustrated in a flow chart as occurring in a particular order, no particular ordering is required unless specifically stated, or required because an act is dependent on another act being completed prior to the act being performed.

As shown in <FIG>, method <NUM> comprises an act <NUM> of identifying a computing scenario. In embodiments, act <NUM> includes identifying a computing scenario having a scenario beginning and a scenario ending, the scenario being covered by a plurality of traces corresponding to execution of a plurality of instances of the scenario. In an example, the scenario identification component <NUM> identifies computing scenario <NUM>. In embodiments, the scenario identification component <NUM> makes this identification based on user input, based on analysis of traces <NUM>, etc..

Method <NUM> also comprises an act <NUM> of identifying a plurality of scenario phases. In embodiments, act <NUM> includes identifying a plurality of scenario phases between the scenario beginning and the scenario ending, each phase having a corresponding phase beginning and a corresponding phase ending, each phase being covered by one or more corresponding traces of the plurality of traces, the one or more corresponding traces being usable to identify one or more wait operations that occurred during a prior execution of a prior instance of the phase in connection with execution of a prior instance of the scenario. Continuing the foregoing example, the phase identification component <NUM> identifies phases <NUM> from the computing scenario <NUM> identified in act <NUM>. In embodiments, the phase identification component <NUM> makes this identification based on user input, based on analysis of traces <NUM>, etc..

As discussed, in embodiments, each phase is independent of one another-such that the duration of one phase does not affect the duration of another phase. Thus, in some embodiments of act <NUM>, a duration of each phase in the plurality of phases is independent from a duration of each other phase in the plurality of phases.

Method <NUM> also comprises an act <NUM> of extracting critical paths from the phases. In embodiments, act <NUM> includes, for each prior instance of each phase, identifying, based at least on the one or more corresponding traces, a corresponding wait chain comprising a series of wait operations between the corresponding phase beginning and the corresponding phase ending. Still continuing the example, for each of phases <NUM>, the critical path extraction component <NUM> analyzes one or more corresponding traces of that phase from traces <NUM>. Based at least on these traces, the critical path extraction component <NUM> identifies a wait chain set characterizing the wait operations identified from the trace. Thus, in the context of examples 300a/300b, the critical path extraction component <NUM> identifies wait chain set 303a for phase 302a, wait chain set 303b for phase 302b, etc..

As discussed, in embodiments, the critical path extraction component <NUM> also identifies one or more attributes of each wait operation, such as one or more of a first call stack of a first thread that initiated a given wait operation, a second call stack of a second thread that terminated the wait operation, a type of the wait operation, or context information for a thread involved in the wait operation. Thus, in some embodiments of act <NUM>, identifying each corresponding wait chain comprises identifying, for each wait operation in the wait chain, one or more attributes of the wait operation, including identifying one or more of (i) a first call stack of a first thread that initiated the wait operation, (ii) a second call stack of a second thread that terminated the wait operation, (iii) a type of the wait operation, or (iv) thread context information.

Method <NUM> also comprises an act <NUM> of identifying wait signatures. In embodiments, act <NUM> includes identifying one or more signatures, each signature configured to match a subset of wait operations in one or more wait chains. Still continuing the example, the signature identification component <NUM> identifies signatures <NUM> which, in embodiments, comprises one or more regular expressions and/or one or more conditional statements that are configured to match attributes of one or more wait operations in the wait chain sets <NUM> that were identified by the critical path extraction component <NUM>. Thus, in some embodiments of act <NUM>, each signature comprises at least one of (i) one or more regular expressions that are configured to match attributes of one or more wait operations, or (ii) one or more conditions that are configured to match attributes of one or more wait operations. Furthermore, in some embodiments of act <NUM>, each signature is configured to match a subset of wait operations in one or more wait chains based on matching one or more of (i) a sequence of call stacks that lead to a particular wait, (ii) a sequence of call stacks that terminated the particular wait, or (iii) a type of the particular wait.

Method <NUM> also comprises an act <NUM> of calculating coverage. In embodiments, act <NUM> includes calculating one or more coverages, including calculating a signature coverage that characterizes one or more portions of the identified corresponding wait chains that have one or more wait operations that are matched by the one or more signatures. Still continuing the example, the coverage calculation component <NUM> calculates one or more coverages, including calculating a signature coverage based on matching the signatures <NUM> accessed in act <NUM> to the wait chain sets <NUM> extracted in act <NUM>. In embodiments, the signature coverage of a given wait chain is calculated based on a percentage of total wait operations in the wait chain to which at least one signature matches (or, in the inverse, a percentage of total wait operations in the wait chain to which at least one signature does not match). However, other embodiments quantify coverage in other ways, such as an absolute number of matching or non-matching wait operations in each wait chain.

As discussed, in some embodiments the coverage calculation component <NUM> also calculates at least one of a phase coverage that characterizes how much of a computing scenario is covered by identified phases, or a wait coverage that characterizes how much of the identified phases have identified corresponding wait chains. Thus, in some embodiments of act <NUM>, calculating the one or more coverages includes calculating at least one of (i) a phase coverage that characterizes how much of the computing scenario is covered by the plurality of phases, or (ii) a wait coverage that characterizes how much of the plurality of phases have identified corresponding wait chains.

Method <NUM> also comprises an act <NUM> of triggering analysis of trace(s) having unknown wait states. In embodiments, act <NUM> includes based on calculating the one or more coverages, triggering an analysis of one or more of the plurality of traces as having unknown wait states based on the identified one or more traces corresponding to a larger amount of uncovered wait chain portions than one or more others of the plurality of traces. Still continuing the example, the analysis component <NUM> uses the coverages illustrated in example 300b (and computed by the coverage calculation component <NUM>) to identify at least one of traces <NUM> that has a lesser amount of signature coverage by signatures <NUM>, and to trigger a further analysis of that trace (e.g., by a computer and/or by a human). In one particular example, the analysis component <NUM> identifies, and triggers analysis of, a first trace corresponding to the illustrated wait chain in wait chain set 303d, since that wait chain has the lowest percent (i.e., <NUM>%) of signature coverage. In another particular example, the analysis component <NUM> identifies, and triggers analysis of, a second trace corresponding to the illustrated wait chain in wait chain set 303c, since that wait chain has a greatest number (i.e., three) of uncovered wait operations.

As mentioned, triggering the analysis of one or more of the of traces may result in one or more additions to the signatures <NUM>, and that the analysis component <NUM> may use these new signature(s) against existing traces <NUM> to determine how widespread a newly-identified performance issue was, or may use these new signatures against new traces <NUM> validate if a newly-identified performance issue has actually been fixed and properly deployed in, to identify a later regression of the performance issue, etc. Thus, in some embodiments, method <NUM> also comprises identifying a signature that matches at least one of the unknown wait states, and determining at least one of (i) whether the signature matches at least one of the plurality of traces, or (ii) whether the signature matches at least one additional trace not in the plurality of traces.

As also mentioned, the analysis component <NUM> may utilize the data generated by the coverage calculation component <NUM> to perform a "worst case scenario" analysis that determines if one or more scenario goals would be met using only uncovered wait chain durations. Thus, in some embodiments, method <NUM> comprises, based on calculating the one or more coverages, determine if one or more scenario goals would be met using only uncovered durations.

Accordingly, the embodiments described herein perform an automated wait chain-based analysis of trace data in order to identify traces that contain relatively larger durations unknown wait events. These embodiments identify a computing scenario that has a definable beginning and ending, as well as statistically independent phases of this scenario (i.e., in which the duration of one phase is an independent variable to the duration of another phase). For each phase, the embodiments herein perform a critical path analysis of different traces of that phase (e.g., each trace corresponding to a different execution of the phase across one or more computers). An output of the critical path analysis of a trace is a wait chain, which identifies a chain of wait operations between the beginning of the phase and the ending of the phase. Based on a collection of signatures that match known wait chain patterns (i.e., known/identified performance issues), these embodiments determine a signature coverage over the identified wait chains. Wait chains with a higher degree of signature coverage are considered to have known and/or common performance issues, while wait chains with a lesser degree of signature coverage are considered to have unknown and/or rare performance issues. Thus, the embodiments herein are usable to identify-and trigger an analysis of-traces having wait chains with lesser degrees of signature coverage for further analysis to identify/fix rare performance issues.

Claim 1:
A computer system comprising:
at least one processor (<NUM>); and
at least one computer-readable medium (<NUM>) having stored thereon computer-executable instructions that are executable by the at least one processor to cause the computer system to identify a trace (<NUM>) based on wait chain coverage analysis (<NUM>), the computer-executable instructions including instructions that are executable by the at least one processor to cause the computer system to perform at least the following:
identify (<NUM>) a computing scenario having a scenario beginning and a scenario ending, the scenario being covered by a plurality of traces corresponding to execution of a plurality of instances of the scenario;
identify (<NUM>) a plurality of scenario phases between the scenario beginning and the scenario ending, each phase having a corresponding phase beginning and a corresponding phase ending, each phase being covered by one or more corresponding traces of the plurality of traces, the one or more corresponding traces being usable to identify one or more wait operations that occurred during a prior execution of a prior instance of the phase in connection with execution of a prior instance of the scenario;
for each prior instance of each phase, identify (<NUM>), based at least on the one or more corresponding traces, a corresponding wait chain comprising a series of wait operations between the corresponding phase beginning and the corresponding phase ending;
identify (<NUM>) one or more signatures, each signature configured to match a subset of wait operations in one or more wait chains;
calculate (<NUM>) one or more coverages, including calculating a signature coverage that characterizes one or more portions of the identified corresponding wait chains that have one or more wait operations that are matched by the one or more signatures; and
based on calculating the one or more coverages, trigger (<NUM>) an analysis of one or more of the plurality of traces as having unknown wait events based on the identified one or more traces corresponding to a larger amount of uncovered wait chain portions than one or more others of the plurality of traces, wherein unknown wait events are operations that have not matched to one or more of the signatures.