Machine learning based software correction

Techniques and solutions are described for training and using a classifier based on a machine learning model to analyze performance information to assist in correcting a software bug or performance issue. The performance information can be processed prior to submission to a trained classifier, such as to remove, modify, or format data. A classification result provided by the classifier can be compared with a database to determine whether a solution or target is associated with the classification result. User feedback can be used to provide more accurate suggestions of solutions or targets, as well as to improve the accuracy of the classifier.

FIELD

The present disclosure generally relates to automated or semi-automated correction of software coding errors, or bugs, or performance issues. Particular implementations relate to the training or use of a machine learning-based classifier.

BACKGROUND

Determining and correcting the source of software errors or performance issues, commonly referred to as debugging, remains a major problem in software development. Several studies have estimated that debugging consumes over fifty percent of software development time and costs. Although software configuration management systems and formalized debugging approaches can assist in debugging efforts, debugging can remain a tedious, time consuming task.

Further, even after release bugs can arise, or performance issues identified. In some cases, a program may function, but it may suffer from performance issues (e.g., unexpectedly or undesirably high execution time, processor use, memory use, network use, or the like). Or, a program may function as desired in some scenarios, but fail to function, or suffer from performance issues, in other scenarios. Detecting and resolving these bugs or performance issues can be time consuming. Accordingly, room for improvement exists.

SUMMARY

Techniques and solutions are described for training and using a classifier based on a machine learning model to analyze performance information to assist in correcting a software bug or performance issue. The performance information can be processed prior to submission to a trained classifier, such as to remove, modify, or format data. A classification result provided by the classifier can be compared with a database to determine whether a solution or target is associated with the classification result. User feedback can be used to provide more accurate suggestions of solutions or targets, as well as to improve the accuracy of the classifier.

A method is provided for obtaining classification results, such as in the form of a performance indicator that may be associated with a solution or target for correcting a performance issue or bug. Performance information is received. The performance information is associated with the performance issue or bug and with software execution during the occurrence of the performance issue or bug. The performance information includes at least one performance parameter (e.g., execution time or resource use) and at least one software feature (e.g., program, program component, program operation or process, etc.) associated with the at least one performance parameter.

At least a portion of the performance information, including the at least one software feature, is formatted for submission to a classifier using a machine learning model. The formatted performance information is provided to the classifier. A classification result is received. It is determined if a solution is associated with the classification result, such as by querying a database. Based on the determining, at least one identified solution, or an indication that a solution was not identified, is returned.

According to another embodiment, a method is provided for automatically analyzing performance information. At least one software performance metric is monitored. It is determined whether at least one metric violates a threshold. Performance information, including at least one performance parameter, is caused to be gathered. A type associated with the performance information is determined. An extraction rule associated with the type is received. At least one performance parameter is extracted from the performance information using the extraction rule. Performance information that includes the performance parameter is formatted. The formatted performance information is provided to a classifier or a clustering engine. A classification or clustering result is received. A solutions database is queried for a solution using the classification or clustering result. At least one identified solution or an indication that a solution was not identified is returned.

According to a further aspect, a method is provided for training a classifier based on a machine learning algorithm. A plurality of training data objects are received. The plurality of training data objects include performance information associated with software associated with a performance issue or bug and with software execution during the occurrence of the performance issue or bug. The performance information includes at least one performance parameter and at least one software feature associated with the at least one performance parameter. The training data objects include at least one performance indicator, which is associated with a solution to the performance issue or bug. The training data objects further include a solution to the performance issue or bug.

For the plurality of training data objects, at least a portion of the performance information is formatted for submission to a machine learning algorithm. A machine learning model is trained with the formatted performance information to provide a trained classifier. The trained classifier outputs a performance indicator in response to an input test data object. The input test data object includes performance information. The output performance indicator can be submitted to a solutions database to provide a suggested solution or an indication that a solution was not identified.

The present disclosure also includes computing systems and tangible, non-transitory computer readable storage media configured to carry out, or including instructions for carrying out, an above-described method. As described herein, a variety of other features and advantages can be incorporated into the technologies as desired.

DETAILED DESCRIPTION

Overview

Software is ubiquitous in today's society. Software is being incorporated into an increasing number of devices, developed and improved for an ever increasing number of applications, and is growing ever more complex. Determining and correcting the source of software errors or performance issues, commonly referred to as debugging, remains a major problem in software development. Several studies have estimated that debugging consumes over fifty percent of software development time and costs.

As used herein, a bug, also referred to as a coding error or software defect or fault, refers to code behavior that produces unintended or undesirable behavior. Bugs can result from failure to follow the correct syntax of a programming language. Bugs can also result from improper code semantics or code design, such as mathematical operator errors (e.g., division by zero), variables that may go out of bounds (e.g., a variable that causes an array to go out of bounds), memory leaks, security flaws, or bugs that result from improper interaction of software modules or components. Categories of bugs can include arithmetic errors, logic errors, syntax errors, resource errors, threading errors, interface errors, or errors resulting from the interaction of different programmers. Bugs can be associated with these and other types, and in some cases error messages or codes, including error messages from exception handling routines. Further, performance information, such as logs or traces, can include performance parameters (e.g., execution time or resource use) that can be associated with a code error or code that can be improved.

Even after software is released, bugs can be identified, or performance issues identified, typically requiring software developers to determine the source of an error and deploy an appropriate correction. This problem can be compounded when a user, such as a business, requests customization to a standard software program, or customizes the software themselves. In some cases, a program may function, but it may suffer from performance issues (e.g., unexpectedly or undesirably high execution time, processor use, memory use, network use, or the like). Or, a program may function as desired in some scenarios, but fail to function, or suffer from performance issues, in other scenarios. Performance issues can arise even if code technically does not contain a bug (e.g., the program does not crash, and may not even invoke error handling routines, but has suboptimal operation). Detecting and resolving these bugs or performance issues can be time consuming.

Software development can be very specialized. As developers work on a particular software program, they may be able to more efficiently determine the source of a software bug or performance issue and to develop and deploy a correction. However, if the developer leaves or is reassigned, the knowledge can be lost, which can result in work being redone to fix a bug or performance issue that previously occurred, or more time and resources being needed to fix a new error. Even if the developer is still in the same position, that developer may not assigned to fix the issue when it arises again, and a different developer assigned to the task may have to redevelop the solution.

Particularly for programs that have been deployed (e.g., have been installed and are in use), the complexity of identifying and correcting bugs and performance issues can be even greater. For example, a program, such as an application, may operate using one or more frameworks, program libraries, and services, such as accessing a database. If a bug or performance issues arises, a first step may be to determine whether the bug or performance issue is being cause by the application, one of the frameworks, a component of a program library, a service, or a combination of these. It can be difficult to find a developer with expertise in every software (or hardware) component involved in program execution, which can lead to significant time being spent passing the issue between multiple developers, or a single developer having to learn the operation of multiple components, also resulting in further expense.

Many software programs have logging capabilities, which can be automatically maintained or can be selectively activated, to track program execution. For example, undesired program behavior can result in a log entry. An exception handling routine, in some cases, can generate a log entry as part of the handling. Logs can be reviewed by developers to help identify the presence, or possible cause, of a bug or performance issue.

In some cases, a bug or performance issue may not be documented in a log, or the information in the log may not be sufficient to identify its cause. Some programs (or programs that can monitor other programs) can generate program traces that can track aspects of program execution. For example, a database system may have tracing functionality to monitor query execution, such as monitoring the execution time of various operations or suboperations of query execution. Reviewing a trace can identify operations that are associated with the bug or performance issue. For example, a particular SQL operation may be implicated if it executes for an abnormally long time.

Other components can provide similar functionality for recording performance information. Some programming languages or frameworks can allow for the monitoring of particular components of program execution, such as execution of particular method calls, including the time taken for their execution. A framework or operating system can also monitor parameters, such as resource use of particular programs (e.g., CPU, memory, or network use). A user interface component can track the time taken to generate user interface screens or the time taken to execute a request for data.

Even though software components may provide logging or tracing functionality, as mentioned above, a particular developer may not know how to access or interpret the logs, or to activate and interpret the tracing. This problem is compounded when multiple programs are involved. Even after the problem is identified, a solution still needs to be developed and deployed.

The present disclosure provides various technological innovations that can provide improved detection and resolution of bugs and performance issues. In one aspect, the present disclosure provides a tool that can automatically detect performance issues. For example, rules can be defined that measure key performance indicators defined for a program or software component. If one or more key performance indicators indicate a performance issue, such as being higher or lower relative to a threshold, the tool can automatically cause performance measurements, such as those reflected in logs or traces, to be analyzed, and optionally generated. In other cases, the generation or analysis of performance information can be triggered in response to error detection, including as part of an error handling routine. The analysis can be used to automatically generate a code correction, suggest a resolution to a developer, or alert the developer to a particular program, program component, or code fragment that may be causing a bug or performance issues.

In other cases, a developer or administrator can manually cause the generation and analysis of performance information, such as log and trace files, upon observing or learning of a performance issue or bug. In further cases, a developer or administrator can manually add a log or trace to be automatically analyzed. For example, a saved log or trace file can be submitted for analysis.

Analyzing performance information can include categorizing the performance information. Categorization can be based on various criteria, such as by software component or function. For example, the performance information can be classified as representing user interface components, application components, or database components. The performance information can be subcategorized, if desired, including using automatic classification tools, such as deep natural language processing. Subcategorization can be used to suggest particular subcomponents that might be responsible for a performance issue or bug. For instance, for performance information initially categorized as being in the database layer, the subclassification may suggest that the issue lies with a particular database engine, such as a JOIN engine, OLAP engine, row engine, or column engine.

Categorized information or subcategorized performance information can be analyzed, such as by being filtered, to narrow down a potential issue to particular operations reflected in the performance information. In some cases, this can result in particular categories, subcategories, or individual performance information items being selected for further analysis, or being excluded from further analysis. For instance, if performance information is received for multiple categories or subcategories, but only performance information for one category is shown as violating a particular KPI or metric, only that performance information may be further processed. In other cases, a greater amount, including all, performance information is processed. For example, performance information from operationally linked components may be further processed.

In particular aspects, performance information can be formatted prior to further analysis. For example, the performance information can be converted to a common or generic format, which can allow comparisons between code or other program components that carry out similar functions, and may generate similar bugs or performance issues, although some specific details (e.g., particular variables, method calls, tables accessed, etc.) may differ between the code or other program components.

The performance information can be analyzed using a machine learning technique. For example, the performance information can be converted to one or more word vectors and analyzed using embedded wording techniques such as Continuous Bag of Words or Skip Gram. The analysis can generate a label or contextual code or operations that are likely associated with the performance issue or bug.

The code, operations, or other output of the machine learning algorithm can be used to search a database of possible solutions. If a solution is identified, it can be presented to a user who can then evaluate and potentially implement the solution. In other cases, the solution may be automatically implemented. Feedback can be provided to the system as to whether a suggested solution was correct, which feedback can be used to further refine the machine learning classifier or adjust solution rankings in the database.

If a solution is not identified, one or more potential code or program features, also referred to as targets or “hotspots,” may be suggested to a user. The user can then analyze the suggested hotspot and may develop a solution to the issue. Feedback can be provided by the user as to whether the suggested hotspot was accurate, which can be used to further refine the machine learning classifier or adjust hotspot rankings in the database. If a solution was identified, the solution can be added to the database.

If no solution or hotspot was identified, but the user identifies a solution or hotspot, the solution or hotspot may be added to the database.

In at least some cases, an organization may have a large amount of historical data that can be used to train a classifier. For instance, a significant portion of software available from SAP, SE, of Walldorf, Germany, is written in the ABAP programming language. Particularly as it is at least partially an interpreted language, ABAP code can be updated by applying correction instructions to source code, rather than replacing entire source code files or replacing object or executable code files. For instance, updates, such as those in the form of correction instructions, can be created and manipulated using the ABAP WORKBENCH of SAP SE and propagated and applied using the SAP HANA TRANSPORT FOR ABAP system, also of SAP SE. The correction instructions, as well as performance information and performance indicators (e.g., programs, program components, or program operations) can be used to train a machine learning model.

Historical information from prior incidents of bugs or performance issues can be used to train a machine learning algorithm to provide a classifier. The historical information can include performance information associated with the incident, identifications of the source of the associated bug or performance issue (e.g., the particular code or operators, etc.), and solution information for correcting the issue (e.g., ABAP correction instructions or notes).

Thus, disclosed technologies can provide advantages in identifying and correcting software bugs and performance issues, which can reduce the manual effort needed to both find and correct bugs and performance issues. The disclosed technologies can improve the operation of a computer by correcting such bugs and performance issues, which can improve execution time and the use of computing resources, such as CPU, memory, and network use. Disclosed technologies can address the technical problem of software bugs and performance issues through the technical solution of a trained classifier that can be used to provide suggested solutions or target areas to investigate to locate the source of the bug or performance issue. Disclosed technologies can be implemented as a centralized resource, such that the feedback of multiple users can be aggregated and made available, and can be used to improve the quality of the suggested solutions and target areas, to add new solutions or target areas to a database that can be accessed using the results of the classifier, and to provide training data that can be used to improve the accuracy of a classifier. Technical solutions also include features for classifying performance information and formatting performance information to be input to the classifier.

Example Processing and Use of Performance Information

FIG. 1is a block diagram illustrating a scenario100of how performance information can be provided to a classifier to provide a recommended solution, or investigation target, for a bug or performance issue. The environment100includes raw performance information108. The raw performance information108can include one or more log files110, one or more trace files112, or other types of information. Although referred to as files, the log, trace, or other performance information can be provided in other ways than in the form of a file. For example, the information can be obtained directly from memory.

The raw performance information108may relate to a common software component or function, or relate to a plurality of components or functions. For instance, the performance information108may be provided from one or more applications (e.g., applications that are directly accessible by a user, such as an end user), a user interface that accesses an application, one or more frameworks (e.g., programs that support the operation, such as providing services to an application), one or more operating systems, and applications or other services accessed by an application, such as a database system. The raw performance operation can include values associated with various parameters of execution of the components or functions, such as execution time or resource use (e.g., CPU, memory, network, etc.).

In a processing step116, the raw performance information108is processed to provide processed performance information118, including processed log files120and processed trace files122. In at least some cases, at least some of the raw performance information108is not processed and is used in its raw form in subsequent analysis steps.

The processing116can include removing particular types of performance information. For instance, desired information can be extracted, or undesired information can be removed from a log or trace, such as removing individual log or trace entries. Undesired information can include information that is not used in subsequent processing, such as performance metrics for particular software components, or particular types of performance metrics.

In a specific example, information is excluded from subsequent processing that relates to software components that are not modifiable by a developer, or which do not reflect (e.g., provide information that can be used to analyze) potential bugs or performance issues of software components that are modifiable by a developer. As an example, components of an operating system, library, or repository provided by a third party (or which are otherwise not modifiable by a developer) are excluded. In a specific example, in C++, elements of core C++ or C libraries (e.g., the C standard library) may be excluded.

A first level classification step124can be applied to the processed performance information118, the raw performance information108, or a combination thereof. The classification124is shown as providing categorized performance information128, which can include categories such as “UI Issue”130, “Application Issue”132, “Database Issue”134, and other categories136.

The first level classification124can employ one or more rules, which can be maintained in a rule repository (not shown). The rules can specify criteria for classifying performance information. Classification criteria can include classifying performance information via file extension or file type. For instance, files with a .har or .hwl extension may be classified as “user interface” files with .plv or .csv extensions may be classified as “database,” and files with .txt or .csv extensions may be classified as “application.”

Classification criteria can also be based on metadata associated with the performance information (e.g., metadata indicating a type of performance information, a source program, a program that generated a log or trace, etc.). Classification criteria can also be based on contents of the performance information, such as whether the contents include programming language elements (e.g., elements associated with ABAP, C++, JAVA, JavaScript, etc.), database operations (e.g., SQL statements or tokens, database operators, or database suboperators), or user interface elements (e.g., JAVA classes providing user input functionality). Classification criteria can also be based on comparing contents of the performance information, such as execution times, to one or more thresholds, where, for example, execution times that meet a threshold are associated with a first classification and execution times that do not meet the threshold are associated with a second classification. In at least some cases, classification at124can occur prior to the processing116, which can allow information to be used for classification that is not needed for subsequent steps. Or, the rule can be used to define an extraction process.

If two extensions, or other classification criteria, have the same value, other classification criteria can be used to determine a final classification. For example, if two classifications might be associated with a .csv file extension, criteria such as execution time or the presence or absence of SQL operators or ABAP elements may be used to distinguish between two categories, such as “application” and “database.” Or, execution time alone can be used to determine a classification (or subclassification) type.

In at least some implementations, classification, at least for certain categories, can include a multilevel classification. A multilevel classification can be carried out in a similar manner to the classification at124, where the presence of particular keywords, for example, can be used to classify all or a portion of performance information into a subcategory. In some cases, an entire source of performance information (e.g., an entire log or trace) can be subclassified. In other cases, portions of performance information can be subclassified, and can be maintained with performance information having a different subclassification or can be extracted into a separate group having the particular subclassification. If desired, a tag, such as a metadata identifier, can be applied to categorized or subcategorized information.

In a particular aspect, subcategorization can be carried out, at least to a portion of the performance information (typically the categorized information128, but in other cases, the raw performance information108or the processed performance information118, in which case the classification can be a first level classification, a second level classification, a higher level classification, or a combination thereof) using a machine learning algorithm in process140. The machine learning algorithm can be, for example, a deep natural language processing algorithm.

The process140(or another subclassification process) can provide clustered performance information144. The clustered performance information144can be distributed among different categories146and optionally one or more subcategories148. For example, the “user interface” category146can include subcategories148relating to coding issues, issues with a document object model, JavaScript issues, and CSS issues. A “database” category can include subcategories148relating to various database engines, such as a JOIN engine, an OLAP engine, a column engine, or a row engine.

As described, classification or subclassification can be automated. In other cases, the raw performance information108and the processed performance information118can be classified or subclassified in another manner, such as manually according to user input or by a metadata field associated with the performance information.

The clustered performance information144can be formatted using process152to provide formatted input156for a classifier160based on a machine learning model. Formatting can include removing certain portions or replacing certain portions of the performance information. As an example, the classifier160may have been trained using training data where SQL operators were left in the training data, but information as to specific fields and tables accessed by the operators was removed. In this manner, performance information can be more directly compared. In many cases, a problem may lie with how the operators are used, or with other software code or components that is functionally associated with the operators. Even if the problem lies with the data being processed, this can be revealed using training and classification data that does not include the tables and fields (e.g., a possible solution may be to reformat a particular table or to confirm table data is in the correct format).

The classifier160can use any suitable machine learning technique. Generally, the machine learning technique is a supervised machine learning technique, such as decision trees, artificial neural networks, instance-based learning, Bayesian methods, reinforcement learning, inductive logic programming, genetic algorithms, support vector machines, or combinations thereof. In a particular example, the classifier160uses a word embeddings based technique, such as Continuous Bag of Words or Skip Gram. The formatting152can include formatting clustered (or other) performance information144for use in the classifier160(e.g., converting the performance information to one or more word vectors).

In some cases, the classifier160can include, or can be replaced by, a clustering engine (e.g.,FIG. 1can include a clustering engine160). When included, the clustering engine can use machine leaning or non-machine learning based approaches. The clustering engine can provide a clustering result, which can otherwise be used in a similar manner as a classification result from the classifier160.

The classifier160produces an output, or classification result,164. The output164can be code, code fragments, programs, program components, or other output that can be used to identify a likely source of a performance issue or bug. In some cases, the output164can represent a target, such as code or components that a developer may investigate as a problem source.

The output164can be supplied to a recommendation engine168. The recommendation engine168can search a database172for information related to the output164, such as to identify one or more possible solutions to the bug or performance issue, or to suggest one or more targets that may be responsible for the bug or performance issue. Possible solutions or targets, and other information, such as the output164, can be provided to a user through a user interface176.

The user interface176can also allow a user to provide feedback to the recommendation engine168, the database172, or the classifier160. For example, new solutions can be added to the database172, or provided to the recommendation engine168and used to provide more accurate target and solution rankings. If the user determines that the output164was accurate or inaccurate, that feedback can be used to update weights used by the classifier160. Similarly, new associations can be created between formatted input156and input received through the user interface, such as backpropagating the user input as output164into the classifier160. Thus, the recommendation engine168and the classifier160can become more accurate and comprehensive as the environment100is used.

Example Classification and Identification of Solution or Target

FIG. 2presents a flowchart of a method200for analyzing and correcting performance issues or program bugs using one or more disclosed technologies. The method200can be carried out using various computing environments, including environments disclosed herein, such as the environment100.

In some cases, the method200can begin by automatically detecting a violation of a key performance indicator (KPI) at208. In some cases, a KPI can also be, or be associated with, one or more performance parameters of performance information. A KPI indicator can, for example, be a threshold execution time or threshold use of computing resources. For example, if execution of a database query, or rendering of a user interface screen, exceeds a threshold, a KPI violation may occur and be detected at208. Instead of, or in addition to, a KPI, error detection or error handling routines may be used to automatically or manually trigger the processing of method200, which can otherwise proceed as described for a KPI.

At212, performance information, such as logs or traces, is automatically gathered. The performance information gathered at212can be tailored to a particular KPI violation or a software component associated with the KPI violation. For instance, a rule can be defined that indicates actions to be taken when the KPI violation occurs. The actions can include retrieving logs or generating traces.

In a particular example, KPI violations can be specified with respect to ranges, including ranges for particular application types, applications, or operations. For instance, for a user-interface action that is associated with “instant” feedback, a violation can occur if the response time (e.g., from the time the user requests an action to the time when the user interface provides the appropriate response) is greater than 150 milliseconds, and a “high” violation can occur if the response time is greater than 500 milliseconds. A KPI for “instant feedback” can also be specified with respect to a number of roundtrips (e.g., from a user interface component to an application component, a database component, or a plurality of components) that is greater than 0 for both violation and high violation (e.g., the user interface should handle the action without calling another component). For a “simple” action, a violation may be indicated if the response time is greater than 1,000 milliseconds, and a high violation may be indicated if the response time is greater than 3,000 milliseconds. A violation for “simple” actions can also be specified as more than 1 roundtrip occurring for a violation, with a high violation being indicated if more than 2 roundtrips occurred. For a “complex” action, a violation may be indicated if the response time is greater than 3,000 milliseconds, with a high violation occurring if the response time is greater than 5,000 milliseconds. In terms of roundtrips, a violation may be indicated for a complex action if more than two roundtrips occur, and a higher violation indicated if more than three roundtrips occur. A user can define KPI types and values for various types of actions, as desired.

The actions at212can also include running tests or taking other actions to reproduce a bug or performance issue. As an example, if a user interface element or screen is taking longer than a threshold time to render,212can include activating tracing functionality and then reloading the screen or user interface element, where the trace will capture information about computing operations occurring while the bug or performance issue is occurring.

Alternatively, the method200can begin at216by receiving a user request to analyze a bug or performance issue, where the request includes user input of performance information, such log or trace files to be analyzed. In other cases, the user can initiate an analysis request, but the performance information gathering, including optionally reproducing the bug or performance issue (or executing a test) can be carried out as in212.

Whether initiated at208or216, the method200can process performance information at220. Processing performance information can include removing certain types of information from the performance information, such as information relating to components that are not modifiable or which are otherwise designated as not being candidates for corrective action. Processing at220can also include removing information that does not inform a possible source, or corrective action, for the bug or performance issue.

At224, the processed performance information is categorized, such as at a first level. The categorization224can be automatic or manual based on user input. Automatic categorization224can be carried out using information such as a file extension of a log or trace file, metadata associated with the performance information, or the contents of the performance information. If desired, the categorized information can be subcategorized at228. Subcategorization228, in some cases, can be carried out in a similar manner as the categorization224. In other aspects, subcategorization228can be carried out by semantically analyzing the performance information, such as using a machine learning algorithm. In a particular example, the semantic analysis is carried out using deep natural language processing.

The subcategorized performance information (or, in other cases, categorized information or even uncategorized information) is formatted at232. Formatting can include modifying the performance information for use by a classifier based on a machine learning technique. For example, formatting can be used to convert the performance information to a common format, including by removing specific information (e.g., database table identifiers or database table field identifiers) from the performance information, and optionally substituting a generic term for the specific term (e.g., TABLE for a specific table identifier, FIELD for a specific field, ROW for a specific row, or COLUMN for a specific column) In other aspects the formatting232can be omitted.

At236the performance information (typically formatted) is submitted to a classifier based on a machine learning model. The formatting at232can include formatting the performance data for use with the classifier. For example, the classifier can use a word embeddings based technique, such as Continuous Bag of Words or Skip Gram techniques. Formatting at232can include forming suitable input word vectors.

After submission to the classifier, the classifier provides output. The output can be one or more code segments, operations, programs, or program components that may be associated with the performance issue or bug. The output can be used to search a database for a potential solution at240. At244it is determined whether a solution has been identified. In some cases, more than one potential solution can exist. In such cases, the most likely solution can be returned, or a ranked list of possible solutions can be returned. If one or more solutions have been identified, they are provided, such as to a user through a user interface or in return to an API or RPC call, at248.

At252, it is determined whether the solution is accepted, such as in response to user input indicating whether the solution solved a performance issue or bug. If the solution is indicated as appropriate (e.g., solving the performance issue or bug), the ranking of the solution in the database, and optionally the classifier, can be updated accordingly (e.g., to rank the solution higher in future analyses or to provide a stronger correlation between the input and output in the classifier) at254. The method200can then end at256.

If the solution is not accepted, the ranking in the solution database, and optionally the classifier, can be updated at258to reflect that the solution was not accepted (e.g., providing a lower ranking of the solution in future analyses or weakening a correlation between the input and output for the classifier). At262, it can be determined whether the performance issue or bug was resolved, such as by a user employing an alternative solution of the database (e.g., a lower ranked solution) or creating a new solution. If the issue was resolved, the database can be updated with the new solution, or the ranking altered for an alternative solution at266. If the solution is a new solution, the database can be updated to include the solution at270. Optionally, at274, the classifier can be updated to reflect a positive correlation between the input and the output (e.g., the output was positively correlated with the input, even though the solution of the solution database was not positively correlated). The method200can then end at276. If the issue was not resolved at262, the method can end at276.

If, at244, a solution was not identified, it can be determined at278whether a target or “hotspot” was identified. If not, the method200can proceed to262and the method can continue as previously described. If a target was identified at278, the target can be provided, such as through a user interface or as a value in response to an API or RPC call, at280. The method200can determine at282whether the target was accepted, such as in response to user input. If the target was not accepted, the database can be updated at258to modify the ranking of the target and, optionally, the classifier updated to weaken a correlation between the input and output. If the target was accepted at282, at290, the method200can update the database to modify the ranking of the target and, optionally, the classifier update to strengthen a correlation between the input and output. After258or290, the method200can proceed to262and can continue as previously described.

Example Training of Machine Learning Model

FIG. 3is a diagram illustrating how training data310can be used to train a machine learning model to provide a classifier that can be used to automatically determine features associated with performance issues or bugs, and automatically suggest or provide a resolution. The training data310can represent a performance issue or bug, information useable to identify the performance issue or bug, and an identified solution to the performance issue or bug.

The training data310can include a performance identifier314. The performance identifier314be, or include, a label that identifies a source of a performance issue or bug. For instance, the performance identifier314may be, or include, a problem description such as “data not formatted correctly,” or “improper ordering of query terms.” The performance identifier may be, or include, an identifier of one or more program components that may be responsible for a performance issue or bug, such as identifying particular code segments, particular methods, or particular data types, data structures, or other program features or processes.

The training data310can include performance information318, such as log or trace information. The performance data can specify parameter values (e.g., execution time or resource use, such as CPU, memory, network, etc.) for particular operations, processes, or other components of program operation. The performance information318can be edited or formatted, such as to remove information that may not be relevant, to replace certain specific features with more generic features (e.g., removing references to particular tables or fields thereof), or to add metadata tags to performance information. The training data310can include a solution description322, which can be a description of how to solve the performance issue represented by the training data310, or can include an actual, implementable solution, such as replacement program files, program code, specifications of database operations, or the like.

The performance identifier314can represent the type of output that will be provided by a classifier formed from a machine learning model326after the model is trained (or the classifier receives additional training) The performance identifier314and the solution description322can be stored in association with one another in a solution database330. In addition to, or instead of, a solution description (or solution), the performance identifier314can be stored in association with one or more targets or hotspots.

In some cases, a performance identifier314can be associated with multiple potential solution descriptions322or targets. When a solution description322(or target) is provided by the solution database330, the solution description (or target) can be provided based on a ranking, such as a ranking based on a probability that the solution description will resolve the performance issue or bug (or that the target is associated with the performance issue or bug). In at least some cases, multiple solution descriptions322(or targets) can be provided in response to an analysis request, such as providing a ranked list.

The performance information314can be used to construct one or more word vectors334for use in the machine learning model326. The word vectors334can be “one hot” vectors, where index positions in the array represent a vocabulary and the indicated word (or other token) is identified by setting a value for the word as a first value (e.g., ‘1’) and setting other index values to a second, different value (e.g., ‘0’). The word vectors334, or other input to the machine learning model326, can include input other than “words.” For example, at least one input can be, or can include, numerical values, including numerical values representing performance parameters (e.g., execution time, CPU use, memory use, network use, etc.).

The word vectors334(an optionally other input) and performance identifier314are used to train the machine learning model326.

Example Use of Classifier

FIG. 4is a diagram illustrating how classification data410can be submitted to a trained classifier414to provide a suggested solution418or target422for a performance identifier426returned by the classifier. The classification data410can include performance information430, such as log or trace information. The classification data410can be processed in a manner analogous to that described for the performance information318ofFIG. 3. In at least some cases, the performance information430and the performance information318are processed in an identical manner

The performance information430can be converted to word vectors434, including in an analogous manner to that described for the performance information318and word vectors334ofFIG. 3. As with the input to the machine learning model326, the input to the classifier414can include information in addition to the word vectors434, or the word vectors can include additional information (e.g., values for performance parameters).

The classifier414produces the performance identifier426as output. The performance identifier426can be submitted to a solutions database438. The solutions database438can return the suggested solution418or target422. The solution418or target422can be associated with a confidence value or returned as part of a ranked list. In some cases, the performance identifier426itself may serve as a target or hotspot, in which case additional information may not be returned from the solutions database438if a solution is not identified.

Example Computing Environment

FIG. 5is a diagram of a computing environment500in which disclosed technologies can be implemented. The computing environment500can include a production environment510which may be subject to performance issues or bugs, and can be monitored, including being automatically monitored. For example, components of the production environment510can be monitored for execution time and resource use, and an analysis can be triggered if one, a plurality, or a specified combination of measured values violate a specified threshold.

The production environment510can include one or more components that can be associated with performance issues or bugs. The components can include a user interface514, an application518, and a database522. However, it should be appreciated that the components are representative only, and a production environment510can include more, fewer, or different components than shown.

Each of the components of the production environment510can be associated with performance information530, such as logs or traces. In at least some cases, all or a portion of the performance information of components of the production environment510can be different. For example, a file type, data format, or data content may differ. In addition, names or metadata information of the performance information530can differ, and the procedure used to activate or obtain the performance information can be different for different components. The components of the production environment510may have other differences, such as being written in different programming languages, and being operationally or physically separate from another.

The components of the production environment510can be used at least partially independently from one another, at least in some aspects. For instance, the application518have some functionality that does not require the use of the database522, the application may be useable with other databases, or may be used with additional user interfaces other than the user interface514. Similarly, the database522can be configured to operate with applications other than the application518, or the database can include its own applications.

The production environment510can be in communication with a performance analyzer534. In some aspects, the performance analyzer534can be associated with multiple production environments. In further aspects, the performance analyzer534can be part of a production environment510.

The performance analyzer534can include a monitor component538. The monitor component538can monitor components of the production environment510. In some cases, the monitor component538can generate an alert or start an analysis if a performance issue or bug is detected, such as if a KPI violation is detected or if an error handling routine is triggered. In at least some cases, analysis may be triggered if a sum of KPIs for different components exceeds a threshold, in which case performance information for the involved components can be further analyzed.

The monitor component538can also allow a user to view performance parameters, where a user can institute an analysis if desired. In some case, the monitor component538can be omitted. For example, other features of the performance analyzer534can be triggered by a user, or the production environment510can include a monitor component that triggers or calls functionality of the performance analyzer. The monitor component538can include stored operations for retrieving the performance information530if a rule violation is detected, or if a user request an analysis.

Although not shown, the performance analyzer534can include rule definitions that can include parameters to be monitored and values for determining whether a rule is violated. The performance analyzer534can include functionality for allowing a user to view, create, or modify such rules, such as providing a user interface or an API that can be called by a program or a user.

The performance analyzer534can include an extractor component542. The extractor component542can be used to extract (or remove) information from performance information530provided from the production environment510. For example, the extractor component542can include methods for removing unneeded data from performance information530, such as data relating to software components that are not alterable by the user or in conjunction with the performance analyzer (e.g., library files or files provided by a third party) or data that is not informative as to the source of a performance issue or bug, or how such a performance issue or bug might be resolved.

The performance analyzer534can include a categorizer546. The categorizer546can apply rules or analysis (for instance using semantic analysis, such as using deep natural language processing) to classify or categorize performance information530(which can be performance information having been extracted by the extractor component542). The categorizer546can be used to categorize performance information530, such as identifying the performance information as associated with the user interface514, the application518, or the database522. The categorizer546may also categorize the performance information530in a subcategory, such as, for the database522, whether the information is associated with a JOIN engine, an OLAP engine, a column engine, or a row engine.

In some aspects, the categorizer546can analyze the performance information530, such as to determine components or subcomponents of an aspect of the production environment510that may be giving rise to a performance issue or bug. Accordingly, categorizing by the categorizer546can include determining that particular sources of performance information530do or do not give rise to a violation (e.g., an application is running slowly, but the problem is identified as occurring in the database522rather than the user interface514or the application518). For particular sources of performance information530that may associated with a problem, the performance information can be subclassified as belonging to a particular component (e.g., function, code segment, operations) of the production environment510. Further processing can be carried out based on the categorization or subcategorization applied by the categorizer546.

The performance analyzer534can include a formatter550. The formatter550can format performance information530for use to train a machine learning model to provide a classifier, to provide additional training for a classifier, or to be used by a classifier. The formatter550can select performance information530(including performance information processed by the extractor component542) for further processing. For example, based on the categorization by the categorizer546, some performance information may no longer be needed for an analysis.

The formatter550can also modify performance information530, including by converting the performance information to a standard format, which can include removing some of the performance information or replacing some of the performance information, such as by abstracting certain details in the performance information. As an example, if performance information530involves SQL operations, identifiers for specific tables, columns, rows, fields, etc. can be replaced by a generic label (e.g., a specified row identifier is replaced with “ROW”). The formatter550can format performance information530for use by a classifier554. For example, if a classifier554uses word vectors as input, the formatter550can convert the performance information530to one or more suitable word vectors.

In at least some cases, a performance analyzer534can access, or include, a plurality of classifiers554, where a suitable classifier can be selected, for example, based on categorizations (or subcategorizations) applied by the categorizer546. In such cases, a formatter550can apply different operations depending on the classifier554to be used. For instance, different classifiers554can include different vocabularies, which can affect how a word vector is constructed. Or, different classifiers554may use different supplemental input (e.g., numerical values), including not using supplemental input. The classifiers554can also be based on different machine learning models.

In some cases, the use of multiple classifiers554can provide more accurate results, or simplify data preparation (e.g., by the formatter55). For instance, performance issues or bugs associated with the user interface514may not be relevant, or as relevant, to performance issues associated with the database522. In other cases, the computing environment500can include a single classifier554, where the classifier accounts for limited relevance between issues/solutions between different components of the production environment510(e.g., weightings of a hidden layer of a machine learning model will be lower for issues/solutions between different components than issues/solutions positively correlated with other particular components or subcomponents).

Example Sources of Performance Issues or Bugs

FIG. 6illustrates how performance issues or bugs can relate to application or system layer programs or program components, and how different types of performance information can be used to identify programs or components that may be associated with a performance issue or bug.FIG. 6can represent issues that may arise using products of SAP, SE, of Walldorf, Germany, including a database, such as the HANA database, an application server, such as an ABAP server, and a user interface component, such as the Fiori user interface.

Performance issues or bugs can be associated with an application layer610or a system layer614. Performance information for application layer610database components618can be provided by all or a portion of a Performance Trace, such as a SQL Trace (ST05). The information in the performance information for the database components618provided by the Performance Trace can include, for individual SQL statements, how many times the statement executed, how many records were selected or changed, and statement execution time. Performance information for the database components618can also be provided by the PlanViz (ST05) report, which can provide information regarding a SQL execution plan, including tables used, a number of table entries processed by each SQL operator in the plan, and number of records input and output.

The performance information for the database components618can be used to identify issues such as improper modelling used with Core Data Services (e.g., of the HANA database system of SAP SE of Walldorf, Germany) For example, the performance information for the database components618can be used to indicate that application level logic is not being pushed down to the database layer (for example, application level logic that would restrict the amount of data to be processed at the database level). Or, the performance information for the database components618may indicate a problem with the quality of the source data.

In one scenario, a problem with the quality of the source data can be that the amount of source data is insufficient to provide a level of detail requested, such as in a Core Data services view. Database views can be categorized by size (e.g., small, medium, large, etc.), where a size category may include a minimum number of rows in order to meet the category definition. For example, a size category can be provided as an annotation (e.g., metadata) for a Core Data Services view. Performance issues can arise when the quantity of data is not sufficient to meet criteria defined for a database view. In the event a performance issues is identified, such as by available data being a threshold amount for a size category, such as an average size for the category (the average of the minimum and maximum number of rows, or other data units, for the category), a user can device whether to proceed using the limited dataset available in the database, supplying additional data to satisfy size criteria, or redefining the view to a different category size.

The SQL Trace, particularly if executed using the “extended” option, can also be used to identify system layer614database components622. For example, the performance issues or bugs can be associated with issues in a secure development kit. System layer issues614in the database components622can also involve range restriction issues, such as whether SQL operations are executed against current information and at least a portion of historical data.

Performance information for application layer610application components628, such as with or using an ABAP server, can be identified using the ABAP Runtime Analysis (transaction SAT) of SAP SE of Walldorf, Germany. The performance information for the application components628includes metrics for particular program parts (e.g., particular classes, programs, or function groups) and can be further specified by particular code blocks (e.g., methods, events, function modules), type of code (e.g., flow logic or message handling), and whether, and what types, of database access operations should be analyzed. The performance information can include execution time for particular operations of particular programs or program components.

Similarly, the STAD transaction of SAP SE of Walldorf, Germany, can provide information regarding execution of particular transactions or program “jobs”, including statistical information regarding database processing involved with the transaction, the use of stored database procedures, RFC statistics, and information about the source of a job. The statistical information can include execution time, memory use, CPU use, number of database records processed, and amount of information returned to a source of a transaction or job. The performance information of the application components can be used to analyze issues related to data processing time and potential errors in how various logical data objects (e.g., instances of abstract data types) are processed.

At the system layer614, application components632can be investigated using performance information provided by the SM50 transaction and the ST02 transaction. The SM50 transaction can provide information about executed applications, including the status (running, aborted), error messages, cumulative CPU use, and elapsed running time. The ST02 transaction can provide information regarding buffer and memory configuration and use, including buffer use, number of buffered objects swapped to page area, data transferred from a database to a particular buffer, and amounts of memory allocated and used. The performance information for application components632can be used to identify problems arising from having too many background jobs, a high number of swaps between a buffer and a page area, and low buffer utilization.

Performance information, such as a HTTP trace, can be used to identify issues with user interface component638at the application layer610and user interface components642at the system layer614. For example, the components638can be associated with slow data rendering or slow graph rendering, while the components642can be associated with issues in a framework that provides data visualizations or issues with JavaScript code.

Relationships between performance information and program components, including whether the programs are at an application layer610or a system layer614can be used to categorize a performance issue or bug for further analysis. For example, performance information for a number of application layer610components, system layer614components, or a combination thereof can be obtained and analyzed. Metrics can be used to determine which components are likely involved in the performance issue or bug, such as parameters of performance issues where performance is at a value that violates a threshold. Depending on which components are implicated, certain performance information can be excluded from subsequent analysis, used to determine a classifier to be used, and used to determine how data should be formatted for use with the classifier.

Example Identification and Correction of SQL Issue

This Example 8 illustrates how disclosed technologies can be used to identify and resolve performance issues or bugs. For instance,FIG. 7illustrates an example SQL statement710. The SQL statement710is identified as having a performance issue, such as exceeding a set threshold execution time. In response to manually or automatically detecting that the threshold has been exceeded, performance information can be automatically or manually captured.

FIG. 8illustrates performance information800associated with query execution of the SQL statement710. In particular, the performance information800lists operators810associated with a JOIN engine used in the query. For each operator810, the performance information includes execution time814, CPU time818, tables processed822with the operator, a number of input rows828, and a number of output rows832.

In a particular aspect, the execution times814can be compared with a threshold. In this example, the operator836, JERequestedAttributes is identified as taking316milliseconds, which is determined as exceeding the threshold. Determining that the operator836exceeds the threshold can be used for a number of purposes. The performance issue or bug can be classified as involving the database, and the JOIN engine in particular. This classification can be used to select performance information (e.g., particular data or data sources) for further analysis, a classifier to be used, and how data should be formatted for use with the classifier. For instance, the performance information800can be used to identify the performance issue as involving the JOIN engine, and using a classifier for the JOIN engine.

The operator836is associated with operations720ofFIG. 1. The operations720can be initially be formatted for use with a classifier using the format (OPERATOR NAME, EXECUTION TIME, GROUP NUMBER, TABLES PROCESSED, MAX ROWS, FILTER PROCESSED). With the operations720, the initially formatted performance data can be expressed as formatted data730. The group number can be determined by segmenting input rows and output rows by groups based on row number. For instance, for a scenario with 300 input rows and 300 output rows, the input and output rows can be distributed in groups of 100. Input rows can be represented as X1(rows1-100), X2(rows101-200), and X3(rows201-300). Similarly, output rows can be represented as Y1(rows1-100), Y2(rows101-200), and Y3(rows201-300).

Groups can be formed by combining permutated selections (i.e., Cartesian product) of input and output rows, such as having Group 1 of X1,Y1, Group 2 of X1, Y2, Group 3 of X1, Y3, Group 4 of X2,Y1, etc. So, in the case of an operator associated with 298 input rows and 10 output rows, the operator would be assigned to Group 7 (X3,Y1). The maximum number of rows would be the largest of 298 and 10, or 298.

In particular cases, the groups of input and output rows can be used to identify performance issues. For example, a JOIN operation can be predicted to use a particular number of input rows and produce a particular number of output rows. If more input rows or output rows are present than would be predicted, a performance error can be detected, which can be associated, for example, with incorrect processing. Solutions can include filtering data, applying operators to data, or clustering data. Thus, grouping by input/output rows can provide insights as to the source of a performance issue, and potential solutions.

At least in some cases, at least a portion of the formatted data730can be replaced with abstracted information. For example, the particular tables processed by a statement may not been particularly informative as to the source of a performance issue or bug, and may obscure a comparison with similar performance issues that happen to involve different tables. Accordingly, formatted data740has table and field names replaced with generic identifiers (e.g., “Table”, “Field,” “FieldA”).

The formatted data740can be converted into a suitable word vector. For example, each element of the tuple can be converted into a word vector (e.g., a one-hot vector corresponding to the word), and the collection of vectors can be used as input for a classifier. In some cases, not every element of the formatted data740is converted to a word vector. For example, the EXECUTION TIME element can be left as a numerical (e.g., integer) value. Or the word vectors may be formed in another manner. For example, rather than one-hot word vectors, the word vectors can include an ordered sequence of attribute values according to a sequence of attributes defined for a “word”. For example, word vectors can be constructed using values for each attribute of (OPERATOR NAME, EXECUTION TIME, GROUP NUMBER, TABLES PROCESSED, MAX ROWS, FILTER PROCESSED). Thus, the formatted data itself can serve as a word vector.

Word vectors can be constructed differently, including using different attributes, depending on a particular component being analyzed, or a particular report providing performance information. For example, in analyzing application or user interface components, rather than “input rows” and “output rows,” other features can be used in word vectors. As an example, application logs (including, in products of SAP SE of Walldorf, Germany, the ST12 or SAT traces, or the example information ofFIG. 9) can provide attributes such as net and gross execution time. For a statement, “net time” can identify how much time was taken to execute the statement, and how many times the statement was executed. Statements of the report can include programmatic keywords (e.g., keywords in the ABAP programming language), such as LOOP.

As a more particular example, consider a statement including a LOOP keyword that executed 5000 times and took 5 seconds to complete. A cluster for the application, application log (i.e., particular performance information), the LOOP operator, or including the LOOP operator can be used to identify that this particular statement is causing a performance issue or bug. In some cases, a classifier can be used to return information (e.g., contexts words or a particular target word) that can be submitted to a solutions database to identify a potential target or solution for the problem (or, the output of the classifier can itself be a target). Or, the classifier can position the performance information within the cluster, and the position in the cluster can be used to identify a potential target or solution.

As another example, consider performance information indicating that a particular READ statement was called (and executed) by multiple program modules. The performance information submitted to the classifier can indicate which function modules are invoking the READ statement and how many time the READ statement was invoked. The performance information, such as in the form of a word vector (e.g., a word vector of particular attributes of the performance information) can be submitted to a classifier for a cluster for, or including, the READ statement or operation (e.g., for READ statements, generally). The classifier may return particular modules that are associated with a performance issue or bug, including a number of times the read statement is invoked by the function module. A solution can include features such as caching, for example in a temporary table, the values from a first READ statement, which is then used to provide results to subsequent invocations of the READ statement.

As can be seen, the attributes used in a word vector (or clustering technique) can be tailored to particular performance information and particular clusters. In additional examples, database performance can be measured using attributes such as operation (or call), number of times an operation was executed, execution time (net per operation or gross for an overall process, or gross for an operation executed multiple times), number of records selected, total duration, server identifier, length, table type, table name, source of an operation (e.g., program or function module), type (e.g., ABAP, SQL, system operation), indications (or a number) of identical operations (e.g., table READ operations), number of records accessed or returned, buffer type (or, buffered or not buffered), an object name (e.g., a table, pool, cluster, or view name), or SQL statement or portion thereof.

For user interface layer performance information, clusters can be associated with document object model, cascading style sheet, or JavaScript issues. The clusters can be further divided into subclusters, such as having cascading style sheet issues be further divided into clusters for rendering issues or property manipulation issues. Keywords of these user interface categories can be used to identify the clusters. Attributes can include operation type, program module, and execution time. In the example of application layer issues, subclusters can be based on programmatic keywords, such as keywords in the ABAP programming language. Attributes can include various attributes mentioned above.

In at least some cases, including when application and user interface performance issues or bugs are analyzed, the classifier step (or, at least a classifier step that includes word vectors) can be omitted. That is, the performance information can be analyzed and assigned to a cluster, and the cluster can be used to identify particular solutions or targets. As an example, clusters can be associated with user defined functions. If the performance information includes calls to a user defined function (where an end user supplies actual code to be executed when the function is called), then a cluster for user defined function can be indicated.

Clustering algorithms, such as implemented using a clustering engine, can be used to further position performance information within a cluster. For example, a clustering algorithm can consider one or more, typically a plurality, attributes of the performance information. The position of attributes of the performance information, including for particular statements or elements of the performance information, within the cluster can be used to identify particular targets or potential solutions in a manner than can be otherwise similar to when a classifier is used. A clustering engine can use techniques such as connectivity models, centroid models, distribution models, density models, subspace models, group models, or graph-based models. Or, a neural model, such as a trainer classifier, can be used (e.g., a machine learning based clustering technique), but the classifier may be based on a technique other than Continuous Bag of Words or-Skip Gram. In such cases, the input to the clustering engine can be a word vector or collection of attributes as discussed above, or can be another type of input. The clustering engine can apply hard clustering or soft clustering.

In particular cases, a collection of word vectors for a particular cluster, such as an operational cluster, can be submitted as context words in a Skip-Gram technique. For example, operators associated with a JOIN engine can be a cluster, while operators for a row engine, a column engine, and an OLAP engine can form other clusters. In further cases, clusters may be determined in another manner Typically, clusters are defined, or context words otherwise selected, based on an expected or actual functional relationship between operators.

In particular implementations of a Continuous Bag of Words technique, each operator can be used to form a word as described above. The word can be submitted to a classifier that can return context words that are in a similar format. The context words can provide insights into the source of a performance issue or bug. The context words can be cross referenced to a solutions database that may include potential solutions, or target areas, to address the performance issue or bug.

The classifier can return an output that identifies one or more performance indicators, or other output (according to the training of the classifier) that can be associated with a performance issue or bug. In this case, the output can be used to determine that the CASE statement in the SQL statement710is resulting in the filter push down (e.g., of the operations to the database layer) not occurring.FIG. 7illustrates the identified solution750. The solution750is a suggested, generic correction of the FILTER PROCESSED element of the tuple of formatted data740. A developer can implement the solution to provide the modified SQL statement760.

The developer can identify the solution750as correct, which can result in additional training of the classifier, and can be used to update the ranking of the solution in the database.

Example Application Performance Information

FIG. 9illustrates performance information900that can be obtained from an application trace. The performance information900includes particular program operations (e.g., method calls, database operations)910, net execution time914, and an identifier918of a program associated with a program operation. The performance information900can be analyzed to determine program operations exceeding a threshold net execution time914. Any program operations exceeding the threshold can then be formatted and submitted to a classifier, including generally as described for the performance information710.

Example Operations for Use in Classifying Performance Information or Sources of Performance Issues or Bugs

FIG. 10illustrates a table1000of various logical operators1010and their corresponding physical operators1016, an identifier1020of a database engine that applies the operator1010, and identifiers1024of any related plan operators (POPs). The logical operators1010, physical operators1016, engine identifiers1020, and POP identifiers1024can be used to subcategorize performance information prior to submitting performance information for training a machine learning model or to a classifier for analysis. For example, different classifiers may be used for different database engines. The logical operators1010, physical operators1016, engine identifiers1020, and POP identifiers1024can also be used as labels or tags for potential targets (e.g., classifier output), such as to identify whether a performance issue or bug is associated with a particular physical operator or POP identifier.

Example Vocabulary and Word Vectors

FIG. 11illustrates how one-hot word vectors1110,1112,1114can be formed from a vocabulary1120. The illustrated vocabulary1120has 8 “words,” each of which can be indicated by an index position1122in an array. So, for example, in the vocabulary1120, the word “JOIN” is represented by index 0, “SQL” by index 1, etc. The word “JOIN” can then be specified using the vector1110, where index 0, corresponding to the array index associated with that word, is set to “1” and all other values are set to “0.” Similarly, the “word” “CASE WHEN,” having an index of 4 in the vocabulary1120, can be represented by the vector1112, where index 4 has a value of “1” and all other values are set to “0.” At least in some cases, some positions of the vocabulary are not associated with a word, but with another value. For example, index position 3 of the vocabulary1120can represent execution time, where a numerical value is entered when the “execution time” is to be specified. For instance, vector1114includes a value of “366” indicating, for example, a 366 millisecond execution, in index position 3, with other values in the array set to “0.”

Example Word Vectors for Continuous Bag of Words Technique

FIG. 12illustrates how a plurality of one-hot word vectors, such as described with respect toFIG. 11, can be used in the Continuous Bag of Words machine learning model. The input to a hidden layer1210of a machine learning model (e.g., using the Continuous Bag of Words technique) is a N×V matrix1214, where N is the number of context words and V is the number of words in the vocabulary. The matrix1214is formed from N context words, represented by one-hot word vectors1218, each having V elements.

The matrix1214can be averaged by the number of context words and applied to the hidden layer1210. The hidden layer produces a single output word vector1220.

According to an aspect of the present disclosure, the one-hot word vectors1218provide the context for an error indicated by the output word vector. AlthoughFIG. 12illustrates a plurality of context word vectors1218, in some aspects a single context word vector can be used. When a machine learning model is being trained, error between a predicted result (e.g., output word vector1220) and the correct result can be backpropagated to adjust weights in the hidden layer1210and improve the accuracy of the model.

Example Word Vector for Skip Gram Technique

FIG. 13illustrates how a plurality of one-hot word vectors, such as described with respect toFIG. 11, can be used in the Skip-Gram model. The input to a hidden layer1310of a machine learning model (e.g., using the Skip-Gram technique) is an array1314having V elements, where V is the number of words in a vocabulary. The array1314is applied to the hidden layer to provide a matrix1318corresponding to C context words, which can be represented as C word vectors1320, where C is a number of context words specified during training.

According to an aspect of the present disclosure, the context vectors1320provide the context for an error indicated by the output word vector. AlthoughFIG. 13illustrates a plurality of context word vectors1320, in some aspects a single context word vector can be used. When a machine learning model is being trained, error between a predicted result (e.g., context vectors1320) and the correct result can be backpropagated to adjust weights in the hidden layer1310and improve the accuracy of the model.

Example Methods of Training and Using Classifier

FIG. 14Ais a flowchart of an example method1400of obtaining classification results, such as in the form of a performance indicator that may be associated with a solution or target for correcting a performance issue or bug. The method1400can be carried out using the computing environment100ofFIG. 1or the computing environment500ofFIG. 5. At1404, performance information is received. The performance information is associated with the performance issue or bug and with software execution during the occurrence of the performance issue or bug. The performance information includes at least one performance parameter (e.g., execution time or resource use) and at least one software feature (e.g., program, program component, program operation or process, etc.) associated with the at least one performance parameter.

At least a portion of the performance information, including the at least one software feature, is formatted at1408for submission to a classifier using a machine learning model. At1412, the formatted performance information is provided to the classifier. A classification result is received at1416. At1420, it is determined if a solution is associated with the classification result, such as by querying a database. Based on the determining, at1424, at least one identified solution, or an indication that a solution was not identified, is returned.

FIG. 14Bis a flowchart of an example method1428of automatically analyzing performance information. The method1428can be carried out using the computing environment100ofFIG. 1or the computing environment500ofFIG. 5.

At1432, at least one software performance metric is monitored. It is determined at1436whether at least one metric violates a threshold. Performance information, including at least one performance parameter, is caused to be gathered at1440. At1444, a type associated with the performance information is determined. An extraction rule associated with the type is received at1448. At1452, at least one performance parameter is extracted from the performance information using the extraction rule. Performance information that includes the performance parameter is formatted at1456. The formatted performance information is provided to a classifier or a clustering engine at1460. In some cases, the classifier can carry out clustering, or the clustering engine can use machine learning techniques. At1464, a classification or clustering result is received. A solutions database is queried for a solution using the classification or clustering result at1468. At1472, at least on identified solution or an indication that a solution was not identified is returned.

FIG. 14Cis a flowchart of an example method1480of training a classifier based on a machine learning algorithm. The method1480can be carried out using the computing environment100ofFIG. 1or the computing environment500ofFIG. 5.

At1484, a plurality of training data objects are received. The plurality of training data objects include performance information associated with software associated with a performance issue or bug and with software execution during the occurrence of the performance issue or bug. The performance information includes at least one performance parameter and at least one software feature associated with the at least one performance parameter. The training data objects include at least one performance indicator, which is associated with a solution to the performance issue or bug. The training data objects further include a solution to the performance issue or bug.

For the plurality of training data objects, at least a portion of the performance information is formatted at1488for submission to a machine learning algorithm. At1492, a machine learning model is trained with the formatted performance information to provide a trained classifier. The trained classifier outputs a performance indicator in response to an input test data object. The input test data object includes performance information. The output performance indicator can be submitted to a solutions database to provide a suggested solution or an indication that a solution was not identified.

Computing Systems

FIG. 15depicts a generalized example of a suitable computing system1500in which the described innovations may be implemented. The computing system1500is not intended to suggest any limitation as to scope of use or functionality of the present disclosure, as the innovations may be implemented in diverse general-purpose or special-purpose computing systems.

With reference toFIG. 15, the computing system1500includes one or more processing units1510,1515and memory1520,1525. InFIG. 15, this basic configuration1530is included within a dashed line. The processing units1510,1515execute computer-executable instructions, such as for implementing components of the computing environment100ofFIG. 1or the computing environment500ofFIG. 5. A processing unit can be a general-purpose central processing unit (CPU), processor in an application-specific integrated circuit (ASIC), or any other type of processor. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power. For example,FIG. 15shows a central processing unit1510as well as a graphics processing unit or co-processing unit1515. The tangible memory1520,1525may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two, accessible by the processing unit(s)1510,1515. The memory1520,1525stores software1580implementing one or more innovations described herein, in the form of computer-executable instructions suitable for execution by the processing unit(s)1510,1515.

A computing system1500may have additional features. For example, the computing system1500includes storage1540, one or more input devices1550, one or more output devices1560, and one or more communication connections1570. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing system1500. Typically, operating system software (not shown) provides an operating environment for other software executing in the computing system1500, and coordinates activities of the components of the computing system1500.

The tangible storage1540may be removable or non-removable, and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium which can be used to store information in a non-transitory way and which can be accessed within the computing system1500. The storage1540stores instructions for the software1580implementing one or more innovations described herein.

The input device(s)1550may be a touch input device such as a keyboard, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to the computing system1500. The output device(s)1560may be a display, printer, speaker, CD-writer, or another device that provides output from the computing system1500.

In various examples described herein, a module (e.g., component or engine) can be “coded” to perform certain operations or provide certain functionality, indicating that computer-executable instructions for the module can be executed to perform such operations, cause such operations to be performed, or to otherwise provide such functionality. Although functionality described with respect to a software component, module, or engine can be carried out as a discrete software unit (e.g., program, function, class method), it need not be implemented as a discrete unit. That is, the functionality can be incorporated into a larger or more general purpose program, such as one or more lines of code in a larger or general purpose program.

Cloud Computing Environment

FIG. 16depicts an example cloud computing environment1600in which the described technologies can be implemented. The cloud computing environment1600comprises cloud computing services1610. The cloud computing services1610can comprise various types of cloud computing resources, such as computer servers, data storage repositories, networking resources, etc. The cloud computing services1610can be centrally located (e.g., provided by a data center of a business or organization) or distributed (e.g., provided by various computing resources located at different locations, such as different data centers and/or located in different cities or countries). The cloud computing services1610are utilized by various types of computing devices (e.g., client computing devices), such as computing devices1620,1622, and1624. For example, the computing devices (e.g.,1620,1622, and1624) can be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablet computers or smart phones), or other types of computing devices. For example, the computing devices (e.g.,1620,1622, and1624) can utilize the cloud computing services1610to perform computing operators (e.g., data processing, data storage, and the like).

Implementations

For clarity, only certain selected aspects of the software-based implementations are described. Other details that are well known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any specific computer language or program. For instance, the disclosed technology can be implemented by software written in C, C++, C#, Java, Perl, JavaScript, Python, Ruby, ABAP, SQL, XCode, GO, Adobe Flash, or any other suitable programming language, or, in some examples, markup languages such as html or XML, or combinations of suitable programming languages and markup languages. Likewise, the disclosed technology is not limited to any particular computer or type of hardware. Certain details of suitable computers and hardware are well known and need not be set forth in detail in this disclosure.