Patent Description:
Symptoms of software regression may be apparent to users and software developers. However, determining what causes of software regression may be difficult. Operating systems, complex software applications, and other types of software products are typically developed by a large number of development teams organized into a hierarchy. Each team may submit or "check in" one or more changes and/or configuration modifications (collectively referred to herein as "payloads") to source code of a software application at a corresponding level in the hierarchy daily, weekly, bi-weekly, monthly, or in other time intervals. Subsequently, the payloads checked in can be propagated to higher levels in the hierarchy which may also include payloads from other development teams. Periodically, a version of the source code of the software product with various payloads can be compiled into executable instructions as a distinct "build" of the software product. One or more computing devices (e.g., servers in a testing lab or client devices of users signed up for testing) can then execute the build of the software product and collect various performance metrics for analysis. Examples of performance metrics can include power consumption, processor load, memory consumption, execution latency, and/or other suitable types of metrics. The collected metrics can be transmitted to a regression detector, for instance, hosted on a server for regression analysis.

Ideally, if each build only includes one change to the source code, then an impact of the payload to the performance of the software product can be easily detected. However, compiling source code with only one payload is neither efficient nor feasible due to complexity of today's software products. In practice, each build of a software product typically includes a large number of payloads. For instance, individual builds of an operating system (e.g., iOS® provide by Apple, Inc. of Mountain View, California) can include hundreds or even thousands of payloads. As such, determining impact of one of these payloads to the measured performance metrics of the software product can be rather difficult. In addition, as time goes on, payloads from lower levels can be propagated into higher levels of the hierarchy with additional payloads from other development teams. Thus, the additional payloads can mask impact of the payloads from the lower levels and render software regression detection difficult.

Several embodiments of the disclosed technology are directed to techniques for detecting software regression by analyzing performance metrics of various builds of a software product denominated by various payloads included in each build. In certain embodiments, a regression detector can be configured to receive data representing measured performance metrics of multiple builds of a software product and a list of payloads included in each of the builds. In certain implementations, each payload can be assigned a unique identification number or other suitable forms of identification and tracked for the multiple builds of the software product. For example, the regression detector can be configured to receive the following partial example dataset listed in the table below:.

In the table above, four builds (i.e., Build No. <NUM>-<NUM>) have been compiled and executed to collect corresponding metric values (i.e., "<NUM>," "<NUM>," "<NUM>," and "<NUM>"). Each of the four builds can include a combination of various payloads with a corresponding identifier (i.e., "Payload A," "Payload B," and "Payload C"). Presence of a payload in a build is indicated with "<NUM>" while absence of a payload in a build is indicated with "<NUM>. " In other examples, the identification and/or the presence/absence of the payloads can be represented with other values or in other suitable manners.

The regression detector can be configured to perform statistical analysis of the received dataset using the identification and presence/absence of each of the payloads as a denominator. For instance, the regression detector can be configured to apply multiple linear regression to the received dataset to generate a set of regression coefficients for the respective payloads. In a dataset with one dependent variable Yi (e.g., the "Metric") and multiple independent variables Xip where p corresponds to, e.g., different payloads (e.g., "Payload A," "Payload B," and "Payload C"), a linear relationship between the dependent variable and the independent variables can be modeled using a disturbance term or error variable ε, as follows: <MAT> where i = <NUM>, <NUM>,. , n
The coefficients (i.e., β<NUM>, β<NUM>,. , βp) can be estimated using various techniques, such as least squares estimation, maximum likelihood estimation, ridge regression, least absolute deviation, etc. As such, by estimating the coefficients using a suitable technique, a multiple linear model of the received dataset can be obtained.

In certain embodiments of the disclosed technology, using the obtained multiple linear model, the regression detector can be configured to determine which one or more of the payloads have statistically significant impact (i.e., above a noise level) to the performance metrics in the received dataset. For instance, example coefficients for the foregoing payloads may be as follows:.

In the above example, the estimated coefficients for Payload A and Payload B are -<NUM> and <NUM>, respectively. The estimated coefficient for Payload C is <NUM> while an intercept (β<NUM>) for the model is <NUM>. As such, the multiple linear model for the received dataset can be expressed as follows: <MAT> where i = <NUM>, <NUM>,.

In certain embodiments, the regression detector can be configured to detect, identify, or select one or more payloads as impacting the performance metrics of the software product (referred to herein as a "significant payloads") from the multiple linear model based on a preset threshold. For instance, in the example above, the regression detector can be configured to select "Payload C" as a significant payload based on a threshold of ±<NUM>. In certain implementations, the threshold can be set corresponding to a noise level in the received dataset. In other implementations, the threshold can be set in other suitable manners with other suitable values.

In further embodiments, the regression detector can be configured to force one or more of the estimated coefficients (i.e., β<NUM>,. , βp) corresponding to the payloads to be zero. As such, only a subset of the payloads is included in the multiple linear model. For instance, the regression detector can be configured to apply the Least Absolute Shrinkage and Selection Operator ("LASSO") when estimating the coefficients corresponding to the payloads in the multiple linear model. According to LASSO, a sum of absolute values of the coefficients can be forced to be less than a fixed threshold during estimation. Such limitation on the sum of the absolute values of coefficients can force certain one(s) of the coefficients to be set to zero, for instance, those corresponding to "Payload A" and "Payload B" in the above example. As such, a simpler model based on only "Payload C" can be obtained, as follows: <MAT> where i = <NUM>, <NUM>,. , n
In further examples, the limitation on the sum can be adjusted such that a number of the multiple payloads is less than another threshold (e.g., five, ten, fifteen, etc.).

Upon detecting one or more significant payloads, the regression detector can be configured to perform one or more remedial actions. In one example, the regression detector can be configured to generate and transmit a message to a development team associated with submitting the significant payload to indicate a presence of a bug or other types of software defect in the significant payload. In another example, the regression detector can also be configured to supply a build of the software product that does not include the significant payload for deployment in a computing system. In yet another example, the regression detector can be configured to automatically cause the significant payload to be reversed in source code of the software product, by, for instance, compiling another build without the significant payload. In further examples, the regression detector can be configured to notify users having builds that include the significant payload, create a patch for reversing the significant payload, and/or perform other suitable remedial actions.

Several embodiments of the disclosed technology can thus "unscramble" contributions of various payloads in multiple builds to performance metrics of the software product. Without being bound by theory, it is believed that a significant payload can propagate from lower levels in the hierarchy to higher levels. As the significant payload propagates, a corresponding contribution to the performance metrics is detected and represented by the measured performance metrics. As such, by using each payload as a denominator, analysis of the performance metrics can reveal a coefficient of the payload to the measured performance metrics in various builds. Based on such a coefficient, the regression detector can thus detect which one or more of the payloads have statistically significant impact on the measured performance metrics.

Certain embodiments of systems, devices, components, modules, routines, data structures, and processes for detecting software regression in computing systems are described below. In the following description, specific details of components are included to provide a thorough understanding of certain embodiments of the disclosed technology. A person skilled in the relevant art will also understand that the technology can have additional embodiments. The technology can also be practiced without several of the details of the embodiments described below with reference to <FIG>.

As used herein, the term "software regression" or "software performance regression" generally refers to certain negative effects in performance related to one or more features of a software product after applying software updates, patches, or other types of changes to the software product. For example, a software product executing on a computing device can receive an update from version <NUM> to version <NUM>. However, after installing version <NUM>, the computing device can experience a significantly increase in power consumption under similar conditions as prior to applying the update. In other examples, the applied update may render certain features of the software product to malfunction, may render the computing device to consume more memory or other types of computing resources, or even crash the computing device. Also used herein, a "software product" can refer to a standalone software application (e.g., a word processor, a spreadsheet application, a web browser, etc.), an operating system (e.g., iOS provided by Apple, Inc. of Mountainview California), a component of a standalone application or operating system (e.g., a device driver, an add-in module to a web browser, etc.), a software platform (e.g., Amazon Web Services® provided by Amazon, Inc. of Seattle, Washington), or other suitable types of software programs.

Also used herein a "payload" generally refers to any one or more of a change to source code of a software application, an enablement of a feature of the software application, or a configuration change to execution of the software product. Operating systems and other complex software products are typically developed by a large number of developers organized into development teams in a hierarchy. Each development team may submit or "check in" one or more payloads to a software product. The checked in payloads can then be propagated to higher levels in the hierarchy in the development cycle. In accordance with embodiments of the disclosed technology, each checked in payload can be tracked using a unique identifier, such as an alphanumerical value or other suitable identifications.

Periodically, a version of the source code of the software product with various payloads from one or more levels in the hierarchy can be compiled into executable instructions as a distinct "build" of the software product associated with a distinct build identifier. Due to different date/time of checking in payloads by different development teams and the propagation of the checked in payload to higher and/or lower levels of the hierarchy, different builds of the software product can each include a unique combination of payloads from the various development teams. For instance, a first build can include payloads A, B, and C; a second build can include payloads A, C, and D; and a third build can include payloads B, C, and D.

The various builds can then be tested by, for instance, being provided as a beta version of the software product to users signed up for testing. One or more computing devices (e.g., servers in a testing lab or client devices of the users) can then execute the various builds of the software product and values of various "performance metrics" can be collected for regression analysis. Examples of performance metrics can include power consumption, processor load, memory consumption, execution latency, and/or other suitable types of metrics. The collected values of the performance metrics can include or otherwise being associated with a build identifier, identifications of payloads included in a build corresponding to the build identifier, or other suitable information.

Also used herein "multiple variable model" generally refers to a mathematical model in which values of a dependent variable depend on two, three, four, or any suitable numbers of multiple dependent variables. One example multiple variable model is a "multiple linear model" in which values of a dependent variable is linearly proportional to values of multiple dependent variables as follows: <MAT> where i = <NUM>, <NUM>,. , n
The coefficients (i.e., β<NUM>, β<NUM>,. , βp) can be estimated using various techniques, such as least squares estimation, maximum likelihood estimation, ridge regression, least absolute deviation, etc. Other examples of multiple variable model can also include parabolic, sinusoidal, or other suitable non-linear models.

Symptoms of software regression may be apparent to users and software developers but determining what causes of software regression may be difficult due to multiplicity of payloads included in each build of a software product. Ideally, if each build only includes one payload, then an impact of the payload to the performance of the software product can be easily detected. However, compiling source code with only one payload is neither efficient nor feasible due to complexity of today's software products. In practice, each build of a software product typically includes hundreds or even thousands of payloads. As such, determining impact of one of these payloads to the measured performance metrics of the software product can be rather difficult. In addition, as time goes on, payloads from lower levels can be propagated into higher levels of the hierarchy with additional payloads from other development teams. Thus, the additional payloads can mask impact of the payloads from the lower levels and render software regression detection difficult.

Several embodiments of the disclosed technology are directed to techniques for detecting software regression by analyzing performance metrics of various builds of a software product based on payloads included in the builds. In certain implementations, a dataset can be compiled to include multiple entries each containing data representing an identification of multiple payloads included in a build of a software product executed at a computing device and a corresponding value of a performance metric of executing the build at the computing device. Upon accessing the dataset, at the computing system, a set of coefficients individually corresponding to one of the multiple payloads can be estimated using a multiple variable model with the performance metric as a dependent variable and the multiple payloads as independent variables. Based on the multiple variable model, one of the multiple payloads with a corresponding estimated coefficient whose absolute value is greater than a preset threshold can be identified as impacting the performance metrics of the software product. As such, by using each payload as a denominator, analysis of the performance metrics can reveal a coefficient of the payload to the measured performance metrics in various builds, as described in more detail below with reference to <FIG>.

<FIG> and <FIG> are schematic diagrams illustrating certain operational stages of detecting software regression in a computing environment <NUM> in accordance with embodiments of the disclosed technology. As shown in <FIG>, the computing environment <NUM> can include client devices <NUM> corresponding to users <NUM> and development teams <NUM>, an application server <NUM>, and a regression server <NUM> communicatively coupled to one another via a computer network <NUM>. Though particular components of the computing environment <NUM> is shown in <FIG>, in other implementations, the computing environment <NUM> can also include web servers, database servers, and/or other suitable components in addition to or in lieu of those shown in <FIG>.

The client devices <NUM> can each include a computing device that facilitates the users <NUM> and the development teams <NUM> to communicate with the application server <NUM> and/or the regression server <NUM> via the computer network <NUM>. In the illustrated embodiment, the client devices <NUM> each includes a desktop computer. In other embodiments, the client devices <NUM> can also include servers, laptop computers, tablet computers, smartphones, or other suitable computing devices. Though three users <NUM> and three development teams <NUM> are shown in <FIG> for illustration purposes, in other embodiments, the computing environment <NUM> can facilitate communications of other suitable numbers of users <NUM> and development teams <NUM>.

The computer network <NUM> can include any suitable types of network. For example, in one embodiment, the computer network <NUM> can include an Ethernet or Fast Ethernet network having routers, switches, load balancers, firewalls, and/or other suitable network components implementing TCP/IP or other suitable types of protocol. In other embodiments, the computer network <NUM> can also include an InfiniBand network with corresponding network components. In further embodiments, the computer network <NUM> can also include a combination of the foregoing and/or other suitable types of computer networks.

The application server <NUM> can be configured to receive various payloads <NUM> (shown as first, second, and third payloads 103a-103c, respectively) to a software product submitted by the corresponding development teams <NUM> daily, weekly, monthly, or in other suitable time intervals. Periodically, the application server <NUM> can be configured to compile a current version of software product having various payloads, such as the first, second, and third payloads 103a-103c, into a distinct build <NUM>. As such, as time goes on, the application server <NUM> can compile and generate various builds <NUM> of the software product each having different combinations of the payloads <NUM>. Though the application server <NUM> is shown as a single entity in <FIG>, in other implementations, the application server <NUM> can include multiple servers individually configured for perform one or more of receiving the payloads <NUM>, compile the builds <NUM>, providing the builds <NUM> to the users <NUM>, or other suitable functions.

Upon generating the builds <NUM> of the software product, the application server <NUM> can also be configured to provide the various builds <NUM> to the users <NUM> for testing, for instance, as beta versions of the software product. While executing the builds <NUM>, the client devices <NUM> of the users <NUM> can generate and collect performance metrics <NUM> (<FIG>) from the software product and/or the client devices <NUM>. As shown in <FIG>, the client devices <NUM> can then transmit the collected performance metrics <NUM> to the regression server <NUM> for software regression analysis.

The regression server <NUM> can include a regression detector <NUM> (shown in <FIG>) that is configured to analyze the received data of performance metrics <NUM> from the client device <NUM> of the users <NUM> and identify one or more payloads <NUM> (<FIG>) from those included in the various builds <NUM> (<FIG>) that likely have caused software regression in the software product. As shown in <FIG>, upon receiving the performance metrics <NUM> from the client devices <NUM>, the regression server <NUM> can be configured to assemble the received performance metrics <NUM> into a dataset <NUM> having identification of multiple payloads <NUM> included in a build <NUM> of the software product executed at the computing devices <NUM> and a corresponding value of a performance metric <NUM> of executing the build <NUM> at the computing devices <NUM>. The regression server <NUM> can then access the assembled dataset <NUM> and identify the one or more payloads <NUM> that impact the performance metrics <NUM> by applying a multiple variable model to the dataset <NUM>. Example components of the regression detector <NUM> are described in more detail below with reference to <FIG>.

Upon detecting that one or more payloads <NUM> impact the performance metrics <NUM> of the software product, the regression server <NUM> can be configured to perform one or more remedial actions. In one example as shown in <FIG>, the regression server <NUM> can be configured to generate and transmit a message (shown as "Bug Alert <NUM>") to a development team <NUM> (e.g., the second development team 103b) associated with submitting the payload <NUM> to indicate a likely presence or probability of a bug or other types of software defect in the payload <NUM>. In another example, the regression server <NUM> can also be configured to instruct the application server <NUM> (<FIG>) to only supply builds <NUM> of the software product without the identified payload <NUM> for deployment in a computing system. In yet another example, the regression server <NUM> can be configured to automatically causing the identified payload <NUM> be reversed in source code of the software product, by, for instance, generating another build <NUM> by compiling the source code of the software product without the identified payload <NUM>. In further examples, the regression detector can be configured to notify users having builds <NUM> that include the identified payload <NUM>, create a patch for reversing the identified payload <NUM>, and/or perform other suitable remedial actions.

Several embodiments of the disclosed technology can thus "unscramble" contributions of various payloads <NUM> in multiple builds <NUM> to performance metrics <NUM> of the software product. Without being bound by theory, it is believed that a payload <NUM> can propagate from lower levels in a hierarchy of development teams to higher levels. As the payload propagates, a corresponding contribution to the performance metrics <NUM> can be detected and represented by the measured performance metrics <NUM>. As such, by using payloads <NUM> as denominators, analysis of the performance metrics <NUM> can reveal a coefficient of the payloads <NUM> to the measured performance metrics <NUM> in various builds <NUM>. Based on such a coefficient, the regression server <NUM> can thus detect which one or more of the payloads <NUM> have statistically significant impact on the measured performance metrics <NUM>. As a result, troubleshooting the software product can be more focused and efficient when compared to analyzing the performance metrics <NUM> as a whole.

<FIG> is a schematic diagram illustrating example components of a regression detector <NUM> executing on a regression server <NUM> in accordance with embodiments of the disclosed technology. In <FIG> and in other Figures herein, individual software components, objects, classes, modules, and routines may be a computer program, procedure, or process written as source code in C, C++, C#, Java, and/or other suitable programming languages. A component may include, without limitation, one or more modules, objects, classes, routines, properties, processes, threads, executables, libraries, or other components. Components may be in source or binary form. Components may include aspects of source code before compilation (e.g., classes, properties, procedures, routines), compiled binary units (e.g., libraries, executables), or artifacts instantiated and used at runtime (e.g., objects, processes, threads).

Components within a system may take different forms within the system. As one example, a system comprising a first component, a second component and a third component can, without limitation, encompass a system that has the first component being a property in source code, the second component being a binary compiled library, and the third component being a thread created at runtime. The computer program, procedure, or process may be compiled into object, intermediate, or machine code and presented for execution by one or more processors of a personal computer, a network server, a laptop computer, a smartphone, and/or other suitable computing devices.

Equally, components may include hardware circuitry. A person of ordinary skill in the art would recognize that hardware may be considered fossilized software, and software may be considered liquefied hardware. As just one example, software instructions in a component may be burned to a Programmable Logic Array circuit or may be designed as a hardware circuit with appropriate integrated circuits. Equally, hardware may be emulated by software. Various implementations of source, intermediate, and/or object code and associated data may be stored in a computer memory that includes read-only memory, random-access memory, magnetic disk storage media, optical storage media, flash memory devices, and/or other suitable computer readable storage media excluding propagated signals.

As shown in <FIG>, the regression server <NUM> can include a processor <NUM> operatively coupled to the datastore <NUM>. The processor <NUM> can be configured to execute suitable instructions to provide the regression detector <NUM>. In the illustrated embodiment, the regression detector <NUM> can include an input/output component <NUM>, an analysis component <NUM>, and a control component <NUM>. Though particular components of the regression detector <NUM> are shown in <FIG>, in other embodiments, the regression detector <NUM> can also include command, database, or other suitable types of components in addition to or in lieu of those shown in <FIG>.

The input/output component <NUM> can be configured to receive the performance metrics <NUM> from the client devices <NUM> and assemble the received performance metrics <NUM> into a dataset <NUM>. For example, a partial example dataset <NUM> can be as follows:.

As shown above, four builds (i.e., Build No. <NUM>-<NUM>) have been compiled and executed to collect corresponding metric values (i.e., "<NUM>," "<NUM>," "<NUM>," and "<NUM>"). Each of the four builds can include a combination of various payloads with a corresponding identifier (i.e., "Payload A," "Payload B," and "Payload C"). Presence of a payload in a build is indicated with "<NUM>" while absence of a payload in a build is indicated with "<NUM>. " In other examples, the identification and/or the presence/absence of the payloads can be represented with other values or in other suitable manners. The input/output component <NUM> can also be configured to receive additional performance metrics <NUM> from the client devices <NUM> or other suitable data sources and update the dataset <NUM> accordingly.

The analysis component <NUM> can be configured to perform statistical analysis of the dataset <NUM> using the identification and presence/absence of each of the payloads <NUM> (<FIG>) as a denominator. In certain embodiments, the statistical analysis can include fitting data of the dataset <NUM> into a multiple variable model. For instance, the regression detector can be configured to apply multiple linear regression to the received dataset to generate a set of regression coefficients for the respective payloads. In a dataset <NUM> with one dependent variable Yi (e.g., the "Metric") and multiple independent variables Xip where p corresponds to, e.g., different payloads <NUM> (e.g., "Payload A," "Payload B," and "Payload C"), a linear relationship between the dependent variable and the independent variables can be modeled using a disturbance term or error variable ε, as follows: <MAT> where i = <NUM>, <NUM>,. , n
The coefficients (i.e., β<NUM>, β<NUM>,. , βp) can be estimated using various techniques, such as least squares estimation, maximum likelihood estimation, ridge regression, least absolute deviation, etc. As such, by estimating the coefficients using a suitable technique, a multiple linear model of the dataset <NUM> can be obtained.

In certain embodiments of the disclosed technology, the analysis component can be configured to determine which one or more of the payloads <NUM> have statistically significant impact (i.e., above a noise level) to the performance metrics <NUM> in the dataset <NUM> using the obtained multiple linear model. For instance, example coefficients for the foregoing payloads <NUM> may be as follows:.

The analysis component <NUM> can then be configured to detect, identify, or select one or more payloads <NUM> as impacting the performance metrics <NUM> of the software product from the multiple linear model based on a preset threshold. For instance, in the example above, the analysis component <NUM> can be configured to select "Payload C" as a significant payload based on a threshold of ±<NUM>. In certain implementations, the threshold can be set corresponding to a noise level in the received dataset <NUM>. The noise level can be calculated, for instance, by the input/output component <NUM> or the analysis component <NUM>, as a standard deviation, an entropy, or other suitable indication of noise levels. In other implementations, the threshold can be set in other suitable manners with other suitable values.

In further embodiments, the analysis component <NUM> can be configured to force one or more of the estimated coefficients (i.e., β1,. , βp) corresponding to the payloads to be zero. As such, only a subset of the payloads is included in the multiple linear model. For instance, the analysis component <NUM> can be configured to apply the Least Absolute Shrinkage and Selection Operator ("LASSO") when estimating the coefficients corresponding to the payloads <NUM> in the multiple linear model. According to LASSO, a sum of absolute values of the coefficients can be forced to be less than a fixed threshold during estimation. Such limitation on the sum of the absolute values of coefficients can force certain one(s) of the coefficients to be set to zero, for instance, those corresponding to "Payload A" and "Payload B" in the above example. As such, a simpler model based on only "Payload C" can be obtained, as follows: <MAT> where i = <NUM>, <NUM>,. , n
In further examples, the limitation on the sum can be adjusted such that a number of the multiple payloads is less than another threshold (e.g., five, ten, fifteen, etc.).

Upon detecting one or more significant payloads, the analysis component <NUM> can be configured to instruct the control component <NUM> to perform one or more remedial actions. In one example, the control component <NUM> can be configured to generate and transmit a message (shown as "Bug Alert <NUM>") to a development team <NUM> associated with submitting the identified payload <NUM> to indicate a likely presence or probability of a bug or other types of software defect in the payload <NUM>. In another example, the control component 126r can also be configured to instruct the application server <NUM> (<FIG>) to supply a build of the software product that does not include the identified payload <NUM> for future deployment in a computing system. In yet another example, the control component <NUM> can be configured to automatically causing the identified payload to be reversed in source code of the software product, by, for instance, compiling another build <NUM> without the identified payload <NUM>. In further examples, the control component <NUM> can be configured to notify users <NUM> (<FIG>) having builds <NUM> that include the identified payload <NUM>, create a patch for reversing the identified payload <NUM>, and/or perform other suitable remedial actions.

<FIG> is a schematic diagram illustrating an example data schema <NUM> for a performance metric dataset <NUM> in <FIG> in accordance with embodiments of the disclosed technology. As shown in <FIG>, the data schema <NUM> can include a record ID field <NUM>, a Build ID field <NUM>, one or more payload fields <NUM> (shown as "Payload <NUM><NUM>," "Payload <NUM><NUM>," and "Payload n <NUM>"), and a performance metric field <NUM>. The record ID field <NUM> can be configured to contain data representing an identification (e.g., an integer, an alphanumeric string, etc.) associated with an entry in the dataset <NUM>. The Build ID field can be configured to contain data representing a distinct identifier, such as a build number, associated with the performance metric <NUM>. The payload fields <NUM> can be configured to individually contain a distinct identifier, such as an alphanumeric string, of a payload included in the build corresponding to the value in the Build ID filed. The performance metric field <NUM> can be configured to contain data representing a numerical value and an associated unit of measurement corresponding to a performance metric collected during execution of the build corresponding to the value in the Build ID filed.

<FIG> are example plots of measured performance metrics <NUM> of executing various builds <NUM> of a software product in accordance with embodiments of the disclosed technology. As discussed above with reference to <FIG>, various builds <NUM> (<FIG>) can be executed during different days and corresponding performance metric <NUM> can be collected and plotted as shown in <FIG>. Each of the builds <NUM> can include multiple payloads <NUM>, and effects of differences between a pair of consecutive builds may be within a noise level. As such, though the performance metric <NUM> shows a clear upward trend (represented by the dashed line) in <FIG>, determining which one(s) of the payload <NUM> that cause the upward trend on the performance metric <NUM> may be difficult. As shown in <FIG>, by applying the technique described above with reference to <FIG>, one payload <NUM> (as represented by stars in <FIG>) can be identified as causing the upward trend in the performance metric <NUM>. For instance, in the example shown in <FIG>, as the payload <NUM> propagates, the performance metric <NUM> increases accordingly. As such, the payload <NUM> can be a cause of deviation in the performance metric <NUM>.

<FIG> are flowcharts illustrating processes of detecting software regression in accordance with embodiments of the disclosed technology. Though embodiments of the processes are described below in the context of the computing environment <NUM> of <FIG> and <FIG>, embodiments of the processes can be implemented in other computing environments with additional and/or different components.

As shown in <FIG>, a process <NUM> for detecting software regression can include accessing a dataset at stage <NUM>. As described above with reference to <FIG>, the dataset can include multiple entries each containing data representing an identification of multiple payloads included in a build of a software product executed at a computing device and a corresponding value of a performance metric of executing the build at the computing device. The process <NUM> can then include estimating a multiple variable model using the accessed dataset at stage <NUM>. In one embodiment, estimating the multiple variable model can include fitting data in the dataset into a multiple linear model and estimate a set of linear coefficients each corresponding to one of the payloads included in the various builds. In other embodiments, estimating the multiple variable model can also include fitting the data in the dataset into other suitable types of multiple variable models. The process <NUM> can then include identifying one or more payloads that have statistically significant impact on the performance metric at stage <NUM>. Example operations of identifying the one or more payloads are described below with reference to <FIG>. The process <NUM> can further include performing remedial actions based on the identified one or more payloads at stage <NUM>. Example remedial actions are described above with reference to <FIG>.

As shown in <FIG>, the example operations for identifying one or more payloads can include receiving model coefficients individually corresponding to one of the payloads at stage <NUM>. The operations can then include a decision stage <NUM> to determine whether a coefficient corresponding to a payload is above a threshold. In response to determining that the coefficient is not above the threshold, the operations include indicating that the corresponding payload does not have statistically significant impact on the performance metric at stage <NUM>. Otherwise, the operations proceed to indicating that the corresponding payload does have statistically significant impact on the performance metric at stage <NUM>. The operations can then include another decision stage <NUM> to determine whether additional coefficients are present int eh received model coefficients. In response to determining that additional coefficients are present int eh received model coefficients, the operations revert to determining whether an additional coefficient exceeds the threshold at stage <NUM>. Otherwise, the operations proceed to indicating that the payload identification operation is complete at stage <NUM>.

<FIG> is a computing device <NUM> suitable for certain components of the computing system <NUM> in <FIG>. For example, the computing device <NUM> can be suitable for the client devices <NUM>, the application server <NUM>, or the regression server <NUM> of <FIG>. In a very basic configuration <NUM>, the computing device <NUM> can include one or more processors <NUM> and a system memory <NUM>. A memory bus <NUM> can be used for communicating between processor <NUM> and system memory <NUM>.

Depending on the desired configuration, the processor <NUM> can be of any type including but not limited to a microprocessor (µP), a microcontroller (µC), a digital signal processor (DSP), or any combination thereof. The processor <NUM> can include one more level of caching, such as a level-one cache <NUM> and a level-two cache <NUM>, a processor core <NUM>, and registers <NUM>. An example processor core <NUM> can include an arithmetic logic unit (ALU), a floating-point unit (FPU), a digital signal processing core (DSP Core), or any combination thereof. An example memory controller <NUM> can also be used with processor <NUM>, or in some implementations memory controller <NUM> can be an internal part of processor <NUM>.

Depending on the desired configuration, the system memory <NUM> can be of any type including but not limited to volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.) or any combination thereof. The system memory <NUM> can include an operating system <NUM>, one or more applications <NUM>, and program data <NUM>. This described basic configuration <NUM> is illustrated in <FIG> by those components within the inner dashed line.

The computing device <NUM> can have additional features or functionality, and additional interfaces to facilitate communications between basic configuration <NUM> and any other devices and interfaces. For example, a bus/interface controller <NUM> can be used to facilitate communications between the basic configuration <NUM> and one or more data storage devices <NUM> via a storage interface bus <NUM>. The data storage devices <NUM> can be removable storage devices <NUM>, non-removable storage devices <NUM>, or a combination thereof. Examples of removable storage and non-removable storage devices include magnetic disk devices such as flexible disk drives and hard-disk drives (HDD), optical disk drives such as compact disk (CD) drives or digital versatile disk (DVD) drives, solid state drives (SSD), and tape drives to name a few. Example computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. The term "computer readable storage media" or "computer readable storage device" excludes propagated signals and communication media.

The system memory <NUM>, removable storage devices <NUM>, and non-removable storage devices <NUM> are examples of computer readable storage media. Computer readable storage media include, but not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other media which can be used to store the desired information and which can be accessed by computing device <NUM>. Any such computer readable storage media can be a part of computing device <NUM>. The term "computer readable storage medium" excludes propagated signals and communication media.

The computing device <NUM> can also include an interface bus <NUM> for facilitating communication from various interface devices (e.g., output devices <NUM>, peripheral interfaces <NUM>, and communication devices <NUM>) to the basic configuration <NUM> via bus/interface controller <NUM>. Example output devices <NUM> include a graphics processing unit <NUM> and an audio processing unit <NUM>, which can be configured to communicate to various external devices such as a display or speakers via one or more A/V ports <NUM>. Example peripheral interfaces <NUM> include a serial interface controller <NUM> or a parallel interface controller <NUM>, which can be configured to communicate with external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device, etc.) or other peripheral devices (e.g., printer, scanner, etc.) via one or more I/O ports <NUM>. An example communication device <NUM> includes a network controller <NUM>, which can be arranged to facilitate communications with one or more other computing devices <NUM> over a network communication link via one or more communication ports <NUM>.

The network communication link can be one example of a communication media. Communication media can typically be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and can include any information delivery media. A "modulated data signal" can be a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), microwave, infrared (IR) and other wireless media. The term computer readable media as used herein can include both storage media and communication media.

The computing device <NUM> can be implemented as a portion of a small-form factor portable (or mobile) electronic device such as a cell phone, a personal data assistant (PDA), a personal media player device, a wireless web-watch device, a personal headset device, an application specific device, or a hybrid device that include any of the above functions. The computing device <NUM> can also be implemented as a personal computer including both laptop computer and non-laptop computer configurations.

Claim 1:
A method for software regression detection in a computing system (<NUM>), comprising:
accessing (<NUM>) a dataset (<NUM>) having multiple entries each containing data representing an identification of multiple payloads (<NUM>) included in a build (<NUM>) of a software product executed at a computing device (<NUM>) and a corresponding value of a performance metric (<NUM>) of executing the build at the computing device, the payloads individually representing a source code change, a feature enablement, or a configuration modification of the software product; and
upon accessing the dataset, at the computing system,
estimating (<NUM>) a set of coefficients individually corresponding to one of the multiple payloads using a multiple variable model with the performance metric as a dependent variable and the multiple payloads as independent variables;
identifying (<NUM>) at least one of the multiple payloads with a corresponding estimated coefficient whose absolute value is greater than a preset threshold; and
generating and transmitting (<NUM>), from the computing system, a message to a development team associated with submitting the at least one of the multiple payloads, the message indicating to the development team that a software defect that impacts the performance metric of the software product is likely present in the at least one of the payloads.