Reliable automatic estimation of test flakiness

A test flakiness system retrieves, from a repository, a software test and a software module. The test flakiness system performs the flakiness test against the software module, determining a flakiness value for the software test. On a condition that a difference between the flakiness value and a set of historical flakiness values exceeds a threshold, the test flakiness system creates a defect record.

TECHNICAL FIELD

Aspects of the present disclosure relate to software testing, and more particularly, to determining problems introduced by code changes that can be identified by an increase in tests' flakiness.

BACKGROUND

Software development can involve large, complex applications. Release of these applications can involve continuous testing of the application code. Changes to the code base can introduce defects into the applications. Changes to supporting infrastructure can also introduce problems into an application system. Tests performed against application systems can sometimes fail for reasons other than code defects. Continuous testing attempts to detect issues as quickly as possible by running tests upon a change in the software code. Many tests can be automated.

DETAILED DESCRIPTION

A flaky test is a test for which a result is non-deterministic, e.g., in software, a test that can both pass and fail without any changes to the underlying code. Put another way, a flaky test sometimes fails, but if retried enough times, it passes. Alternatively, a flaky test usually passes, but if retried enough times, it fails. Such tests can be problematic because they can mask newly introduced coding errors that can be difficult to identify for subsequent investigation. As a codebase and its number of tests increases, flaky tests can be likely, especially for end-to-end tests involving both software and hardware components. However, some defects that are introduced by code changes and masked by flaky tests can be identified. Different strategies can be applied against flaky tests. First, one can run a flaky test several times and declare success if it passes at least once and failure if all runs fail. Another strategy is to completely ignore flaky tests' results. Yet another strategy is to segregate flaky tests in their own test suite until (and if) they demonstrate deterministic behavior.

Some tests can be complex and involve many prerequisites and dependencies. However, a test failure is not always indicative that the function under test has a defect. For example, tests involving networking can fail because of network issues. Alternatively, software tests can fail because of a real problem in the software, but because the test doesn't consistently indicate that something is wrong, rather occurring every now and then, the source of the problem can be hard to identify. One can suspect that the fault lies with infrastructure or with some other dependency upon which the function relies, when the problem actually lies with the software.

This phenomenon can be more relevant for some types of software than for others. For example, software using the cloud can rely on replicated communication paths that can lead to race conditions. For example, if multiple entities talk to one another, over different communication paths, those entities should not depend on a particular message delivery order. Otherwise, a function may usually work—but sometimes fail—depending on how the network decides to deliver specific messages. Such a test can be referred to as a “flaky” test. Sometimes it succeeds and sometimes it fails. However, the test can still provide value.

Absent a test flakiness system, a developer likely gets a notification that a test has failed. However, after investigation, the developer may conclude that the failing test is not due to a code change. The developer retries the test and it succeeds. However, one-time success of a flaky test does not guarantee future success. An increase in a flaky test's failure from 10 failures in 1000 test executions to 30 failures in 1000 test executions represents a 300% increase in failures. A human, however, likely discerns no difference in a “mostly succeeds” test result.

Flaky tests are likely an unavoidable reality for test sets at a large scale. Flaky tests can compromise a fragile balance of workflow and quality. Bugs can hide under the disguise of a flaky test, e.g., a test loses its purpose if the developer just ignores the failure due to its flaky nature. Flaky tests also take up valuable resources when running them and increase execution cost. Ultimately, flaky tests can reduce continuous testing/integration stability, increase time-to-market, reduce developer trust, and impact developer experience.

Flaky tests can be expensive to repair, particularly if a developer has become inured to a test failure and the root cause of the flakiness manifests itself in a production environment and results in a customer's outage or impaired operations. An assumption can often be made that failure of a flaky test doesn't indicate a real bug—it's just a flaky test. Another issue can be a poorly written test itself, that manifests itself as a false alert and distraction to the development team.

Aspects of the present disclosure address the above-noted and other deficiencies by providing a flakiness testing system. The testing system monitors a measure of flakiness and pinpoints changes (commits) to a codebase that may have affected flakiness. Benefits of the embodiments of the present disclosure for a flakiness testing system include first determining a flakiness value for each of a set of software tests and then monitoring the results of these software tests over time for any increase in flakiness. Such monitoring can improve software and system quality by identifying potential issues which, while significant, are too infrequent to be identified by a human. Such monitoring further allows engineers to investigate not only software code, but supporting infrastructure, for defects and/or impediments.

A flakiness test system, as part of an automated testing system, can also provide automated creation of defect records and other types of submissions to defect (or bug) tracking systems. Automated testing allows software tests and other sets of repeatable tasks to be performed without human interaction. Furthermore, these tests can run with varying frequency to ensure that an application continuously performs as expected. Problems frequently occur whenever the source code is updated. A benefit of automated testing is that it can increase accuracy. Indeed, automated testing is less likely to be affected by human error. Incorporating flakiness testing into an automated testing system can be beneficial when dealing with a large codebase or when new features are added. In addition, automated testing helps ensure that errors or defects in the code are identified and fixed as quickly as possible. Extending automated testing to include flakiness testing can improve testing coverage of software as well as extend the bounds of the testing beyond the application and to other servers and components in the application environment, e.g., networking elements.

Automated testing can also lead to reduced costs. When tests are automated, the need for manual testers is reduced. In addition, the time needed to execute tests is reduced, leading to savings in terms of both time and money.

Moreover, automated flakiness tests can help reduce the cost of software development by detecting errors earlier in the process and allowing them to be fixed. They can also help reduce the cost of supporting an application, as automated tests can require less time to identify defects. This is particularly relevant to defects identified by flaky tests, as correlating a defect that may manifest itself in less than one percent of tests with a change to an application or other element in the cloud can consume large amounts of time to identify, let alone correct. If automated flakiness tests are incorporated into a continuous testing system, the tests can be configured to automatically execute each time a new feature or change is introduced into the application or application infrastructure. This can help ensure that any issues in the recent changes are identified as quickly as possible so that they can be fixed as quickly as possible.

Automated flakiness testing can help to improve collaboration between developers and infrastructure engineers. By incorporating tests focused on flaky behavior, both developers and information technology engineers can rely on them during the implementation of new changes or features as well as the roll-out of new hardware and hosts. This can improve coordination between different members of a team in identifying and resolving issues.

As discussed in greater detail below, a flakiness testing system may include a collection of servers that provide one or more services to one or more client devices. The flakiness testing system may retrieve, from a repository, a software test and a software module. The flakiness testing system may then perform the software test against the software module. The flakiness testing system may then determine a flakiness value for the software test and compare that flakiness value with a set of historical flakiness values. If the difference between the flakiness value and the historical flakiness values exceeds a threshold, the flakiness testing system may create a defect record in a bug tracking system.

Although aspects of the disclosure may be described in the context of continuous testing, embodiments of the disclosure may be applied to any computing system that performs testing and evaluates the results of the tests.

FIG.1is an illustrative example of a flakiness testing architecture100, in accordance with some embodiments of the disclosure. However, other flakiness testing architectures100are possible, and the implementation of a computer system utilizing examples of the disclosure are not necessarily limited to the specific architecture depicted byFIG.1. As shown inFIG.1, continuous testing system106includes a test flakiness system108. In some embodiments, the test flakiness system receives a request104from a client device102to perform a software test against a software module. In some embodiments, the test flakiness system determines that a new version of an application is available for testing. In some embodiments, a continuous testing system106determines that a new version of a software module112is available. In some embodiments, flaky tests are run periodically against a software application and supporting infrastructure to identify defects or other issues that may have been introduced by changes to a cloud infrastructure.

Upon such a determination, the test flakiness system108can retrieve a software module112and a software test114from a repository110. In some embodiments, the repository110can be a code control system or a version control system. In some embodiments, the repository can be part of a continuous testing system.

In some embodiments, the test flakiness system108applies the software test114against the software module112. In some embodiments, the test flakiness system108may repeat the software test114a number of times in order to obtain a flakiness value for the software test.

In some embodiments, the test flakiness system108is part of a continuous testing system, such as continuous testing system106. In some embodiments, any change to a software module results in a full build of an application and execution of an entire suite of tests. In some embodiments, a test can be applied against a single module, for a single element of functionality, e.g., a unit test. In some embodiments, tests are periodically run across an entire application and assessed for an increase in flakiness that exceeds a threshold.

In an embodiment, the test flakiness system108compares the current flakiness value of the test to historical flakiness values116. In some embodiments, if the difference between the current flakiness value and the historical flakiness values116exceeds a threshold, a defect record118is created and stored in the repository. In some embodiments, the defect record may be added to a defect, or bug, tracking system. In some embodiments, a trouble ticket may be added to an infrastructure maintenance system to solicit information on infrastructure changes, such as network changes or host updates, that may contribute to an increase in flakiness of a test. In some cases, simply alerting stakeholders that “something” changed on a particular date, perhaps at a particular time, can cause the question, “could that increase be due to the new software we installed on our network switches,” to be more deeply considered. Adding the defect to a bug tracking system allows it to be triaged against other competing demands.

In some embodiments, p can be defined to be a measure of a test's flakiness before some change C, and p′ (p-prime) to be the measure of that test's flakiness after the change. At least four scenarios can be contemplated.

In a first scenario, a test has a low measure of flakiness, e.g., p<0.001, before a change C, and the test's flakiness increases only slightly after the change C, e.g., |p′−p|<0.001. Such cases can be hard to detect using simple statistical methods.

In a second scenario, no test failures are observed before a code change C (which does not mean the test is not flaky), but after the change, the test fails once. Formally, this may be indistinguishable from the first scenario, but can be easy to detect and may not require special treatment.

In a third scenario, a test's flakiness p is already large before a change C, e.g., p>0.1. In this scenario a small change can be much harder to detect because as C→0, (|p′−p|/p′)→0. In layman's terms, the flakier the test, the more the test can mask any new failures associated with a change.

In a fourth scenario, a test has a measure p of flakiness, and after a change C, its flakiness p′increases such that, for example, |p′−p|»0.001. In some embodiments such cases can be identified and associated with underlying changes to code or to an environment in which the code executes.

Some embodiments use an automated test system that performs builds, as part of which the test system runs a collection of individual tests. Then, in some embodiments, for each test, during each build, the test system can record the success or failure of a test. In some embodiments, the test system can periodically, e.g., every N days, calculate the value of p′, an estimate of flakiness for the test.

FIG.2is a graph200of an illustrative example of test flakiness over time, in accordance with some embodiments of the disclosure.

In an embodiment, graph200shows the execution of a test executed weekly over a period of ten weeks. The graph200indicates a flakiness value (p) for the test of approximately 0.025 through week five. While the flakiness is slightly below 0.025 for weeks one, three, and five, the test results indicate that the flakiness is relatively stable.

However, commencing with week six, the flakiness of the test increases by approximately 0.075, in increments of 0.025 per week, reaching almost 0.1 at week 8. While the measure of change in flakiness for weeks nine and ten approximates that of weeks two and three, the flakiness has almost quadrupled since week five. An examination of the graph suggests that, in some embodiments, changes occurred to the system under test during weeks six, seven, and eight that significantly increased the flakiness of the test. While the test may not unequivocally indicate a code defect, engineers can examine any changes applied to the code base in weeks six, seven, and eight, as well as any changes that may have been made to the system's infrastructure. Notably, the change from 0.025 to 0.1 would likely be undetectable to a human observer yet is statistically significant.

To calculate p′, a measure of test flakiness, i.e., the probability of a test's failure, an experiment can be performed every N days. Perform n iterations of a test observing ν failures and calculate p′n=ν/n, then perform the test an additional m times, checking the convergence criterion, e.g., |p′n−p′n+m|ε. The goal of the convergence criterion is that, as a sequence progresses, the probability of an “unusual” outcome becomes smaller and smaller. Continue until the convergence criterion is met. In some embodiments, in order to detect small values of p′, more repetitions of tests can be performed.

In some embodiments, an objective is to perform an experiment until the change in the number of failures converges to zero. For example, if a test is performed twice and fails once, one explanation is that p is 0.5 (a 50% failure rate). A more likely explanation, for a flaky test, is that an insufficient number of samples have been taken. Alternatively, the test can be executed 100 times, followed by an additional 50 executions, and see if the estimates based on 100 iterations and 50 iterations differ more than some epsilon (F), where epsilon is the mean distance between the iterations expressed in terms of a number of standard deviations.

In some embodiments, an alternative convergence criterion can be derived using error propagation. Error propagation is the effect of variables' uncertainties (or errors, and more specifically random errors) on the uncertainty of a function based on them. When the variables are the values of experimental measurements, they have uncertainties due to measurement limitations, e.g., instrument precision, which can propagate due to the combination of variables in the function.

The uncertainty u can be expressed in a number of ways. It may be defined by the absolute error Δx. Uncertainties can also be defined by a relative error (Δx)/x, which is often written as a percentage. The uncertainty on a quantity can also be quantified in terms of the standard deviation, a, which is the positive square root of the variance. The value of a quantity and its error can then be expressed as an interval x±u. If the statistical probability distribution of the variable is known or can be assumed, confidence limits can be derived to describe the region within which the true value of the variable may be found. For example, the 68% confidence limits for a one-dimensional variable belonging to a normal distribution are approximately ±one standard deviation a from the central value x, which means that the region x±σ will cover the true value in roughly 68% of cases. In some embodiments, the probability can be calculated using ν/n, the standard deviation, and a measure of uncertainty is σp2=(ν2/n)((σν/ν)2(σn/n)2, with a stopping criterion σp2<ε.

In some embodiments, an estimate of flakiness p can be calculated every N days, and the results stored. For each test, a probability of the observed test flakiness obeying p′ can be calculated and any new detectable flakiness that may have been introduced since the last calculation of p′ can be identified.

In some embodiments, using the terminology of statistical hypothesis testing, a null hypothesis Ho asserts that no new detectable flakiness is introduced after p′; an alternative hypothesis HA is that the test flakiness p′ changed to p″ at some later point in time.

A t-test is a statistical test that is used to compare the means of two groups. It is often used in hypothesis testing to determine whether a process or treatment actually has an effect on the population of interest, or whether two groups are different from one another. Student's t-test is a method of testing hypotheses about the mean of a small sample drawn from a normally distributed population when the population standard deviation is unknown. In some embodiments, Student's t-test can be used to determine whether two sets of data, e.g., model and observation, are significantly different from each other. In some embodiments, one can assume that both p and p′ are distributed normally with an unknown (and equal) variance. In such embodiments, the null hypothesis holds that the means are equal, e.g., HØ: μ(p)=μ(p′). In some embodiments, the t-test uses the t-statistic (where t-statistic is abbreviated from “hypothesis test statistic”): T=(μ(p′)−(p))/(σ(p′)/√{square root over (n)}))

where σ(p′) is an estimate of the standard deviation and n is the number of samples, i.e., test runs. In some cases, T follows a t-distribution with n−1 degrees of freedom and can be reasonably approximated by the standard normal distribution N(0,1) when n>10. With the significance level chosen at α=5%, this yields an acceptance criterion for the null hypothesis of: −z.025<T<Z.025 ⇒−1.96<T<1.96, where a negative value specifies reduced flakiness. A high rate of type II errors, when n is small, can be a disadvantage of the t-test. A type II error is a statistical term used within the context of hypothesis testing that describes an error that occurs when one fails to reject a null hypothesis that is actually false. A type II error produces a false negative, also known as an error of omission. In some embodiments, deeming flakiness as unchanged when it actually has changed can constitute a type II error. Often, there can be a trade-off between a cost of running a large number of tests, e.g., a large number for n, and the resulting flakiness indicators.

A trade-off can exist regarding a frequency of comparing observations with a recorded p. If the comparison is too infrequent, a change in test flakiness may go undetected for an unacceptable amount of time, during which time customers are impacted and support organizations are attempting to respond to issues. In some embodiments, multiple comparison windows may be employed. For example, a test flakiness value may be compared with historical values of the last 5, 10, or 30 days.

FIG.3is an example table300illustrating the recording of test flakiness over time, in accordance with some embodiments of the disclosure. The example table, “Table 1,” includes historical flakiness values pTn associated with a test T. In some embodiments, the example table is similar to the flakiness values116ofFIG.1. In the example, the test recorded a flakiness value of pT1 at time T1, and a flakiness value of pT2 at time T2. In some embodiments, these historical flakiness values can be compared with a current flakiness value to indicate whether some change has statistically changed the amount of flakiness associated with the test.

FIG.4is a flow diagram of an example method400of determining test flakiness, in accordance with some embodiments of the disclosure. Method400may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, a processor, a processing device, a central processing unit (CPU), a system-on-chip (SoC), etc.), software (e.g., instructions running/executing on a processing device), firmware (e.g., microcode), or a combination thereof. In some embodiments, at least a portion of method400may be performed by continuous testing system106ofFIG.1.

With reference toFIG.4, method400illustrates example functions used by various embodiments. Although specific function blocks (“blocks”) are disclosed in method400, such blocks are examples. That is, examples are well suited to performing various other blocks or variations of the blocks recited in method400. It is appreciated that the blocks in method400may be performed in an order different than presented, and that not all of the blocks in method400may be performed.

Method400begins at block410, where the processing logic causes the test flakiness system to retrieve a software test and a software module. In some embodiments, the processing logic may cause the test flakiness system to retrieve multiple software tests to be run in combination against one or more software modules. In some embodiments, the software test or tests and the software module or modules may be stored in a test repository. In some embodiments, the test repository may be similar to the repository110inFIG.1. In some embodiments, the retrieval of the software test and the software module may be obtained as a result of a continuous testing system receiving a notification that a change has been made to one or more software modules of a software application. In some embodiments, a request may come from a client device such as client device102inFIG.1. In some embodiments, the continuous testing system106may actively monitor an execution environment of a development pipeline to determine that a new release of an application is ready for testing. In some embodiments, the test flakiness system108may monitor tests determined to be flaky.

At block420, the processing logic performs the software test against the software module. In some embodiments the software test is repeated multiple times. In some embodiments multiple runs of a test are performed. In some embodiments, a first series of executions of the test is performed. In some embodiments, a second series of executions of the test is performed and the results compared with the first series of executions to determine if the two series of test executions converge. In some embodiments, the frequency of testing is between 5 and 30 days. In some embodiments, the testing may be performed daily. In some embodiments, testing may be performed upon a change to a software module. In some embodiments, testing may be performed based on a combination of time and software changes. In some embodiments, an occurrence of other events may influence testing.

At block430, the processing logic determines a flakiness value for the software test. In some embodiments, the flakiness value can be a success rate for multiple occurrences of the software test. For example, if a test is performed twice and fails once, one explanation is that the test is experiencing a 50% failure, or flakiness, rate. However, if an additional 1000 executions of the test are all successful, the flakiness rate drops to 0.0999% or 0.000999.

At block440, the processing logic compares the flakiness value with historical flakiness values. In some embodiments, the historical flakiness values are for the same software test and the same software module. In an embodiment, if the historical flakiness value of a test is 2% and the current flakiness value is 2%, even though the test occasionally fails, the frequency of failure has remained rather stable. However, should the flakiness value rise to 5%, representing a more than 2λ increase in failures, engineering resources may be alerted to the change and assigned to investigate.

At block450, if the current flakiness value exceeds the value of the historical flakiness values for a particular test, the processing logic creates a defect record. In some embodiments, creation of a defect record is dependent on a difference between the flakiness value and the historical flakiness values exceeding a threshold. In some embodiments, a notification of the creation of a defect record is sent to a client device. In some embodiments, the notification may be part of a report generated by the test flakiness system. In some embodiments, a current flakiness value exceeding the value of the historical flakiness values for a particular test may result in a rollback of a software change. In some embodiments,

In some embodiments, processing logic updates the historical flakiness values. In some embodiments, this update is recorded in a set of historical flakiness values associated with a software test and a software module. In some embodiments, this set of historical flakiness values is updated after every set of test executions. In some embodiments, this record may be similar to the table ofFIG.3.

FIG.5is a block diagram depicting an example environment500for a test flakiness architecture, in accordance with some embodiments. The environment500includes test flakiness system506. Test flakiness system506, which may correspond to test flakiness system106ofFIG.1, contains processing device508and memory510. Example environment500also includes client device502, which may correspond to client device102ofFIG.1. Example environment500also includes repository508, which contains software test516, software module518, and historical flakiness values520. Repository508may correspond to repository110ofFIG.1. Test flakiness system506further includes flakiness value512and threshold514. It should be noted that request504, flakiness value514, threshold516, software test518, software module520, historical flakiness values522, and defect record524are shown for illustrative purposes only and are not physical components of test flakiness system506.

The processing device508of test flakiness system506receives one or more requests504. The processing device508retrieves a software test516and a software module518from the repository508. The processing device508performs the software test516. The processing device508determines a flakiness value514for the software test516. The processing device508compares the flakiness value514with a set of historical flakiness values522. The processing device508updates the set of historical flakiness values522. On a condition that a difference between the flakiness value514and the set of historical flakiness values522exceeds a threshold516, the processing device508creates a defect record524.

The example computing device600may include a processing device602, e.g., a general-purpose processor, a programmable logic device (PLD), a main memory604, e.g., synchronous dynamic random-access memory (DRAM), read-only memory (ROM), static memory606, e.g., flash memory, and a data storage device618, which may communicate with each other via a bus630.

Computing device600may further include a network interface device608that may communicate with a network620. The computing device600also may include a video display unit610, e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT), an alphanumeric input device612, e.g., a keyboard, a cursor control device614, e.g., a mouse, and an acoustic signal generation device616, e.g., a speaker. In one embodiment, video display unit610, alphanumeric input device612, and cursor control device614may be combined into a single component or device, e.g., an LCD touch screen.

Data storage device618may include a computer-readable storage medium628on which may be stored one or more sets of instructions625that may include instructions for a test flakiness system108for carrying out the operations described herein, in accordance with one or more aspects of the present disclosure. The test flakiness system108may correspond to the test flakiness system108ofFIG.1. Instructions625may also reside, completely or at least partially, within main memory604and/or within processing device602during execution thereof by computing device600, main memory604and processing device602also constituting computer-readable media. The instructions625may further be transmitted or received over a network620via network interface device608.

The methods and illustrative examples described herein are not inherently related to a particular computer or other apparatus. Various general-purpose systems may be used in accordance with the teachings described herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear as set forth in the description above.