Patent Publication Number: US-2023161687-A1

Title: System and method for test selection according to test impact analytics

Description:
FIELD OF THE INVENTION 
     The present invention is of a system and method for determining test selection according to test impact analytics and in particular, of such a system and method for applying test timing and history to determine test analytics for per test coverage. 
     BACKGROUND OF THE INVENTION 
     Various methods are known in the art for determining test impact analytics. Test impact analytics relates to the impact of a particular test on determining the behavior of code, and in particular, whether such code is executing in an expected manner according to the expected parameters and is therefore showing expected behavior. 
     Not all tests are necessary to determine whether code shows such expected behavior. For example, tests that only cover code that has not changed may not be necessary to run. However it may be difficult to determine which tests are relevant for which aspects of the code, and even more difficult to determine which tests are of relatively greater importance. 
     BRIEF SUMMARY OF THE INVENTION 
     The present invention overcomes the drawbacks of the background art by providing a system and method for selecting one or more tests according to test impact, for example as determined with regard to test history and/or test timing. By “test history” it is meant which tests were executed at which time, for example in order to determine which tests are relevant for coverage of particular code. 
     According to at least some embodiments, there is provided a system and method for statistically determining a relative importance of running a particular test within a plurality of such tests. The solution is based on mapping of a test method to a list of methods/functions which are executed upon execution of a test, or overall execution of a test itself. Preferably, once the relative importance of running a particular test has been determined, it is possible to determine whether a particular test should be run at a particular time, or indeed whether it should be run at all. The relative importance may be determined for example according to a characteristic of the code, a characteristic of the test, a characteristic of who or what caused the code to change, when the code was changed and the context of other changes to the code, or a combination thereof. 
     For example in relation to an importance of the code, characteristics include but are not limited to code usage in execution by the end user, interactions between a section of the code and other sections (with increased interactions relating to increased importance), an importance of the code overall for execution in production (that is, being “mission critical”), code with many changes or new commits, otherwise modified code, and so forth. Optionally such importance may be determined according to build scanning of the binary files and commit history as described herein, and/or other methods. 
     In relation to an importance of the test, characteristics include but are not limited to tests that apply to modified or added code, or code that is deemed important, tests that have failed previously, new and modified tests, and so forth. The relationship between a test and the code that is tested may be determined statistically, for example. As a non-limiting example, the statistical relationship may relate to a test execution time frame relative to execution time frame of a particular section of code. 
     Implementation of the method and system of the present invention involves performing or completing certain selected tasks or steps manually, automatically, or a combination thereof. Moreover, according to actual instrumentation and equipment of preferred embodiments of the method and system of the present invention, several selected steps could be implemented by hardware or by software on any operating system of any firmware or a combination thereof. For example, as hardware, selected steps of the invention could be implemented as a chip or a circuit. As software, selected steps of the invention could be implemented as a plurality of software instructions being executed by a computer using any suitable operating system. In any case, selected steps of the method and system of the invention could be described as being performed by a data processor, such as a computing platform for executing a plurality of instructions. 
     Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The materials, methods, and examples provided herein are illustrative only and not intended to be limiting. 
     An algorithm as described herein may refer to any series of functions, steps, one or more methods or one or more processes, for example for performing data analysis. 
     Implementation of the apparatuses, devices, methods and systems of the present disclosure involve performing or completing certain selected tasks or steps manually, automatically, or a combination thereof. Specifically, several selected steps can be implemented by hardware or by software on an operating system, of a firmware, and/or a combination thereof. For example, as hardware, selected steps of at least some embodiments of the disclosure can be implemented as a chip or circuit (e.g., ASIC). As software, selected steps of at least some embodiments of the disclosure can be implemented as a number of software instructions being executed by a computer (e.g., a processor of the computer) using an operating system. In any case, selected steps of methods of at least some embodiments of the disclosure can be described as being performed by a processor, such as a computing platform for executing a plurality of instructions. 
     Software (e.g., an application, computer instructions) which is configured to perform (or cause to be performed) certain functionality may also be referred to as a “module” for performing that functionality, and also may be referred to a “processor” for performing such functionality. Thus, processor, according to some embodiments, may be a hardware component, or, according to some embodiments, a software component. 
     Further to this end, in some embodiments: a processor may also be referred to as a module; in some embodiments, a processor may comprise one or more modules; in some embodiments, a module may comprise computer instructions—which can be a set of instructions, an application, software—which are operable on a computational device (e.g., a processor) to cause the computational device to conduct and/or achieve one or more specific functionality. 
     Some embodiments are described with regard to a “computer,” a “computer network,” and/or a “computer operational on a computer network.” It is noted that any device featuring a processor (which may be referred to as “data processor”; “pre-processor” may also be referred to as “processor”) and the ability to execute one or more instructions may be described as a computer, a computational device, and a processor (e.g., see above), including but not limited to a personal computer (PC), a server, a cellular telephone, an IP telephone, a smart phone, a PDA (personal digital assistant), a thin client, a mobile communication device, a smart watch, head mounted display or other wearable that is able to communicate externally, a virtual or cloud based processor, a pager, and/or a similar device. Two or more of such devices in communication with each other may be a “computer network.” 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The invention is herein described, by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of the preferred embodiments of the present invention only, and are presented in order to provide what is believed to be the most useful and readily understood description of the principles and conceptual aspects of the invention. In this regard, no attempt is made to show structural details of the invention in more detail than is necessary for a fundamental understanding of the invention, the description taken with the drawings making apparent to those skilled in the art how the several forms of the invention may be embodied in practice. In the drawings: 
         FIG.  1    relates to a non-limiting, exemplary system for statistically determining a relative importance of running a particular test; 
         FIG.  2 A  shows a non-limiting, exemplary build history map, showing the mapping between execution of tests and their footprint in a file or per method; 
         FIG.  2 B  shows a non-limiting exemplary statistical model, again operating per file or per method; 
         FIG.  3 A  shows a non-limiting, exemplary system for analyzing builds to determine which tests are relevant; 
         FIG.  3 B  shows a non-limiting, detailed example of a build mapper; 
         FIG.  4    shows an additional non-limiting, exemplary build mapper; 
         FIGS.  5 A and  5 B  relate to non-limiting exemplary systems and flows for providing information to an artificial intelligence system with specific models employed and then analyzing it; 
         FIG.  6    shows a non-limiting, exemplary system and flow for ensemble learning; and 
         FIG.  7    shows a non-limiting, exemplary method for training a machine learning model. 
     
    
    
     DESCRIPTION OF AT LEAST SOME EMBODIMENTS 
     Test impact analytics are an important aspect of testing code. Simply running all available tests may take a great deal of time and slow down development. On the other hand, failing to run an important test may cause code to be inadequately tested, and even lead to failures in production or execution. 
     New defects in the system are mainly created by code changes. Therefore, it is preferable that areas of code that were modified since the last text execution are tested. The tests that are run on code that was recently modified have a much higher probability of failure. 
     On the other hand, tests that have been previously run on code that was not modified should not suddenly start failing. If such a test is failing without a reason, it may be defective and so this test result would not be trusted. 
     However if a test fails, it is most probable that the reason for the failure is due to the code change that was done prior to the occurrence of the failure. 
     Given the above, when a full regression cycle for every given code change is performed, many tests may be run that have a very low probability of failure. As noted above, this may cause the delivery pipeline to be very slow and the feedback for the developers may be very long. In addition, the infrastructure cost of running a full regression is very high, the waiting queue for a test environment may then expand to a catastrophic level. In such a situation, sometimes test jobs need to be canceled urgently in order to push some important fixes or changes earlier. However, making such changes without prior analysis will not lead to the most efficient outcome—such as canceling only unnecessary tests. 
     Some manual test operations are built on the practice of having a test architect select the required tests to execute for a given build in order to optimize the test execution time. This selection is based on the knowledge of what the tests are doing and what the code change is, and how the code change may impact specific tests. Through such knowledge, the tests that are selected are hopefully the ones with a higher probability to fail. However, this is a manual selection done based on an individual&#39;s knowledge and subject to human mistakes. Such a manual selection inherently lacks visibility of how the changes are actually impacting the tests. 
     Preferably, instead of relying on fallible human selection, automatic and scientific methods are performed for test impact analytics and for selecting important code to test. Various methods may be used for the determination of code importance and/or test impact analytics. These test impact analytics consider the impact of the code change on tests and/or of the test on the code, in order to build a list of tests to be performed. Optionally the list is built according to an order of relative test importance, which may in turn be determined according to an importance of the test, an importance of the code to be tested or a combination thereof. For example, tests may be preferentially implemented on code that the end user is actively using, or causing to execute. 
     A user may also implement a manual rule or condition for running one or more tests. Additionally or alternatively, a user may also determine a degree of sensitivity vs accuracy. For example, the user may prefer that analysis of a relationship between code components and tests tends to favor running a test rather than not running the test. On the other hand, the user may prefer to only run tests that are clearly related to, or associated with, particular code. 
     Turning now to the Figures,  FIG.  1    relates to a non-limiting exemplary system for statistically determining a relative importance of running a particular test. The solution is based on mapping of a test method to a list of methods in source files or generated artifacts by a build system (JAR, WAR, .NET assemblies, transpiled javascript code, etc) which are executed during the test method run, or execution of a test. Preferably, once the relative importance of running a particular test has been determined, it is possible to determine whether a particular test should be run at a particular time, or indeed whether it should be run at all. 
     In a system  100 , the list of tests to run or exclusion of such tests is determined by a backend  102  by analyzing code changes and the status of various tests in the previous run (failed tests or new/modified tests). Backend  102  may for example correspond to the previously described cloud system. Backend  102  determines the effect of such code changes. Backend  102  receives the status of the tests being run, as well as the results from previously run tests, for example from an agent or other software monitoring the underlying test framework. Preferably backend  102  then applies statistical analysis to determine which tests are to be run, and more preferably to determine a relative importance of running one or more tests. For example and without limitation, backend  102  may determine that certain tests are required according to one or more test criteria, while other tests are less important in relation to the required tests. Optionally, input may be obtained from an end user or from another source, such as a software source for parameters for example, to determine which tests are more relevant and/or which tests are required. 
     Code changes are preferably collected by backend  102  after being reported by a build scanner  104 . Build Scanner  104  preferably scans tests classes in the build map or artefact folder to allow management of test changes. Build scanner  104  preferably monitors changes in the build, and in particular changes in the code, which may be used to determine which tests are relevant by backend  102 . Build scanner  104  may optionally cause or define the build, but preferably listens to a build server that actually causes the build to be constructed (not shown). Upon detecting that a new build has been made, or upon being notified of such a new build being made, build scanner  104  preferably then determines any differences from a previous build, optionally and preferably ignoring any extraneous information, such as comments on the code. Alternatively, build scanner  104  only receives the differences in the code from a previous build. Also optionally and alternatively, build scanner  104  determines such differences in the code from a previous build. 
     A test listener  106  may for example operate according to the previously described test agent. Preferably test listener  106  is able to modify test methods and to determine which should be excluded. Test listener  106  also preferably defines a new footprint format. Optionally, the context for this footprint, for example in terms of which tests are applied, the headers for communication for the tests, and so forth is also sent. Such context is preferably sent for uni-test implementations, optionally in place of timing or time based information. Alternatively, the test context is not sent. 
     Optionally and preferably test listener  106  collects code coverage from the application being tested while tests are running; collects test information and test results; and forces the test framework to run the recommended test list and skip the other tests not in the list (the recommendation process is described in greater detail below). 
     Back end  102  is optionally able to determine coverage by already applied test methods in relation to the previous build of the code. Optionally, back end  102  is able to predict code coverage of available tests on the new test build. Optionally and preferably back end  102  determines test coverage according to the test results or may receive such test coverage from a test agent as described in greater detail below. Optionally, test coverage is determined according to one or more of detection of at least one change in the code from a previous build to a current build and analysis of at least one test to determine whether such change has been tested; assessment of at least one previously failed test; assessment of at least one test coverage hole; or a combination thereof. As previously noted, a test coverage hole may optionally be determined in the case that at least a portion of the code has not adequately been tested by test(s) that have been run, which may optionally include but is not limited to zero test coverage, in which no test that has been run tests that portion of the code. 
     In some cases (for example, when using test frameworks for tests running) the test listener  106  collects footprints in a test execution time frame and packs together test identifier and collected footprints—so-called colored footprints. In this case optionally back end  102  does not perform any additional mechanism for associating between test and tested code. 
     For example, in order to determine test coverage, optionally and preferably the number of methods tested is divided by the total number of methods in each component(s) to determine the percent coverage. Preferably also code branches and code lines are also considered in the calculation for the percent coverage. This determination is optionally and preferably made per environment and also across all environments. 
     The test cycle of a build may also optionally be divided into test environment and coverage may optionally be calculated as described above both per test environment and also across the entire build. Additionally and/or alternatively, unit-test results may also optionally be considered, unit-test results only provide coverage results across an entire build. The results of these tests may optionally show 100% coverage for each individual method within the build, as the tests may provide coverage such that each method is tested once, even if not all of the other test environments have 100% coverage. 
     It is also possible to skip a test, which may then optionally be noted separately, to show how many tests were skipped in general in terms of the coverage, and/or in comparison to what was checked in the previous build. 
     If tests are executed in stages, then optionally the determination is made for each test stage, for example to select one or more tests to be performed at each stage. 
     Backend  102  preferably associates footprints to a particular test according to time of footprints collection for a preferable time based solution. Optionally, backend  102  reviews tests that were performed, the code that was executed or reviewed at the time of performing the tests, and then compares such information. Preferably a statistical analysis is used to determine the likelihood of a particular test being associated with particular code. 
     Backend  102  preferably then generates a test to method mapping  108 , by updating the build history on each build (shown as a build history  110 ) and by updating a statistical model  112  through a timer. Wherever the term “test to method mapping” is described, optionally a test may be mapped to a file or to another code component. Each of these two different types of mapping is explained in greater detail below. 
     Build History Mapping  110  is a matrix for each test, which contains affected files and methods for particular execution identifier. Up to N executions may be supported for each test. This mapping is created based on test execution data (footprints, etc.). Optionally, the above build map information may be fed to a machine learning model, to further analyze and refine test mapping to particular parts of code according to the build history. Also optionally, the above build map is fed to a statistical model to determine a further statistical map.  FIG.  2 A  shows a non-limiting, exemplary build history map, showing that for each test, the map determines which files or methods have been covered per execution. This example refers to a scenario with  3  tests, N executions and M files, which may optionally be a method or any other suitable code component.  FIG.  2 A  shows a mapping between an execution of tests to a file or a method in a file, or to any other suitable code component. Each row indicates that a file/method has been executed in the current execution. 
     A Statistical Model map  112  is a matrix of all known tested and test affected files (or methods). This mapping is calculated on the basis of historical mapping.  FIG.  2 B  shows a non-limiting exemplary statistical model, showing that for each file or method, which tests should run is determined. This example refers to a scenario with N tests and M files (or methods). Again, instead of a file, optionally a method or any other suitable code component may be used. 
       FIG.  3 A  shows a non-limiting, exemplary system for analyzing builds to determine which tests are relevant. 
     In the system  300 , executable code processor  354  is executing a test  356 . The test agent  336  monitors the behavior of unit under test  356  and causes one or more tests to be performed. These tests may perform through a test engine server  332  and a test framework  334 . Test framework  334  preferably determines the code coverage for new, modified, or existing code and may also receive information from cloud system  322  regarding previous tests. Test framework  334  may then calculate the effect of a new test as to whether or not it will increase code coverage or whether it will not increase test code coverage, and in particular test coverage for specific code. 
     Test information is sent first to a storage manager  342  and then to analysis engine  320 . Analysis engine  320  determines whether or not test code coverage should be updated, how it should be updated, whether any code has not been tested, and so forth. This information is stored in database  328  and is also passed back to gateway  324 . 
     As shown, the test listener functions of  FIG.  1    may be performed by test agent  336 , alone or in combination with analysis engine  320 . 
     A build mapper  302  preferably determines the relevance of one or more tests, according to whether the code that is likely covered by such tests has changed. Such a determination of likely coverage and code change in turn may be used to determine which tests are relevant, and/or the relative relevance of a plurality of tests. 
     Build mapper  302  preferably receives information about a new build and/or changes in a build from a build scanner  312 . Alternatively, such functions may be performed by analysis engine  320 . Build mapper then preferably receives information about test coverage, when certain tests were performed and when different portions of code were being executed when such tests were performed, from test agent  336  and/or analysis engine  320 . 
     Build mapper  302  preferably comprises a footprint correlator  304 , for determining which tests relate to code that has changed, or that is likely to have changed, as well as for receiving information regarding code coverage. Footprint correlator  304  in turn preferably communicates such information to a history analyzer  306  and a statistical analyzer  308 . History analyzer  306  preferably assigns likely relevance of tests to the new or changed code, based on historical information. Such likely relevance is then based to statistical analyzer  308 . Statistical analyzer  308  preferably determines statistical relevance of one or more tests to one or more sections of code, preferably new or changed code. For example, such statistical relevance may be determined according to the timing of execution of certain tests in relation to the code that was being executed at the time. Other relevance measures may also optionally be applied. Information regarding the results of the build history map and/or statistical model are preferably stored in a database  310 . 
     For determining the relative or absolute impact of a test, and optionally for selecting one or more preferred tests, a test impact analyzer  360  is shown as part of cloud system  322 . Test impact analyzer  360  may have part or all of its functions incorporated in build mapper  302  or at another location (not shown). Test impact analyzer  360  is preferably able to recommend one or more tests to be performed, for example according to a policy set by the user, according to one or more rules or a combination thereof. Tests may be given a preference, relative or absolute, according to such a policy or such rules. 
     Non-limiting examples of such rules include a preference for impacted tests, previously used at least once, that cover a footprint in a method that changed in a given build. Also preferably recently failed tests may be performed again. New tests that were added recently or modified tests may be performed again. Tests that were recommended in the past but were not executed since then may have a preference to be performed. 
     Tests that are covering code that is being used in production may be preferred, particularly in case of inclusion of one of the above rules. Other code related rules may include but are not limited to tests that are covering code that was modified multiple times recently, and/or tests that are covering code that is marked manually or automatically as high risk code. 
     A non-limiting example of a user determined policy rule is the inclusion of an important test recommended by the user so that it is always executed in each selective run of the tests. 
     When a user selects the option of running only the recommended tests, it is done by defining a selective run policy that may include information about the occasions when to run the selective tests list or full test list, that can be based on number or tests executions in a day/week/month, or every number of test executions, or between a certain time frame. 
     Test impact analyzer  360  preferably performs test impact analytics, based on the above. Test Impact Analytics is based on the correlation between a test coverage and code changes. It may for example include giving a preference to the tests that have footprints in methods that were modified in a given build. Test impact analyzer  360  is supported by calculations of the code coverage for each test that was executed in any given test environment. The per test coverage is calculated based on a statistical correlation between a given time frame of the tests that were executed with the coverage information being collected during this time frame, as described above and in greater detail below. 
     Also as described in greater detail below, a machine learning system is preferably used to refine the aforementioned correlation of tests to coverage data. Such a system is preferably applied, due to the fact that test execution is not deterministic by nature and the fact that tests may run in parallel, which may render the results even more non-deterministic. 
     Optionally, test impact analyzer  360  determines that a full list of tests is to be run, for example but without limitation under the following conditions: 1. User selection of full run. 2. When server bootstrapping code has been modified. 3. Configuration or environment variables have been modified. 
       FIG.  3 B  shows a non-limiting, detailed example of a build mapper, as operated by an exemplary computational device. Optionally any of the software modules of  FIG.  3 A  may be operated by the same or a different computational device, also featuring such a processor and memory. As shown, a computational device  360  features a processor  362  and a memory  364 , for operating build mapper  302  as previously described. 
     Functions of processor  362  preferably relate to those performed by any suitable computational processor, which generally refers to a device or combination of devices having circuitry used for implementing the communication and/or logic functions of a particular system. For example, a processor may include a digital signal processor device, a microprocessor device, and various analog-to-digital converters, digital-to-analog converters, and other support circuits and/or combinations of the foregoing. Control and signal processing functions of the system are allocated between these processing devices according to their respective capabilities. The processor may further include functionality to operate one or more software programs based on computer-executable program code thereof, which may be stored in a memory, such as a memory  364  in this non-limiting example. As the phrase is used herein, the processor may be “configured to” perform a certain function in a variety of ways, including, for example, by having one or more general-purpose circuits perform the function by executing particular computer-executable program code embodied in computer-readable medium, and/or by having one or more application-specific circuits perform the function. 
     Also optionally, memory  364  is configured for storing a defined native instruction set of codes. Processor  362  is configured to perform a defined set of basic operations in response to receiving a corresponding basic instruction selected from the defined native instruction set of codes stored in memory  364 . For example and without limitation, memory  364  may store a first set of machine codes selected from the native instruction set for receiving information from build scanner  312  (not shown) about a new build and/or changes in a build; a second set of machine codes selected from the native instruction set for receiving information about test coverage, when certain tests were performed and when different portions of code were being executed when such tests were performed, from test agent  336  and/or analysis engine  320  (not shown); and a third set of machine codes from the native instruction set for operating footprint correlator  304 , for determining which tests relate to code that has changed, or that is likely to have changed, as well as for receiving information regarding code coverage. 
     Memory  364  may store a fourth set of machine codes from the native instruction set for communicating such changed code and/or code coverage information to a history analyzer  306 , and a fifth set of machine codes from the native instruction set for assigning likely relevance of tests to the new or changed code, based on historical information. Memory  364  may store a sixth set of machine codes from the native instruction set for communicating such changed code and/or code coverage information to a statistical analyzer  308 , and a seventh set of machine codes from the native instruction set for determining statistical relevance of one or more tests to one or more sections of code, preferably new or changed code. 
       FIG.  4    shows a non-limiting, exemplary build mapper, in an implementation which may be used with any of  FIGS.  1 - 3    as described above, or any other system as described herein. The implementation may be operated by a computational device as described with regard to  FIG.  3 B . Components with the same numbers as in  FIG.  3    have the same or similar function. A build mapper  400  features a machine learning analyzer  402 , which preferably receives information from history analyzer  306  and statistical analyzer  308 . Machine learning analyzer  402  then preferably applies a machine learning model, non-limiting examples of which are given in  FIGS.  5 A and  5 B , to determine the relative importance of a plurality of tests to particular code, files or methods. 
     More preferably, an output correlator  404  receives information from history analyzer  306  and statistical analyzer  308 , and transmits this information to machine learning analyzer  402 . Such transmission may enable the information to be rendered in the correct format for machine learning analyzer  402 . Optionally, if history analyzer  306  and statistical analyzer  308  are also implemented according to machine learning, or other adjustable algorithms, then feedback from machine learning analyzer  402  may be used to adjust the performance of one or both of these components. 
     Once a test stage finishes executing, optionally with a “grace” period for all agents to submit data (and the API gateway to receive it), then preferably the following data is available to machine learning analyzer  402 : a build map, a test list, and time slices. A Build map relates to the code of the build and how it has changed. For example, this may be implemented as a set of unique IDs+code element IDs which are persistent across builds. The test list is a list of all tests and their start/end timing. Time slices preferably include high-time-resolution slicing of low-coverage-resolution data (e.g. file-level hits [or method hits] in 1-second intervals). 
     The first step is to process the data to correlate the footprint per test (or a plurality of tests when tests are run in parallel). The second step is model update for the machine learning algorithm. Based on the build history, the latest available model for a previous build is loaded (ideally this should be the previous build). 
     If no such model exists, it is possible to assume an empty model with no data, or an otherwise untrained machine learning algorithm. The model consists of a set of test+code element id mapping (which are the key) and a floating point number that indicates the correlation between the test and the code element id. Such correlation information is preferably determined by statistical analyzer  308 . For example, a “1.0” means the highest correlation, whereas a  0  means no correlation at all (the actual numbers will probably be in between). 
     For any test+code element id, the method preferably updates each map element, such as each row, according to the results received. For example, updating may be performed according to the following formula: 
     NewCorrelation[test i,code element id j]=OldCorrelation[test i,code element id j] *0.9+(0.1 if there is a hit, 0 otherwise). This type of updating is an example of a heuristic which may be implemented in addition to, or in place of, a machine learning algorithm. Preferably these coefficients always sum up to 1.0, so there is effectively a single coefficient that relates to the speed (number of builds). For example, it is possible to do a new statistical model after each set of tests run, optionally per build. 
     Next preferably a cleanup step is performed where old correlations are deleted for code elements that no longer exist in the new build. Optionally a further cleanup step is performed where old tests are deleted, and methods that are very uncorrelated with tests (e.g. &lt;0.1). 
     Optionally tests are selected for implementation according to a variety of criteria, once the statistically likely relationship between a particular test that is executed and the related code has been established. Such criteria may be determined according to test impact analytics. These test impact analytics consider the impact of the code and/or of the test on the code, in order to build a list of tests to be performed. Optionally the list is built according to the above described order of relative importance. The list may also comprise a plurality of lists, in which each list may contain a preferred order of tests. Such lists may be assigned for performance in a particular order and/or according to a particular time schedule. For example and without limitation, one list of a plurality of tests may be executed immediately, while another such list may be executed at a later time, which may for example and without limitation be a particular time of day or day of the week. 
     Another consideration for the creation of one or more lists is the implementation of a minimal set of tests as opposed to a full test review. For example, a list may contain a minimal necessary set of tests. Alternatively or additionally, at least one list may comprise a full set of tests only on code that the end user is actively using, or causing to execute. Such a list may be preferentially implemented, while a full set of tests on code that the user is not actively using may not be preferentially implemented, may be implemented with a delay or when resources are free, or may not be implemented at all. 
     The test impact analytics may include such criteria as preferred implementation of new and modified tests; tests selected by the user, failed tests, and/or tests that are selected according to code that actual end users cause to execute when the code is in production. Other important criteria for the code that may influence test selection include highly important code sections or code that is considered to be mission critical, as well as code that has required significant numbers of tests according to past or current criteria, with many interactions with other sections of code and/or with many new commits in the code. 
       FIGS.  5 A and  5 B  relate to non-limiting exemplary systems and flows for providing information to an artificial intelligence system with specific models employed and then analyzing it. 
     Turning now to  FIG.  5 A  as shown in a system  500 , information from the output correlation  404  of  FIG.  4    is preferably provided at  502 . This information is then fed into an AI engine in  506  and a test relevance ranking is provided by the AI engine in  504 . In this non-limiting example, AI engine  506  comprises a DBN (deep belief network)  508 . DBN  508  features input neurons  510  and neural network  514  and then outputs  512 . 
     A DBN is a type of neural network composed of multiple layers of latent variables (“hidden units”), with connections between the layers but not between units within each layer. 
       FIG.  5 B  relates to a non-limiting exemplary system  550  with similar or the same components as  FIG.  5 A , except for the neural network model. In this case, a neural network  562  includes convolutional layers  564 , neural network  562 , and outputs  512 . This particular model is embodied in a CNN (convolutional neural network)  558 , which is a different model than that shown in  FIG.  5 A . 
     A CNN is a type of neural network that features additional separate convolutional layers for feature extraction, in addition to the neural network layers for classification/identification. Overall, the layers are organized in 3 dimensions: width, height and depth. Further, the neurons in one layer do not connect to all the neurons in the next layer but only to a small region of it. Lastly, the final output will be reduced to a single vector of probability scores, organized along the depth dimension. It is often used for audio and image data analysis, but has recently been also used for natural language processing (NLP; see for example Yin et al, Comparative Study of CNN and RNN for Natural Language Processing, arXiv:1702.01923v1 [cs.CL] 7 Feb. 2017). 
       FIG.  6    shows a non-limiting, exemplary system and flow for ensemble learning. Such a system features a combination of different models of which two non-limiting examples are shown. In this non-limiting example, in the system  600  a plurality of AI inputs  610  are provided from a plurality of AI models  602 , shown as AI models  602 A and  602 B. Such models may be any suitable models as described herein. The outputs of these AI models are then provided to AI engine  606 , operating an ensemble learning algorithm  608 . Ensemble learning algorithm  608  may feature any type of suitable ensemble learning method, including but not limited to, a Bayesian method, Bayesian optimization, a voting method, and the like. Additionally and alternatively, ensemble learning algorithm  608  may feature one or more additional neural net models, such as for example, without limitation a CNN or an encoder decoder model. Transform models in general may also be used as part of ensemble learning algorithm  608 . 
     Outputs  612  from AI engine  606  may then be provided as test relevance ranking  604 , as previously described. 
       FIG.  7    shows a non-limiting, exemplary method for training a machine learning model. As shown in a flow  700 , the training data is received in  702  and it is processed through the convolutional layer of the network in  704 . This is if a convolutional neural net is used, which is the assumption for this non-limiting example, as shown for example with regard to  FIG.  5 B . After that the data is processed through the connected layer in  706  and adjusted according to a gradient in  708 . Typically, a steep descent gradient is used in which the error is minimized by looking for a gradient. One advantage of this is it helps to avoid local minima where the AI engine may be trained to a certain point but may be in a minimum which is local but it&#39;s not the true minimum for that particular engine. The final weights are then determined in  710  after which the model is ready to use. 
     In terms of provision of the training data, preferably balanced and representative training data is used. The training data is preferably representative of the types of actual data which will be processed by the machine learning model. 
     It is appreciated that certain features of the invention, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the invention, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable sub-combination. 
     Although the invention has been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims. All publications, patents and patent applications mentioned in this specification are herein incorporated in their entirety by reference into the specification, to the same extent as if each individual publication, patent or patent application was specifically and individually indicated to be incorporated herein by reference. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present invention.