Validation of a system using a design of experiments processing technique

A validation system includes a test block that operates to apply a set of inputs to a system under test, such as a test system or an executable test algorithm, and receive from said system under test a first set of outputs produced by operation of the system under test in response to application of the set of inputs. The first set of outputs, as well as a second set of outputs reflecting output produced by operation of a reference system or executable reference algorithm in response to application of the same set of inputs, is processed to make a validation determination. A validation processing block compares the first and second sets of outputs to validate the system under test as an equivalent to the reference system.

TECHNICAL FIELD

The present invention relates to a system (such as a System on Chip (SoC)) which executes an algorithm and, in particular, to a method and apparatus for validating a new version of that algorithm on the system using a Design of Experiments (DoE) processing technique.

BACKGROUND

Those skilled in the art understand that a System on Chip (SoC) combines the components of a complex computing or electronic system on a single integrated circuit. The included components of the system comprise both hardware components and software components. Hardware components include: timers, power supplies and regulators, input/output interfaces (analog and digital), microprocessors (or microcontrollers) and volatile/non-volatile memories. Software components include: an operating system, applications and drivers. A block diagram of an exemplary SoC10is provided inFIG. 1.

It is common, over the course of the operating life of an SoC, for an update of the SoC software components to occur. A new version of an algorithm may be provided to replace an existing version of the algorithm. Such a new version may fix problems with the existing version (such as “bugs”), present a more efficient implementation of functionality supported by the existing version, or add new functionality to that provided by the existing version. Validation of the new version of the algorithm is important to ensure that the new version can be used in place of the existing version without adversely affecting operation of the SoC. If validation is confirmed, the existing version may be deleted and the new version thereafter executed in its place.

SUMMARY

The present invention is directed to method and apparatus for validating a new version of an algorithm in regard to an existing or reference algorithm. A comparison of the algorithms is performed by executing both algorithms, modifying predetermined inputs to both algorithms and comparing the resulting outputs. Proper configuration of the inputs will allow, using the method of Design of Experiments (DoE), the determination of whether the differences in output are due to significant functional differences between the subject algorithms. The result of the comparison process leads to a determination of whether the new algorithm is basically the same as the existing or reference algorithm, and may further lead to a characterization of the new version as better or worse performing than the existing or reference algorithm.

The present invention is also directed to method and apparatus for validating a new system in regard to an existing or reference system. A comparison of the systems is performed by operating both systems, modifying predetermined inputs to both systems and comparing the resulting outputs. Proper configuration of the inputs will allow, using the method of Design of Experiments (DoE), the determination of whether the differences in output are due to significant functional differences between the subject systems. The result of the comparison process leads to a determination of whether the new system is basically the same as the existing or reference system, and may further lead to a characterization of the new system as better or worse performing than the existing or reference system.

In an embodiment, a system comprises: software components including a reference algorithm; hardware components including a processing component operable to execute the reference algorithm; and a validation system configured to test execution of a test algorithm in comparison to execution of said reference algorithm and in response thereto validate the test algorithm to replace said reference algorithm for execution by said processing component.

In an embodiment, a system comprises: software components including a test algorithm; hardware components including a processing component operable to execute the test algorithm; and a built-in-self-test system configured to test execution of the test algorithm in comparison to execution of a reference algorithm and in response thereto validate that execution of test algorithm by said processing component conforms with execution of said reference algorithm.

In an embodiment, a method comprises: accessing a reference algorithm; executing the accessed reference algorithm in response to a set of inputs to produce a first set of outputs; accessing a test algorithm; executing the accessed test algorithm in response to the same set of inputs to produce a second set of outputs; and comparing the first and second sets of outputs to validate the test algorithm as an equivalent to the reference algorithm.

In an embodiment, a validation system comprises: a test block configured to apply a set of inputs to a system under test and receive from said system under test a first set of outputs produced by operation of the system under test in response to application of the set of inputs; a memory storing the first set of outputs and further storing a second set of outputs, said second set of outputs reflecting output produced by operation of a reference system in response to application of the same set of inputs; and a validation processing block configured to compare the first and second sets of outputs to validate the system under test as an equivalent to the reference system.

In an embodiment, a method comprises: applying a set of inputs to a system under test; receiving from said system under test a first set of outputs produced by operation of the system under test in response to application of the set of inputs; storing the first set of outputs; storing a second set of outputs, said second set of outputs reflecting output produced by operation of a reference system in response to application of the same set of inputs; and comparing the first and second sets of outputs to validate the system under test as an equivalent to the reference system.

DETAILED DESCRIPTION OF THE DRAWINGS

Updating of the software components of an SoC is a conventional activity. Here, the phrase “software components” is understood to mean and refer to any algorithmic-based process executed by the SoC including, without limitation, firmware, FPGA code, software, microcode, driver code, application code, operating system code, and the like.

Regression testing is an important element in the course of software component improvement and evolution. Regression testing is performed to give a level of assurance that already working systems are not compromised by any unanticipated adverse effects from planned innovative changes and fixes. Embodiments described herein focus on implementation of a method and system for a generic, relatively automated means of regression testing that can be made a ready attribute of a software system, and in particular the software system of an SoC. The method and system are further applicable to testing a software system of an SoC to ensure proper operation in a test mode, such as built-in-self-test (GIST) operation. The method and system are further applicable to testing an overall system in the context of making a replacement of a reference system with a new system.

Reference is now made toFIG. 2. An SoC10, of the type shown inFIG. 1, is functionally enhanced over prior art SoC implementations to include an on-board validation engine12. The validation engine12would typically be included in the firmware of the SoC10. In an alternative implementation, the validation engine12is provided within the SoC10using a micro-controller device. In yet another implementation, the validation engine12is provided using a software driver of the SoC10that is specific to the validation purpose. The validation engine12may, for example, comprise a component of a BIST or other test mode function for the SoC10.

In the scenario of an algorithm update, the validation engine12functions to implement a validation process which seeks to determine whether a new version (test) algorithm for an SoC software component can be used by the SoC in place of an existing (reference) algorithm. In the scenario of a test mode validation, the validation engine12functions to implement a validation process which seeks to determine whether the current version (test) algorithm is functioning properly in comparison to the functionality of a reference algorithm.

A number of input settings are provided in accordance with a Design of Experiments (DoE) methodology, and both the reference and test software components are executed in response to those input settings. Numerical output from each execution of the software components is collected and processed to tabulate means and variances. A table of DoE effects is generated from the tabulated means and variances as intervals to the desired level of confidence concerning whether the software component provided by the test algorithm can be used in place of, or operates equivalently to, a software component provided by the reference algorithm. A validation determination with respect to the test algorithm is made based on a selected confidence interval (CI). If the CI is approximately centered on zero, the validation determination is that there is no significant difference between the test algorithm and the reference algorithm (and thus the test algorithm can be used by the SoC in place of the reference algorithm). However, if the CI is not approximately centered on zero, the validation determination is there is a significant difference between the test algorithm and the reference algorithm (and thus the test algorithm cannot be used by the SoC in place of the reference algorithm). The CI value may be output as quantitative value indicative of the degree of difference between the test algorithm and the reference algorithm.

Reference is now made toFIG. 3which illustrates a functional block diagram of the validation engine12ofFIG. 2. The validation engine12is a wrapping control algorithm over two software components: the reference algorithm and the test algorithm.

The validation engine12includes an algorithm block14. The algorithm block14stores both the reference algorithm16and the test algorithm18. In another implementation, the testing of the reference algorithm16as described herein is a historical action completed in the past with the resulting test data stored, and thus the algorithm block14includes just the test algorithm18.

The validation engine12further includes a test block20. The test block20comprises an execution block22and an output block24. The execution block20functions to execute an algorithm selected from the algorithm block14through algorithm application programming interface (API)26. Execution of the selected algorithm comprises applying certain input data to the selected algorithm, having the selected algorithm process the input data, and further having the selected algorithm generate output data. The input data comprise certain reference Design of Experiments (DoE) treatments as a parameter set. A plurality of parameter sets28(1)-28(M) are loaded through a test interface30into the execution block22.

Reference is now made toFIG. 4which illustrates a flow diagram for operation of the validation engine12. The reference algorithm is selected in step50and loaded in step52for execution by the execution block20. Setup is initialized in step54for a parameter set28. The reference algorithm is then executed in step56with the parameter set from the step54initialization. Data output from the execution of the reference algorithm is then formatted in step58and saved in step60. The steps54-60are then repeated62as required so that each parameter set28is evaluated and corresponding output data is collected.

Again, the testing of the reference algorithm may comprise a historical action completed in the past. The output data from such testing is formatted in step58and saved in step60for use in making a validation determination with respect to the test algorithm as will now be described.

Next, the test algorithm is selected in step70and loaded in step72for execution by the execution block20. Setup is initialized in step74for a parameter set28. The test algorithm is then executed in step76with the parameter set from the step74initialization. Data output from the execution of the test algorithm is then formatted in step78and saved in step80. The steps74-80are then repeated82as required so that each parameter set28is evaluated and corresponding output data is collected.

It will be noted that the same plurality of parameter sets28(1)-28(M) are used in the execution of the reference algorithm and in the execution of the test algorithm.

Reference is once again made toFIG. 3. The execution block22of the validation engine12includes input data storage for storing the plurality of parameter sets28(1)-28(M). This input data storage need not be separate from the volatile/non-volatile memory of the SoC. The output block24of the validation engine12produces output data32(1)-32(M), corresponding to execution of the parameter sets28(1)-28(M) by the selected algorithm, and collects that data in a dataset34which contains all data generated by execution of the algorithm in response to the plurality of parameter sets28(1)-28(M). Thus, there is a first output dataset34(R) produced containing the output data resulting from execution of the reference algorithm. Again, the first output dataset34(R) may comprise historical data. There is also a second output dataset34(T) produced containing the output data resulting from execution of the test algorithm. The output datasets34are stored in a data store36as DoE formatted data38. This output data store need not be separate from the volatile/non-volatile memory of the SoC.

In the event that each parameter set includes multiple samples for each parameter, the algorithm is executed on those samples and the output data32from execution by the execution block22will be tabulated by the execution block22to calculate mean and variance data which will comprise the output datasets34.

Referring again toFIG. 4, after the output data has been collected from execution of the parameter sets by both the reference algorithm and the test algorithm, DoE processing is performed in step90and DoE analysis for making the validation decision is performed in step92.

With reference once again toFIG. 3, the validation engine12still further includes a DoE processing block40which processes the DoE formatted data38and performs the operations of steps90and92ofFIG. 4. The DoE processing block40functions to generate a table of DoE effects from the DoE formatted data38(for example, from the tabulated means and variances) as intervals to the desired level of confidence (reference42; step90ofFIG. 4). This table data is then processed by the DoE processing block40to make a validation decision (reference44) based on a selected confidence interval (CI) (step92ofFIG. 4). The validation results are provided at validation output46. The validation output data may comprise, for example, pass/fail information for the test algorithm (i.e., “pass” if the test algorithm is can be used by the SoC in place of the reference algorithm, and “fail” if not). The validation output data in such an implementation may function as a control signal which drives SoC10acceptance of a proposed test algorithm. If the validation output46is “fail”, the SoC10will continue to use the reference algorithm. However, the validation output46is “pass”, the SoC10will replace the reference algorithm with the test algorithm.

The validation decision (reference44) based on a selected confidence interval (CI) may comprise an evaluation as to whether the CI is approximately centered on zero, which would indicate that there is no significant difference between the test algorithm and the reference algorithm. Such a CI suggests that the test algorithm can be used by the SoC10in place of the reference algorithm. A CI other than approximately centered on zero, however, would indicate that there is a significant difference between the test algorithm and the reference algorithm. In such case, the test algorithm should not be used by the SoC10in place of the reference algorithm.

In an alternative implementation, the validation output46may comprise the CI value itself as a quantitative value indicative of the degree of difference between the test algorithm and the reference algorithm.

The validation engine12may be installed within the SoC10and function in test mode, such as a BIST mode, to evaluate test algorithm in comparison to stored (historical) DoE information relating to the reference algorithm. Such a test mode may be initiated at start-up of the SoC10, or initiated on a periodic basis. If the test mode execution indicates a negative validation decision (reference144), with a validation output146of “fail”, the SoC10may be disabled from operation, or generate an error report, or be enabled for limited operation. However, if the validation output146is “pass”, the SoC10will be enabled for full operation upon exit from test mode.

The processing performed by the validation engine12may be summarized as follows:

1) Generate output data from execution of the reference algorithma. Initialize the system for execution of reference algorithmi. Set statistical significance level and other setup parameters and configurations1. Do N times:a. apply each input parameter set of the DoEb. generate reference algorithm output value

2) Generate output data from execution of the test algorithma. Initialize the system for execution of test algorithmi. Set statistical significance level and other setup parameters and configurations1. Do N times:a. apply each input parameter set of the DoEb. generate test algorithm output value

3) Tabulate mean and variance of output based on N samples for each parameter of the parameter set

4) Generate a table of DoE effectsa. As intervals to the desired level of confidence (e.g., 99%), using means and variances as generated

5) Generate validation: go/no-go based on the selected confident interval (CI)a. i.e., “Validation DoE fails to find significant difference” if the CI is approximately centered on zero, otherwiseb. “Validation DoE finds significant difference”i. And yields the CI as a quantitative value of the difference

A more specific summary of the configuration of and processing performed by the validation engine12is as follows:

1) The SoC device contains:a. Parameter set(s) stored in memory that are pre-configured with associated validation valuesb. Memory storage partitioned to store datai. Reference algorithm output data partition1. Mean and standard deviation for each DoE treatment pre-generated by a reference SoC for each applicable sub-system, using the same on-board Firmware Diagnostic Algorithm (FDA)ii. Test algorithm output data partitioniii. Partition for validation report byte(s)/registers to identify DoE effect status

2) Initialize Systema. Load mean and standard deviation of pre-generated reference algorithm DoE treatmentsi. These data are specific to a selected target sub-system for which the test algorithm is being evaluatedii. These are loaded as a default configuration of parameters

4) Using the reference algorithm and test algorithm means and standard deviations, generate a table of DoE effectsa. Store the DoE effects as CI ranges in memory to a predetermined level of significance (e.g., 99%)

5) Generate validation result (for example, go/no-go) based on predetermined significance level from DoE effectsa. e.g., “Validation DoE fails to find significant difference between test and reference algorithm” if the CI is approximately centered on zero, otherwiseb. “Validation DoE finds significant difference”

6) Repeat starting at step2for another sub-system as required

With reference toFIG. 4, the steps50-62are performed in the above summarization in step1as an initialization operation. As an alternative, and as described above, the steps50-62for the reference algorithm may instead be performed in the context of evaluating the test algorithm.

Reference is now made toFIG. 5. There may occur instances when a given reference system is to be replaced by a new system, or where a new system must be evaluated in comparison to the reference system. In connection with such a system replacement or evaluation, a validation process may be performed by a validation engine112to ensure that the new system functions in a manner which is consistent with the reference system (with respect to common functionalities). A number of input settings are provided in accordance with a Design of Experiments (DoE) methodology, and both the reference and new systems are operated in response to those input settings. Numerical output from each operation of the systems is collected and processed to tabulate means and variances. A table of DoE effects is generated from the tabulated means and variances as intervals to the desired level of confidence concerning whether the new system can be used in place of the reference system. A validation determination with respect to the new system is made based on a selected confidence interval (CI). If the CI is approximately centered on zero, the validation determination is that there is no significant difference between the new system and the reference system (and thus the new system can be used in place of the reference system). However, if the CI is not approximately centered on zero, the validation determination is there is a significant difference between the new system and the reference system (and thus the new system cannot be used in place of the reference system). The CI value may be output as quantitative value indicative of the degree of difference between the new system and the reference system.

FIG. 5illustrates a functional block diagram of the validation engine112acting on a test bed114. In one implementation, the test bed includes both the reference system116and the new system118. In another implementation, the testing of the reference system116as described herein is a historical action completed in the past with the resulting test data stored, and thus the test bed includes just the new system118.

The validation engine112includes a test block120. The test block120comprises an execution block122and an output block124. The execution block120functions to operate a system selected from the test bed114through a system interface126. Operation of the selected system comprises applying certain input data to the selected system, having the selected system process or operate on the input data, and further having the selected system generate output data. The input data comprise certain reference Design of Experiments (DoE) treatments as a parameter set. A plurality of parameter sets128(1)-128(M) are loaded through a test interface130into the execution block122.

Reference is now made toFIG. 6which illustrates a flow diagram for operation of the validation engine112. The reference system is selected in step152for operation under the control of the execution block120. Setup is initialized in step154for a parameter set128. The reference system is then operated in step156with the parameter set from the step154initialization. Data output from the reference system in response to the test input is then formatted in step158and saved in step160. The steps154-160are then repeated162as required so that each parameter set128is evaluated and corresponding output data is collected.

Again, the testing of the reference system may comprise a historical action completed in the past. The output data from such testing is formatted in step158and saved in step160for use in making a validation determination with respect to the new system as will now be described.

Next, the new system is selected in step172for operation under the control of the execution block120. This selection could be made, for example, in connection with a test mode or built-in-self-test (GIST) functionality supported by the system. Setup is initialized in step174for a parameter set128. The new system is then operated in step176with the parameter set from the step174initialization. Data output from the new system in response to the test input is then formatted in step178and saved in step180. The steps174-180are then repeated182as required so that each parameter set128is evaluated and corresponding output data is collected.

It will be noted that the same plurality of parameter sets128(1)-128(M) are used in the test operation of the existing (reference) system and in the test operation of the new system.

Reference is once again made toFIG. 5. The execution block122of the validation engine112includes input data storage for storing the plurality of parameter sets128(1)-128(M). The output block124of the validation engine112produces output data132(1)-132(M), corresponding to execution of the parameter sets128(1)-128(M) by the selected system, and collects that data in a dataset134which contains all data generated by operation of the system in response to the plurality of parameter sets128(1)-128(M). Thus, there is a first output dataset134(R) produced containing the output data resulting from execution of the reference system. Again, the first output dataset134(R) may comprise historical data. There is also a second output dataset134(N) produced containing the output data resulting from execution of the new system. The output datasets134are stored in a data store136as DoE formatted data138.

In the event that each parameter set includes multiple samples for each parameter, the systems are operated in response to those samples and the output data132from execution by the execution block122will be tabulated by the execution block122to calculate mean and variance data which will comprise the output datasets134.

Referring again toFIG. 6, after the output data has been collected from operation of the system(s) in response to the parameter sets, DoE processing is performed in step190and DoE analysis for making the validation decision is performed in step192.

With reference once again toFIG. 5, the validation engine112still further includes a DoE processing block140which processes the DoE formatted data138and performs the operations of steps190and192ofFIG. 6. The DoE processing block140functions to generate a table of DoE effects from the DoE formatted data138(for example, from the tabulated means and variances) as intervals to the desired level of confidence (reference142; step190ofFIG. 6). This table data is then processed by the DoE processing block140to make a validation decision (reference144) based on a selected confidence interval (CI) (step192ofFIG. 6). The validation results are provided at validation output146. The validation output data may comprise, for example, pass/fail information for the new system (i.e., “pass” if the operation of the new system is consistent with that of the reference system, and “fail” if not). The validation output data in such an implementation may function as a control signal which enables of the new system for operation outside of a test mode (i.e., enabled for normal operation). If the validation output146is “fail”, the new system may be disabled from operation, or generate an error report, or be enabled for limited operation. However, the validation output146is “pass”, the new system will be enabled for full operation.

The validation decision (reference144) based on a selected confidence interval (CI) may comprise an evaluation as to whether the CI is approximately centered on zero, which would indicate that there is no significant difference between the operation of the new system and the operation of the reference system. Such a CI suggests that the new system can be used in place of, or is functionally and operationally equivalent to, the reference system. A CI other than approximately centered on zero, however, would indicate that there is a significant difference between operation of the new system and the reference system. In such case, the new system should not be used in place of the reference system.

In an alternative implementation, the validation output146may comprise the CI value itself as a quantitative value indicative of the degree of difference between the new system and the reference system.

The validation engine112may, in an embodiment, be installed within the new system itself and function in test mode, such as a BIST mode, to evaluate new system operation in comparison to stored (historical) DoE information relating to the reference system. Such a test mode may be initiated at start-up of the system, or initiated on a periodic basis. If the test mode execution indicates a negative validation decision (reference144), with a validation output146of “fail”, the new system may be disabled from operation, or generate an error report, or be enabled for limited operation. However, if the validation output146is “pass”, the new system will be enabled for full operation upon exit from test mode.