Test apparatus for performing a test on a device under test and data set filter for filtering a data set to obtain a best setting of a device under test

A test apparatus for performing a test on a device under test includes a data storage unit being configured to store sets of input data applied to the device under test during the test and to store the respective output data of the device under test, the output data being obtained from the device under test as a response to the input data including values of setting variables related to settings of the device under test and values of input variables including further information, each set of input data representing one test case; and a data processor configured to process the data stored in the data storage unit such that a best combination of setting variables of the device under test is determined for one or more combinations of the input variables to obtain an optimized setting of the device under test for the one or more combinations of the input variables.

BACKGROUND OF THE INVENTION

The present invention relates to a test apparatus for performing a test on a device under test and a data set filter for filtering a data set to obtain a best setting of the device under test. The data set filter may be part of the device under test. According to embodiments, an optimization in a test case database is shown, wherein the test case database or a data storage unit may comprise multiple sets of input data and respective output data.

Calibration/correction & FW (firmware) optimization shall find the best device settings (y) as a function of known information (x). This is a multivariate optimization problem. The usual method changes y iteratively, and measures the optimization target (g) for all values of x in each iteration. Known optimization algorithms operate on algorithmic representations of functions that shall be optimized.

Therefore, classification algorithms may calculate a function of best setting variables dependent on the input of known information (x). Deriving good results from these classification or regression algorithms involves a deep knowledge of the used algorithm to find the optimal result and not to remain in a local minimum only representing a non-optimal result. Moreover, these algorithms are computationally complex and therefore very time consuming.

SUMMARY

According to an embodiment, a test apparatus for performing a test on a device under test may have: a data storage unit configured to store sets of input data applied to the device under test during the test and to store the respective output data of the device under test, wherein the output data is acquired from the device under test as a response of the device under test to the input data, wherein the input data has values of setting variables related to settings of the device under test and values of input variables having further information, wherein each set of input data represents one test case; and a data processor configured to process the data stored in the data storage unit such that a best combination of setting variables of the device under test is determined for one or more combinations of the input variables to acquire an optimized setting of the device under test for the one or more combinations of the input variables.

Another embodiment may have a data set filter for filtering a data set to acquire a best setting of a device under test for current input variables using an optimization vector, the optimization vector representing a relation between an index of input variables and an index of a combination of setting variables indicating the best setting of the device under test.

According to another embodiment, a method for performing a test on a device under test may have the steps of: storing sets of input data applied to the device under test during the test; storing the respective output data of the device under test derived from the device under test as a response of the device under test to the input data, wherein the input data has values of variables related to settings of the device under test and values of input variables having known information, wherein each set of input data represents one test case; and processing the data stored in the data storage unit such that the best setting of the device under test for one or more combinations of the further variables is determined.

According to another embodiment, a method for filtering a data set to acquire a best setting of the device under test for current input variables using an optimization vector may have the step of: forming the optimization vector representing a relation between an index of input variables and an index of a combination of setting variables indicating the best setting of the device under test.

Another embodiment may have a non-transitory digital storage medium having a computer program stored thereon to perform the method for performing a test on a device under test, the method having the steps of: storing sets of input data applied to the device under test during the test; storing the respective output data of the device under test derived from the device under test as a response of the device under test to the input data, wherein the input data comprises values of variables related to settings of the device under test and values of input variables having known information, wherein each set of input data represents one test case; and processing the data stored in the data storage unit such that the best setting of the device under test for one or more combinations of the further variables is determined, when said computer program is run by a computer.

Another embodiment may have a non-transitory digital storage medium having a computer program stored thereon to perform the method for filtering a data set to acquire a best setting of the device under test for current input variables using an optimization vector, the method having the step of: forming the optimization vector representing a relation between an index of input variables and an index of a combination of setting variables indicating the best setting of the device under test, when said computer program is run by a computer.

Embodiments of the present invention show a test apparatus for performing a test on a device under test. The test apparatus comprises a data storage unit and a data processor. The data storage unit is configured to store sets of input data applied to the device under test during the test and to store the respective output data of the device under test. The output data may be obtained from the device under test as a response of the device under test to the input data, wherein the input data comprises values of setting variables related to settings of the device under test and values of input variables comprising further information. Each set of input data represents one test case. Moreover, the data processor is configured to process the data stored in the data storage unit such that a best combination of setting variables of the device under test is determined for one or more combinations of the input variables to obtain an optimized setting of the device under test for the one or more combinations of the input variables.

The present invention is based on the finding that it is advantageous to derive the optimal settings of the device under test directly from the database, i.e. the data storage unit, by omitting the cumbersome step of deriving a function representing the setting parameters of the device under test based on current input data. This speeds up the total optimization process and is furthermore less error prone. The proposal here is to create and store a large number of test cases in a test case database, where the optimization target has been measured for many different values of x and y. This database is then analyzed to find the optimal function y(x) and to verify the optimization result.

Therefore, if not already present, a countable number of possible values of input variables and a countable number of possible variable values of setting variables is determined, for example, by quantizing input and/or setting variables having too many potential values when operating the device under test. This may be real or integer variables, etc. Moreover, according to embodiments, the data storage unit is transformed into a new (further, second) database (table, matrix, list) where all possible combinations of input variables are applied or located along a first dimension of the database and all possible combinations of setting variables are located along a second dimension of the further database. Entries of the further database may be an index of the current test case to link the respective input variables and setting variables of a common test case and furthermore to link the original or non-quantized data of the database or the data storage unit to the quantized values of the further database.

According to further embodiments, the indices of test cases within the entries or fields of the further database are weighted based on the optimization target (g). Each input of the non-quantized values of the input variables and setting variables of a single test case result in a (unique) value of the optimization function (g) or a derivative thereof, such as for example the figure of merit (f). For each value or combination of the first dimension, the best result of the optimization function within the second dimension is determined. A position of the determined result indicates the (quantized) combination of values of setting variables representing a best setting of the device under test for each possible combination of input variables.

The filtering of the data set may be performed by a data set filter to obtain a best setting of a device under test for current input variables using an optimization vector. The optimization vector represents a relation between an index of input variables and an index of a combination of setting variables indicating the best setting of the device under test. Embodiments show that the data set filter is part of the test apparatus. Therefore, the optimization vector p may be calculated using the data processor or the test apparatus.

DETAILED DESCRIPTION OF THE INVENTION

In the following, embodiments of the invention will be described in further detail. Elements shown in the respective figures having the same or a similar functionality will have associated therewith the same reference signs.

FIG. 1shows a schematic block diagram of the test apparatus2for performing a test on a device under test. The test apparatus2comprises a data storage unit4and a data processor6. The data storage unit is configured to store sets of input data8applied to the device under test3(drawn with dashed lines since it is not part of the test apparatus) during the test and to store the respective output data10of the device under test. The output data10is obtained from the device under test as a response of the device under test to the input data8. The input data8comprises values of setting variables related to settings of the device under test and values of input variables comprising further information. Each set of input data represents one test case. Moreover, the data processor6is configured to process the data12stored in the data storage unit4such that a best combination of setting variables of the device under test is determined for one or more combinations of the input variables to obtain an optimized setting of the device under test for the one or more combinations of the input variables. The further information may comprise at least one of a stimulus waveform, an input pattern, a stimulus signal, an environment condition, or an environment status.

FIG. 2shows a schematic flow diagram with a sequence of exemplary tables202,204,206,208representing results of selected steps which may be performed by the data processor to derive the best settings11of the device under test, e.g. represented by the parameter vector16″. The best setting of the device under test may refer to a (optimal or best) combination of values of the setting variables for a current set of values of input variables. The first table202shows a short extract or a short excerpt of the data storage unit4showing four different test cases14. Moreover, the data storage unit comprises input variables x1and x28aand setting variables y1and y28b. Output data of the device under test has been omitted for simplification.

In other words, given is a table (database, data storage unit)4with K test cases14, k=1 . . . K with multiple variables.

The goal is to find the (best) values of Q variables y(k)=y1(k), . . . , yQ(k) as a function of P variables x=x1(k), . . . , xP(k) that optimize variable values g(k). Optionally, test cases with the optimum combination of x variables and y variables are selected.

Variables xpand yqcan be real-valued, integer, Boolean, categorical, or of any other suitable data type.

The user can specify whether a good value of variable g isa. small orb. large orc. deviates little from a desired value g0ord. deviates a lot from an undesired value g0.

The user can chose to find theA. best worst case orB. best average orC. best (best) case

According to embodiments, the data processor6is configured to perform a quantization of the input data8of at least one input variable8aand/or setting variable8bto form a discrete representation8′ of the input data. Therefore, the data processor may process the discrete representation8′ of the input data8such that the best setting of the device under test3for the one or more combinations of the input variables8ais determined. This is advantageous, since it forms a countable or discrete number or amount of possible value combinations of all input variables8aand/or setting variables8b.

In other words, if variables xp(k) or yq(k) take on too many values in k=1 . . . K (e.g. when they are real valued or integer), their values are quantized to Npdiscrete values for variables xpand Mqdifferent discrete values for variables yq. For Boolean or categorical variables, quantization is not necessary.
xp,1,xp,2, . . . ,xp,Npforp=1 . . .P
yq,1,yq,2, . . . ,yq,Mqforq=1 . . .Q(1)

Variables values xp(k) and yq(k) can now be represented as indices, 1≤np(k)≤Np, and 1≤mq(k)≤Mq.
np(k):xp(k)=xp,np(k)
mq(k):yq(k)=yq,mq(k)(2)

To simplify notation, from now on, xp(k) and yq(k), refer to the quantized variable values8′.

The second table204shows the storage unit with quantized values as a result of the quantization of the first table202representing the storage unit4. The quantization to integer values is exemplary and may be further applied to any other quantization pattern or quantization steps.

According to a further embodiment, all possible combinations of input variables8aand setting variables8bmay be assigned or allocated to a (unique) placeholder or value or representative, such as for example an integer value. In other words, after quantization, there are N=N1·N2· . . . ·NPdifferent combinations xnof variable values. For each test k, the combination x(k) is described by an index n(k)=1 . . . N, such that
xn(k)=x(k),  (3)
which can e.g. be computed as

Similar for variables y:

In other words, the data processor6is configured to assign a unique value to each possible combination of input variables and/or to each possible combination of setting variables. More specifically, the data processor6is configured to assign the unique value to each possible combination of input variables8ausing formula (4), where k is the current test case, n(k) is the unique value of each possible combination of input variables, np(k) is an index representing possible values of the input variable p, P is the total number of input variables p, and Niis the total number of possible values of input variable i. Additionally or alternatively, the data processor6is configured to assign the unique value to each possible combination of setting variables using formula (7) where k is the current test case, m(k) is the unique value of each possible combination of setting variables, mp(k) is an index representing possible values of the setting variable p, P is the total number of setting variables p, and Mjis the total number of possible values of setting variable j.

According to further embodiments, the data processor6may assign indices representing the test cases14to a respective combination of input variables8aof each possible combination of input variables and to a respective combination of setting variables8bof each possible combination of setting variables. The aforementioned assignment may be formed or applied in a table206such that a position along the first dimension of the table indicates a unique value18aof a current input variable8awithin all possible combinations of input variables and/or a position along the second dimension of the table206indicates a unique value18bof a current setting variable8bwithin all possible combinations of setting variables. This is advantageous, since each test case representing the used non-quantized values, is directly related to the quantized representation thereof and therefore, an output or a result of the optimization value with non-quantized values may be calculated and assigned directly to the quantized representation and is easily accessible by means of computer programming languages.

In other words, test cases for combinations of x and y shall be found. This may be applied by identifying the sets(n,m) of test cases for all combinations n=1 . . . N of variables x and all combinations m=1 . . . M of variables y.
(n,m)={k:(x(k)=xn){circumflex over ( )}(y(k)=ym)}  (9)

Note, this set (or each field) can contain no or one or multiple test cases or indices thereof, which causes no problem in the following steps. No test cases may be applied or derived, if the test does not cover all possible variable combinations, e.g. if the test is aborted before the test is completely performed. Multiple test cases may be applied or derived due to the quantization of the variables of the test cases. This may be seen in the context of a possible application scenario.

For example, a device under test may be tested using the randomly created test cases. For large data sets, it is most likely that the duration of the test, started at the end of a working day, exceeds one night or even a weekend and that therefore, the test is still running in the morning of the next working day and therefore, results are not present. The same counts if the test is interrupted during processing, e.g. due to an error in the test routine or a power outage. Nonetheless, in contrast to known methods which fill a parameter space covering the results of the test in an organized, sorted or regular manner, for example one dimension after the other, the proposed device already covers a huge variety of the parameter space. However, the coverage may be not as dense as in the known approaches. This counts for those dimensions, where the parameter space is already filled regularly by the known approaches, wherein the coverage of those areas of the variable space, which is not already regularly filled by the known approaches may be much denser. However, according to the invention, it is possible to determine dependencies or relations between all used variables of the test cases at an early stage of the test. Using one of the known approaches, dependencies on one or more of the used variables may not have been examined at the same test stage. Moreover, the proposed random test case generation provides the same results, only derived in a different order, than a deterministic approach, if the whole test is performed. Nonetheless, a huge amount of tests is aborted or interrupted during processing. In this case, the random test case generation outperforms the deterministic approach since variables of the random test case generation are nevertheless varied or have a high variation wherein in a classical nested loop for example, the variable of the outmost loop is varied comparably slowly.

Moreover, the data processor6may determine a best setting of the device under test3based on an optimization target16, which may be a function of output variables, wherein values of the output variables form the output data10. In other words, e.g. to form a quality metric such as the (processed) optimization target or figure of merit, the optimization target may weight different output variables derived from the device under test3to receive a single function which can be optimized. Based on the optimization target, the data processor may generate a standardized or processed optimization target, such that a small value of the standardized optimization target refers to a good setting of the device under test. The data processor may therefore determine the best setting11of the device under test based on the standardized optimization target. This is advantageous, since by using the standardized optimization target, the optimization targets are easily exchangeable or replaceable by other possible standardized optimization targets, since a minimum of the chosen standardized optimization target is in all cases the best case. Otherwise, by replacing a (non-standardized) optimization target, the best achievable value or limit of the optimization target is obviously also to be replaced.

In other words, to simplify optimization, it is convenient to transform variable g(k) into a figure of merit f(k) for each test that can be minimized, i.e. where small is good.

Letters a, b, c, d refer to the options for a good value of variable g described above.

According to further embodiments, the data processor6may assign a result of an optimization target16for each test case to a respective combination of input variables of each possible combination of input variables8aand to a respective combination of setting variables8bof each possible combination of setting variables, wherein, if multiple results of test cases are assigned to the same combination of input variables and the same combination of output variables, a single value20representing the multiple results is determined. The single value20representing the multiple results of the optimization target may be calculated using the maximum value of the multiple results of the optimization target, the maximum value of the multiple results of the optimization target, or an average value of the multiple results of the optimization target. Embodiments show that the assignment of the result of the optimization target may be applied in a further table. This is advantageous, since each result of the (standardized) optimization target derived using e.g. the non-quantized values of a test case14, is directly related to the respective combination of (quantized) values of input variables8aand the respective combination of (quantized) variables of setting variables8b. Therefore, an output or a result of the optimization value with non-quantized values may be calculated and assigned directly to the quantized representation and is easily accessible by means of computer programming languages.

Therefore, the data processor may assign, for each non-empty field of a table, where indices representing the test cases are stored, a result of an optimization target based on the output values referring to the one or more indices stored in the field to a corresponding field of a further table, wherein, if multiple indices are related to one field, a single value representing the multiple results of the optimization target is assigned to the corresponding field of the further table. In other words, the searched optimization function y(x)16, is essentially also a function of indices, m(n). Because there can be multiple test cases (or one or none)(n,m) (cf. table206) for each pair (n,m), the figure of merit is now consolidated to F(n,m) (cf. table208) across all test cases in set(n,m).

When(n,m) is empty, F(n,m) is set to infinity so that it does not affect minimization. Letters A, B, C refer to the options described above.

According to further embodiments, the data processor may determine an index of the best combination of setting variables for the one or more combinations of input variables within at least one test case, wherein the index indicates a unique value of the best combination of setting variables within all possible combinations of values of setting variables. This is advantageous, since similar values of input variables8amay be quantized to the same representation8′ thereof and, in the context of probably not having a complete test set, not tested with the same combinations of values of input variables. However, it is most likely, that the combination of setting variables which performs best for one test case, performs best or at least near the optimum for all further test cases related to the same quantized values of input variables.

In other words, to find the optimum function y(x), for each combination n=1 . . . N of variables x8a, the best combination μ(n)16″ across all combinations m=1 . . . M of variables y8bthat minimize the consolidated figure of merit F(n,m) shall be found. Note that a small value of F(n,m) is good.

This defines the optimum combination μ(n)16″ of indices for variable values y8bas a function of combination n of variable values x. In terms of variables x and y, the optimum function is
yμ(n)(xn)  (13)

To find the optimal test cases, the setoptof test cases that satisfy the optimality condition is given by
opt={k:m(k)=μ(n(k))}.  (14)

One way to show the effectiveness of optimization is to show the distribution of g(k) for test cases inoptcompared to all test cases.

Performing optimization, or calibration, is thus equivalent to applying a data set filter based on an expression that is true, when
m(k)=μ(n(k))  (15)

Therefore, a data set filter for filtering a data set may obtain a best setting11of a device under test for current input variables using an optimization vector16″, the optimization vector representing a relation between an index18aof input variables8aand an index18bof a combination of setting variables8bindicating the best setting of the device under test.

A schematic representation of the data set filter100is shown inFIG. 3. Therefore, input to the optimization vector may be values of input variables8a, e.g. from the data storage unit4. Therefore, a currently tested device under test, e.g. not the same but of the same type where the optimization vector was derived from, may be evaluated using the optimization vector16″. This is advantageous, since the finding of the best variable combinations has been performed beforehand and the best value combination may be obtained by only pointing on the corresponding field in the data storage unit. Therefore, and according to further embodiments, the data set filter100may be implemented in the device (the earlier device under test now in use) such that the setting variables are set, in real-time, meaning that the output of the device is obtained using the best value combination of setting variables derived from the optimization vector, based on the currently applied values of input variables. Moreover, the data set filter may be part of the test apparatus2, for example, implemented in the data processor6or derived from the data processor6. However, further embodiments show that the data processor6calculates the optimization vector μ(n) which is used by the data set filter100to determine the best setting of the device under test. In other words, the test apparatus comprises the data set filter, wherein the data processor is configured to form the optimization vector.

According to embodiments applicable test cases for all combinations of x and y are extracted, where both have been quantized to obtain a tractable number of combinations. Then, for each combination of x, the best combination of y is determined that minimizes a consolidated (e.g. worst case) optimization metric across the applicable test cases. Overall this defines the optimal function ŷ(x). Performing optimization or calibration translates to applying a filter that returns test cases where y(k)=ŷ(x(k)). To verify the effectiveness of optimization or calibration, the behavior of those test cases that satisfy the optimality criteria can be compared against all test cases, e.g. by comparing their distributions of optimization target variable g(k).

FIG. 4shows a flowchart of a method300for performing a test on a device under test3. The method300comprises a step302of storing sets of input data applied to the device under test during test, a step304of storing the respective output data of the device under test derived from the device under test as a response of the device under test to the input data, wherein the input data comprises values of variables related to settings of the device under test and values of input variables comprising known information, wherein each set of input data represents one test case, and a step306of processing the data stored in the data storage unit such that the best setting of the device under test for one or more combinations of the further variables is determined.

FIG. 5shows a schematic flowchart of a method400for filtering a data set to obtain a best setting of a device under test for current input variables using an optimization vector. The method400comprises a step402of forming the optimization vector representing a relation between an index of input variables and an index of a combination of setting variables indicating the best setting of the device under test.

Although the present invention has been described in the context of block diagrams where the blocks represent actual or logical hardware components, the present invention can also be implemented by a computer-implemented method. In the latter case, the blocks represent corresponding method steps where these steps stand for the functionalities performed by corresponding logical or physical hardware blocks.

Further embodiments of the invention relate to the following examples:

1. Optimization algorithm operating on a fixed subset of (x,y) pairs. Optionally, x and y are multiple variables, e.g. vectors. Further optionally, x and y variables can be a mix of numeric and categorical data. Therefore, numeric variables may be quantized.
2. Moreover, the calibration (of a device under test) may be implemented as a dataset filter.

A further embodiment of the inventive method is, therefore, a data carrier (or a non-transitory storage medium such as a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and/or non-transitory.

A further embodiment of the invention method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may, for example, be configured to be transferred via a data communication connection, for example, via the internet.