Patent Publication Number: US-2023161688-A1

Title: Intelligent test method for dynamically generating test case according to test performance of tested system

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     The present application claims priority to Chinese Patent Application No. 202111405653.4, filed on Nov. 24, 2021, the content of which is incorporated herein by reference in its entirety. 
     TECHNICAL FIELD 
     The present disclosure relates to the field of automatic driving simulation training, in particular to an intelligent test method for dynamically generating a test case according to test performance of a tested system. 
     BACKGROUND 
     Compared with the training in real scenes, simulation is more efficient and low-cost, so it is widely used in the process of training system. Case generation, as an important link, can facilitate training of various situations that the tested system may encounter in the simulation environment, and simulating unusual special situations to improve the training efficiency. In order to test the performance of the system under test in various scenarios, the existing case generation methods cover various real situations by generating a large number of cases, but this method requires higher requirements for computing power and time cost. How to dynamically generate effective cases more intelligently according to the existing performance of an automatic training system is an urgent problem to be solved. 
     SUMMARY 
     In view of the shortcomings of the prior art, the present disclosure provides an intelligent test method for dynamically generating test cases according to the test performance of a tested system. The method takes into account the performance of the trained agent in the previous test cases in the simulation, and dynamically generates the cases that are more in line with the test requirements according to the test results. 
     The purpose of the present disclosure is realized by the following technical solution: 
     An intelligent test method for dynamically generating test cases according to the test performance of a tested system comprises following steps: 
     S 1 , selecting a test template test_case_template containing N variables var. 
     S 2 , sampling values in a value range of each variable var, and forming a sampling set Q with all of the sampled values. 
     S 3 , selecting a value from a sampling set Q corresponding to each variable to form a template variable numerical sampling vector g=(s 1 , s 2 , . . . , s N ), s 1 ∈Q 1 , s 2 ∈Q 2 , . . . , s N ∈Q N , and then generating a set G_init from g. 
     S 4 , starting a k th  round of test, wherein when k=1, the set G k =G_init; traversing each template variable numerical sampling vector in the set G k , obtaining a category label(g) corresponding to each template variable numerical sampling vector g according to the test template when traversing, and obtaining a test performance data set D k  corresponding to the set G k . 
     S 5 , determining whether k is equal to 1. If k=1, training a decision tree T k  whose classification error rate is less than a preset threshold epsilon k  by the test performance data set D k , predicting the label(g) by the decision tree T k  according to any g; and putting a sample with wrong classification into a data set D k+1 . If k&gt;1, classifying and verifying the samples in the test performance data set D k  by using decision trees T 1 , T 2 , . . . , T k−1 , and carrying out weighted average on a classification result of each sample for each decision tree to calculate an error rate. If the error rate is greater than or equal to a preset threshold epsilon k−1 , training a decision tree T k  by the test performance data set D k , predicting the label(g) by the decision tree T k  according to any g, putting the samples with wrong classification into the data set D k+1  and going to S 6 . If the error rate is less than the preset threshold epsilon k−1 , ending testing the test template test_case_template; 
     S 6 , finding all areas in which the classification label of g is predicted to be the first class by T k  in a N-dimensional space where g is located according to all leaf nodes classified as a first class in T k , and taking a union of the areas, wherein the union is denoted as R k ; 
     S 7 , sampling a batch of new vectors g_new in the N-dimensional space where g is located, keeping a vector distribution density positively correlated with a distance between g_new and a boundary of R k ; forming a set G_new by g_new, letting k=k+1, if k is less than a threshold of preset rounds threshold, letting G k =G_new, going to S 4 , and starting a next round of testing; otherwise, ending testing the test template test_case_template. 
     Further, in S 4 , in order to obtain the data set D k  corresponding to the set G k , first, a copy of the test template test_case_template is generated, the var i  in the copy is replaced by s i  in g, and the replaced copy of the test template is defined as a test case and denoted as H(g); the tested system is tested with H(g) to obtain a test score score(g), and then a two-class category label(g) for the test case H(g) is provided according to the score(g), and finally, a binary group (g, label(g)) is put into the test performance data set D k . 
     Further, the step of keeping a vector distribution density positively correlated with a distance between g_new and a boundary of the area R in S 7  is specifically that the vector in G_new satisfying the following conditions: 
     (1) The closer to the boundary of R k  is, the denser the vector distribution is. 
     (2) A certain distance exists between a newly sampled vector and a tested vector. 
     Further, the distance between the newly sampled vector g_new and the tested vector g_old satisfies: 
     An absolute value of projection of g_new−g_old in the j dimension is greater than or equal to d jk , where g_old is an arbitrary vector belonging to the union of G 1 , G 2 , . . . , G k , and d jk  is the threshold of the distance between g_new and g_old in the j dimension in the k th  round. 
     Further, the category label(g) is LOW or NORMAL, the category label(g) is LOW when the test score(g) is less than a preset score threshold, and the category label(g) is NORMAL when the test score(g) is greater than or equal to the preset score threshold. 
     Further, d jk  satisfies the following condition: 
         d   jk =alpha* d   jk-1    
     where alpha is a given normal number, 0&lt;alpha&lt;1. 
     The present disclosure has the following beneficial effects: 
     In the simulation process, the present disclosure makes full use of the results of previous test cases, and by constructing a plurality of decision trees, the algorithm can accurately predict the performances of trained agents in different cases, and learn the spatial division of variables that will lead to different results in the simulation test. According to this spatial division, the case distribution characteristics of the next stage are set to improve the effectiveness of case generation and training efficiency. In addition, the whole process can be iterated online in real time, and the update of the decision tree is completed in parallel with the case test. At this time, this method can significantly improve the efficiency of simulation training, and it is universal and suitable for simulation case testing in various scenarios, and excellent performance can be obtained in many application scenarios that need virtual simulation training. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         FIG.  1    is a flow chart of an intelligent test method for dynamically generating test cases according to the test performance of a tested system. 
     
    
    
     DESCRIPTION OF EMBODIMENTS 
     The purpose and effect of the present disclosure will become more apparent from description of the present disclosure in detail according to the following drawings and preferred embodiments. It should be understood that the specific embodiments described here are only for explaining the present disclosure, but not for limiting the present disclosure. 
     The method of the present disclosure as shown in  FIG.  1    is applied to the simulation test process of autonomous driving virtual reality, so as to realize the intelligent test of dynamically generating cases according to the test performance of the tested system. In such a scenario, this embodiment specifically includes the following steps: 
     S 1 , among the cases that are prone to accidents in automatic driving, a test template case test_case_template was selected for the tested automatic driving agent vehicle to overtake the preceding vehicle. The test template contained six variables Var={A_x, A_y, A_v, B_x, B_y, B_v}, where (A_x, A_y) is the current coordinate of the tested agent vehicle with a unit of m; A_v is the current speed of the measured vehicle with a unit of km/h; (B_x, B_y) is the coordinate unit of the preceding vehicle of a unit of m; B_v is the speed of the preceding vehicle with a unit of km/h. 
     Each variable has a definite value range: A_x belongs to [2,4], A_y belongs to [0,3], B_x belongs to [1,5], B_y belongs to [4,10], A_v belongs to [30,50] and B_v belongs to [30,60]. 
     S 2 , values were sampled for 100 times in the value range for each variable var, variables were evenly distributed in the value range thereof during sampling, and a sampling set Q is formed with all the sampled values of each variable var. 
     S 3 , a value was selected from a sampling set Q corresponding to each variable; a vector g was obtained after each time of sampling, and then a set G_init={g 1 , g 2 , . . . g 100 } was generated from g, where g i  is called a template variable numerical sampling vector, and the value of the j th  dimension of g i  corresponds to the numerical sampling of the j th  variable. 
     S 4 , the k th  round of test was started. When k=1, the set G k =G_init; d k  is the threshold value of the distance between g and g_old in the k th  round, d 1 =0.1. When k&gt;1, d k =0.9d k−1 , and the preset threshold of the error rate epsilon=0.1. 
     Each template variable numerical sampling vector in the set G k  was traversed, and the category label label(g) corresponding to each template variable numerical sampling vector g was obtained according to the test template in the traversal process, and the test performance data set D k  corresponding to the set G k  was obtained. The specific operation was as follows: 
     (1) A copy of test_case_template of autopilot was generated, var i  in the copy was replaced with s i  in G. The test template copy after replacement is called as a test case and denoted as H(g). 
     (2) The tested system was tested with H(g) to get a test score score(g), and then a two-class category label(g) was provided for the test case H(g) according to the score(g). In an embodiment, the category label(g) was LOW or NORMAL. When the test score(g) was less than the preset score threshold, the category label(g) was LOW, and when the test score(g) was greater than or equal to the preset score threshold, the category label(g) was NORMAL. 
     (3) The binary group (g, label(g)) was put into the test performance data set D k . 
     In the embodiment of automatic driving, the i th  dimension numerical sampling g i   j  in g i  was used to replace the i th  template variable var i  in the test template copy, where i=1, 2, . . . , N. The test template copy after replacement is called an automatic driving test case and denoted as H(g i ), where H represents the mapping relationship from the template variable numerical sampling vector to the generated test case. The system under test was tested by the automatic driving test case H(g i ), and the test score of the agent unmanned vehicle was score(g i ). Then, a test result label label(g i ) was set for the test case H(g) according to score(g i ). The rules for setting the classification label were as follows: if score(g i ) was less than 60, then label(g i ) was set to low performance LOW, and if score(g i ) was in a range of [60,100], then label(g i ) was set as normal performance NORMAL, and the binary group (g i , label(g i )) was put into the test performance data set D k . 
     S 5 , whether k is equal to 1 was determined. If k=1, D k  was used to train a decision tree T k  whose classification error rate is less than the preset threshold epsilon k , the label(g) was predicted by the decision tree T k  according to any g, and sample with wrong classification was put into the data set D k+1 . If k&gt;1, decision trees T 1 , T 2 , . . . , T k−1  were used to classify and verify the samples in D k , and weighted average was carried out on a classification result of each sample for each decision tree to calculate an error rate. If the error rate was greater than or equal to a preset threshold epsilon k−1 , a decision tree T k  would be trained by D k , the label(g) would be predicted by T k  according to any g, the samples with wrong classification would be put into the data set D k+1  and then S 6  would be executed. If the classification error rate was less than the preset threshold epsilon k−1 , the test of the test template test_case_templat would be ended. 
     S 6 , according to all leaf nodes classified as a first class in T k , all areas in which the classification label of g was predicted to be the first class by T k  are found in the N-dimensional space where g was located, and a union of the areas was taken. The union was denoted as R k . 
     In this embodiment of automatic driving, all the leaf nodes classified as LOW in Tk are found in the N-dimensional space where G is located, and all the areas predicted by Tk as LOW by the category label of G are merged and marked as Rk. The category label of any vector outside Rk is NORMAL. 
     S 7 , a batch of new vectors g_new was sampled in the N-dimensional space where g is located, and a vector distribution density was kept positively correlated with a distance between g_new and a boundary of R k ; g_new formed a set G_new. Let k=k+1, and if k was less than a threshold of preset rounds, then G k =G_new, S 4  would be executed, and a next round of testing would be started; otherwise, the test of the test template test_case_template would be ended. 
     The vectors in the set G_new satisfied the following conditions: 
     (1) In order to make the new test cases approximate to the boundary as much as possible, the closer the R k  boundary was, the denser the vector distribution was. 
     (2) In order to reduce the repeated testing of similar cases, there was a certain distance between the newly sampled vector and the tested vector, and the following conditions were met: 
     The absolute projection value of g−g_old in the j th  dimension was greater than or equal to d jk , where g_old is an arbitrary vector belonging to the union of G 1 , G 2 , . . . , G k , and d jk  is the threshold of the distance between g and g_old in the j th  dimension in the k th  round. 
     As the number of iteration rounds increased, the accumulated vectors gradually increased. In order to reduce the amount of data calculation, d jk  was set to be a gradually decreasing value, namely: 
         d   jk =alpha* d   jk−1    
     where alpha is a given normal number, 0&lt;alpha&lt;1. 
     In an embodiment of automatic driving, alpha is 0.9. 
     The intelligent test method of the present disclosure is applied to the automatic driving scene, which can influence the algorithm logic of automatic driving case selection, so that the case that is more likely to be tested as the junction of low and normal is selected, and the training efficiency of automatic driving in the virtual training ground is increased. 
     It can be understood by those skilled in the art that the above examples are only preferred examples of the present disclosure, and are not intended to limit the present disclosure. Although the present disclosure has been described in detail with reference to the above examples, it is still possible for those skilled in the art to modify the technical solutions described in the above examples or equivalently replace some of the technical features. Modifications, equivalents and substitutions made within the spirit and principle of the present disclosure shall be included in the scope of protection of the present disclosure.