Electronic control unit testing optimization

A computer-implemented method for implementing electronic control unit (ECU) testing optimization includes capturing, within a neural network model, input-output relationships of a plurality of ECUs operatively coupled to a controller area network (CAN) bus within a CAN bus framework, including generating the neural network model by pruning a fully-connected neural network model based on comparisons of maximum values of neuron weights to a threshold, reducing signal connections of a plurality of collected input signals and a plurality of collected output signals based on connection weight importance, ranking importance of the plurality of collected input signals based on the neural network model, generating, based on the ranking, a test case execution sequence for testing a system including the plurality of ECUs to identify flaws in the system, and initiating the test case execution sequence for testing the system.

BACKGROUND

Technical Field

The present invention relates to electronic control units (ECUs), and more particularly to ECU testing optimization.

Description of the Related Art

Modern vehicles (e.g., automobiles) can include tens of electronic control units (ECUs). More technologically advanced vehicles such as, e.g., autonomous driving vehicles, may have at least ECUs. ECUs are responsible for much of the functionality of automobiles, including powertrain, vehicle safety, comfort, infotainment and telematics. The software within ECUs is becoming increasingly complex as more functionality is added into ECUs. Accordingly, because ECUs are responsible for a vast array of vehicle functionality, extensive and thorough testing can be needed to ensure proper operation.

SUMMARY

According to an aspect of the present invention, a computer-implemented method is provided for electronic control unit (ECU) testing optimization. The computer-implemented method includes capturing, within a neural network model, input-output relationships of a plurality of ECUs operatively coupled to a controller area network (CAN) bus within a CAN bus framework, including generating the neural network model by pruning a fully-connected neural network model based on comparisons of maximum values of neuron weights to a threshold, reducing signal connections of a plurality of collected input signals and a plurality of collected output signals based on connection weight importance, ranking importance of the plurality of collected input signals based on the neural network model, generating, based on the ranking, a test case execution sequence for testing a system including the plurality of ECUs to identify flaws in the system, and initiating the test case execution sequence for testing the system. The test case execution sequence executes a first test case corresponding to a first collected input signal earlier than a second test case corresponding to a second collected input signal having a lower ranking than the first collected input signal.

According to another aspect of the present invention, a system is provided for electronic control unit (ECU) testing optimization. The system includes a memory device storing program code and at least one processor device operatively coupled to the memory device. The at least one processor device is configured to execute program code stored on the memory device to capture, within a neural network model, input-output relationships of a plurality of ECUs operatively coupled to a controller area network (CAN) bus within a CAN bus framework by generating the neural network model by pruning a fully-connected neural network model based on comparisons of maximum values of neuron weights to a threshold, reduce signal connections of a plurality of collected input signals and a plurality of collected output signals based on connection weight importance, rank importance of the plurality of collected input signals based on the neural network model, generate, based on the ranking, a test case execution sequence for testing a system including the plurality of ECUs to identify flaws in the system, and initiate the test case execution sequence for testing the system. The test case execution sequence executes a first test case corresponding to a first collected input signal earlier than a second test case corresponding to a second collected input signal having a lower ranking than the first collected input signal.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

In accordance with embodiments of the present invention, systems and methods are provided for electronic control unit (ECU) testing optimization by automatically selecting an optimal set of test cases based on an importance of each individual test case and generating a sequence for execution. More specifically, the optimization can be based on an input-output analysis of ECUs based on neural network modeling for understanding nonlinear relationships that discover the linkage between the input signals into the ECUs and the output measurements. If a combination of input signals does not produce different output measurements, then those test cases with the combination of input signals will be ignored because they fail to generate different output measurement and reveal any new insights. The embodiments described herein can produce equivalent segments of continuous input signals so that any one random sample within the segment will be sufficient to represent this segment for functionality testing. Therefore, the embodiments described herein can produce and select an optimal set of test cases by modeling the input-output relationship for discrete input signals and discovering the equivalent segments for continuous signals. Furthermore, the embodiments described herein can be used to generate the execution sequence based on the importance ranking of input signals. The embodiments described herein can be used to reveal ECU faults at early stages while also reducing the amount of time and/or computational resources spent on testing.

Referring now in detail to the figures in which like numerals represent the same or similar elements and initially toFIG.1, a high-level diagram is provided illustrating an exemplary environment100. The environment100includes a vehicle110. For example, the vehicle110can be an autonomous or self-driving vehicle. In this illustrative embodiment, the vehicle110is an automobile and, more particularly, a car. However, the vehicle110can be any suitable automobile or other type of vehicle in accordance with the embodiments described herein.

As further shown, the environment100includes an electronic control unit (ECU) testing mechanism120. In this illustrative embodiment, the ECU testing mechanism120is located within the vehicle110. However, in other embodiments, the ECU testing mechanism120can be located outside of the vehicle110and in communication with the vehicle110over a network.

In one embodiment, the ECU testing mechanism120is a hardware-in-the-loop (HiL) ECU testing mechanism implementing HiL simulation. In general, HiL simulation refers to a technique in which signals from a controller are fed into to a test system including hardware and/or software that replaces a machine or physical part of a system (“plant”), and provides virtual stimuli that emulates the response of the plant. The test system and the plant can be connected through components such as, e.g., actuators and sensors. For example, in accordance with the embodiments described herein, an HiL ECU testing mechanism replaces an engine with a test system including hardware and/or software that interacts with the ECU. HiL simulation can provide benefits over testing using the plant itself by increasing the scope of testing, providing a safer and lower risk testing environment, enabling compliance with time-to-market restrictions, reducing cost and risk, etc.

As will be described in further detail below with reference toFIG.2, the ECU testing mechanism120can include a plurality of components that use the real ECU hardware in the testing environment and apply the vehicle process model. In this manner, the real ECU hardware is tested for its functionality while a correct vehicle model interacts with the ECU hardware for ease of testing.

Referring now toFIG.2, a high-level overview of an electronic control unit (ECU) testing mechanism200is illustratively depicted in accordance with one embodiment of the present invention. The ECU testing mechanism200can be illustratively implemented within the environment described above with reference toFIG.1(e.g., ECU testing mechanism120).

More specifically, in one embodiment, the ECU testing mechanism200is a hardware-in-the-loop (HiL) ECU testing mechanism, as described in further detail above with reference toFIG.1. For example, as shown, ECU testing mechanism200can include a plurality of components including vehicle sensors210, signal conditioner220, testing procedure environment230, output drivers240, vehicle actuators250, and vehicle process model260.

The vehicle sensors210are in communication with the vehicle process model260. The vehicle process model260simulates the real vehicle process for certain ECU functionalities and generates signals output for the vehicle sensors210. The sensor signals may include multiple variables corresponding to different sensors.

Once the signals are generated from the vehicle sensors210, the signals can then be preprocessed by the signal preprocessor220to generate preprocessed signals. Preprocessing can include, but is not limited to, signal conditioning, noise removal, outlier detection, outlier removal, etc.

Each preprocessed signal is then passed to the testing procedure environment230. The testing procedure environment230is configured to choose which sensor signals and which range and/or value to use to test an ECU or a group of ECUs working together. The testing procedure environment230can also generate the sequence of execution order that will reveal issues or faults as early as possible. Further details regarding the testing procedure environment230will be described below with reference toFIG.3.

The output driver240sare configured to gather the ECU output and pass the ECU output along to the vehicle actuators250. The vehicle actuators250are configured to receive the feedback from the ECUs about the signals generated from the vehicle process model260. Based on the feedback, the vehicle actuators250can generate signals to cause the vehicle process model260to make certain changes and/or measurements. The changes/measurements can then be compared against the desired output to check the correctness of the test.

Referring now toFIG.3, a schematic of a testing procedure environment300is illustratively depicted in accordance with one embodiment of the present invention. The testing procedure environment can be illustratively implemented within the system described above with reference toFIG.2(e.g., testing procedure environment230).

As shown, the testing procedure environment300includes controller area network (CAN) bus system310including a plurality of ECU's312-1through312-5in communication with a CAN bus314. The number of ECUs shown in the CAN bus framework310is for illustrative purposes only and should not be considered limiting. The input and outputs signal of the plurality of ECUs312-1through312-5are directly linked to the CAN bus314.

The plurality of ECUs312-1through312-5can communicate with each other without complex dedicated wiring in between through the CAN bus system310. More specifically, a CAN-connected ECU can receive messages transmitted from any of the ECUs in the CAN bus system310. Each individual ECU will decide relevance and act accordingly. If a signal is intended for a particular ECU, then only the particular ECU will respond to the sending signal. Such a design allows easy modification and inclusion of additional nodes.

When it is desired to test certain functionality of the vehicle (e.g., the visual recognition and response subsystem in an autonomous vehicle), multiple ECUs and one or more sensors320can be linked through CAN bus314. For example, the one or more sensors320can include one or more cameras. A central gateway (CGW) component330is also linked so that signals can be directed to a computing device340configured to perform inspection.

Some ECUs have discrete values and other ECUs have continuous values. The procedures that will be described in further detail below with reference toFIG.4andFIG.5can generate a subset of testing cases for each individual ECU. More specifically,FIG.4describes the procedure for the ECUs with discrete values andFIG.5explains the procedure for the ECUs with continuous values.

Referring now toFIG.4, a block/flow diagram is provided illustrating a system/method400for implementing electronic control unit (ECU) testing optimization. For example, the system/method400can be implemented by a testing procedure environment, such as the testing procedure environment230ofFIG.2. The system/method400can be used to optimize testing of ECUs in, e.g., a vehicle, as described above with reference toFIGS.1-3. For example, the vehicle can be an autonomous or self-driving vehicle.

At block410, input-output relationships of a plurality of electronic control units (ECUs) is captured within a neural network model. The plurality of ECUs can be operatively coupled to a controller area network (CAN) bus within a CAN bus framework. For example, capturing the input-out relationships can include generating the neural network model can include collecting a plurality of input signals and a plurality of output signals. Furthermore, signal connections of the plurality of input signals and the plurality of output signals can be reduced based on connection weight importance at block410.

More specifically, generating the neural network model can including obtaining a fully-connected neural network model corresponding to a deep feed-forward neural network model including an input layer, at least one hidden layer and an output layer. In this embodiment, the number of hidden layers can be set by users. In practice, two hidden layers may be sufficient to quantify the input-output relationship of the plurality of ECUs. Since ECUs are physical systems and exhibit non-linear input-output relationships, a linear model may not be adequate to characterize the input-output relationship of the plurality of ECUs. The training of the deep feed-forward neural network can be performed by a backpropagation gradient descent technique to update neuron weights so that the input-output relationship can be established with reasonable accuracy.

The input-output relationships of the plurality of ECUs can be used to identify the subset of test cases that are representative of the relationship, which should be smaller than the exhaustive combinatorial number. Thus, in one embodiment, generating the neural network model can further include pruning the fully-connected neural network model based on comparisons of maximum values of neuron weights to a threshold. The pruning of the neural network model makes the neural network sparse while maintaining a certain level of accuracy, thereby enabling selection of a small subset of unique test cases with maximum coverage. Accordingly, the neural network model can be obtained to reduce the number of test cases without losing any important or necessary testing input space.

As an illustrative example, for a deep feed-forward neural network model including two hidden layers, the following procedure outlines a pruning process that can generate a neural network model that is sparse and maintains accuracy at a certain predefined level without loss of generalization in accordance with an illustrative embodiment:1. Train the neural network model until a predetermined accuracy rate is met and for each correctly classified example the condition is satisfied: max absolute sample error <, whereis a user defined parameter corresponding to a target classification rate.2. For each wlm, each dhmand each νpm, and if

maxh⁢maxp⁢vpm⁢dhm<ϵ
then remove νpmand dhm, where wlmrepresents the neuron weight from the input layer of the neural network model to the first hidden layer of the neural network model, dhmrepresents the neuron weight from the first hidden layer to the second hidden layer of the neural network model, νpmrepresents the neuron weight from the second hidden layer to the output layer, the subscripts l, h and p represent weight indices, the superscript m marks the variables as the neuron weights in the neural network model, and ∈ is a pre-defined threshold having a value between 0 and 1, inclusive. The value ∈ controls the sparseness and coverage of the final pruned model. More specifically, the higher the value of E, the sparser and less accurate the network model that is generated.3. For each νpmand dhm, if max|νpm|<∈, remove νpm; if max|dhm|<∈, remove dhm.4. If no weights satisfy conditions (2) & (3), then remove wlmwith the smallest|wlmνpmdhm|5. Retrain neural network model; if the classification rates fall below the target, stop; otherwise go to step 2.

At block420, importance of the plurality of input signals is ranked based on the neural network model. As will be described in further detail below, the ranking of the plurality of input signals by importance can be used to generate a test case execution sequence for executing test cases each corresponding to an input signal. More specifically, test cases corresponding to high-ranking input signals may be executed earlier than test cases corresponding to low-ranking input signals.

The following procedure outlines a process that can be used to rank input signal importance based on the neural network model in accordance with an illustrative embodiment:1. Given the (parsed) neural network model, for each input to hidden layer weight wlm, compute a product ΣhΣp|wlmνpmdhm|.2. Sort input nodes in order of the product in ascending order to generate a sorted list.3. Assign ranks to the inputs according to positions within the sorted list.

As mentioned above, the neural network model generated at block410can be used to reduce the number of test cases with discrete input signals. To further reduce the number of test cases for continuous input signals, at block430, a segmentation for continuous input signals can be produced. More specifically, an input range can be segmented into equivalent groups. For each group, any test case for a specific value within the group can be denoted as a representative test case selected to reduce the total number of test cases. Accordingly, any one random sample within a segment can be sufficient to represent the segment for functionality testing.

As an illustrative example, consider a set of ECUs including ECU1and ECU2that produce multiple selections of discrete values. For example, ECU1can have values of on and off, and ECU2can have values of 10, 20 and 30. Each combination of different values for individual ECUs can generate an output value. It is possible that the combination of two ECUs with ECU1of value on and ECU2of value 10 and 20 can produce very similar output signal values. This indicates that testing cases with 10 and 20 for ECU2are redundant. Therefore, we can safely select either 10 or 20 for the ECU2when ECU1is of value on. However, it is time consuming and labor intensive if this relationship is discovered in a manual approach.FIG.4describes an automated procedure to discover this relationship. By going through many different combinations, the system/method ofFIG.4can collect this set of input and output value pairs. Then, this set of input output value pairs can be used for training the neural network model described in block410. Since a subset of testing case combinations is selected and ranked while at the same time keeping the coverage of testing, block420can prune the neural network model by removing the connections between different neurons. The result will make the neural network model sparse without sacrificing the accuracy.

The system/method ofFIG.4describes a procedure for discrete input signals. However, ECUs with continuous values may require different approach to further reduce the testing cases. An exemplary procedure that can be used to produce the equivalent segmentation for continuous input signals at block430will now be described in further detail below with reference toFIG.5.

Referring now toFIG.5, a block/flow diagram is provided illustrating a system/method for producing an equivalent segmentation for continuous input signals.

At block510, an input signal variable (node) xicorresponding to an input signal is selected from a neural network model. More specifically, the neural network model can be a neural network model modeling a non-linear input-output relationship of a plurality of ECUs. Since there are multiple input signals, the input signal variable is selected one at a time.

At block520, an output signal variable (node) corresponding to an output signal is selected from the neural network model. Since there are multiple output signals, the output signal variable is selected one at a time.

At block530, a clustering analysis is performed to cluster the input signal into a plurality of segments. Each of the plurality of segments can be equivalent in terms of testing the relationship between the input signal variable and the output signal variable. More specifically, for the given input signal variable xiand other remaining input signal variables xk, if both xiand xkproduce respective output signal variables yiand ykand the difference in yiand ykis less than a threshold (e.g., |yi−yk|<μ where μ is the threshold), then xiand xkare determined as belonging to the same cluster.

At block540, an activation value of each segment is represented as an average of activation values within the segment. The activation value determines a representative test case. Therefore, instead of testing the whole range, only a handful of representative test cases need to be selected while maintaining coverage and accuracy.

At block550, it is determined if all output signal variables have been enumerated through. If not, the procedure will revert back to block520to select another output signal variable from the neural network model. If so, the procedure will revert back to block510to select another input signal variable from the neural network model.

In an exemplary use case, suppose that ECU3can have any value between 10 and 100 representing a sensor reading for pressure. It is not possible to test all values within this range. It is highly possible that any value between 10 and 30 will produce a very close output value, and so on for other ranges. Therefore, any value between 10 and 30 may be sufficient to cover the testing range of 10 and 30. In this approach, the procedure described inFIG.5can achieve the goal of reducing the testing cases for the ECUs of continuous values. For example, starting with ECU3and one output neuron corresponding to one measurement, given a value of 10 for ECU3, the value of output measurement is recorded. Then, a next value is experimented with, e.g. 10.5, in an incremental of 0.5, and the output value is measured again. If the two output measurements are within a pre-defined threshold, then the procedure will make a segmentation of between 10 and 10.5 so that any value between 10 and 10.5 will generate the same effect for that particular output measurement. This procedure will continue and enlarge the segment range until the whole range is clustered into non-overlapping segments. Any value inside a segment is a representative for that segment. Therefore, the system/method ofFIG.5can reduce the required testing cases for ECUs with continuous input values.

Referring back toFIG.4, at block440, a test case execution sequence is generated and initiated based on the ranking. The test case execution sequence is used to test a system including the plurality of ECUs to identify flaws in the system. More specifically, a subset of relevant input signal variables sufficient to test the functionality between the input and the given output can be obtained for a given output measurement. Furthermore, a subset of representative test inputs for continuous signals can be generated based on the clustering analysis, thereby reducing the number of test cases. The ranking performed at block420offers guidance regarding how to generate the test case execution sequence based on importance. Therefore, instead of testing the whole range, only a subset of representative test cases needs to be selected while maintaining coverage and accuracy.

The system/method400automatically selects an optimal set of test cases based on the importance of each individual test case and generate a sequence for execution. If a certain combination of input signals does not produce different output measurements, then those test cases with the combination of input signals will be ignored because they do not generate different output measurement and reveal any new insights. The system/method400can produce equivalent segments of continuous input signals so that any one random sample within a given segment will be sufficient to represent the given segment for functionality testing. Therefore, system/method400produces an optimal set of testing cases by modeling the input-output relationship for discrete input signals and discovering the equivalent segments for continuous signals, and generates a test case execution sequence based on the importance ranking of input signals.

Referring now toFIG.6, an exemplary computer system600is shown which may represent a server or a network device, in accordance with an embodiment of the present invention. The computer system600includes at least one processor (CPU)605operatively coupled to other components via a system bus602. A cache606, a Read Only Memory (ROM)608, a Random-Access Memory (RAM)610, an input/output (I/O) adapter620, a sound adapter630, a network adapter690, a user interface adapter650, and a display adapter660, are operatively coupled to the system bus602.

A first storage device622and a second storage device629are operatively coupled to system bus602by the I/O adapter620. The storage devices622and629can be any of a disk storage device (e.g., a magnetic or optical disk storage device), a solid state magnetic device, and so forth. The storage devices622and629can be the same type of storage device or different types of storage devices.

A speaker632may be operatively coupled to system bus602by the sound adapter630. A transceiver695is operatively coupled to system bus602by network adapter690. A display device662is operatively coupled to system bus602by display adapter660.

A first user input device652, a second user input device659, and a third user input device656are operatively coupled to system bus602by user interface adapter650. The user input devices652,659, and656can be any of a sensor, a keyboard, a mouse, a keypad, a joystick, an image capture device, a motion sensing device, a power measurement device, a microphone, a device incorporating the functionality of at least two of the preceding devices, and so forth. Of course, other types of input devices can also be used, while maintaining the spirit of the present invention. The user input devices652,659, and656can be the same type of user input device or different types of user input devices. The user input devices652,659, and656are used to input and output information to and from system600.

Electronic control unit testing (ECUT) component670may be operatively coupled to system bus602. ECUT component670is configured to perform one or more of the operations described above. ECUT component670can be implemented as a standalone special purpose hardware device, or may be implemented as software stored on a storage device. In the embodiment in which ECUT component670is software-implemented, although shown as a separate component of the computer system600, ECUT component670can be stored on, e.g., the first storage device622and/or the second storage device629. Alternatively, ECUT component670can be stored on a separate storage device (not shown).