Patent Description:
Modern vehicles, including those capable of autonomous driving, include partially autonomous driver assistance systems, for example, lane keeping assistance, adaptive cruise-control, collision mitigation, self-parking, and the like. Some vehicles require a human driver to maintain a level of control over the vehicle, even in a fully autonomous operating mode, by, for example, keeping their hands-on the steering wheel. In order to ensure compliance with such requirements, some vehicles analyze sensed values from the steering system to automatically detect when a driver has removed his hands from the steering wheel.

<CIT> discloses a vigilance monitoring system and method of monitoring the alertness or wakefulness of a driver. The monitored parameters include cardiac, respiratory and movement parameters. Sensors are located in various locations of the driver side section to detect the vigilance of a driver. These sensors include pressure sensors embedded in the seat and pedals, and a head band for monitoring EEG, EMG and EOG signals.

<CIT> discloses a method and apparatus for controlling operation of an autonomous vehicle based on driver operation profiles. The method may include capturing, from a plurality of vehicle systems, driver-vehicle interaction data from the plurality of vehicle systems indicative of a plurality of driving characteristics of a driver of the autonomous vehicle. The method may also include determining, by a processing system, when the driver-vehicle interaction data deviates from a driver operation profile stored in the memory. Furthermore, the method may include controlling, by the processing system operation of at least one system of the autonomous vehicle in response to determining that the driver-vehicle interaction data deviates from the driver operation profile.

<CIT> discloses a system and method for determining whether a driver is holding a vehicle steering wheel. The vehicle will include an electric power steering system and may further include autonomous or semi-autonomous driving features, such as Lane Centering Control or Lane Keeping Assist. The system includes a passive detection technique which monitors steering torque and steering angle, determines a resonant frequency of oscillation of the steering system from the measured data, and compares the resonant frequency to a known steering system natural frequency to make a hands-on/off determination. If the passive technique results are below a confidence threshold, then an active technique is employed which provides a steering angle perturbation and measures the frequency response, where the perturbation signal has characteristics which are prescribed based on the results of the passive technique. A steering torque greater than a threshold value is also an indication of the driver holding the steering wheel.

<CIT> discloses a method for testing a hands-off detection algorithm for a vehicle, the method comprising determining a plurality of system behavior test conditions for the hands-off detection algorithm based on the vehicle, generating, for each of the plurality of hands-off detection test cases, an expected test outcome; and for each of the plurality of hands-off detection test cases, conducting a test of the hands-off detection algorithm with the vehicle based on the orthogonal array to generate a plurality of actual test outcomes.

<CIT> discloses a method for "hands-on" identification on a steering system having two subsystems connected to one another by an elastic connection. The elastic connection has a static friction state and a sliding friction state for a respective set of external state variables. The steering system is excited by an excitation vibration, which is generated by a controllable vibration generator and has a respective excitation amplitude and a respective excitation frequency, for a respective set of external state variables, in which the respective excitation amplitude and the respective excitation frequency for the currently present set of external state variables are taken from a prescribed table and the vibration generator is controlled with them. A reaction torque to the excitation vibration is measured using a sensor. A phase difference between the excitation vibration and the reaction torque is calculated to identify a "hands-off" state as well as a "hands-on" state.

Vehicles capable of autonomous driving, include those providing partially autonomous driver assistance systems, may still require a human driver to maintain a minimum level of control over the vehicle by, for example, keeping their hands-on a steering wheel. In some partially autonomous functions, (for example, highly automated parking) a driver must release the steering wheel in order for the function to operate effectively. In both cases, the vehicle may automatically determine whether the driver's hands are on or off the steering wheel. In order to ensure compliance with such requirements, some vehicles analyze sensed values from the steering system to automatically detect when a driver has removed his hands from the steering wheel. This capability is known as "Hands-off Detection" (HOD).

Hands-off detection is performed by vehicle control systems by monitoring steering torque, steering angle, and/or other aspects of the vehicle's steering system to infer whether a driver is grasping the steering wheel. Some vehicles provide hands-off detection algorithms, which continuously or periodically monitor the steering system and make either a hands-off or hands-on determination. The hands-off detection algorithm is be executed by an electronic control unit on the vehicle, which provides a "hands-on" or "hands-off" indication to other vehicle systems, for example, via a Controller Area Network (CAN™) bus.

As vehicles become more complex, manufacturer requirements for accurately testing features have increased. Hands-off detection algorithms take into account vehicle steering system conditions, but vehicle usage and conditions have not been fully defined in such algorithms. Incomplete testing of hands-off detection can lead to algorithms producing false negative results (for example, not detecting hands-off) or false positive results (for example, detecting hands-off when driver is engaged to the steering system). These errors lead to quality issues and complaints from manufactures as well as end user customers, decreasing overall satisfaction. Testing must capture both these types of problems to ensure effective vehicle application.

Embodiments presented herein address these problems using system behavior testing to allow a minimum number of tests to be run while providing a broad coverage of vehicle usage conditions to detect any issues with the system. Providing broad coverage in testing of all vehicle usage conditions leads to reduced determination errors, which in turn increases the quality of the steering system performance for the vehicle. Using embodiments presented herein, tests can sweep a variety of vehicle usage and environmental conditions to discover problems due to interactions between these variables. Such embodiments reduce the number of required tests by orders of magnitude, but still provide effective testing of the hands-off detection algorithm across a large variety of conditions.

Using such embodiments, failure conditions are quickly identified, and corrective actions are taken to improve the hands-off detection algorithm. In some embodiments, testing, identification, and correction are iterated until an array is produced that correctly recognizes a hands-off detection in all cases. The results of the array can then be implemented in the vehicle's control systems to improve hands-off detection. Such improvements are impractical using conventional testing.

The invention provides a method for testing a hands-off detection algorithm for a vehicle. The method includes determining a plurality of system behavior test conditions for the hands-off detection algorithm based on the vehicle. The method includes selecting an orthogonal array defining a plurality of hands-off detection test cases based on the plurality of system behavior test conditions. The method includes generating, for each of the plurality of hands-off detection test cases, an expected test outcome. The method includes for each of the plurality of hands-off detection test cases, conducting a test of the hands-off detection algorithm with the vehicle based on the orthogonal array to generate a plurality of actual test outcomes. The method includes generating a response table based on the plurality of actual test outcomes. The response table includes a plurality of system behavior test condition interactions. The method includes determining, for each of the plurality of system behavior test condition interactions, a result rating based on the expected test outcomes for the plurality of hands-off detection test cases and the plurality of actual test outcomes. The method includes identifying, within the response table and based on the result rating for each of the plurality of system behavior test condition interactions, which one or more of the plurality of system behavior test conditions exhibits a high failure condition.

The invention also provides a system for testing a hands-off detection algorithm for a vehicle. The system includes an electronic processor. The electronic processor is configured to determine a plurality of system behavior test conditions for the hands-off detection algorithm based on the vehicle. The electronic processor is configured to select an orthogonal array defining a plurality of hands-off detection test cases based on the plurality of system behavior test conditions. The electronic processor is configured to generate, for each of the plurality of hands-off detection test cases, an expected test outcome. The electronic processor is configured to receive a plurality of actual test outcomes generated by conducting, for each of the plurality of hands-off detection test cases, a test of the hands-off detection algorithm with the vehicle based on the orthogonal array. The electronic processor is configured to generate a response table based on the plurality of actual test outcomes. The response table includes a plurality of system behavior test condition interactions. The electronic processor is configured to determine, for each of the plurality of system behavior test condition interactions, a result rating based on the expected test outcomes for the plurality of hands-off detection test cases and the plurality of actual test outcomes. The electronic processor is configured to identify, within the response table and based on the result rating for each of the plurality of system behavior test condition interactions, which one or more of the plurality of system behavior test conditions exhibits a high failure condition.

The invention also provides a non-transitory computer-readable medium including instructions executable by an electronic processor to perform a set of functions. The set of functions includes determining a plurality of system behavior test conditions for the hands-off detection algorithm based on the vehicle. The set of functions includes selecting an orthogonal array defining a plurality of hands-off detection test cases based on the plurality of system behavior test conditions. The set of functions includes generating, for each of the plurality of hands-off detection test cases, an expected test outcome. The set of functions also includes receiving a plurality of actual test outcomes generated by conducting, for each of the plurality of hands-off detection test cases, a test of the hands-off detection algorithm with the vehicle based on the orthogonal array. The set of functions includes generating a response table based on the plurality of actual test outcomes, the response table including a plurality of system behavior test condition interactions. The set of functions also includes determining, for each of the plurality of system behavior test condition interactions, a result rating based on the expected test outcomes for the plurality of hands-off detection test cases and the plurality of actual test outcomes. The set of functions also includes identifying, within the response table and based on the result rating for each of the plurality of system behavior test condition interactions, which one or more of the plurality of system behavior test conditions exhibits a high failure condition.

The terms "mounted," "connected" and "coupled" are used broadly and encompass both direct and indirect mounting, connecting, and coupling. Further, "connected" and "coupled" are not restricted to physical or mechanical connections or couplings, and can include electrical connections or couplings, whether direct or indirect. Also, electronic communications and notifications may be performed using any known means including wired connections, wireless connections, etc..

It should also be noted that a plurality of hardware and software based devices, as well as a plurality of different structural components may be utilized to implement the invention. It should also be noted that a plurality of hardware and software based devices, as well as a plurality of different structural components may be used to implement the invention. In addition, it should be understood that embodiments of the invention may include hardware, software, and electronic components or modules that, for purposes of discussion, may be illustrated and described as if the majority of the components were implemented solely in hardware. However, one of ordinary skill in the art, and based on a reading of this detailed description, would recognize that, in at least one embodiment, the electronic based aspects of the invention may be implemented in software (for example, stored on non-transitory computer-readable medium) executable by one or more processors. As a consequence, it should be noted that a plurality of hardware and software based devices, as well as a plurality of different structural components may be utilized to implement the invention. For example, "control units" and "controllers" described in the specification can include one or more electronic processors, one or more memory modules including non-transitory computer-readable medium, one or more input/output interfaces, and various connections (for example, a system bus) connecting the components. It should be understood that although certain drawings illustrate hardware and software located within particular devices, these depictions are for illustrative purposes only. In some embodiments, the illustrated components may be combined or divided into separate software, firmware and/or hardware. For example, instead of being located within and performed by a single electronic processor, logic and processing may be distributed among multiple electronic processors. Regardless of how they are combined or divided, hardware and software components may be located on the same computing device or may be distributed among different computing devices connected by one or more networks or other suitable communication links.

For ease of description, some or all of the example systems presented herein are illustrated with a single exemplar of each of its component parts. Some examples may not describe or illustrate all components of the systems. Other example embodiments may include more or fewer of each of the illustrated components, may combine some components, or may include additional or alternative components.

<FIG> is a block diagram of one example embodiment of a system <NUM> for testing hands-off detection algorithms used in vehicle control systems, for example, those found in autonomous vehicles. It should be noted that, in the description that follows, the term "autonomous vehicle" should not be considered limiting. The systems and methods described herein may be used with autonomous vehicles, partially autonomous vehicles, and conventional non-autonomous motor vehicles. The term "autonomous vehicle" is used in an inclusive way to refer to an autonomous or partially autonomous motor vehicle, which possesses varying degrees of automation (that is, the vehicle is configured to drive itself with limited, or in some cases no, input from a driver). In should also be noted that the term "driver," as used herein, generally refers to an occupant of an autonomous vehicle, who operates the controls of the vehicle or provides control input to the vehicle to influence the autonomous or partially-autonomous operation of the vehicle.

In the example illustrated, the system <NUM> includes a testing computer <NUM> and the vehicle <NUM>. As described in detail herein, in some embodiments, the testing computer <NUM> and systems of the vehicle exchange commands and data to perform tests on aspects of the control systems of the vehicle <NUM>, in particular, hands-off detection algorithms.

In some embodiments, the testing computer <NUM> is a computer server including at least an electronic processor, a memory, and a communication interface. As described in detail herein, in some embodiments the testing computer <NUM> operates to receive and analyze hands-off detection algorithm test data from the vehicle <NUM>. In some embodiments, the testing computer <NUM> uses one or more machine learning methods to analyze test data to train hands-off detection algorithms to reduce false negative and false positive determinations. Machine learning generally refers to the ability of a computer program to learn without being explicitly programmed. In some embodiments, a computer program (for example, a learning engine) is configured to construct an algorithm based on inputs. Supervised learning involves presenting a computer program with example inputs and their desired outputs. The computer program is configured to learn a general rule that maps the inputs to the outputs from the training data it receives. Example machine learning engines include decision tree learning, association rule learning, artificial neural networks, classifiers, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representation learning, similarity and metric learning, sparse dictionary learning, and genetic algorithms. Using these approaches, a computer program can ingest, parse, and understand data and progressively refine algorithms for data analytics.

In the example embodiment illustrated in <FIG>, the vehicle <NUM> is an autonomous vehicle that includes an electronic controller <NUM>, a steering system <NUM>, vehicle control systems <NUM>, vehicle sensors <NUM>, a GNSS (global navigation satellite system) system <NUM>, a transceiver <NUM>, and a human machine interface (HMI) <NUM>. The components of the vehicle <NUM>, along with other various modules and components are electrically and communicatively coupled to each other by or through one or more control or data buses (for example, the bus <NUM>), which enable communication therebetween. The use of control and data buses for the interconnection between, and communication among, the various modules and components would be known to a person skilled in the art in view of the invention described herein. In some embodiments, the bus <NUM> is a Controller Area Network (CAN™) bus. In some embodiments, the bus <NUM> is an automotive Ethernet™, a FlexRay™ communications bus, or another suitable wired bus. In alternative embodiments, some or all of the components of the vehicle <NUM> may be communicatively coupled using suitable wireless modalities (for example, Bluetooth™ or near field communication).

The electronic controller <NUM> (described more particularly below with respect to <FIG>) operates with the steering system <NUM>, the vehicle control systems <NUM> and the sensors <NUM> to autonomously control the vehicle. The electronic controller <NUM> receives sensor telemetry from the sensors <NUM> and determines path data (for example, vehicle trajectories) and other control commands for the vehicle <NUM>. The electronic controller <NUM> transmits the vehicle path data and control commands to, among other things, the steering system <NUM> and the vehicle control systems <NUM> to drive the vehicle (for example, by generating braking signals, acceleration signals, steering signals). In some embodiments, the electronic controller <NUM> implements partially autonomous control of the vehicle <NUM> (for example, adaptive cruise control, lane keeping assistance, lane centering assistance, highly automated parking, and the like).

The steering system <NUM> is configured to direct the vehicle <NUM> by moving steering components connected to a front axle of the vehicle <NUM> based on a steering command (for example, a movement of a steering wheel of the steering system <NUM> by a driver or a command from the electronic controller <NUM>). The steering system <NUM> includes sensors for measuring steering torque (for example, the torque force being applied to the steering wheel), steering angle, and steering angular speed. These measurements are reported to the electronic controller <NUM>, which uses one or more of the measurements to, among other things, detect a hands-off condition using a hands-off detection algorithm as described herein.

The vehicle control systems <NUM> include controllers, sensors, actuators, and the like for controlling aspects of the operation of the vehicle <NUM> (for example, acceleration, braking, shifting gears, and the like). The vehicle control systems <NUM> are configured to send and receive data relating to the operation of the vehicle <NUM> to and from the electronic controller <NUM>.

The sensors <NUM> determine one or more attributes of the vehicle and its surrounding environment and communicate information regarding those attributes to the other components of the vehicle <NUM> using, for example, electrical signals. The sensors <NUM> may include, for example, vehicle control sensors (for example, sensors that detect accelerator pedal position and brake pedal position, wheel speed sensors, vehicle speed sensors, yaw sensors, force sensors, odometry sensors, and vehicle proximity sensors (for example, camera, radar, LIDAR, and ultrasonic). In some embodiments, the sensors <NUM> include one or more cameras or other imaging devices configured to capture one or more images of the environment surrounding the vehicle <NUM>.

In some embodiments, the vehicle <NUM> includes, in addition to the sensors <NUM>, a GNSS (global navigation satellite system) system <NUM> The GNSS system <NUM> receives radiofrequency signals from orbiting satellites using one or more antennas and receivers (not shown). The GNSS system <NUM> determines geo-spatial positioning (for example, latitude, longitude, altitude, and speed) for the vehicle based on the received radiofrequency signals. The GNSS system <NUM> communicates this positioning information to the electronic controller <NUM>. The electronic controller <NUM> may use this information in conjunction with or in place of information received from some of the sensors <NUM> when autonomously controlling the vehicle <NUM>.

The transceiver <NUM> includes a radio transceiver communicating data over one or more wireless communications networks (for example, cellular networks, satellite networks, land mobile radio networks, etc.) including the communications network <NUM>. The communications network <NUM> is a communications network including wireless connections, wired connections, or combinations of both. The communications network <NUM> may be implemented using a wide area network, for example, the Internet (including public and private IP networks), a Long Term Evolution (LTE) network, a <NUM> network, a <NUM> network, <NUM> network and one or more local area networks, for example, a Bluetooth™ network or Wi-Fi™ network, and combinations or derivatives thereof.

The transceiver <NUM> also provides wireless communications within the vehicle using suitable network modalities. Accordingly, the transceiver <NUM> communicatively couples the electronic controller <NUM> and other components of the vehicle <NUM> with networks or electronic devices both inside and outside the vehicle <NUM>. For example, the electronic controller <NUM>, using the transceiver <NUM>, can communicate with the testing computer <NUM> to send and receive data, commands, and other information (for example, testing conditions and results).

In some embodiments, the electronic controller <NUM> controls aspects of the operation of the vehicle <NUM> based on commands received from the human machine interface (HMI) <NUM>. The HMI <NUM> provides an interface between the components of the vehicle <NUM> and a driver of the vehicle <NUM>. The HMI <NUM> is configured to receive input from the driver, receive indications of vehicle status from the system's controllers (for example, the electronic controller <NUM>), and provide information to the driver based on the received indications. The HMI <NUM> provides visual output, such as, for example, graphical indicators (for example, fixed or animated icons), lights, colors, text, images, combinations of the foregoing, and the like. The HMI <NUM> includes a suitable display mechanism for displaying the visual output, such as, for example, an instrument cluster, a mirror, a heads-up display, a center console display screen (for example, a liquid crystal display (LCD) touch screen, or an organic light-emitting diode (OLED) touch screen), or other suitable mechanisms. In alterative embodiments, the display screen may not be a touch screen. In some embodiments, the HMI <NUM> displays a graphical user interface (GUI) (for example, generated by the electronic controller and presented on a display screen) that enables a driver or passenger to interact with the autonomous vehicle <NUM>. The HMI <NUM> may also provide audio output to the driver such as a chime, buzzer, voice output, or other suitable sound through a speaker included in the HMI <NUM> or separate from the HMI <NUM>. In some embodiments, HMI <NUM> provides haptic outputs to the driver by vibrating one or more vehicle components (for example, the vehicle's steering wheel and the seats), for example, using a vibration motor. In some embodiments, HMI <NUM> provides a combination of visual, audio, and haptic outputs.

<FIG> illustrates an exemplary embodiment of an electronic controller <NUM>, which includes an electronic processor <NUM> (for example, a microprocessor, application specific integrated circuit, etc.), a memory <NUM>, and an input/output interface <NUM>. The memory <NUM> may be made up of one or more non-transitory computer-readable media and includes at least a program storage area and a data storage area. The program storage area and the data storage area can include combinations of different types of memory, such as read-only memory ("ROM"), random access memory ("RAM"), electrically erasable programmable read-only memory ("EEPROM"), flash memory, or other suitable memory devices. The electronic processor <NUM> is coupled to the memory <NUM> and the input/output interface <NUM> (for example, via a control and data bus <NUM>). The electronic processor <NUM> sends and receives information (for example, from the memory <NUM> and/or the input/output interface <NUM>) and processes the information by executing one or more software instructions or modules, capable of being stored in the memory <NUM>, or another non-transitory computer readable medium. The software can include firmware, one or more applications, program data, filters, rules, one or more program modules, and other executable instructions. The electronic processor <NUM> is configured to retrieve from the memory <NUM> and execute, among other things, software for performing methods as described herein. In the embodiment illustrated, the memory <NUM> stores, among other things, an autonomous driving engine <NUM> and a hands-off detection algorithm <NUM>.

The input/output interface <NUM> transmits and receives information from devices external to the electronic controller <NUM> (for example, over one or more wired and/or wireless connections), for example, components of the vehicle <NUM> via the bus <NUM>. The input/output interface <NUM> receives user input, provides system output, or a combination of both. As described herein, user input from a driver the vehicle <NUM> may be provided via, for example, the steering system <NUM> and the HMI <NUM>. The input/output interface <NUM> may also include other input and output mechanisms, which for brevity are not described herein and which may be implemented in hardware, software, or a combination of both.

It should be understood that although <FIG> illustrates only a single electronic processor <NUM>, memory <NUM>, and input/output interface <NUM>, alternative embodiments of the electronic controller <NUM> may include multiple processing units, memory modules, and/or input/output interfaces. It should also be noted that the vehicle <NUM> may include other electronic controllers, each including similar components as, and configured similarly to, the electronic controller <NUM>. In some embodiments, the electronic controller <NUM> is implemented partially or entirely on a semiconductor (for example, a field-programmable gate array ["FPGA"] semiconductor) chip. Similarly, the various modules and controllers described herein may be implemented as individual controllers, as illustrated, or as components of a single controller. In some embodiments, a combination of approaches may be used.

As noted, certain autonomous or partially autonomous functions of autonomous vehicles require a driver to have his or her hands on the steering wheel. Such functions may not activate or may terminate when a hands-off state is detected. False positive or false negative results for hands-off detection may lead to inefficient operation of the vehicle and result in driver annoyance. Accordingly, <FIG> illustrates an example method <NUM> for testing hands-off detection algorithms used by such vehicles (for example, the hands-off detection algorithm <NUM> of the vehicle <NUM>). Although the method <NUM> is described in conjunction with the system <NUM> as described herein, the method <NUM> could be used with other systems and vehicles. In addition, the method <NUM> may be modified or performed differently than the specific example provided. By way of example, the method <NUM> is described as being performed in part by the testing computer <NUM>. However, it should be understood that in some embodiments, portions of the method <NUM> may be performed in other ways or by or with other devices (for example, the electronic controller <NUM>).

At block <NUM>, the testing computer <NUM> determines a plurality of system behavior test conditions for the hands-off detection algorithm based on the vehicle. System behavior test conditions can be characteristics of the environment in which the vehicle <NUM> is operating, characteristics of the vehicle <NUM>, an operational status of the vehicle <NUM>, a hands-on condition, and the like. <FIG> presents a table <NUM>, which includes a number of system behavior test conditions (referred to in the table <NUM> as variables <NUM>). As illustrated in <FIG>, each of the system behavior test conditions may have two or more state values <NUM>. One example of a system behavior test condition is a road type/conditions (for example, whether the road surface is smooth or rough, dry or wet, snowy, etc.). Another example of a system behavior test condition is tire balance (for example, whether one or more tires on the vehicle are in balance or out of balance by a particular amount). Another example of a system behavior test condition is a tire pressure (for example, whether the tire pressure is too high, too low, or nominal for the vehicle being tested). Another example of a system behavior test condition is an ambient temperature or ambient temperature range (for example, above or below freezing, above 80F, etc.). Another example of a system behavior test condition is vehicle speed at the time of testing (for example, expressed in terms of an absolute speed or a range). Another example of a system behavior test condition is an initial hands-on wheel value (for example, for how long and at what force hands are applied to the steering wheel prior to each testing phase). Another example of a system behavior test condition is a vehicle weight (for example, whether the vehicle is empty or is loaded, either according to a testing standard or an absolute weight value). The table <NUM> presents an example group of system behavior test conditions and potential state values useful for testing during the spring, summer, or fall seasons. As illustrated in <FIG>, which presents a table <NUM>, different state values may be evaluated during a winter test.

Returning to <FIG>, at block <NUM>, the testing computer <NUM> selects an orthogonal array defining a plurality of hands-off detection test cases based on the plurality of system behavior test conditions. For example, as illustrated in table <NUM> of <FIG>, an orthogonal array is presented in columns A through G, representing eighteen hands-off detection test cases <NUM> through <NUM> (presented in detail in table <NUM>). As illustrated in the table <NUM>, each of the plurality of hands-off detection test cases includes a state value for each of the plurality of system behavior test conditions.

In some embodiments, the method <NUM> includes modifying the orthogonal array by adjusting a degree of freedom for the orthogonal array based on a desired granularity level for the test. For example, providing more than eighteen hands-off detection test cases will provide for more possible interactions between the variables.

Returning to <FIG>, at block <NUM> the testing computer <NUM> generates, for each of the plurality of hands-off detection test cases, an expected test outcome. For example, as illustrated in <FIG>, six tests (three with hands-on and three with hands-off) will be performed for each hands-off detection test case. The expected results (for example, hands-on detected or hands-off detected) for each test of each test case are determined.

At block <NUM>, for one of the plurality of hands-off detection test cases, a test of the hands-off detection algorithm is performed with the vehicle. For example, a driver operates the vehicle according to the state values for the hands-off detection test case. For example, for hands-off detection test case <NUM> (see <FIG>), the vehicle <NUM> is driven at 1mph on a smooth and dry road at an ambient temperature under 50F. The tires are in balance and have nominal pressure. The vehicle weight complies with the UNR79 specification and the test commences with the driver's hands exerting <NUM>. 5N of force on the steering wheel for <NUM> seconds. Under these conditions, the driver operates the vehicle and performs six tests. Three of the tests are performed during hands-off time periods and three of the tests are performed during hands-on time periods. In one test, the driver's hands are removed from the steering wheel for <NUM>. In a second test, the driver's hands are removed from the steering wheel for <NUM>. In a third test, the driver's hands are removed from the steering wheel for <NUM>. In a fourth test, driver's hands exert <NUM>. 05N of force on the steering wheel for <NUM>. In a fifth test, driver's hands exert <NUM>. 1N of force on the steering wheel for <NUM>. In a sixth test, driver's hands exert <NUM>. 05N of force on the steering wheel for <NUM>. For each of the hands-off time periods and hands-on time periods, the output of the hands-off detection algorithm is recorded (for example, transmitted to the testing computer <NUM> by the electronic controller <NUM> via the transceiver <NUM>). These outputs represent a plurality of actual test outcomes.

At block <NUM>, if hands-off detection test cases of the plurality of hands-off detection test cases remain to be tested, the next ands off detection test case in the orthogonal array is performed (at block <NUM>).

After all tests have been completed (at block <NUM>), the testing computer <NUM> generates a response table based on the plurality of actual test outcomes (at block <NUM>). An example data set for a plurality of actual test outcomes is presented in the table <NUM> of <FIG>. Using the plurality of actual test outcomes, a response table is generated that includes a plurality of system behavior test condition interactions. An example response table <NUM> is illustrated in <FIG>. Each of the plurality of system behavior test condition interactions is associated with two distinct system behavior test conditions of the plurality of system behavior test conditions, each having a state value. The system behavior test condition interactions are determined by the orthogonal array, as described above. Each of these system behavior test condition interactions represents a test (for example, a hands-off detection test case performed for a particular hands-off or hands-on time period) where the two condition/state value combinations were in effect. As illustrated in the table <NUM>, each possible interaction is presented once and represented by one cell in the table <NUM>.

Returning to <FIG>, at block <NUM>, the testing computer <NUM> determines, for each of the plurality of system behavior test condition interactions, a result rating based on the expected test outcomes for the plurality of hands-off detection test cases and the plurality of actual test outcomes (for example, by comparing those of the plurality of actual test outcomes generated for the hands-off detection test case to the expected test outcomes for the hands-off detection test case). In the example illustrated, where the test passed (for example, the hands-off or hands-on condition was determined correctly by the hands-off detection algorithm) for all instances of that system behavior test condition interaction, that interaction is assigned a result rating of <NUM>. Where the test failed (for example, the hands-off or hands-on condition was determined incorrectly by the hands-off detection algorithm) for all instances of that system behavior test condition interaction, that interaction is assigned a result rating of <NUM>. Where the test failed for some instances of that system behavior test condition interaction, but passed for others, that interaction is assigned a result rating of <NUM>.

At block <NUM>, the testing computer <NUM> identifies, within the response table and based on the result rating for each of the plurality of system behavior test condition interactions, which one or more of the plurality of system behavior test conditions exhibits a high failure condition. A high failure condition is label, which indicates that it is likely that the system behavior condition so labeled contributes to testing failures for the hands-off detection algorithm. For example, as illustrated in table <NUM>, test failures are indicated by a <NUM>. In some embodiments, the number of failures (<NUM>) is used to determine whether the system behavior test conditions exhibits a high failure condition. In one example, a high failure condition is determined using an absolute threshold (for example, where three or more failures indicate high failure condition). In that example, the testing computer <NUM> may determine, based on the results presented in table <NUM>, that the system behavior test conditions corresponding to A3, B1, B2, D1, E3, F2, and G3 exhibit high failure conditions. In another example, a high failure condition is determined as a relative value indicative of how much higher one system behavior test condition failure rate is from the next nearest system behavior test condition failure rate, or above a median or average failure rate for all system behavior test condition that indicate failures. In another example, a high failure condition is determined only for those system behavior test conditions that exhibit the highest number of failures. In that example, the testing computer <NUM> may determine, based on the results presented in table <NUM>, that only the system behavior test condition corresponding to A3 would be labeled as exhibiting a high failure condition.

In some embodiments, system behavior test conditions exhibiting a high failure condition may be used to adjust the hands-off detection algorithm to generate an adjusted hands-off detection algorithm. For example, the portions of the algorithm that incorporate sensor data or make determinations based on those conditions (for example, tire pressure, wheel speed, etc.) may be analyzed or re-programmed to address the failure rates. In another example, the response table or portions thereof may be used as training data for a machine learning hands-off detection algorithm to re-train the algorithm.

In some embodiments, the vehicle <NUM> is configured with the adjusted hands-off detection algorithm and the method <NUM> is performed iteratively. For example, a second pass of the method <NUM> on an adjusted hands-off detection algorithm tuned based on the response table <NUM> may yield an improved response (as illustrated in the table <NUM> of <FIG>). In some embodiments, the method <NUM> is performed iteratively until the improved test results are acceptable. In some embodiments, the method <NUM> may be performed with successive adjusted hands-off detection algorithms until an algorithm is developed that passes in all test cases.

Thus, the embodiments presented herein provide, among other things, systems and methods for testing hands-off detection algorithms for autonomous and partially autonomous vehicles. The results of the tests can then be implemented in a vehicle's control systems to improve hands-off detection. Such improvements are impractical using conventional testing. For example, to test all possible combinations (using a full factorial) of the conditions described herein would require <NUM>,<NUM> tests to be run <NUM> times. This would require months of testing to develop a single set of results. Iteratively testing algorithms based on such test results would be nearly impossible, given that automobile models and specifications change from year to year.

Claim 1:
A computer-implemented method for testing a hands-off detection algorithm for a vehicle, the method comprising:
determining (<NUM>) a plurality of system behavior test conditions for the hands-off detection algorithm based on the vehicle;
selecting (<NUM>) an orthogonal array defining a plurality of hands-off detection test cases based on the plurality of system behavior test conditions;
generating (<NUM>), for each of the plurality of hands-off detection test cases, an expected test outcome;
for each of the plurality of hands-off detection test cases, conducting (<NUM>) a test of the hands-off detection algorithm with the vehicle based on the orthogonal array to generate a plurality of actual test outcomes;
generating (<NUM>) a response table based on the plurality of actual test outcomes, the response table including a plurality of system behavior test condition interactions;
determining (<NUM>), for each of the plurality of system behavior test condition interactions, a result rating based on the expected test outcomes for the plurality of hands-off detection test cases and the plurality of actual test outcomes; and
identifying (<NUM>), within the response table and based on the result rating for each of the plurality of system behavior test condition interactions, which one or more of the plurality of system behavior test conditions exhibits a high failure condition.