Patent Description:
This disclosure relates to vehicles (e.g., automobiles, trucks, buses, and other road vehicles) with an autonomous driving (a. , self-driving) capability.

Vehicles capable of autonomous driving (i.e., self-driving), which are drivable without human control (e.g., by steering, accelerating, and/or decelerating themselves autonomously) during at least part of their use, are becoming more prevalent.

For example, automobiles, trucks, and other road vehicles may be characterized by various level of driving automation (e.g., any one of levels <NUM> to <NUM> of SAE J3016 levels of driving automation), from partial driving automation using one or more advanced driver-assistance systems (ADAS) to full driving automation.

Computerized perception by these vehicles of their environment and of themselves (e.g., their egomotion), based on various sensors (e.g., cameras, lidar (light detection and ranging) devices, radar devices, GPS or other location sensors, inertial measurement units (IMUs), etc.), is used to autonomously drive them, by determining where and how to safely move them and controlling actuators (e.g., of their powertrain, steering system, etc.) to move them accordingly.

While it has greatly advanced, the computerized perception by these vehicles may remain underutilized in some cases, and this may lead to suboptimal performance, safety, and/or other attributes of autonomous driving of these vehicles.

<CIT> discloses a method of controlling an autonomous vehicle. The method includes receiving an indication of a sensor anomaly, determining one or more sensor recovery strategies based on the sensor anomaly, and executing a course of action that ensures that the autonomous vehicle system operates within accepted parameters. In particular, the autonomous vehicle includes a logic generating a 3D map data, and a localizer integrating and analyzing the data to determine a local pose (position) of the autonomous vehicle, and a planner receiving perception data from a perception engine and generating numerous trajectories. Furthermore, a sensor anomaly may be detected upon the autonomous vehicle determining that sensor data gathered from the sensor includes a range of laser return intensities that are a result of the reflectivity properties various phenomena.

<CIT> discloses a method of controlling an active suspension. The method includes determining a dimension of a road abnormality ahead of the vehicle and comparing the dimension with a vehicle dimension. Responsive to the comparison, the abnormality is classified as one type of a plurality of predetermined types. Responsive to a dimension of the abnormality, the abnormality is further classified as having one of a small, medium, and large severity. The suspension is controlled responsive to the type and severity. Such a method may be used for detecting an abnormal road condition, predicting the severity of the condition, and operating actuatable elements of the suspension system in response to the predicted type and severity of the abnormal road condition.

<CIT> discloses a method of pavement detection. The method includes obtaining a ground point cloud on a road surface to be detected; based on a grating map representing the ground point cloud, determining two-dimensional ground points that are covered by each grid in the grating map; obtaining height values of the two-dimensional ground points covered by each grind; and based on the obtained height values, determining flatness of the road surface to detected.

<CIT> discloses a method of determining sensor degradation by actively controlling an autonomous vehicle. The method includes obtaining sensor readings from a sensor of an autonomous vehicle, determining baseline state information from the obtained sensor readings. A movement characteristic of the autonomous vehicle, such as speed or position, may then be changed. The sensor may then obtain additional sensor readings, and second state information may be determined from these additional sensor readings. Expected state information may be determined from the baseline state information and the change in the movement characteristic of the autonomous vehicle. A comparison of the expected state information and the second state information may then be performed. Based on this comparison, a determination may be made as to whether the sensor has degraded.

<NPL>, discloses a weather recognition method which detects raindrops on the windshield from in-vehicle camera images. A raindrop template is created in advance as a pattern before detection.

For these and other reasons, there is a need for improvements directed to vehicles with an autonomous driving capability.

According to various aspects, this disclosure relates to autonomous driving or various levels of driving assistance of a vehicle in which computerized perception by the vehicle, including of its environment and of itself (e.g., its egomotion), is used to autonomously drive the vehicle. Additionally, the computerized perception can also be used and processed to provide feedback to the autonomous driving, which provide a solution to enhance performance, safety, and/or other attributes of autonomous driving of the vehicle (e.g., when certain conditions affecting the vehicle are determined to exist by detecting patterns in or otherwise analyzing what is perceived by the vehicle), such as by adjusting autonomous driving of the vehicle, conveying messages regarding the vehicle, and/or performing other actions concerning the vehicle.

For example, according to one aspect, this disclosure relates to a system for autonomous driving or various levels of driving assistance of a vehicle. The system comprises an interface configured to receive data from sensors of the vehicle that include a camera and a lidar sensor, among others. The system also comprises a processing entity comprising at least one processor and configured to: provide perception information, the perception information comprising a 3D environmental model of an environment of the vehicle and information about a position of the vehicle; generate control signals for autonomously driving the vehicle based on the 3D environmental model of the environment of the vehicle and the information about the position of the vehicle; and process the perception information to determine whether a predefined condition affecting the vehicle and independent of any other vehicles, pedestrians and other objects of interest detected in the environment of the vehicle exists, wherein the perception information is processed by detecting a pattern in the perception information indicative of the predefined condition, the pattern arising from a combination of different ones of the sensors (<NUM>) and being undetectable from any of the different ones of the sensors (<NUM>) individually and, if so, perform an action concerning the vehicle based on the predefined condition.

According to another aspect, this disclosure relates to non-transitory computer-readable media comprising instructions executable by a processing apparatus for autonomous driving or various levels of driving assistance of a vehicle, wherein the instructions, when executed by the processing apparatus, cause the processing apparatus to: receive data from sensors of the vehicle that include a camera and a lidar sensor, among others; provide perception information, the perception information comprising a 3D environmental model of an environment of the vehicle and information about a position of the vehicle; generate control signals for autonomously driving the vehicle based on the 3D environmental model of the environment of the vehicle and the information about the position of the vehicle; and process the perception information to determine whether a predefined condition affecting the vehicle and independent of any other vehicles, pedestrians and other objects of interest detected in the environment of the vehicle exists, wherein the perception information is processed by detecting a pattern in the perception information indicative of the predefined condition, the pattern arising from a combination of different ones of the sensors and being undetectable from any of the different ones of the sensors individually and, if so, perform an action concerning the vehicle based on the predefined condition.

According to another aspect, this disclosure relates to a method for autonomous driving or various levels of driving assistance of a vehicle. The method comprises: receiving data from sensors of the vehicle that include a camera and a lidar sensor, among others; providing perception information, the perception information comprising a 3D environmental model of an environment of the vehicle and information about a position of the vehicle; generating control signals for autonomously driving the vehicle based on the 3D environmental model of the environment of the vehicle and the information about the position of the vehicle; and processing the perception information to determine whether a predefined condition affecting the vehicle and independent of any other vehicles, pedestrians and other objects of interest detected in the environment of the vehicle exists, wherein the perception information is processed by detecting a pattern in the perception information indicative of the predefined condition, the pattern arising from a combination of different ones of the sensors and being undetectable from any of the different ones of the sensors individually, and, if so, performing an action concerning the vehicle based on the predefined condition.

These and other aspects of this disclosure will now become apparent to those of ordinary skill upon review of a description of embodiments in conjunction with accompanying drawings.

A detailed description of embodiments is provided below, by way of example only, with reference to accompanying drawings, in which:.

It is to be expressly understood that the description and drawings are only for purposes of illustrating some embodiments and are an aid for understanding. They are not intended to and should not be limiting.

<FIG> show an embodiment of a vehicle <NUM> capable of autonomous driving (i.e., self-driving) in an environment <NUM> of the vehicle <NUM>. In this embodiment, the vehicle <NUM> is a road vehicle and its environment <NUM> includes a road <NUM>. The vehicle <NUM> is designed to legally carry people and/or cargo on the road <NUM>, which is part of a public road infrastructure (e.g., public streets, highways, etc.). In this example, the vehicle <NUM> is an automobile (e.g., a passenger car).

The vehicle <NUM> is capable of autonomous driving in that, for at least part of its use, it is drivable without direct human control, including by steering, accelerating, and/or decelerating (e.g., braking) itself autonomously, to travel towards a destination. Although it can drive itself, in some embodiments, the vehicle <NUM> may be controlled or supervised by a human driver in some situations. The vehicle <NUM> can thus be characterized by any level of driving automation or assistance (e.g., any one of levels <NUM> to <NUM> of SAE J3016 levels of driving automation), from partial driving automation using one or more advanced driver-assistance systems (ADAS) to full driving automation.

As further discussed below, in this embodiment, computerized perception by the vehicle <NUM>, including of its environment <NUM> and of itself (e.g., its egomotion), is used to autonomously drive the vehicle <NUM> and, additionally, can also be used to provide feedback to enhance performance, safety, and/or other attributes of autonomous driving of the vehicle <NUM> (e.g., when certain conditions affecting the vehicle <NUM> are determined to exist by detecting patterns in or otherwise analyzing what is perceived by the vehicle <NUM>), such as by adjusting autonomous driving of the vehicle <NUM>, conveying messages regarding the vehicle <NUM>, and/or performing other actions concerning the vehicle <NUM>.

In this embodiment, the vehicle <NUM> comprises a frame <NUM>, a powertrain <NUM>, a steering system <NUM>, a suspension <NUM>, wheels <NUM>, a cabin <NUM>, and a control system <NUM> that is configured to operate the vehicle <NUM> autonomously (i.e., without human control) at least for part of its use.

The powertrain <NUM> is configured to generate power for the vehicle <NUM>, including motive power for the wheels <NUM> to propel the vehicle <NUM> on the road <NUM>. To that end, the powertrain <NUM> comprises a power source (e.g., a prime mover) that includes one or more motors. For example, in some embodiments, the power source of the powertrain <NUM> may comprise an internal combustion engine, an electric motor (e.g., powered by a battery), or a combination of different types of motor (e.g., an internal combustion engine and an electric motor). The powertrain <NUM> can transmit power from the power source to one or more of the wheels <NUM> in any suitable way (e.g., via a transmission, a differential, a shaft engaging (i.e., directly connecting) a motor and a given one of the wheels <NUM>, etc.).

The steering system <NUM> is configured to steer the vehicle <NUM> on the road <NUM>. In this embodiment, the steering system <NUM> is configured to turn front ones of the wheels <NUM> to change their orientation relative to the frame <NUM> of the vehicle <NUM> in order to cause the vehicle <NUM> to move in a desired direction.

The suspension <NUM> is connected between the frame <NUM> and the wheels <NUM> to allow relative motion between the frame <NUM> and the wheels <NUM> as the vehicle <NUM> travels on the road <NUM>. For example, the suspension <NUM> may enhance handling of the vehicle <NUM> on the road <NUM> by absorbing shocks and helping to maintain traction between the wheels <NUM> and the road <NUM>. The suspension <NUM> may comprise one or more springs, dampers, and/or other resilient devices.

The cabin <NUM> is configured to be occupied by one or more occupants of the vehicle <NUM>. In this embodiment, the cabin <NUM> comprises a user interface <NUM> configured to interact with one or more occupants of the vehicle and comprising an input portion that includes one or more input devices (e.g., a set of buttons, levers, dials, etc., a touchscreen, a microphone, etc.) allowing an occupant of the vehicle <NUM> to input commands and/or other information into the vehicle <NUM> and an output portion that includes one or more output devices (e.g., a display, a speaker, etc.) to provide information to an occupant of the vehicle <NUM>. The output portion of the user interface <NUM> may comprise an instrument panel (e.g., a dashboard) which provides indicators (e.g., a speedometer indicator, a tachometer indicator, etc.) related to operation of the vehicle <NUM>.

The control system <NUM> is configured to operate the vehicle <NUM>, including to steer, accelerate, and/or decelerate (e.g., brake) the autonomous vehicle <NUM>, autonomously (i.e., without human control) as the vehicle <NUM> progresses towards a destination along a route on the road <NUM>. More particularly, the control system <NUM> comprises a controller <NUM> and a sensing apparatus <NUM> to perform actions controlling the vehicle <NUM> (e.g., actions to steer, accelerate, decelerate, etc.) to move it towards its destination on the road <NUM>, notably based on a computerized perception of the environment <NUM> of the vehicle <NUM> and of the vehicle <NUM> itself (e.g., its egomotion).

While its control system <NUM> enables it to drive itself, the vehicle <NUM> may be controlled by a human driver, such as an occupant in the cabin <NUM>, in some situations. For example, in some embodiments, the control system <NUM> may allow the vehicle <NUM> to be selectively operable either autonomously (i.e., without human control) or under human control (i.e., by a human driver) in various situations (e.g., the vehicle <NUM> may be operable in either of an autonomous operational mode and a human-controlled operational mode). For instance, in this embodiment, the user interface <NUM> of the cabin <NUM> may comprise an accelerator (e.g., an acceleration pedal), a braking device (e.g., a brake pedal), and a steering device (e.g., a steering wheel) that can be operated by a human driver in the cabin <NUM> to control the vehicle <NUM> on the road <NUM>.

The controller <NUM> is a processing apparatus configured to process information received from the sensing apparatus <NUM> and possibly other sources in order to perform actions controlling the vehicle <NUM>, including to steer, accelerate, and/or decelerate the vehicle <NUM>, towards its destination on the road <NUM>. In this embodiment, the controller <NUM> comprises an interface <NUM>, a processing entity <NUM>, and memory <NUM>, which are implemented by suitable hardware and software.

The interface <NUM> comprises one or more inputs and outputs (e.g., an input/output interface) allowing the controller <NUM> to receive input signals from and send output signals to other components to which the controller <NUM> is connected (i.e., directly or indirectly connected), including the sensing apparatus <NUM>, the powertrain <NUM>, the steering system <NUM>, the suspension <NUM>, and possibly other components such as the user interface <NUM>, a communication interface <NUM> configured to communicate over a communication network (e.g., a cellular or other wireless network, for internet and/or other communications) and/or with one or more other vehicles that are near the vehicle <NUM> (i.e., for inter-vehicle communications), etc. The controller <NUM> may communicate with other components of the vehicle <NUM> via a vehicle bus <NUM> (e.g., a Controller Area Network (CAN) bus or other suitable vehicle bus).

The processing entity <NUM> comprises one or more processors for performing processing operations that implement functionality of the controller <NUM>. A processor of the processing entity <NUM> may be a general-purpose processor executing program code stored in the memory <NUM>. Alternatively, a processor of the processing entity <NUM> may be a specific-purpose processor comprising one or more preprogrammed hardware or firmware elements (e.g., application-specific integrated circuits (ASICs), electrically erasable programmable read-only memories (EEPROMs), etc.) or other related elements.

The memory <NUM> comprises one or more memory elements for storing program code executed by the processing entity <NUM> and/or data (e.g., maps, vehicle parameters, etc.) used during operation of the processing entity <NUM>. A memory element of the memory <NUM> may be a semiconductor medium (including, e.g., a solid-state memory), a magnetic storage medium, an optical storage medium, and/or any other suitable type of memory. A memory element of the memory <NUM> may include a read-only memory (ROM) element and/or a random-access memory (RAM) element, for example.

In some embodiments, the controller <NUM> may be associated with (e.g., comprise and/or interact with) one or more other control units of the vehicle <NUM>. For example, in some embodiments, the controller <NUM> may comprise and/or interact with a powertrain control unit of the powertrain <NUM>, such as an engine control unit (ECU), a transmission control unit (TCU), etc..

The sensing apparatus <NUM> comprises sensors <NUM> configured to sense aspects of the environment <NUM> of the vehicle <NUM>, including objects <NUM> (e.g., people; animals; other vehicles; inanimate things; traffic-management devices such as traffic lights and traffic signs; other obstacles; lanes; free drivable areas; and/or any other tangible static or dynamic objects) in that environment, sense aspects of a state of the vehicle <NUM> including a position (e.g., a location, an orientation, and/or motion) of the vehicle <NUM>, and generate data indicative of these aspects that is provided to the controller <NUM> which can process it to determine actions to be autonomously performed by the vehicle <NUM> in order for the vehicle <NUM> to continue moving towards its destination.

The sensors <NUM> may include any suitable sensing device. For example, in some embodiments, the sensors <NUM> may comprise:.

The vehicle <NUM> may be implemented in any suitable way. For example, in some embodiments, the vehicle <NUM>, including its control system <NUM>, may be implemented using technology as described in prior arts, https://wavmo. com/tech/ and https://waymo. com/safetyreport/, <CIT>, or <CIT>, or using any other suitable automated driving technology (e.g., one or more advanced driver-assistance systems (ADAS)).

With continued reference to <FIG>, in this embodiment, the controller <NUM> comprises a plurality of modules to autonomously drive (e.g., accelerate, decelerate, steer, etc.) and otherwise control the vehicle <NUM> on the road <NUM> towards its destination, including a perception module <NUM> and a driving module <NUM>. These modules may be implemented in any suitable way in various embodiments (e.g., such as described, for instance, in a prior art,<NPL>, or in any known manner).

The perception module <NUM> is configured to provide information <NUM> regarding perception of the environment <NUM> of the vehicle <NUM> and the state of the vehicle <NUM> in real-time based on data from the sensors <NUM>. This information <NUM>, which will be referred to as "perception information", conveys knowledge of the environment <NUM> of the vehicle <NUM> and the vehicle's state (e.g., position, egomotion, etc.) and is used by the driving module <NUM> to autonomously drive the vehicle <NUM>.

More particularly, in this embodiment, the perception module <NUM> is configured to generate a 3D model of the environment <NUM> of the vehicle <NUM> based on data from the sensors <NUM>.

This 3D model, which will be referred to as a "3D environmental model", comprises information providing a representation of the environment <NUM> of the vehicle <NUM>, including objects <NUM> in that environment. The 3D environmental model may include characteristics of these objects <NUM>, such as their class (i.e., type), their shape, their distance to the vehicle <NUM>, their velocity, their position with relation to certain reference points, etc. The perception module <NUM> can detect and potentially classify various objects <NUM> in a scene of the environment <NUM> of the vehicle <NUM> using any suitable known techniques, such as frame-based processing, segmentation, deep-learning or other machine-learning algorithms using deep neural networks or other artificial neural networks, etc..

In some embodiments, as shown in <FIG>, the perception module <NUM> may include a sensor data fusion module <NUM> configured to fuse, i.e., perform data fusion to combine, integrate, and process, data from respective ones of the sensors <NUM>, including from the camera <NUM>, the lidar sensor <NUM>, and possibly others such as the radar sensor <NUM>. Such data fusion may be implemented in any suitable way (e.g., such as described, for instance, in a prior art <CIT>, or in any other known manner).

The perception module <NUM> is also configured to generate information about the position of the vehicle <NUM> in its environment <NUM> by performing localization of the vehicle <NUM> to determine its position and motion, based on data from the sensors <NUM>, such as from the location sensor <NUM>, the vehicle speed sensor <NUM>, and the IMU <NUM>. This information, which will be referred to as "positional information", is indicative of the position (e.g., the location and the orientation) of the vehicle <NUM> and/or one or more other parameters depending on the position of the vehicle <NUM>, such as its motion (e.g., speed, acceleration, etc.) and/or other kinematic aspects of the vehicle <NUM>, which may be specified as its egomotion.

Thus, in this embodiment, the perception information <NUM> provided by the perception module <NUM> includes the 3D environmental model and the positional information for the vehicle <NUM> and may include other information derived from the sensors <NUM>, including the data from the sensors <NUM> itself.

For example, in some embodiments, the perception module <NUM> may be implemented by a LeddarVision™ unit available from LeddarTech® (e.g., https://leddartech. com/leddarvision/) or any other commercially available technology.

The driving module <NUM> is configured to determine how to drive (e.g., accelerate, decelerate, and/or steer) the vehicle <NUM> based on the perception information <NUM> provided by the perception module <NUM>, including the 3D environmental model and the positional information for the vehicle <NUM>, and possibly other information, and to control the vehicle <NUM> accordingly by sending control signals to actuators <NUM>, such as of the powertrain <NUM>, the steering system <NUM>, and/or other components of the vehicle <NUM>, which control motion and/or other operational aspects of the vehicle <NUM>.

For instance, in this embodiment, the driving module <NUM> may implement a planning module <NUM> to plan a safe path for the vehicle <NUM>, such as by applying driving policies, respecting traffic rules, making predictions about trajectories of the vehicle <NUM> and other objects in its environment <NUM> (e.g., to avoid collisions), and/or performing other suitable operations, and a control module <NUM> to generate control signals sent to the actuators <NUM> for autonomously moving the vehicle <NUM> along that path.

In this embodiment, the controller <NUM> comprises a condition detection module <NUM> configured to determine whether one or more predefined conditions affecting the vehicle <NUM> exist based on the perception information <NUM> provided by the perception module <NUM> and, if so, generate information <NUM> regarding existence of the predefined condition(s) affecting the vehicle <NUM>. This information, which will be referred to as "detected condition information", can be used by the driving module <NUM> to perform one or more actions concerning the vehicle <NUM>, such as adjust autonomous driving and/or other operation of the vehicle <NUM>, convey a message regarding the vehicle <NUM>, and/or otherwise act to enhance performance, safety, and/or other attributes of autonomous driving of the vehicle <NUM>. In some cases, this may provide feedback to the driving module <NUM> which may otherwise be unavailable and/or may allow more rapid adjustment of autonomous driving of the vehicle <NUM>.

A given one of the predefined conditions affecting the vehicle <NUM> that can be detected by the condition detection module <NUM> and indicated by the detected condition information <NUM> may be environmental, i.e., external to the vehicle <NUM> and resulting from the environment <NUM> of the vehicle <NUM> and generally independent from objects of interest in the scene that the driving module <NUM> uses to determine commands that are sent to the actuators <NUM>. Examples of objects of interest include adjacent vehicles and pedestrians, among others. For instance, in some embodiments, an environmental one of the predefined conditions affecting the vehicle <NUM> may relate to:.

Alternatively, the detected condition information may be indicative of conditions associated with the vehicle <NUM> and not directly associated with the environment <NUM> in which the vehicle <NUM> operates. Those conditions that can be detected by the condition detection module <NUM> and indicated by the detected condition information <NUM> are vehicular, i.e., intrinsic to the vehicle <NUM> and resulting from one or more components of the vehicle <NUM>, such as the powertrain <NUM>, the steering system <NUM>, the suspension <NUM>, the wheels <NUM>, and/or any other component of the vehicle <NUM>. For example, in some embodiments, a vehicular one of the predefined conditions affecting the vehicle <NUM> may relate to:.

The detected condition information <NUM> generated by the condition detection module <NUM> and indicative of one or more predefined conditions affecting the vehicle <NUM> may thus be maintenance-related and indicative of malfunctions or need for maintenance or adjustment.

For instance, the perception information <NUM> provided by the perception module <NUM> may be conceptually viewed as implementing two detection streams, namely: a main or direct one which performs detection of objects of interest and the output of which is used by the driving module <NUM> to determine short-term actuator commands in order to provide motion control of the vehicle <NUM> into the 3D environmental model; and an ancillary one which looks for predefined conditions in the environment <NUM> that are generally independent of the objects of interest or at least independent of the characteristics of the objects of interest that determine the short-term motion control. In some embodiments, such detection streams are both carried on information conveyed at least by the lidar sensor <NUM> and the camera <NUM>. In other words, information gathered by the lidar sensor <NUM> and by the camera <NUM> is used to look for both objects of interest for short-term motion control and also for the predefined conditions that influence longer-term driving policy and/or vehicle maintenance.

Thus, the perception information <NUM> provided by the perception module <NUM> can be further processed, other than for generating the control signals for motion control in the 3D environmental model, in order to detect one or more predefined conditions affecting the vehicle <NUM>.

In this embodiment, in order to determine whether one or more predefined conditions affecting the vehicle <NUM> exist, the condition detection module <NUM> is configured to detect one or more patterns in the perception information <NUM> output by the perception module <NUM> that are indicative of existence of one or more predefined conditions. Each of these patterns, which will be referred to as a "perception fingerprint", is indicative of a predefined condition affecting the vehicle <NUM> such that the detected condition information <NUM> generated by the condition detection module <NUM> conveys or is otherwise based on that perception fingerprint.

In various examples, a given one of these perception fingerprints may reflect a pattern in the 3D environmental model (e.g., indicative of a predefined condition related to the road <NUM>, weather, illumination, and/or another aspect of the environment <NUM> of the vehicle <NUM>), a pattern in the positional information for (e.g., egomotion of) the vehicle <NUM> (e.g., indicative of a predefined condition related to malfunction of the vehicle <NUM>, such as a worn-out or deflated tire of a wheel <NUM>, a steering anomaly in the steering system <NUM>, anomalous vibration of a motor of the powertrain <NUM>, and/or another aspect of one or more components of the vehicle <NUM>), a pattern in both the 3D environmental model and the positional information for the vehicle <NUM>, or a pattern in neither of the 3D environmental model and the positional information for the vehicle <NUM> (e.g., in the data from the sensors <NUM>). Also, a given one of these perception fingerprints may be a pattern of data from a combination of different ones of the sensors <NUM> that would be undetectable by considering any of these different ones of the sensors <NUM> individually.

More particularly, in this embodiment, the condition detection module <NUM> comprises a perception-fingerprint identification module <NUM> configured to detect one or more perception fingerprints from the perception information <NUM> provided by the perception module <NUM> and cause the detected condition information <NUM> generated by the condition detection module <NUM> to convey or otherwise be based on these one or more perception fingerprints.

The perception-fingerprint identification module <NUM> may implement any suitable algorithm for pattern recognition to detect one or more perception fingerprints from the perception information <NUM> provided by the perception module <NUM>. For example, in this embodiment, the perception-fingerprint identification module <NUM> implements artificial intelligence (Al - sometimes also referred to as machine intelligence or machine learning), such as an artificial neural network, a support vector machine, or any other AI unit, in software, hardware and/or a combination thereof configured to recognize perception fingerprints from the perception information <NUM> provided by the perception module <NUM>.

More specifically, in this embodiment, shown in <FIG>, the perception-fingerprint identification module <NUM> comprises an artificial neural network <NUM> configured to detect one or more perception fingerprints from the perception information <NUM> provided by the perception module <NUM>. The artificial neural network <NUM> may be a deep neural network (e.g., convolutional, recurrent, etc.) and/or implemented using any known kind of neural network technology.

The artificial neural network <NUM> is configured to learn how to detect one or more perception fingerprints from the perception information <NUM> provided by the perception module <NUM>. Learning by the artificial neural network <NUM> may be achieved using any known supervised, semi-supervised, or unsupervised technique.

In some embodiments, the artificial neural network <NUM> may learn during a learning mode by processing "training" data conveying information (e.g., similar to what would be part of the perception information <NUM>) that one is looking for in the 3D environmental model and/or the positional information for the vehicle <NUM>, in particular data including one or more perception fingerprints that are to be detected and thus indicative of one or more predefined conditions affecting the vehicle <NUM>. For instance, a training vehicle with sensors, a perception module, and an artificial neural network similar to the sensors <NUM>, the perception module <NUM>, and the artificial neural network <NUM> of the vehicle <NUM> may be driven in situations characterized by predefined conditions of interest such that the perception module of the training vehicle generates training data that contains perception fingerprints (i.e., patterns) indicative of these predefined conditions and the artificial neural network of the training vehicle learns to identify these perception fingerprints by processing this training data.

For example, in some embodiments, if predefined conditions to be detected include a rough road, a paved road, a slippery road, a sinuous road, strong winds, heavy snow, sleet, artificial light, a worn-out tire, a deflated tire, a motor (e.g., engine) vibrating abnormally, a headlight not working properly, a steering anomaly, anomalous sound, or a combination thereof (e.g., a rough road with strong winds, a slippery road with strong winds, a slippery sinuous road, a slippery sinuous road with strong winds, a slippery road in artificial light, a slippery road with worn-out tires, a rough road with deflated tires, artificial light with a headlight not working, etc.), or any other predefined condition to be detected, the learning mode may involve, for each given one of these predefined conditions, driving the training vehicle in one or more situations characterized by that given predefined condition (e.g., on one or more rough roads, on one or more paved roads, on one or more slippery roads, on one or more sinuous roads, in one or more weather events with strong winds, in one or more weather events with heavy snow, in one or more weather events with sleet, in one or more areas with artificial light, with one or more worn-out tires, with one or more deflated tires, with one or more steering anomalies, with one or more anomalous motor vibrations, with one or more anomalous sounds, etc.) such that the perception module of the training vehicle generates training perception information that contains a perception fingerprint indicative of that given predefined condition and the artificial neural network of the training vehicle learns to identify that perception fingerprint.

In some embodiments, perception fingerprints detectable by the perception-fingerprint identification module <NUM> and predefined conditions affecting the vehicle <NUM> that they are indicative of may thus be maintained in a library or other database. In some cases, the perception-fingerprint identification module <NUM> may attempt to identify a perception fingerprint that has not previously been seen, in which cases, the perception-fingerprint identification module <NUM> may determine if that perception fingerprint is different or anomalous with respect to previously-encountered perception fingerprints. For instance, in a neural network implementation, a perception fingerprint may be a class of information the neural network is trained to detect by looking at the sensor data. With the embodiments in <FIG>, the perception-fingerprint identification module <NUM> may continuously output a perception fingerprint that distinguishes the immediate environment <NUM> in which the vehicle <NUM> operates among other environments the module <NUM> is capable to identify in the perception information <NUM>.

That perception fingerprint can be used as a further input to the driving module <NUM> to condition the signals sent to the actuators <NUM> of the vehicle <NUM>. Accordingly, the driving module <NUM> uses two inputs that both originate from the same nperceptio information <NUM>, in particular object-of-interest information determining short-term motion control and environmental input which conditions the actual rules that determine the short-term motion control. For example, if the environment input indicates that the information produced by the sensors is classified in a fingerprint associated with a slippery road, that input would affect the short-term motion control determined by the driving module <NUM>, for instance steering input, throttle input and brake input would be modulated differently to account for the expected slippery surface of the road.

The artificial neural network <NUM> of the condition detection module <NUM> may be trained to identify a perception fingerprint indicative of a predefined condition affecting the vehicle <NUM> from the perception information <NUM> provided by the perception module <NUM>, even if the sensors <NUM> are not designed to directly measure the predefined condition. For example, in some embodiments, vibration of a motor (e.g., an engine) of the powertrain <NUM> can be identified as an anomalous pattern in the positional information for (e.g., egomotion of) the vehicle <NUM> or a signal from the IMU <NUM> in the perception information <NUM>, as classification of the pattern by the artificial neural network <NUM> indicates the source of the vibration, since the classification will be able to separate or distinguish the vibration with its fingerprint, and natural frequency, from rough road surfaces and other phenomena external to the vehicle <NUM> that may be at play.

With additional reference to <FIG>, in this embodiment, the controller <NUM> may therefore implement a process as follows.

The perception module <NUM> generates the perception information <NUM>, including the 3D environmental model and the positional information for the vehicle <NUM>, based on the data from the sensors <NUM>, and the driving module <NUM> uses the perception information <NUM> to determine how to drive (e.g., accelerate, decelerate, and steer) the vehicle <NUM> and issue signals to the actuators <NUM> (e.g., of the powertrain <NUM>, the steering system <NUM>, etc.) such that the vehicle <NUM> is autonomously driven accordingly.

Meanwhile, the condition detection module <NUM> processes the perception information <NUM> provided by the perception module <NUM> to determine whether it contains one or more perception fingerprints indicative of one or more predefined conditions affecting the vehicle <NUM>. If the condition detection module <NUM> detects one or more perception fingerprints indicative of one or more predefined conditions affecting the vehicle <NUM>, the detected condition information <NUM> generated by the condition detection module <NUM> conveys or is otherwise based on these one or more perception fingerprints.

The driving module <NUM> uses the detected condition information <NUM>, which conveys or is otherwise based on the perception fingerprint(s) indicative of the predefined condition(s) affecting the vehicle <NUM>, to perform one or more actions concerning the vehicle <NUM>.

For example, in some embodiments, the driving module <NUM> may adjust autonomous driving and/or other operation of the vehicle <NUM> based on the perception fingerprint(s) detected by the condition detection module <NUM>. For instance, in some cases, if the detected perception fingerprint(s) indicate(s) that the road <NUM> is rough, slippery, and/or sinuous, there are strong winds, one or more tires of the wheels <NUM> are worn-out or deflated, a motor (e.g., engine) of the powertrain <NUM> vibrates abnormally, there is a steering anomaly in the steering system <NUM>, etc., the driving module <NUM> may adjust the logic to determine the short-term actuator commands and autonomously drive the vehicle <NUM> slower (e.g., reduce the speed of the vehicle <NUM> when going straight and/or turning), reduce the stiffness or increase the damping of the suspension <NUM>, etc. Conversely, if the detected perception fingerprint(s) indicate(s) that the road <NUM> is smooth, dry, and/or straight, there is no strong wind, etc., the driving module <NUM> may adjust the short-term control logic to autonomously drive the vehicle <NUM> faster (e.g., increase the speed of the vehicle <NUM> when going straight and/or turning), increase the stiffness or decrease the damping of the suspension, etc. The driving module <NUM> can issue signals to the actuators <NUM>, such as of the powertrain <NUM>, the steering system <NUM>, and/or the suspension <NUM>, to adjust autonomous driving of the vehicle <NUM> in this way.

As another example, in some embodiments, the driving module <NUM> may convey a message regarding the vehicle <NUM>, such as to an individual (e.g., a user of the vehicle <NUM>) or a computing device, based on the perception fingerprint(s) detected by the condition detection module <NUM>. The message may be indicative of malfunction or another problem with one or more components of the vehicle <NUM>. For instance, in some cases, the driving module <NUM> may convey a notification of maintenance, repair, or other servicing to be performed on the vehicle <NUM> if the detected perception fingerprint(s) indicate(s) that one or more tires of the wheels <NUM> are worn-out or deflated, one or more headlights are not working, a motor (e.g., engine) of the powertrain <NUM> vibrates abnormally, there is a steering anomaly in the steering system <NUM>, etc. In some embodiments, the message regarding the vehicle <NUM> may be conveyed to the user interface <NUM> of the vehicle <NUM>. In other embodiments, the message regarding the vehicle <NUM> may be conveyed to a communication device (e.g., a smartphone or computer) that is distinct (i.e., not part of the vehicle <NUM>, and possibly external to the vehicle <NUM>) via the communication interface <NUM> of the vehicle <NUM>.

The condition detection module <NUM> may be configured to determine whether one or more predefined conditions affecting the vehicle <NUM> exist in various other ways in other embodiments.

For example, in some embodiments, as shown in <FIG>, in order to determine whether one or more predefined conditions affecting the vehicle <NUM> exist, the condition detection module <NUM> may be configured to compare the perception information <NUM> provided by the perception module <NUM> to other information <NUM> available to the controller <NUM> and distinct from the 3D environmental model and the positional information for (e.g., egomotion of) the vehicle <NUM>. This information <NUM>, which will be referred to as "perception-independent reference information", can be obtained from one or more sources independent from the sensors <NUM> used to generate the 3D environmental model and the positional information for the vehicle <NUM>. When determining that the perception information <NUM> does not match the perception-independent reference information <NUM>, the condition detection module <NUM> determines that a predefined condition affecting the vehicle <NUM> exists and generates the detected condition information <NUM> so that it is indicative of that predefined condition, is valid and can be used by the driving module <NUM> to perform one or more actions concerning the vehicle <NUM>, such as adjusting autonomous driving and/or other operation of the vehicle <NUM> or conveying a message regarding the vehicle <NUM>, as discussed previously.

In some embodiments, the perception-independent reference information <NUM> may be derived from data <NUM> representative of expectations related to the vehicle <NUM> (e.g., related to the environment <NUM> of the vehicle and/or one or more operational aspects of the vehicle <NUM>), which may be stored in the memory <NUM> of the controller <NUM>, received via the communication interface <NUM>, or otherwise available to the controller <NUM>.

As an example, in some embodiments, the perception-independent reference information <NUM> may be derived from a map <NUM> (e.g., a high-definition map) representative of a locality of the vehicle <NUM>, and which may be stored in the memory <NUM> of the controller <NUM>, received via the communication interface <NUM>, or otherwise available to the controller <NUM>. The map <NUM> may provide the perception-independent reference information <NUM>, such as a kind of road surface of the road <NUM> that the vehicle <NUM> should expect to encounter at a particular location (e.g., paved road, unpaved road, open country, sandy beach, etc.). The driving module <NUM> may control the vehicle <NUM> based on this information provided by the map <NUM>.

By comparing the perception information <NUM> provided by the perception module <NUM> and the perception-independent reference information <NUM> provided by the map <NUM>, the condition detection module <NUM> can determine whether the surface of the road <NUM> as perceived by the perception module <NUM> (e.g., based on the 3D environmental model and/or the egomotion of the vehicle <NUM>) is indeed as predicted by the map <NUM> and, if not, generate the detected condition information <NUM> so that it is indicative of how the surface of the road <NUM> actually is. The driving module <NUM> may then determine whether and how to adjust autonomous driving of the vehicle <NUM> based on the detected condition information <NUM>. For instance, if the driving module <NUM> determines based on the detected condition information <NUM> that estimated actuator settings of the actuators <NUM> are improper (e.g., suboptimal or insufficient) for smoothness of drive and safety, the driving module <NUM> may send signals to the actuators <NUM> to adjust this accordingly.

As another example, in some embodiments, the perception-independent reference information <NUM> may be derived from a lighting model <NUM> representative of expected lighting (e.g., light and shadow) around the vehicle <NUM>, which may be stored in the memory <NUM> of the controller <NUM>, received via the communication interface <NUM>, or otherwise available to the controller <NUM>.

By comparing actual lighting conveyed by the perception information <NUM> provided by the perception module <NUM> (e.g., based on images from the camera <NUM>) and the expected lighting specified by the lighting model <NUM> of the perception-independent reference information <NUM>, the condition detection module <NUM> can determine whether the actual lighting as perceived by the perception module <NUM> is indeed as predicted by the lighting model <NUM> and, if not, generate the detected condition information <NUM> so that it is indicative of the actual lighting. The driving module <NUM> may then determine whether and how to adjust autonomous driving of the vehicle <NUM> based on the detected condition information <NUM>. For instance, if the driving module <NUM> determines based on the detected condition information <NUM> that settings of the actuators <NUM> are improper (e.g., suboptimal or insufficient) for smoothness of drive and safety, the driving module <NUM> may send signals to the actuators <NUM> to adjust this accordingly. Alternatively or additionally, the driving module <NUM> may send a message indicating that maintenance or other servicing is to be performed on the vehicle <NUM>.

In some embodiments, as shown in <FIG>, the perception-independent reference information <NUM> may be derived from the powertrain <NUM>, the steering system <NUM>, the suspension <NUM>, and/or any other component controlling motion of the vehicle <NUM>. For example, in some embodiments, the perception-independent reference information <NUM> may be indicative of steering movement of steered ones of the wheels <NUM> effected by the steering system <NUM> as reported on the vehicle bus <NUM> (e.g., CAN bus), while the egomotion of the vehicle <NUM> included in the perception information <NUM> provided by the perception module <NUM> can be used to estimate perceived (e.g., past) steering movement of the steered ones of the wheels <NUM>.

By comparing the perceived steering wheel movement with the reported steering wheel movement, the condition detection module <NUM> can determine whether the steering wheel movement as perceived by the perception module <NUM> indeed corresponds to the steering wheel movement as reported on the vehicle bus <NUM> and, if not, generate the detected condition information <NUM> so that it is indicative of what the steering wheel movement actually is. The driving module <NUM> may then determine whether and how to adjust autonomous driving of the vehicle <NUM> based on the detected condition information <NUM>. For instance, if the driving module <NUM> determines based on the detected condition information <NUM> that estimated actuator settings of respective ones of the actuators <NUM> in the steering system <NUM> are improper (e.g., suboptimal or insufficient) for steerability, the driving module <NUM> may send signals to these actuators <NUM> to adjust this accordingly. Alternatively, or additionally, the driving module <NUM> may send a message indicating that maintenance or other servicing is to be performed on the vehicle <NUM>.

As another example, in some embodiments, in order to determine whether one or more predefined conditions affecting the vehicle <NUM> exist, the condition detection module <NUM> may be configured to monitor temporal variation (i.e., variation in time) of the perception information <NUM> provided by the perception module <NUM>. For instance, the condition detection module <NUM> may monitor temporal variation of parameters that depend on the 3D environmental model and, when observing that one or more of these parameters of the 3D environmental model vary in time in a prescribed way deemed to be indicative of a predefined condition affecting the vehicle <NUM>, the condition detection module <NUM> generates the detected condition information <NUM> so that it is indicative of that predefined condition and can be used by the driving module <NUM> to perform one or more actions concerning the vehicle <NUM>, such as adjusting autonomous driving and/or other operation of the vehicle <NUM> or conveying a message regarding the vehicle <NUM>, as discussed previously.

For instance, in some embodiments, the condition detection module <NUM> may monitor a time-dependent statistical behavior of the 3D environmental model. For example, a distribution of "distance to obstacle" or "time to collision" for objects <NUM> in the environment <NUM> of the vehicle <NUM> may be monitored. Desirable behavior within a given driving scenario might be that changes to that distribution are slow and smooth (e.g., below a threshold rate). Control of the vehicle <NUM> by the driving module <NUM> is determined by driving policy, and tracking statistics of the environmental model distribution may help to evaluate different policies and adjust between them.

In another variant, a perception fingerprint may be used solely for vehicle maintenance purposes, without impact on motion control. In such instance, the perception-fingerprint identification module <NUM> may, in addition to camera and lidar data, receive an input from drivetrain sensors configured to detect specific malfunctions or drivetrain conditions. In this instance, the condition detection module <NUM> would provide a higher level of intelligence in fault detection and trigger a maintenance message when the actual impact of a fault condition, reported by a drivetrain sensor, is observed in the 3D environmental model.

While in embodiments considered above the vehicle <NUM> travels on land, the vehicle <NUM> may travel other than on land in other embodiments. For example, in other embodiments, the vehicle <NUM> may fly (e.g., a delivery drone or other unmanned aerial vehicle, a flying car or other personal air vehicle, etc.) or travel on water (e.g., a water taxi or other boat), such that "driving" generally means operating, controlling, and directing a course of the vehicle <NUM>.

Certain additional elements that may be needed for operation of some embodiments have not been described or illustrated as they are assumed to be within a purview of those of ordinary skill. Moreover, certain embodiments may be free of, may lack and/or may function without any element that is not specifically disclosed herein.

Any feature of any embodiment discussed herein may be combined with any feature of any other embodiment discussed herein in some examples of implementation.

In case of any discrepancy, inconsistency, or other difference between terms used herein, meanings of the terms used herein are to prevail and be used.

Claim 1:
A system for autonomous driving of a vehicle (<NUM>), the system comprising:
- an interface (<NUM>) configured to receive data from sensors (<NUM>) of the vehicle (<NUM>) that include a camera (<NUM>) and a lidar sensor (<NUM>); and
- a processing entity (<NUM>) comprising at least one processor and configured to:
- provide perception information, the perception information comprising a 3D environmental model of an environment (<NUM>) of the vehicle (<NUM>) and information about a position of the vehicle (<NUM>);
- generate control signals for autonomously driving the vehicle based on the 3D environmental model of the environment (<NUM>) of the vehicle (<NUM>) and the information about the position of the vehicle (<NUM>);
- characterized in that the processing entity (<NUM>) is configured to:
process the perception information to determine whether a predefined condition affecting the vehicle (<NUM>) and independent of any other vehicles, pedestrians and other objects of interest detected in the environment (<NUM>) of the vehicle (<NUM>) exists, wherein the perception information is processed by detecting a pattern in the perception information indicative of the predefined condition, the pattern arising from a combination of different ones of the sensors (<NUM>) and being undetectable from any of the different ones of the sensors (<NUM>) individually; and,
- if so, perform an action concerning the vehicle (<NUM>) based on the predefined condition.