Patent Description:
Autonomous driving, also referred to as automatic driving, automated driving, or piloted driving, is the movement of vehicles, mobile robots and driverless transport systems that are largely autonomous. There are different degrees of autonomous driving. In Europe, various transport ministries, for example the Federal Institute for Road Systems (Bundesanstalt für StraBenwesen) in Germany, have defined the following autonomous stages:.

A slightly different definition of levels is known from the Society of Automotive Engineers (SAE). In this regard, reference is made to the SAE J3016 standard. Such definitions could also be used instead of the above given definition.

In the field of autonomous driving, it is known to make use of map data as well as data of vehicle sensors in order to control operation of a vehicle. For example, <CIT> discloses a method for operating a vehicle relative to a passable object in a surrounding area of the vehicle. In accordance with the method, map data values from a map are input. The map data values include first object data values representing the passable object. In addition, second object data values representing the surrounding area, including the passable object, are recorded. The input map data values are reconciled with the second object data values in accordance with predefined first comparison criteria. The vehicle is then operated as a function of the reconciliation.

<CIT> discloses a method for verifying the integrity of a sensing system. A vehicle includes an integrated circuit configured to support a message-based protocol between the integrated circuit and a sensor device associated with the vehicle, and send a sensor capability safety support message, as part of the message-based protocol, to determine one or more capabilities of the sensor device. The integrated circuit is also configured to receive, in response to the sensor capability safety support message, identification data corresponding to the sensor device, from the sensor device. The memory is configured to store a plurality of request data corresponding to a plurality of fields supported by the message-based protocol and associated with the integrated circuit and the sensor device capabilities, and store the response, including the identification data, from the sensor device.

One task that needs to be performed for autonomous driving is the recognition of the signalization or signal state of traffic lights. This recognition is at present realized using cameras. However, with the currently available camera systems, recognizing the signal state may not be sufficiently safe for a vehicle that needs to autonomously enter an intersection with traffic lights.

In this regard, <CIT> discloses a method for signaling a state of a traffic light on a display terminal of a vehicle. The traffic light generates signal phase and timing information and transmits this information to a roadside unit. The roadside unit publishes intersection position information, local MAP information and real-time SPaT information. A vehicle infrastructure communication module is used for information data interaction between a terminal device and the roadside unit. The terminal device is used for acquiring, processing and displaying vehicle motion state data and the state of the traffic light.

<CIT> document discloses a method for determining a validity of a message received by a first vehicle in automated vehicle and highway systems. Status information of an intersection where a first vehicle tries to enter and a Signal Phase and Timing (SPaT) message are received, a traveling route of the first vehicle is set on a High-Definition (HD) map generated using the intersection status information, lane information having the same lane status as that of a travel lane of the first vehicle is acquired based on the intersection status information and the SPaT message, and a first validity determination for determining whether the intersection status information and the SPaT message are valid is executed based on the lane information and the HD map.

It is to be expected that the recognition of the signal states of traffic lights will need to fulfill the requirements of ASIL (Automotive Safety Integrity Level).

These high safety requirements could mean that certain functions cannot be implemented in the vehicle.

It is an object of the present invention to provide improved solutions for controlling operation of a vehicle equipped with an automated driving function.

This object is achieved by a method according to claim <NUM>, by a computer program according to claim <NUM>, which implements this method, and by an apparatus according to claim <NUM>. The dependent claims include advantageous further developments and improvements of the present principles as described below.

According to a first aspect, a method for controlling operation of a vehicle equipped with an automated driving function comprises the steps of:.

Accordingly, a computer program comprises instructions, which, when executed by at least one processor, cause the at least one processor to perform the following steps for controlling operation of a vehicle equipped with an automated driving function:.

The term computer has to be understood broadly. In particular, it also includes electronic control units, embedded devices and other processor-based data processing devices.

The computer program code can, for example, be made available for electronic retrieval or stored on a computer-readable storage medium.

According to another aspect, an apparatus for controlling operation of a vehicle equipped with an automated driving function comprises:.

According to the invention, instead of trying to adapt the camera-based recognition of the signal state in such way that it fulfills the high safety requirements, the safety requirements are shifted to other components of the vehicle. For this purpose, such components of the vehicle are selected, which can easily fulfill high safety requirements. Information of these components is then used to validate the determined signal state.

According to the invention, a second signal state of the traffic light is determined from a SPaT message received from a communication infrastructure. The determined first signal state of the traffic light is then validated by comparing the first signal state and the second signal state. A SPaT message describes the current phase at a signalized intersection, together with the residual time of the phase, for every lane of the intersection. By using a redundant system for determining the signal state of the traffic light, the overall reliability of signal state recognition is increased. The operation mode is then set to a normal operation mode in case the first signal state and the second signal state match, to a safe mode in case the first signal state and the second signal state do not match, and to a fallback mode in case no first signal state can be determined. When both determined signal states match, this is a strong indication that the first signal state has been determined correctly and normal operation is possible. When both determined signal states conflict, this is a strong indication that the first signal state may not have been determined correctly. In this case, it is advisable to make use of a safe mode. In case no first signal state can be determined, the automated driving function needs to rely on the information of the SPaT message. In this case, it is advisable to enter a fallback mode. As an example not being part of the invention, the vehicle may know that it is located on one of two straight-ahead lanes, but it is unsure if the traffic light shows green. If in this situation a vehicle in the second / parallel lane enters the intersection, it could cross the intersection a little bit slower or with a slight delay. According to the invention, if the vehicle knows from map data that it is located at a traffic light which sends secured signals, e.g. SPaT and MAP messages, it trusts these signals and slowly pass the intersection, preferably in combination with a sensor detection of cross-traffic.

In an advantageous embodiment, the determined first signal state of the traffic light is further validated by determining if an ego-localization system of the vehicle delivers results with a required accuracy, wherein the operation mode is set to a safe mode when this is not the case. Vehicles with an automated driving function necessarily have an ego-localization system. Such a system will typically be able to perform a self-diagnosis, which can be used to evaluate the accuracy of the system. In case the results of the ego-localization are sufficiently accurate, it may be impossible to determine on which lane of a road with multiple lanes the vehicle is located. As a consequence, the determined first signal state may actually refer to different lane and it is not safe to continue normal operation.

In an advantageous embodiment, the determined first signal state of the traffic light is further validated by determining if information obtained with environment sensors of the vehicle is in agreement with information from a highly detailed map available in the vehicle, wherein the operation mode is set to a safe mode when this is not the case. In order to facilitate the camera-based detection and evaluation of traffic lights, the positions of traffic lights can be included in a highly detailed map available in the vehicle, as this allows reducing the search space of the camera. Current discussions place requirements on these map entries from a perspective of functional safety. In case there is a conflict between the information obtained with the environment sensors and the information from the highly detailed map, this is a strong indication that either the highly detailed map is not accurate or outdated or that the environment sensors deliver incorrect data. In both cases, it is not safe to continue normal operation.

In an advantageous embodiment, the determined first signal state of the traffic light is further validated by determining if information from a highly detailed map available in the vehicle is in agreement with information from a MAP message received from a communication infrastructure, wherein the operation mode is set to a safe mode when this is not the case. A SPaT message is generally used together with a MAP message, which describes the physical geometry of one or more intersections. Generic facilities and infrastructure-centric messaging, such as SPaT and MAP messaging, are developed in ISO/TC <NUM>/WG <NUM> jointly with CEN/TC <NUM>/WG <NUM>. In case there is a conflict between the information from the MAP message and the information from the highly detailed map, this is an indication that the highly detailed map is not accurate or outdated. As a consequence, it is not safe to rely on the highly detailed map for detecting and evaluating traffic lights and to continue normal operation.

In an advantageous embodiment, the determined first signal state of the traffic light is further validated by determining if information from a SPaT message received from a communication infrastructure is in agreement with information from a MAP message received from the communication infrastructure, wherein the operation mode is set to a safe mode when this is not the case. In case there is a conflict between the information from the SPaT message and the information from the MAP message, this is a strong indication that at least one of those messages is not correct. As a consequence, it is not safe to rely on the information from the SPaT message to determine the signal state and to continue normal operation.

In an advantageous embodiment, in the safe mode the vehicle is brought into a safe condition. For example, the vehicle may be securely stopped. This ensures that the vehicle does not perform any action that may cause harm to the surrounding traffic.

Advantageously, a vehicle equipped with an automated driving function comprises an apparatus according to the invention or is configured to perform a method according to the invention for controlling operation of the vehicle. In this way, the vehicle is able to fulfill the high safety requirements of the recognition of the signal states of traffic lights. The vehicle may be any type of vehicle, e.g. a car, a bus, a motorcycle, a commercial vehicle, in particular a truck, an agricultural machinery, a construction machinery, a rail vehicle, etc. More generally, the invention can be used in all vehicles that need to cope with traffic lights.

Further features of the present invention will become apparent from the following description and the appended claims in conjunction with the figures.

Thus, for example, it will be appreciated by those skilled in the art that the diagrams presented herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure.

The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. Moreover, explicit use of the term "processor" or "controller" should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, read only memory (ROM) for storing software, random access memory (RAM), and nonvolatile storage.

In the claims hereof, any element expressed as a means for performing a specified function is intended to encompass any way of performing that function including, for example, a combination of circuit elements that performs that function or software in any form, including, therefore, firmware, microcode or the like, combined with appropriate circuitry for executing that software to perform the function. The disclosure as defined by such claims resides in the fact that the functionalities provided by the various recited means are combined and brought together in the manner which the claims call for. It is thus regarded that any means that can provide those functionalities are equivalent to those shown herein.

<FIG> schematically illustrates a method according to the invention for controlling operation of a vehicle equipped with an automated driving function. In a first step, a first signal state of a traffic light is determined <NUM> from data of at least one vehicle sensor. The determined first signal state of the traffic light is then validated <NUM> based on information available in the vehicle. For this purpose, a second signal state may be determined from a SPaT message received from a communication infrastructure. The first signal state may then be validated by comparing the first signal state and the second signal state. Furthermore, the first signal state may be validated by determining if an ego-localization system of the vehicle delivers results with a required accuracy, by determining if information obtained with environment sensors of the vehicle is in agreement with information from a highly detailed map available in the vehicle, by determining if information from the highly detailed map is in agreement with information from a MAP message received from the communication infrastructure, or by determining if information from the SPaT message is in agreement with information from the MAP message. Of course, also combinations of these validation measures may be used. Finally, an operation mode of the vehicle is set <NUM> as a function of a validation result. Preferably, the operation mode may be set to a safe mode in case any of the validation measures delivers a negative result. In the safe mode, the vehicle may be brought into a safe condition. If all validation measures deliver a positive result, the operation mode may be set to a normal operation mode. In case no first signal state may be determined from the data of the at least one vehicle sensor, the operation mode may be set to a fallback mode, in which the vehicle trusts the determined second signal state.

<FIG> schematically illustrates a block diagram of a first embodiment of an apparatus <NUM> according to the invention for controlling operation of a vehicle equipped with an automated driving function. The apparatus <NUM> has an input <NUM> for receiving data, e.g. messages M1, M2 from a communication infrastructure <NUM>, sensor data SD of at least one vehicle sensor <NUM> or further information INF available in the vehicle. An evaluation module <NUM> is configured to determine a first signal state SS of a traffic light from the sensor data SD. In addition, the evaluation module <NUM> may determine a second signal state SM from a SPaT message received from the communication infrastructure <NUM>. A validation module <NUM> is configured to validate the determined first signal state SS of the traffic light based on the information INF available in the vehicle. For example, the validation module <NUM> may validate the first signal state SS by comparing the first signal state SS and the second signal state SM, by determining if an ego-localization system of the vehicle delivers results with a required accuracy, by determining if information obtained with environment sensors of the vehicle is in agreement with information from a highly detailed map available in the vehicle, by determining if information from the highly detailed map is in agreement with information from a MAP message received from the communication infrastructure, or by determining if information from the SPaT message is in agreement with information from the MAP message. Of course, also combinations of these validation measures may be used. A mode setting module <NUM> is configured to set an operation mode of the vehicle as a function of a validation result. Preferably, the mode setting module <NUM> may set the operation mode to a safe mode in case any of the validation measures delivers a negative result. In the safe mode, the vehicle may be brought into a safe condition. If all validation measures deliver a positive result, the mode setting module <NUM> may set the operation mode to a normal operation mode. In case no second signal state SS may be determined from the sensor data SD, the operation mode may be set to a fallback mode, in which the vehicle accesses further sensor information or trusts the determined second signal state. For setting the operation mode, the mode setting module <NUM> may generate a mode signal M, which may be provided to an automatic driving control unit <NUM> via an output <NUM>. A local storage unit <NUM> is provided, e.g. for storing data during processing. The output <NUM> may also be combined with the input <NUM> into a single bidirectional interface.

The evaluation module <NUM>, the validation module <NUM>, and the mode setting module <NUM> may be controlled by a controller <NUM>. A user interface <NUM> may be provided for enabling a user to modify settings of the evaluation module <NUM>, the validation module <NUM>, the mode setting module <NUM>, or the controller <NUM>. The evaluation module <NUM>, the validation module <NUM>, the mode setting module <NUM>, and the controller <NUM> can be embodied as dedicated hardware units. Of course, they may likewise be fully or partially combined into a single unit or implemented as software running on a processor, e.g. a CPU or a GPU.

A block diagram of a second embodiment of an apparatus <NUM> according to the invention for controlling operation of a vehicle equipped with an automated driving function is illustrated in <FIG>. The apparatus <NUM> comprises a processing device <NUM> and a memory device <NUM>. For example, the apparatus <NUM> may be a computer, an electronic control unit or an embedded system. The memory device <NUM> has stored instructions that, when executed by the processing device <NUM>, cause the apparatus <NUM> to perform steps according to one of the described methods. The instructions stored in the memory device <NUM> thus tangibly embody a program of instructions executable by the processing device <NUM> to perform program steps as described herein according to the present principles. The apparatus <NUM> has an input <NUM> for receiving data. Data generated by the processing device <NUM> are made available via an output <NUM>. In addition, such data may be stored in the memory device <NUM>. The input <NUM> and the output <NUM> may be combined into a single bidirectional interface.

The processing device <NUM> as used herein may include one or more processing units, such as microprocessors, digital signal processors, or a combination thereof.

The local storage unit <NUM> and the memory device <NUM> may include volatile and/or non-volatile memory regions and storage devices such as hard disk drives, optical drives, and/or solid-state memories.

In the following, a preferred embodiment of the invention shall be explained in more detail with reference to <FIG>.

<FIG> illustrates a basic architecture of a V2V (Vehicle-to-Vehicle) and V2X (Vehicle-to-Everything) communication system. Reference numeral <NUM> denotes vehicles, which in this example are a shuttle bus, a car, and a truck. The vehicles <NUM> are equipped with an on-board connectivity module <NUM> including a corresponding antenna such that the vehicles <NUM> can participate in any form of wireless communication service. As shown in <FIG>, the vehicles <NUM> may transmit and receive signals to and from a roadside unit <NUM> or a base station <NUM> of a mobile communication service provider. The roadside unit <NUM> and the base station <NUM> may be connected to a control center computer <NUM> via a network <NUM>. Also shown is a traffic light <NUM> connected to the roadside unit <NUM>. The roadside unit <NUM> may transmit SPaT and MAP messages related to the traffic light <NUM> to the vehicles <NUM> via broadcast / direct communication.

The vehicles <NUM> may also be equipped with means for observing their surroundings. Their sensor systems, which are used to capture the environmental objects, are based on different measuring methods, depending on the application. Widespread technologies are, among others, RADAR, LIDAR, cameras for 2D and 3D image acquisition, and ultrasonic sensors.

Since automated driving is on the rise, a lot more data needs to be exchanged among the vehicles <NUM>, e.g. using V2V communication, and also between the vehicles <NUM> and the network. The communication systems for V2V and V2X communication need to be adapted correspondingly. They may make use of mobile communication technologies, such as LTE or <NUM>, wireless local area networks, or even optical communication technologies.

<FIG> schematically shows an exemplary block diagram of a board electronics system of a vehicle. Part of the board electronics system is an infotainment system, which comprises a touch-sensitive display unit <NUM>, a computing device <NUM>, an input unit <NUM>, and a memory device <NUM>. The display unit <NUM> is connected to the computing device <NUM> via a data line <NUM> and includes both a display area for displaying variable graphical information and an operator interface (touch-sensitive layer) arranged above the display area for inputting commands by a user. The input unit <NUM> is connected to the computing device <NUM> via a data line <NUM>. Reference numeral <NUM> designates a press button that allows a driver to manually request a tele-operated driving session if the vehicle is blocked and the driver wants the support of a tele-operated driving operator to find a way out of the blocking situation. There is no need for a dedicated press button <NUM> if other techniques for manual control are used. This includes selecting an option in a user menu displayed on the display unit <NUM>, detecting the command with speech recognition, or using gesture control means.

The memory device <NUM> is connected to the computing device <NUM> via a data line <NUM>. In the memory device <NUM>, a pictogram directory and/or symbol directory is deposited with pictograms and/or symbols for possible overlays of additional information.

The other parts of the infotainment system, such as a camera <NUM>, radio <NUM>, navigation device <NUM>, telephone <NUM> and instrument cluster <NUM> are connected via a data bus <NUM> with the computing device <NUM>. As data bus <NUM>, the high-speed variant of the CAN (Controller Area Network) bus according to ISO standard <NUM>-<NUM> may be used. Alternatively, an Ethernet-based bus system such as IEEE <NUM>. 03cg can be used. Bus systems implementing the data transmission via optical fibers are also usable. Examples are the MOST Bus (Media Oriented System Transport) or the D2B Bus (Domestic Digital Bus). For inbound and outbound wireless communication, the vehicle is equipped with an on-board connectivity module <NUM>. It can be used for mobile communication, e.g. mobile communication according to the <NUM> standard.

Reference numeral <NUM> denotes an engine control unit. Reference numeral <NUM> denotes an ESC (electronic stability control) unit, whereas reference numeral <NUM> denotes a transmission control unit. The networking of such control units, all of which are allocated to the category of the drive train, typically occurs with a CAN bus <NUM>. Since various sensors are installed in the vehicle and these are no longer only connected to individual control units, such sensor data are also distributed via the bus system <NUM> to the individual control devices.

Modern vehicles may comprise additional components, such as further sensors for scanning the surroundings, like a LIDAR sensor <NUM> or a RADAR sensor <NUM> and additional video cameras <NUM>, e.g. a front camera, a rear camera or side cameras. Such sensors are increasingly used in vehicles for observation of the environment. Further control devices, such as an ADC (automatic driving control) unit <NUM>, etc., may be provided in the vehicle. The RADAR and LIDAR sensors <NUM>, <NUM> may have a scanning range of up to <NUM>, whereas the cameras <NUM>, <NUM> may cover a range from <NUM> to <NUM>. The components <NUM> to <NUM> are connected to another communication bus <NUM>, e.g. an Ethernet-Bus due to its higher bandwidth for data transport. One Ethernet-bus adapted to the special needs of car communication is standardized in the IEEE <NUM>. 1Q specification. Moreover, a lot of information about the environment may be received via V2V communication from other vehicles. Particularly for those vehicles that are not in line of sight to the observing vehicle, it is very advantageous to receive the information about their position and motion via V2V communication.

Reference numeral <NUM> denotes an on-board diagnosis interface, which is connected to another communication bus <NUM>.

For the purpose of transmitting the vehicle-relevant sensor data via the an on-board connectivity module <NUM> to another vehicle, to other infrastructure, or to a control center computer, a gateway <NUM> is provided. This gateway <NUM> is connected to the different bus systems <NUM>, <NUM>, <NUM> and <NUM>. The gateway <NUM> is adapted to convert the data it receives via one bus to the transmission format of another bus so that it can be distributed using the packets specified for the respective other bus. For forwarding this data to the outside, i.e. to another vehicle or to the control central computer, the an on-board connectivity module <NUM> is equipped with a communication interface to receive these data packets and, in turn, to convert them into the transmission format of the appropriate mobile radio standard.

<FIG> shows a preferred embodiment of a method for controlling operation of a vehicle equipped with an automated driving function. According to this preferred embodiment, several verification steps are performed. Depending on the respective verification results, either a decision is made to set the operation mode to normal operation or a decision is made to set the operation mode to a safe mode.

In a first verification step <NUM>, the ego-localization of the vehicle is evaluated. In particular, it is checked if an ego-localization system of the vehicle delivers results with a required accuracy, e.g. sufficiently accurate to locate the vehicle on one of several lanes. If this is not the case, the operation mode is set to safe mode <NUM>, in which the vehicle is brought into a safe condition, e.g. securely stopped. Otherwise, the method proceeds to the second verification step <NUM>. In the second verification step <NUM>, it is checked whether information obtained with environment sensors of the vehicle is in agreement with information from a highly detailed map available in the vehicle. If this is not the case, the operation mode is set to safe mode <NUM>. Otherwise, the method proceeds to the third verification step <NUM>. In the third verification step <NUM>, the information from the highly detailed map is compared with information retrieved from a MAP message. In particular, it is checked whether the information from the highly detailed map is in agreement with the information from the MAP message, e.g. with regard to the number and direction of the lanes. If this is not the case, the operation mode is set to safe mode <NUM>. Otherwise, the method proceeds to the fourth verification step <NUM>. In the fourth verification step <NUM>, it is checked whether information retrieved from a SPaT message is in agreement with the information from the MAP message. If this is not the case, the operation mode is set to safe mode <NUM>. Otherwise, the method proceeds to the fifth verification step <NUM>. In the fifth verification step <NUM>, it is checked whether the signal state of the traffic light detected with the vehicle sensors is in agreement with the information from the SPaT message. If the signal states do not match, the operation mode is set to safe mode <NUM>. If the signal states match, the operation mode is set to a normal operation mode <NUM> and the vehicle will continue its course once this is permitted by the traffic light. It may happen that no signal state can be detected with the vehicle sensors. For this situation, a fallback operation mode <NUM> may be defined, in which the vehicle trusts the determined second signal state.

Claim 1:
A method for controlling operation of a vehicle (<NUM>) equipped with an automated driving function, the method comprising:
- determining (<NUM>) a first signal state (SS) of a traffic light (<NUM>) from data (SD) of at least one vehicle sensor (<NUM>) and a second signal state (SM) of the traffic light (<NUM>) from a SPaT message (M1) received (<NUM>) from a communication infrastructure (<NUM>);
- validating (<NUM>) the determined first signal state (SS) of the traffic light (<NUM>) by comparing the first signal state (SS) and the second signal state (SM); and
- setting (<NUM>) an operation mode of the vehicle (<NUM>) as a function of a validation result, wherein the operation mode is set to a normal operation mode (<NUM>) in case the first signal state (SS) and the second signal state (SM) match and to a safe mode (<NUM>) in case the first signal state (SS) and the second signal state (SM) do not match;
characterized in that the operation mode is set to a fallback mode (<NUM>) in case no first signal state (SS) can be determined (<NUM>),
wherein in the fallback mode (<NUM>) the vehicle (<NUM>) trusts the determined second signal state, if the vehicle (<NUM>) knows from map data that it is located at a traffic light (<NUM>) which sends secured signals.