Patent Description:
Tele-operated Driving (ToD) is a technology that is becoming a key enabler for automated driving, i.e. autonomous driving functions in vehicles. In addition to providing the support of a command center (CC) to solve deadlock situations, that is situation that the automated vehicle (AV) cannot solve by itself, tele-operated driving is becoming a legal requirement for automated driving in an increasing number of countries.

However, high latencies on the remote vehicle side must be expected. They are mainly caused by the delays coming from the controller, actuators and internal communications in the vehicle. These latencies introduce operating delays, especially, when the vehicle is standing still and the command center triggers the start of a drive, e.g. when a traffic light switches from red to green.

Document <CIT> describes such a method for tele-operated driving of a vehicle. The vehicle is also an autonomous vehicle that may perform driving manoeuvers in an automated manner by itself, but if a deadlock situation is detected, the vehicle requests assistance from a remote assistor in a command center. The assistor is only able to select certain autonomous driving manoeuvers, i.e. the assistor is not able to manually perform single driving manoeuvers of an own choice, like, e.g., turning the driving wheel.

Document <CIT> describes a system comprising a tele-operated vehicle, where a steering command for the vehicle may be generated by a remote artificial intelligence unit. A steering command may trigger a sequence of operations or tasks in the vehicle. This implies that when such a steering command is transmitted to the vehicle, the remote control center has to wait until the sequence of operations or tasks is executed, before the next steering command may be transmitted for adapting to the new driving situation resulting from the previous steering command.

Document <CIT> describes an autonomous vehicle operated with guide assistance. The guide assistance may be provided by a human operator who is located remotely from the vehicle and operates a user interface that displays predetermined operations for which the operator can make a selection and the selected operation can then be converted to instructions for the autonomous vehicle. Thus, after selecting an operation, the operator has to wait until the autonomous vehicle has finished all the instructions of the selected operation before the next operation may be selected.

Document <CIT> discloses a method of remote operation of a vehicle, in which data sensor captured by the vehicle are sent via a communication link to a remote operator after having determined a latency time of the link. The data sensor are rendered to an operator display by taking into account of the latency time. In other words, the operator sees the sensor data as taken by the vehicle in a future location. The operator on the basis of said data can remotely control the vehicle by sending steering command through the communication link.

Document <CIT> discloses a method of remote operation of a vehicle, in which the data sensor capture by the vehicle are sent to a remote operate with a priority which is given by the line of sight (to a display) of the operator.

Thus, in the prior art there is the problem that when a human operator triggers a steering command for changing the driving state of a tele-operated vehicle, for example for changing the driving speed and/or changing the driving direction, the set of instructions or tasks that need to be executed for changing the driving state may that numerous and slow that an unwanted or significant latency results from the moment the operator selects the steering command to the moment the new driving state is reached. One important example is the start of the tele-operated vehicle from a standstill, e.g. at a traffic light.

It is an object of the present invention to reduce the amount of time a human operator has to wait from the moment that the operator initiates a steering command for changing a driving state of a tele-operated vehicle until the new driving state is reached.

The object is accomplished by the subject matter of the independent claims. Advantageous developments with convenient and non-trivial further embodiments of the invention are specified in the following description, the dependent claims and the figures.

One aspect of the invention is a method for a tele-operated driving of a vehicle, wherein the vehicle is controlled from a command center, e.g. a remote station, via a communication network. The method comprises that the command center transmits a steering command to the vehicle over the communication network to change a driving state of the vehicle in order to execute a driving maneuver. The change of the driving state means that the steering command changes a driving speed and/or steering angle, e.g. by driving left / right, accelerating, braking the vehicle.

An operator in the control center should not experience a large latency from the moment the steering command is triggered to the successful execution of the steering command, i.e. the change of the driving state of the vehicle.

The invention is based on the following insight. The driving maneuver generally comprises a whole sequence of execution tasks or instructions or operations. One or some of these execution tasks actually do not change the driving state, but rather prepare the vehicle for changing the driving state. For example, before a steering is turned left or right, the vehicle needs to verify that no other traffic participant is close to the vehicle or located along the new driving direction. This can be done by an electronic processing circuitry of the vehicle for observing the environment. In a normal tele-operated driving according to the prior art, after receiving a steering command for turning left or right this would result in the described latency period that is needed for verifying that the environment is clear before the steering is actually turned or changed. This would result in the described latency period.

According to the invention, the execution tasks associated with a single steering command are split into two groups of tasks, namely the "preparation tasks" for initiating or preparing the driving maneuver and the actual "driving tasks" for actually or finally changing the driving state for the driving maneuver. Consequently, according to the inventive method, before the actual steering command is transmitted to the vehicle, the command center transmits a preceding separate "preparation signal" to the vehicle for triggering some or all of those of the execution tasks for which a current driving state of the vehicle remains unchanged (i.e. the preparation tasks that do not change the driving state) and afterwards, the following steering command comprises the at least one remaining final execution task (i.e. the driving tasks that change of steering angle, acceleration, braking) that causes the vehicle to actually change its driving state according to the driving maneuver.

This provides the benefit that when the operator in the control center actually triggers the steering command, the vehicle is able to react in less time or with less latency by immediately changing the driving state, as some or any preparation task (belonging to the driving maneuver) has already been triggered and executed due to the preceding preparation signal. For example, if the driving manoeuver is to turn left or right, the steering command would be the change of the steering angle, but the described verification that the environment to the left or right is clear of any other traffic participant can already be triggered in advance by the preparation signal.

According to the invention, the driving maneuver is a start / acceleration of the vehicle from standstill and the preparation signal triggers one or some or all of the following preparatory execution tasks:.

This is based on the rationale that a start of the vehicle can be a maneuver with many preparatory execution tasks (preparation tasks) that cause an unwanted latency, if they are linked directly to the steering command without the preparation signal.

The invention also comprises embodiments that provide features which afford additional technical advantages.

One embodiment comprises that for generating the steering command the command center receives "steering data" from a user interface that is operated by a human operator. For example, the user interface may provide a steering wheel and/or a brake pedal and/or an acceleration pedal such that the human operator may operate the user interface as though the operator was situated in the vehicle itself. Operating these elements of the user interface generates the steering data. For generating the preparation signal the command center receives "intention data" from the user interface. By means of the intention data the operator may signal, which driving manoeuver is to be performed next, i.e. which change of driving state is intended next. A corresponding preparation signal may be chosen and sent out to the vehicle before the steering data are generated. Thus, the latency can be reduced by means of the preparation signal, the reaction of the vehicle to the operation of the user interface, i.e. the generation of the steering data, is then more prompt than without the use of the preparation signal.

For generating the intention data, one embodiment comprises that the user interface comprises at least one "intention signaling element" that generates the intention data if operated by the operator. The intention signaling element comprises at least one selection button for selecting a respective driving maneuver and if the respective selection button is operated, intention data associated with this selection button are signaled. In other words, by pressing the corresponding button, intention data for a certain driving manoeuver, for example turning left or right or accelerating or braking, may be signaled or generated. When the operator later turns the driving wheel of the user interface and/or operates a pedal, the steering data for triggering the actual steering command follow. Additionally or alternatively to the at least one selection button an observation circuitry detects at least one predefined operator behavior (e.g. looking in the side mirror and/or back mirror) associated with the initiation of the driving maneuver. If such an operator behavior is detected, associated intention data for the driving maneuver are signaled. For example, an operator behavior may be the operator looking in the side mirror or back mirror and/or the turning of the head to one side. This may be interpreted as the intention to turn the vehicle left or right (depending on the direction that the operator is looking at). Likewise, a foot of the operator may be observed for detecting a change of the position of the foot, for example the placement of the foot on the accelerator pedal or the brake pedal. The observation circuitry may comprise a camera with an image processing circuitry and/or a proximity and/or pressure sensor for a pedal. The association of sensor data (e.g. image data) with a specific driving manoeuver may be performed on the basis of a machine learning algorithm that may be implemented on the basis of an artificial neural network.

One embodiment comprises that the "intention data" directly indicate the one specific driving maneuver and the preparation signal triggers one or some or all of the execution tasks that do not change the driving state. In other words, for each driving manoeuver specific or individual intention data are provided.

Alternatively, the indention data only indicate a class of driving maneuvers that have in common at least one common execution task that does not change the driving state. For example, the intention data may prepare the vehicle for heading forward or heading into a left direction or right direction, without specifying a specific driving manoeuver, for example the trajectory. For example, if the class of driving manoeuver is "deceleration", this may comprise several driving manoeuvers, for example braking, rolling out, opening the clutch which all decelerate the vehicle. The preparation signal then triggers the at least one common execution tasks in the vehicle and the vehicle may then wait for the more specific steering command for finalizing the driving maneuver.

One embodiment comprises that the class of driving maneuvers is a selection out of several different pre-defined maneuver classes of driving maneuvers. These maneuver classes comprise at least one of the following: driving forwards, driving backwards, heading left, heading right, overtaking, starting the vehicle from standstill, braking the vehicle into standstill. In other words, each maneuver class defines the basic common execution tasks for driving in a general direction or aiming at a general goal and especially without defining the maneuver details, like, e.g., a trajectory.

The respective specific execution task that is triggered by the preparation signal (i.e. the respective preparation task) can only be triggered as part of a preparation signal, i.e. it is an embedded task. One embodiment comprises that one or some or any execution task that is triggered by the preparation signal (i.e. the preparation tasks) is an embedded task that is not represented on the user interface by a dedicated operating element (e.g. a button).

One embodiment comprises that after the preparation signal has been sent to the vehicle, the command center starts a timer for measuring a predefined waiting time and if the command center does not receive the corresponding steering data for the steering command within the waiting time, an undo signal is transmitted to the vehicle for undoing any execution task that was triggered by the preparation signal. The command center thus transmits the steering command only, if the corresponding user's steering data from the user interface are received within the waiting time. Otherwise, when the timer runs out and no steering data were received, an undo-signal is transmitted to the vehicle for undoing the any execution task (preparation task) that was triggered by the preparation signal. In other words, after the preparation signal has been sent to the vehicle and if no steering command follows, the vehicle is set back into the driving condition before the preparation signal, i.e. the vehicle is set back into the original driving state from before the preparation signal. This prepares the vehicle for the next preparation signal.

The method can be used for resolving a deadlock situation of the vehicle, if the vehicle is an autonomous vehicle or if a driver of the vehicle is unable to cope and asks for help at the command center. One embodiment comprises that the vehicle is controlled in the described way after the vehicle has been driving autonomously or by a driver and has detected a pre-defined deadlock situation causing the vehicle to stop moving and the command center has received a request of the vehicle to assist in resolving the deadlock situation and has started a tele-operated driving session. This provided the benefit that a human operator at the command center is available when the autonomous driving function of the vehicle or the driver is unable to resolve a driving situation itself (i.e. a deadlock situation).

One embodiment comprises that said deadlock situation describes a detection of an obstacle (e.g. a road-blocking vehicle with a breakdown) and/or a road situation (e.g. roadworks) in the surrounding or in a current trajectory of the vehicle, for which an autonomous driving maneuver is not available to the autonomous driving function of the vehicle. Such deadlock situations are possible to define precisely and the can also be reliably detected by the autonomous driving function.

The invention also comprises a command center for a tele-operated driving of a vehicle, wherein the command center comprises an electronic processing circuitry that is designed to perform a method according to the invention. The command center can be a building where a human operator may sit at a workstation that provides the described user interface for remotely controlling the vehicle. The electronic processing circuitry may comprise at least one microprocessor that is linked to a data storage. The data storage may store programming instructions that cause the at least one microprocessor to perform the described method when executed by the at least one microprocessor.

By combining the command center and at least one vehicle of the described kind, a system is obtained that is also part of the invention. The invention also comprises this system with a command center according to the invention and a vehicle that is designed to receive at least one preparation signal and execute the execution tasks associated with the respective preparation signal (i.e. the preparation tasks) and that is further designed to receive at least one steering command and change a driving state, in particular turn a steering and/or accelerate and/or decelerate, according to the steering command.

The invention also comprises the combinations of the features of the different embodiments.

In the following exemplary implementations of the invention are described. The figures show:.

The embodiments explained in the following are a preferred embodiments of the invention. However, in the embodiments, the described components of the embodiment each represent individual features of the invention which are to be considered independently of each other and which each develop the invention also independently of each other and thereby are also to be regarded as a component of the invention in individual manner or in another than the shown combination. Furthermore, the described embodiment can also be supplemented by further features of the invention already described.

In the figures identical reference signs indicate elements that provide the same function.

<FIG> shows a system <NUM> with a vehicle <NUM> and a command center <NUM> (CC) for remotely controlling or operating the vehicle <NUM> for performing driving maneuvers. From the perspective of the command center <NUM>, the vehicle <NUM> therefore is a remote vehicle RV. The vehicle <NUM> may comprise an autonomous driving function <NUM> for driving the vehicle <NUM> autonomously without the support of a human driver (AV - autonomous vehicle). However, when the autonomous driving function <NUM> detects a deadlock situation for which a driving manoeuver is not available in the autonomous driving function <NUM>, the vehicle may request the support of an operator <NUM> in the command center <NUM> by means of a request message R. For a communication between the vehicle <NUM> (e.g. an electronic processing circuitry or central computer of the vehicle <NUM>) and a processing circuitry CPU of the command center <NUM>, a communication link <NUM> may be established. Communication in <NUM> may be based on a wireless connection <NUM>, for example based on a cellular network <NUM> and/or Wi-Fi, and/or an Internet connection over the Internet at <NUM>. Vehicle <NUM> may comprise a corresponding communication unit <NUM> for the wireless connection <NUM>.

The operator <NUM> may handle a workstation <NUM> that provides a user interface <NUM> for receiving input data <NUM> from the operator <NUM>. Such input data <NUM> may comprise steering data <NUM> that describe a driving manoeuver intended by the operator <NUM> and intention data <NUM> that indicate which driving manoeuver the operator <NUM> is planning the next.

For this tele-operated driving of the vehicle <NUM>, the command center <NUM> may perform a method, wherein based on the request message R, the operator <NUM> at the command center <NUM> monitors surrounding <NUM> of the vehicle <NUM> and/or assesses the conflicting situation on the basis of the at least one sensor <NUM> of the vehicle <NUM>. The at least one sensor <NUM> may provide sensor data <NUM> of the surroundings <NUM> and the sensor data <NUM> may be transmitted to the control center <NUM> where they may be presented to the operator <NUM> by the user interface <NUM>. In the control center <NUM> the sensor data <NUM> may be received by an uplink controller UL for receiving, for example, video data. The uplink controller UL may comprise a video codec for presenting your information to the operator <NUM> that may be provided by one or more than one sensor <NUM> of the vehicle <NUM> that may be designed as a camera.

Depending on the observed conflicting situation, the operator <NUM> at the command center <NUM> may decide a respective control strategy of the vehicle <NUM>. The respective control strategy describes a direct control (e.g. a closed loop control) and/or an indirect control (e.g. supervisory control by monitoring and giving high-level commands to the vehicle) of the vehicle <NUM> by the operator <NUM> at the command center <NUM>. The control center <NUM> may comprise a downlink control <NUM> for a downlink connection to the vehicle <NUM>. Depending on the input data <NUM>, the downlink control <NUM> may generate at least one steering command <NUM> for setting a steering angle and/or an acceleration and/or braking force for vehicle <NUM>.

Vehicle <NUM> may comprise a downlink control <NUM> that may receive the steering command <NUM> and may provide for the downlink DL the command to a controller <NUM> that may control at least one actuator <NUM> for driving vehicle <NUM>.

One use case of tele-operated driving ToD is the resolution of said deadlock situation by the command center CC. Two types of controls in the downlink DL may be distinguished: direct and indirect control.

<FIG> also depicts in addition to the ToD architecture of the system <NUM>, its latencies in the downlink DL. In particular, the total latency in the DL is obtained by summing up three latencies: latencies at the remote vehicle Lrv (including latencies of controller, sensors and actuators in the vehicle), latencies in the communication network Lqos and latencies at the command center CC, i.e. Lcc (including reaction time of the human operator <NUM> or driver). Besides the latencies Lcc at the command center CC, the remote vehicle <NUM> contributes significantly to the total latency.

In order to reduce the operational delay, the latency Lrv in the autonomous vehicle <NUM> is reduced. This is done by having the command center CC sending a preceding "early steering signal" or preparation signal <NUM> to the vehicle <NUM> before sending the actual steering command for a specific driving maneuvers.

In particular, a preparation signal <NUM> makes the vehicle <NUM> aware of or prepare for the intended maneuver of the operator <NUM> at command center CC. Receiving this preparation signal <NUM>, the vehicle <NUM> may trigger and execute preparation tasks <NUM> such as:.

that will not change the driving state itself, i.e. the speed and the steering of the autonomous vehicle AV will not change.

After the preparation tasks <NUM> are completed, the vehicle <NUM> is prepared to immediately perform remaining driving tasks <NUM> that actually cause the vehicle <NUM> to change its driving state. The driving tasks <NUM> may be triggered by the steering command <NUM>.

Correspondingly, the following steps may be performed by the system <NUM> for performing a driving manoeuver with the autonomous vehicle <NUM>.

Once the ToD session is started (meaning that the vehicle <NUM> already detected the deadlock, stopped in a safe state and called the CC by means of the request message R), the steps can be:.

Step number <NUM> will be the main impacting step in the delay reduction.

<FIG> illustrates this by a comparison of the reduction of the "Remote Vehicle Latency" (Lrv) by using "early steering signal" or preparation signal <NUM> at the command center CC. The top time line illustrates over time t that at a step S1 the command center <NUM> may send a steering command <NUM> to the vehicle <NUM> without any preceding preparation signal. Once the vehicle <NUM> receives the steering command <NUM> in a step S2, the vehicle <NUM> has to perform all execution tasks <NUM> belonging to the intended driving manoeuver. The execution tasks <NUM> comprise both the preparation tasks <NUM> and the driving tasks <NUM>. At a step S3 the driving manoeuver is finished or completed by executing all execution tasks <NUM> resulting in remove vehicle latency Lrv.

By introducing the preparation signal <NUM>, the bottom timeline results. In a step S4 the command center <NUM> sends the preparation signal <NUM> such that vehicle <NUM> will perform or execute the preparation tasks <NUM> as preparatory execution tasks. When at a step S5 the command center <NUM> sends the actual steering command <NUM> for changing the driving state for the driving manoeuver, the preparation tasks <NUM> belonging to this driving manoeuver have already been executed such that the vehicle only needs to perform the remaining driving tasks <NUM> of the set of execution tasks <NUM> belonging to the driving manoeuver. Thus, after receiving the steering command <NUM> in a step S6, vehicle may react or respond to the steering command <NUM> by finishing the driving manoeuver in a step S7 with a shortened latency Lrv.

In the step S4, in the command center it must be known which preparation signal <NUM> is to be sent or in other words what is the intention of the operator <NUM>. To this end, the workstation <NUM> comprises the ability to generate intention data <NUM> that indicate which preparation signal <NUM> is to be sent, i.e. which at least one preparation task is to be executed by the vehicle <NUM>.

<FIG> illustrates an exemplary workstation <NUM> like it may be provided in the command center <NUM>. For generating the steering data <NUM>, presenting the sensor data <NUM>, at least one display <NUM> may be provided in the workstation <NUM>. For generating the steering data <NUM>, the workstation <NUM> may comprise, for example, a steering wheel <NUM> and/or at least one pedal <NUM> for accelerating and/or braking the vehicle <NUM>.

For generating the preparation signal <NUM>, the corresponding intention data <NUM> may be generated by an intention signaling element <NUM> that may comprise at least one button <NUM> that the operator <NUM> may press to indicate whether the operator <NUM> intends to drive forwards or backwards or left or right as a general maneuver class. Additionally or alternatively, an intention signaling element <NUM> may be provided where the operator <NUM> may indicate a sector <NUM> around the vehicle <NUM> (represented for example by a symbol <NUM>'). Each sector <NUM> may indicate a class of driving manoeuvers that lead or drive the vehicle <NUM> into the respective sector <NUM> (e.g. front, forward left, forward right, backwards, backwards left, backwards right).

Additionally or alternatively an observation circuitry <NUM> may observe the operator <NUM> and may estimate or derive the intention data <NUM> depending on an observed behavior or a movement of the operator <NUM>, for example when the operator <NUM> turns the head and observes a region to the left of the vehicle <NUM>, the intention data <NUM> may indicate that the operator <NUM> intends to perform a driving manoeuver for turning the vehicle <NUM> to the left.

The described elements of the workstation <NUM> constitute the user interface <NUM>.

Claim 1:
Method for a tele-operated driving of a vehicle (<NUM>), wherein the vehicle (<NUM>) is controlled from a command center (<NUM>) via a communication network (<NUM>) and wherein the method comprises that the command center (<NUM>) transmits a steering command (<NUM>) to the vehicle (<NUM>) over the communication network (<NUM>) to change a driving state of the vehicle (<NUM>) in order to execute a driving maneuver, the method being characterised in that:
the driving maneuver comprises a sequence of execution tasks (<NUM>), which are split into preparation tasks for initiating or preparing the driving maneuver such that a current driving state of the vehicle (<NUM>) remains unchanged, and driving tasks for actually or finally changing the driving state of the vehicle (<NUM>) for the driving maneuver, wherein the change of the driving state means that the steering command changes a driving speed and/or a steering angle, and
before the steering command (<NUM>) is transmitted to the vehicle (<NUM>), the command center (<NUM>) transmits a separate preparation signal (<NUM>) to the vehicle (<NUM>) for triggering some or all of those of the preparation tasks for which the current driving state of the vehicle (<NUM>) remains unchanged and the following steering command (<NUM>) comprises at least one remaining final driving task that causes the vehicle (<NUM>) to change its driving state according to the driving maneuver,
wherein the driving maneuver is a start of the vehicle (<NUM>) from standstill and the preparation signal (<NUM>) triggers one or some or all of the following preparation tasks:
- disengage parking brake,
- setup brake pressure,
- change to gear one or reverse gear,
- activate day time running lights or lights,
- turn the steering in a defined direction,
- adjust at least one sensor (<NUM>) of the vehicle (<NUM>) in the defined direction.