Patent ID: 12242266

DETAILED DESCRIPTION

Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. However, in the embodiments described below, when a numerical value such as the number, quantity, amount, range, or the like of each element is mentioned, the idea according to the present disclosure is not limited to the mentioned numerical value except for a case where the numerical value is clearly specified in particular or a case where the numerical value is obviously specified to the numerical value in principle. In addition, a structure or the like described in the following embodiment is not always necessary to the idea according to the present disclosure except for a case where the structure or the like is clearly specified in particular or a case where the structure or the like is obviously specified in principle.

1. Configuration of Remote Assistance System

FIG.1is a configuration diagram of a remote assistance system for an autonomous driving vehicle. The remote assistance system100is a system that assists autonomous driving of the autonomous driving vehicle20having an autonomous traveling function by remote operation of an operator40. The operator40who remotely assists the autonomous driving vehicle20is referred to as a remote operator. The autonomous driving level of the autonomous driving vehicle20targeted for remote assistance is assumed to be level4or level5, for example. Hereinafter, the autonomous driving vehicle20is simply referred to as the vehicle20.

A remote assistance terminal30is used for remote assistance of the vehicle20by the operator40. The remote assistance terminal30is connected to a server10of the management center via a communication network including the Internet. The vehicle20is also connected to the server10of the management center via a communication network including 4G and 5G. A plurality of remote assistance terminals30and a plurality of vehicles20are connected to the server10. The server10receiving the assistance request from the vehicle20selects a person in charge from among the available operators40, and connects the remote assistance terminal30of the operator40in charge with the vehicle20requesting assistance.

The situation in which the vehicle20requests assistance includes, for example, a case of passing a preceding vehicle, a case of passing through a crosswalk, a case of turning right at an intersection, a case of deviating from a lane to avoid an obstacle, and the like. In the remote assistance, at least a part of judgement for autonomous driving by the vehicle20is performed by the operator40. Basic calculations related to recognition, judgement, and operation required for driving are performed in the vehicle20. The operator40judges an action to be taken by the vehicle20based on various kinds of vehicle information transmitted from the vehicle20, and transmits a command to the vehicle20based on the judgement result. The remote assistance command sent from the operator40to the vehicle20includes a command to advance the vehicle20and a command to stop the vehicle20. The remote assistance command includes a command for avoiding an offset with respect to a forward obstacle, a command for passing a preceding vehicle, a command for emergency evacuation, and the like.

FIG.2is a block diagram illustrating an example of a configuration of the vehicle20. The vehicle20includes a computer21. The computer21is an aggregate of a plurality of electronic control units (ECUs) mounted on the vehicle20. The vehicle20includes at least one external sensor22, at least one internal sensor23, at least one actuator24, and a communication device25. These components are connected to the computer21using an in-vehicle network such as a controller area network (CAN).

The computer21comprises at least one processor21a(hereinafter referred to simply as processor21a) and at least one memory21b(hereinafter referred to simply as memory21b) coupled to the processor21a. The memory21bstores at least one program21cexecutable by the processor21a(hereinafter, simply referred to as a program21c) and various pieces of information related thereto. The program21cincludes a plurality of executable instructions. The memory21bstores high-precision three-dimensional map information for autonomous driving.

When the processor21aexecutes the program21c, various processes are executed by the processor21a. The program21cincludes an autonomous driving program for achieving autonomous driving. When the autonomous driving program is executed by the processor21a, the computer21functions as an autonomous driving system that autonomously drives the vehicle20. Hereinafter, the computer21as the autonomous driving system is simply referred to as an autonomous driving system. Further, the program21cincludes a remote assistance program for receiving remote assistance. When the remote assistance program is executed by the processor21a, processing for requesting assistance from the server10and executing the assistance content acquired from the server10in the vehicle20is executed.

The external sensor22includes a camera that images the surroundings of the vehicle20, in particular, the front of the vehicle20. A plurality of cameras may be provided, and may capture images of the side and the rear of the vehicle20in addition to the front. The camera may be shared for autonomous driving and remote assistance by the operator40, or a camera for autonomous driving and a camera for remote assistance may be separately provided. The external sensor22includes a recognition sensor other than the camera. The recognition sensor is a sensor that acquires information for recognizing a situation around the vehicle20. Examples of the recognition sensor other than the camera include a LiDAR and a millimeter wave radar. The external sensor22includes a position sensor that detects a position and a direction of the vehicle20. As the position sensor, a GPS sensor is exemplified. Information obtained by the external sensor22is transmitted to the computer21.

The internal sensor23includes a state sensor that acquires information about the motion of the vehicle20. Examples of the state sensor include a wheel speed sensor, an acceleration sensor, an angular velocity sensor, and a steering angle sensor. As the acceleration sensor and the angular velocity sensor, an IMU may be used. Information obtained by the internal sensor23is transmitted to the computer21.

Hereinafter, the information obtained by the internal sensor23and the information obtained by the external sensor22are collectively referred to as vehicle information. However, in addition to the information acquired by the sensor of the vehicle20, the vehicle information includes a path plan of the vehicle20created by the autonomous driving system and prediction information of an environment surrounding the vehicle20. The autonomous driving system predicts a change in the environment surrounding the vehicle20in a short period of time in the future using information on the situation around the vehicle20obtained by the external sensor22, and creates the path plan based on the prediction information and a target route determined by, for example, a navigation system.

Examples of the actuator24include a steering device that steers the vehicle20, a driving device that drives the vehicle20, and a braking device that brakes the vehicle20. The steering device includes, for example, a power steering system, a steer-by-wire steering system, and a rear wheel steering system. The drive device includes, for example, an engine, an EV system, and a hybrid system. The braking device includes, for example, a hydraulic brake and an electric regenerative brake. The actuator24is operated by a control signal transmitted from the computer21.

The communication device25is a device that controls wireless communication between the vehicle20and the outside. The communication device25communicates with the server10via a communication network. The information processed by the computer21is transmitted to the server10using the communication device25. The information processed by the server10is taken into the computer21using the communication device25. In addition, when vehicle-to-vehicle communication with another vehicle or road-to-vehicle communication with an infrastructure is required for autonomous driving, communication with these external devices is also performed by the communication device25.

FIG.3is a block diagram illustrating an example of a configuration of the remote assistance terminal30. The remote assistance terminal30includes a computer31, a display device32, an input device33, and a communication device35. The display device32, the input device33, and the communication device35are connected to the computer31. The remote assistance terminal30may be installed in the management center or may be installed outside the management center, for example, in the home of the operator40.

The computer31comprises at least one processor31a(hereinafter referred to simply as processor31a) and at least one memory31b(hereinafter referred to simply as memory31b) coupled to the processor31a. The memory31bstores at least one program31cexecutable by the processor31a(hereinafter, simply referred to as a program31c) and various pieces of information related thereto. The program31cincludes a plurality of executable instructions. The memory31bstores high-precision three-dimensional map information for autonomous driving.

When the processor31aexecutes the program31c, various processes are executed by the processor31a. The program31cincludes an operator UI management program that manages a user interface for providing remote assistance to the vehicle20. When the operator UI management program is executed by the processor31a, the computer31functions as an operator UI management system, and executes processing for displaying information necessary for remote assistance on the display apparatus32described later.

The display device32is a device that displays information necessary for the operator40to perform remote assistance. Specifically, the information displayed by the display device32is information corresponding to an image in front of the vehicle20. However, the information displayed on the display device32is not an image acquired by the camera of the vehicle20, that is, information used by the autonomous driving system of the vehicle20.FIG.4Ais a diagram illustrating an example of the image acquired by the camera of the vehicle20.FIG.4Bis an example of a display screen of the display device32. The information displayed on the display device32is a 3D model-based image created by the computer31. In this image, the surrounding environment of the vehicle20is reproduced on a three-dimensional space. A stationary object in the surrounding environment can be spatially reproduced using the high-precision three-dimensional map information and the position information of the vehicle20. The contents of information displayed on the display device32and a method of generating the information will be described in detail later.

The input device33is a device for inputting an operation for remote assistance of the operator40. The information input by the input device33is processed by the computer31and transmitted to the vehicle20. Specific examples of the input device33include a button, a lever, and a touch panel. For example, the vehicle20may be instructed to proceed/stop or to move in the lateral direction by the direction in which the lever is tilted. The movement in the lateral direction includes, for example, offset avoidance with respect to a forward obstacle, lane change, and passing of a preceding vehicle.

The communication device35is a device that controls communication between the remote assistance terminal30and the outside. The communication device35communicates with the server10via a communication network. The information processed by the computer31is transmitted to the server10using the communication device35. The information processed by the server10is taken into the computer31using the communication device35.

2. Overview of Remote Assistance Method

An object of the remote assistance method of the present disclosure is to eliminate the influence of the slip in the assistance timing caused by the delay occurring between the vehicle20and the remote assistance terminal30and to operate the vehicle20at the timing intended by the operator40. In order to achieve this object, in a remote assistance method according to the embodiment of the present disclosure, a future image generated from vehicle information is displayed on the display device32of the remote assistance terminal30instead of an image in front of the vehicle20captured by the camera mounted on the vehicle20. The future image displayed on the display device32is an image predicted to be realized at a future time. The vehicle information used to generate the future image includes the path plan of the vehicle20generated by the autonomous driving system and prediction information of the environment surrounding the vehicle20used to generate the path plan. Since the generation of the path plan and the prediction of the surrounding environment are basic known functions for the autonomous driving system, description thereof will be omitted. The vehicle20transmits the vehicle information to the remote assistance terminal30at least during a period from when the assistance is requested to when the assistance is completed.

In the remote assistance method according to the present embodiment, the remote assistance terminal30transmits the assistance content input by the operator40and the future information of the vehicle20generated by the computer31to the vehicle20via the server10. The future information of the vehicle20generated by the computer31includes information related to a positional relationship between the vehicle20and an object present around the vehicle20at a future time in consideration of a delay time to be described later. Based on the received assistance content and future information, the computer21of the vehicle20determines whether or not the assistance can be executed at a timing when the assistance is actually required.

Hereinafter, an outline of the remote assistance method according to the present embodiment will be specifically described with reference toFIGS.5to7. Here, a situation is assumed in which the vehicle20is about to turn right at the intersection and another vehicle50(here, referred to as the other vehicle) is entering the intersection from the opposite lane.

First, the overall outline ofFIGS.5to7will be described. In the upper part of each ofFIGS.5to7, an image diagram showing the positional relationship between the vehicle20and the other vehicle50recognized from the image of the camera of the vehicle20is drawn. These image diagrams correspond to plan views obtained by converting the camera image shown inFIG.4A. The five image diagrams shown in the upper part are all different in time, and are arranged in time series from left to right in the order of time T0, T11, T21, T31, and T41. However, the image captured by the camera of the vehicle20is a moving image, and the positional relationship between the vehicle20and the other vehicle50is also recognized in a continuous flow of time. Here, the positional relationship at a specific time in the continuous flow of time is merely picked up and shown.

The autonomous driving system of the vehicle20creates a path plan of the vehicle20based on the image of the camera, and calculates a predicted path of the other vehicle50. In each image diagram, a solid line and a dotted arrow line extending from the vehicle20are path plans created by the autonomous driving system. The path plan includes a target position of the vehicle20in a coordinate system centered on the vehicle20or an absolute coordinate system, and a target velocity or a target acceleration at the target position. The solid line is a defined path plan, and the dotted line is an undefined path plan. A dashed arrow line extending from the other vehicle50is a predicted path of the other vehicle50calculated by the autonomous driving system.

An image diagram showing the positional relationship between the vehicle20and the other vehicle50spatially projected on the screen of the display device32of the remote assistance terminal30is depicted in the lower part of each ofFIGS.5to7. These image diagrams correspond to plan views obtained by converting 3D images as shown inFIG.4Bdisplayed on the display device32. The three image diagrams shown in the lower part are all different in time, and are arranged in time series from left to right in the order of time T12, T22, and T32. However, the image displayed on the screen of the display device32is a moving image, and the positional relationship between the vehicle20and the other vehicle50in the screen continuously changes with time. Here, the positional relationship at a specific time in the continuous flow of time is merely picked up and shown.

The computer31of the remote assistance terminal30predicts the positional relationship between the vehicle20and the other vehicle50at a future time on the basis of the path plan of the vehicle20received from the vehicle20and the prediction information of the surrounding environment including the predicted path of the other vehicle50. In each of the image diagrams, a solid line and a dotted arrow line extending from the vehicle20indicate the path plan received from the vehicle20. A solid line indicates a path plan approved by the operator40, and a dotted line indicates a path plan unapproved by the operator40. A dashed arrow line extending from the other vehicle50is a predicted path of the other vehicle50at the future time predicted by the computer31. For example, based on the map information and the velocity and direction of the other vehicle50, the computer31predicts the probability that the other vehicle50will go straight ahead in the current driving lane, the probability that the other vehicle50will turn right, and the probability that the other vehicle50will turn left. When the object around the vehicle20is a pedestrian, the computer31divides the road into grids, for example, and calculates the probability that the pedestrian will pass on each grid using the velocity and direction of the pedestrian, thereby predicting the course of the pedestrian.

FIGS.5to7show examples of remote assistance implemented by the remote assistance method according to the embodiment of the present invention. The examples shown in each figure will be described in order from the example shown inFIG.5.

In the example shown inFIG.5, a request for assistance is issued from the vehicle20at the time T0. In this example, the assistance requested by the vehicle20is to determine whether or not to make a right turn at an intersection ahead. The vehicle20requests assistance not after entering the intersection but before a predetermined distance from the intersection. At the time TO when the vehicle20requests assistance, it is unclear whether the other vehicle50in the opposite lane is going straight, turning left, or turning right. In such a situation, the autonomous driving system of the vehicle20does not make a determination by itself but leaves the determination to the operator40and waits for the determination result to be transmitted. However, while the assistance is being requested, the vehicle20does not completely stop at that location, but moves slowly to the vicinity of the center of the intersection at a low velocity in order to suppress the influence on the following vehicles.

In response to the assistance request from the vehicle20, the server10assigns an operator40, and the vehicle information including the path plan of the vehicle20and the predicted path of the other vehicle50is transmitted together with the assistance request to the remote assistance terminal30of the assigned operator40. The computer31of the remote assistance device30generates a future image in consideration of the delay time based on the vehicle information and displays the generated future image on the display device32at the time T12. It takes time from when the image including the positional relationship between the vehicle20and the other vehicle50is acquired by the camera of the vehicle20to when the vehicle information transmitted from the vehicle20is received by the remote assistance terminal30and the future image can be displayed by the remote assistance terminal30. This time is the delay time in the uplink direction.

The sum of the delay time in the uplink direction and the delay time in the downlink direction, which will be described later, is the delay time to be considered in generation of a future image. The delay time includes a communication delay, a calculation time of the computer21of the vehicle20and a calculation time of the computer31of the remote assistance terminal30. The calculation time may be set to a fixed value, and the delay time may be determined based on the communication delay (for example, Round Trip Time) measured immediately before. Alternatively, the delay time may be given as a fixed value based on the past results. When the delay time is given as a fixed value, the value of the delay time may be changed depending on the time zone in consideration that the communication delay is different depending on the time zone.

The future image displayed on the display device32includes the positional relationship between the vehicle20and the other vehicle50at a future time beyond the current time (time T12). The display device32also displays what kind of assistance the vehicle20is requesting. The operator40determines whether or not the vehicle20may turn right at the intersection based on the positional relationship between the vehicle20, the other vehicle50and the surrounding environment displayed on the display device32of the remote assistance terminal30. The operator40operates the input device33of the remote assistance terminal30according to the determination result of the operator40. In the example shown inFIG.5, the traveling direction of the other vehicle50cannot be specified yet from the image displayed on the display device32at the time T12. Therefore, the operator40inputs No-Go that is suspension of execution of the right turn as the assistance content for the vehicle50.

The assistance content input to the remote assistance terminal30is transmitted from the remote assistance terminal30to the vehicle20. Further, the future information created by the remote assistance terminal30, that is, the information on the positional relationship between the vehicle20and the other vehicle50at the future time is transmitted to the vehicle20together with the assistance content. Here, for simplicity, the time required for the determination by the operator40is ignored, and it is assumed that the assistance content and the future information are transmitted from the remote assistance terminal30to the vehicle20at the time T12. In the remote assistance system, since the determination time of the operator40is not included in the delay time in the uplink direction and the delay time in the downlink direction, the determination time of the operator40can be ignored.

It takes time from when the assistance content and the future information are transmitted from the remote assistance terminal30to when the assistance content and the future information are received by the vehicle20and the assistance content becomes executable by the vehicle20. This time is the delay time in the downlink direction. The time T21is a time that is later than the time at which the vehicle20transmits the vehicle information by the total delay time in the uplink direction and the downlink direction, and the vehicle20can execute the assistance content at the time T21. As described above, the future image displayed on the display device32of the remote assistance terminal30is created in consideration of the delay time. That is, the future image displayed on the display device32at the time T12is created by predicting the actual positional relationship between the vehicle20and the other vehicle50at the time T21. Then, the positional relationship between the vehicle20and the other vehicle50at the time T12predicted at the time T21is transmitted to the vehicle20as future information.

Since the assistance content executed by the vehicle20at the time T21is No-Go, the vehicle20does not execute the right turn and continues to be in the standby state. When the assistance content is the determination of Go, that is, the determination to move the vehicle20actively, the future information received by the vehicle20together with the assistance content is used for final determination of whether or not to perform remote assistance corresponding to the assistance content. Not limited to the scene of the right turn at the intersection, when the assistance content of the operator40with respect to the assistance request is No-Go, the vehicle20continues to be in the standby state regardless of the future information. However, when the determination of the vehicle20and the assistance content of the remote assistance match each other, the future information may not be used to determine whether to perform the remote assistance according to the assistance content. Further, the vehicle20transmits the latest vehicle information at the time T21to the remote assistance terminal30.

After issuing the assistance request at the time T0, the vehicle20continuously transmits the vehicle information to the remote assistance terminal30. Therefore, even at the time T11between the time TO at which the assistance request is issued and the time21at which the assistance content is executed, the vehicle20transmits the vehicle information at the time T11to the remote assistance terminal30.

The computer31of the remote assistance device30generates a future image in consideration of the delay time based on the vehicle information transmitted from the vehicle20at the time T11, and displays the generated future image on the display device32at the time T22. The future image displayed on the display device32at the time T22is an image obtained by predicting the positional relationship between the vehicle20and the other vehicle50at a future time (time T31) beyond the current time (time T22). From the image displayed at the time T22, it can be determined that the other vehicle50is going to turn right at the intersection. When the other vehicle50turns right at the intersection, the vehicle20can safely turn right without causing interference with the other vehicle50. Therefore, the operator40inputs execution of turning right (Go) as the assistance content for the vehicle50.

The vehicle information transmitted from the vehicle20is continuously input to the remote assistance terminal30. At the time T32, a future image generated based on the vehicle information transmitted from the vehicle20at the time T21is displayed on the display apparatus32. The future image displayed on the display device32at the time T32is an image obtained by predicting the positional relationship between the vehicle20and the other vehicle50at a future time (time T41) beyond the current time (time T32). From the image displayed at the time T32, it can be seen that the other vehicle50is turning right at the intersection. If the other vehicle50is also in the middle of turning right, the right turn of the vehicle20can be continued without any problem. Therefore, the operator40inputs execution of turning right (Go) as the assistance content for the vehicle50.

The assistance content input to the remote assistance terminal30by the operator40at the time T22is transmitted to the vehicle20together with the future information created by the remote assistance terminal30at the time T22. The future information received by the vehicle20includes the positional relationship between the vehicle20and the other vehicle50at the time T31predicted at the time T22. Since the assistance content transmitted from the operator40is the determination of Go, the vehicle20refers to the future information and determines whether or not to execute remote assistance according to the assistance content.

In the determination of whether or not the remote assistance can be executed by the vehicle20, it is confirmed whether or not the positional relationship between the vehicle20and the other vehicle50displayed on the display device32when the operator40inputs the assistance content to the remote assistance terminal30is actually realized. As a confirmation method, for example, a degree of coincidence between the positional relationship between the vehicle20and the other50in the future image displayed on the display device32at the time T22(hereinafter referred to as a future positional relationship) and the actual positional relationship between the vehicle20and the other vehicle50realized at the time T31is determined. The future positional relationship is included in the future information transmitted from the remote assistance terminal30. The actual positional relationship between the vehicle20and the other vehicle50can be acquired by the camera of the vehicle20. A method of calculating the degree of coincidence will be described in detail later. When the degree of coincidence is equal to or greater than the predetermined value, the vehicle20determines that it is confirmed that the future positional relationship predicted at the time T22is realized at the time T31.

In response to the confirmation, the vehicle20performs remote assistance according to the assistance content. In the example illustrated inFIG.5, the remote assistance corresponding to the determination of Go of the assistance content is execution of turning right. At the time T31, the autonomous driving system of the vehicle20determines a path plan to turn right at the intersection, and executes the right turn. The delay time in the downlink direction also includes a time for confirming that the predicted future positional relationship is realized.

The assistance content and the future information transmitted from the remote assistance terminal30are continuously input to the vehicle20. The future information transmitted from the remote assistance terminal30at the time T32includes the positional relationship between the vehicle20and the other vehicle50at the time T41predicted at the time T32. Since the assistance content transmitted from the remote assistance terminal30at the time T32is the determination of Go, the vehicle20determines the degree of coincidence between the future positional relationship predicted by the remote assistance terminal30at the time T32and the actual positional relationship between the vehicle20and the other vehicle50realized at the time T41. In the example shown inFIG.5, since the degree of coincidence is equal to or greater than the threshold value, the vehicle20continues to turn right in accordance with the determination of Go by the operator40.

Next, a description will be given of processing in a case where the future positional relationship predicted by the remote assistance terminal30is not actually realized and remote assistance corresponding to the assistance content input by the operator40cannot be executed. In the example shown inFIG.6, the future positional relationship predicted by the remote assistance device30at the time T22does not match the actual positional relationship between the vehicle20and the other vehicle50realized at the time T31. In this case, the vehicle20does not follow the determination of Go of the assistance content transmitted from the remote assistance terminal30at the time T22. That is, the vehicle20does not make a right turn and continues to be in the standby state. In addition, at the time T31, the vehicle20requests the operator40to provide assistance again. In this example, the assistance requested again by the vehicle20is a determination as to whether the vehicle20may continue to make a right turn.

The assistance content and the future information transmitted from the remote assistance terminal30are continuously input to the vehicle20. In the example shown inFIG.6, the future positional relationship predicted by the remote assistance device30at the time T32does not match the actual positional relationship between the vehicle20and the other vehicle50realized at the time T41. Therefore, the vehicle20does not follow the determination of Go by the operator40and continues to be in the standby state. When assistance is requested again from the vehicle20to the operator40at the time T31, new assistance content and future information are input from the remote assistance terminal30to the vehicle20even after the time T41. When the remote assistance terminal30confirms that the predicted future positional relationship is actually realized, the vehicle20makes a right turn in accordance with the determination of Go by the operator40.

In a case where the future positional relationship predicted by the remote assistance terminal30is not realized in reality and remote assistance corresponding to the assistance content input by the operator40cannot be executed, in the example illustrated inFIG.6, assistance is requested again from the vehicle20to the operator40. However, depending on the situation in which the vehicle20is placed, it is not possible to re-request the operator40to provide assistance. For example, in a situation where the color of the traffic light in the direction of travel is likely to change from blue to red, it is not possible to keep waiting in the intersection.

In the example shown inFIG.7, the future positional relationship predicted by the remote assistance device30at the time T22does not match the actual positional relationship between the vehicle20and the other vehicle50realized at the time T31. In this case, the vehicle20does not follow the determination of Go of the assistance content transmitted from the remote assistance terminal30at the time T22. However, there is no situation in which it is possible to continue to be in the standby state in the intersection while re-requesting assistance to the operator40. In this case, the vehicle20selects a path that can ensure safety. In the example shown inFIG.7, at the time T31, the vehicle20abandons the right turn and selects a path that goes straight through the intersection.

The assistance content and the future information transmitted from the remote assistance terminal30are continuously input to the vehicle20. In the example shown inFIG.7, the assistance content and the future information transmitted from the remote assistance terminal30at the time T32are input to the vehicle20. However, since the vehicle20has already selected a path that can ensure safety, the assistance content transmitted from the remote assistance terminal30is not executed. The vehicle20continuously selects a path that can ensure safety even at the time T41.

As described above, according to the remote assistance method of the present embodiment, the remote assistance corresponding to the assistance content input by the operator40is executed in the vehicle20on condition that it is confirmed that the positional relationship between the vehicle20and the other vehicle50displayed on the display device32when the assistance content is input by the operator40is realized. That is, until the above confirmation is obtained, even if the vehicle20receives the assistance content, the remote assistance corresponding to the assistance content is not executed in the vehicle20. According to this, it is possible to operate the vehicle20at the timing intended by the operator40by eliminating the influence of the shift in the assistance timing caused by the delay time including the communication delay. Therefore, it is possible to improve the reliability of the determination of the operator40.

3. Method of Calculating Degree of Coincidence

3-1. First Example

Three examples of a method of calculating the degree of coincidence will be described.FIG.8is a diagram for explaining a first example of a method of calculating the degree of coincidence.FIG.8schematically illustrates the vehicle20and objects50-1,50-2, . . . , and50-N around the vehicle20. Although not shown, it is assumed that there are N objects around the vehicle20. Surrounding objects include all moving objects such as vehicles, pedestrians, and bicycles. The positions of the vehicle20and the peripheral objects50-1,50-2, . . . , and50-N indicated by broken lines are predicted positions referred to by the operator40in the determination of the assistance content, and the positions indicated by solid lines are actual positions when the vehicle20executes the assistance content.

In the first example, the degree of coincidence is expressed using the following function. The following function is configured such that the smaller the error between the predicted position and the actual position, the closer to 0 the value. For example, a calculation formula for the degree of coincidence may be created such that the degree of coincidence is maximized when the value of the following function is 0.
{(×+×|Δ|)+(×+×|Δ|)}/(1+)

In the above function, De represents the amount of change between the current position and the future position of the vehicle20. ΔVe represents the amount of change between the current velocity of the vehicle20and the future velocity predicted from the path plan. Further, Da,i represents the amount of change between the current position and the future position of the object50-i. ΔVa,i represents the amount of change between the current velocity of the object50-iand the future velocity predicted by the autonomous driving system. αe, βe, αa,i, and βa,i are coefficients. In the above function, the term of De and the term of ΔVe related to the vehicle20may be omitted. In this case, the denominator 1+N may be set to N.

The objects50-1,50-2, . . . , and50-N to be calculated in the calculation of the above function are determined in accordance with the determination content of the operator40and the scene. For example, in the example shown inFIG.8, only the object50-1moving in front of the traveling direction the vehicle20may be set as a calculation target. It is also possible to limit the calculation target only to a specific object, for example, a moving object on a road, a vehicle on an opposite lane, a vehicle approaching the ego-vehicle, or a vehicle in general. In addition, it is possible to exclude a specific object, for example, a following vehicle or a pedestrian from the calculation target.

In the examples shown inFIGS.5to7, although an object other than the other vehicle50may be present around the vehicle20, only the other vehicle50affects the determination of the behavior of the vehicle20when turning right at the intersection. Therefore, in the example shown inFIGS.5to7, when the above-described function is used for calculation of the degree of coincidence, only the other vehicle50is set as a calculation target.

It should be noted that the values of the coefficient αa,i and the coefficient βa,i may be changed in accordance with the magnitude of the influence on the determination of the behavior of the vehicle20after adding all objects existing within a certain range from the vehicle20to the calculation target. For example, a coefficient given to an object in a direction approaching the vehicle20may be larger than a coefficient given to an object in a direction away from the vehicle20.

3-2. Second Example

The second example is an example in which Time to Collision (TTC) is used to determine the degree of coincidence. In the second example, with respect to the object interfering with the path of the vehicle20, the predicted TTC in the future positional relationship displayed on the display device32and the actual TTC in the actual positional relationship when the vehicle20executes the assistance content are calculated. When multiple objects are present in the path of the vehicle20, the objects closest to the front and the rear are regarded as the objects interfering with the vehicle20. When the error between the predicted TTC and the actual TTC is within the threshold value, it is determined that the degree of coincidence is high. In the second example, a relative distance or a relative velocity may be used instead of the TTC.

FIG.9is a diagram for explaining the second example of the method of calculating the degree of coincidence in more detail. In the second example, the threshold for determining the degree of coincidence using the assistance content is changed depending on whether the vehicle interfering with the path of the vehicle20is a preceding vehicle50F or a following vehicle50R. The positions of the preceding vehicle50F and the following vehicle50R indicated by broken lines are predicted positions referred to by the operator40in the determination of the assistance content, and the positions indicated by solid lines are actual positions when the vehicle20executes the assistance content. As shown in the graph and schematic diagram inFIG.9, the threshold for determining the degree of coincidence is set higher for the preceding vehicle50F and lower for the following vehicle50R. The relationship between the error between the predicted TTC and the actual TTC and the degree of coincidence may be set linearly as shown in the graph or may be set non-linearly. Even when the relative distance or the relative velocity is used instead of the TTC, the relationship between the error and the degree of coincidence can be set in any manner as long as the degree of coincidence increases as the error decreases.

Here, the “object interfering with the path of the vehicle20” which is the calculation target of the degree of coincidence in the second example will be described in more detail. The preceding vehicle50F and the following vehicle50R shown inFIG.9are objects that are continuously present on the path of the vehicle20. However, in an actual situation, as schematically shown inFIG.10, an object may cross the path of the vehicle20in an extremely short time. In the determination of whether or not the object interferes with the path of the vehicle20, a certain width (for example, a width approximately equal to the width of the vehicle20) is given to the path of the vehicle20.

When an object crosses the path of the vehicle20, whether or not the object interferes with the path of the vehicle20depends on the actual position of the object when the assistance content is executed. For example, the crossing vehicle50C1in which the predicted position referred to by the operator40in the determination of the assistance content interferes with the path as indicated by the broken line and the actual position when the assistance content is executed also interferes with the path as indicated by the solid line is set as the calculation target of the degree of coincidence in the second example. That is, the error between the predicted TTC and the actual TTC is calculated for the crossing vehicle50C1, and the degree of coincidence is calculated from the error according to the graph shown inFIG.10. When the degree of coincidence is equal to or greater than the threshold value, the vehicle20adopts and executes the assistance content. Even in a case where an object crosses the path of the vehicle20, the relative distance or the relative velocity may be used instead of the TTC.

On the other hand, the crossing vehicle50C2in which the predicted position referred to by the operator40in the determination of the assistance content interferes with the path but the actual position at the time of executing the assistance content passes through the path is excluded from the calculation target of the degree of coincidence in the second example. In this case, since the vehicle20and the crossing vehicle50C2do not interfere with each other, the vehicle20adopts and executes the assistance content without calculating the degree of coincidence. On the other hand, the crossing vehicle50C3in which the predicted position referred to by the operator40in the determination of the assistance content interfere with the path but the actual position when the assistance content is executed does not yet interfere with the path is also excluded from the calculation target of the degree of coincidence in the second example. In this case, the vehicle20does not adopt the assistance content without calculating the degree of coincidence.

3-3. Third Example

The third example is an example in which the Mahalanobis distance is used to determine the degree of coincidence. In a case where the vehicle20predicts positions and velocities of peripheral objects and generates a path plan in consideration of uncertainty of the predicted values, it is appropriate to calculate a deviation between the predicted values and the actual measurement values with reference to the uncertainty. In this case, the Mahalanobis distance can be used as a method of calculating the deviation. The Mahalanobis distance M for each object I around the vehicle20is given by the following equation.

M⁡()=

Here, Si is a covariance matrix representing the uncertainty of the predicted value of the object i, and X→i is the deviation between the predicted value given by the exclusive OR of Di and ΔVi and the actual measured value. Even in a case where the deviation X→i between the predicted value and the actual measured value is slightly large, it can be expected that a large problem does not occur in the operation by the autonomous driving system when the uncertainty of the predicted value given by Si is sufficiently large and the uncertainty is included in the path plan of the vehicle20. It can be said that the definition by M(X→i) is an index of the degree of coincidence reflecting how much uncertainty of the predicted value is included in the path plan of the vehicle20.

4. Configuration of Remote Assistance System

A configuration of the remote assistance system according to the embodiment of the present disclosure will be described.FIG.11is a block diagram showing a configuration of a remote assistance system according to the present embodiment. The remote assistance system100includes a vehicle20and a remote assistance terminal30. However, the remote assistance system100may include the server10that relays between the vehicle20and the remote assistance terminal30, and may further include a communication network that connects the vehicle20and the remote assistance terminal30.

First, the functions of the remote assistance terminal30will be described. The remote assistance terminal30includes a vehicle information reception unit301, a future information generation unit302, a remote assistance display unit303, a remote assistance operation unit304, and a command signal transmission unit305. These are realized as functions of the remote assistance device30as the operator UI management system when the program31cstored in the memory31bof the computer31physically constituting the remote assistance device30is executed by the processor31a.

The vehicle information reception unit301communicates with the vehicle20via the communication device35. The vehicle information reception unit301acquires an assistance request and vehicle information for remote assistance issued by the vehicle20.

The future information generation unit302generates future information related to states of the vehicle20and a peripheral object (a vehicle, a pedestrian, and the like) around the vehicle20at a future time beyond the current time based on the vehicle information transmitted from the vehicle20. To generate the future information, specifically, the transmission time at which the vehicle20transmits the vehicle information, the path plan, the prediction information, and the recognition information included in the vehicle information, and the reception time at which the vehicle information reception unit301receives the vehicle information are used. The future time may be a time in the future by the total delay time in the uplink direction and the downlink direction or may be a time in the future by a preset time. Although the time used for the determination by the operator40is not included in the delay time, an average determination time may be added to the future time as a buffer.

The remote assistance display unit303displays the future information generated by the future information generation unit302on the display device32. The future information includes a positional relationship between the vehicle20and a peripheral object around the vehicle20at a future time beyond the current time, and the positional relationship is spatially displayed on the screen of the display device32. The operator40can arbitrarily select between a display by a three-dimensional model as shown inFIG.4Band a display by a two-dimensional image for the display by the display device32. In the display by the three-dimensional model, for example, when it is necessary to grasp an object near a specific place such as near a crosswalk, the viewpoint in the three-dimensional space can be moved and rotated so that the operator40can easily view the vicinity. In addition, when there is a possibility that occlusion exists in the vicinity of a place where assistance is required, the viewpoint can be moved and rotated to a place where occlusion is likely to exist. When the display by the two-dimensional image is used, it is possible to adopt a method of increasing the resolving power of the image, changing the image quality and the number of pixels, enlarging the image, switching to panorama display, switching the display destination to another display apparatus, or the like.

The remote assistance operation unit304receives an operation by the operator40input to the input device33. The operator determines the assistance content for the vehicle20based on the future information displayed on the display device32by the remote assistance display unit303, and inputs the permission determination of Go or No-Go or the detailed content thereof to the input device33. The remote assistance operation unit304transmits the assistance content input to the input device33to the command signal transmission unit305.

The command signal transmission unit305communicates with the vehicle20through the communication device35. The command signal transmission unit305transmits the future information generated by the future information generation unit302and the assistance content determined by the operator40using the future information to the vehicle20.

Next, functions of the vehicle20will be described. The vehicle20includes a command signal reception unit201, a vehicle command change unit202, an autonomous driving system unit203, and a vehicle information transmission unit204. These are realized as functions of the computer21when the program21cstored in the memory21bof the computer21is executed by the processor21a.

The command signal reception unit201communicates with the remote assistance terminal30through the communication device25. The command signal reception unit201receives the assistance content and the future information transmitted from the remote assistance terminal30.

The vehicle command change unit202extracts information that needs to be changed based on the assistance content and the future information obtained by the command signal reception unit201, and converts the extracted information into signal information that can be received by the autonomous driving system unit203. Specifically, when the assistance content obtained by the command signal reception unit201is the determination of Go to permit the action, the vehicle command change unit202compares the positional relationship between the vehicle20and the peripheral object included in the future information with the positional relationship between the vehicle20and the peripheral object when the assistance content is actually executed. As a result of the comparison, when both do not match, that is, when the degree of coincidence between both is less than a threshold value, the vehicle command change unit202determines that execution of the assistance content is not allowed. If the degree of coincidence between both is equal to or greater than the threshold value, the vehicle command change unit202permits execution of the assistance content and transmits the assistance content to the autonomous driving system unit203.

The autonomous driving system unit203is a function as an autonomous driving system of the computer31. During traveling by normal autonomous driving, the autonomous driving system unit203recognizes a peripheral object on the basis of information on the surrounding environment obtained by the external sensor22, and calculates a path along which the vehicle20travels while predicting a behavior of the recognized peripheral object. Then, when it becomes difficult to determine only with the autonomous driving system unit203or when it is likely to become difficult to determine only with the autonomous driving system unit203, a request for remote assistance is output. When receiving the remote assistance, the autonomous driving system unit203performs calculation for executing the assistance content transmitted from the vehicle command change unit202.

The vehicle information transmission unit204communicates with the remote assistance terminal30through the communication device25. When the remote assistance is requested, the vehicle information transmission unit204transmits the assistance request and the vehicle information necessary for the remote assistance to the server10. After the operator40in charge of assisting the vehicle20is assigned by the server10, the vehicle information is transmitted together with the assistance request to the remote assistance terminal30of the operator40via the server10.

5. Processing in Remote Assistance System

The flow of processing in the remote assistance system100configured as described above is shown inFIG.12.FIG.12is a sequence diagram showing a flow of processing among the vehicle20, the remote assistance terminal30, and the operator40by the remote assistance system100.

First, an assistance request and vehicle information are transmitted from the vehicle information transmission unit204of the vehicle20. Specifically, the assistance request is once received by the server10, and is transmitted to the remote assistance terminal30of the operator40together with the vehicle information after the operator40in charge of assisting the vehicle20is assigned. The assistance request and the vehicle information are received by the vehicle information reception unit301of the remote assistance terminal30.

The future information generation unit302of the remote assistance terminal30that has received the assistance request calculates a delay time for calculating the future information. However, when a fixed value is used as the delay time, this processing is unnecessary.

A future information generation unit302of the remote assistance terminal30generates future information on the states of the vehicle20and the peripheral object at a future time beyond the current time based on the delay time and the vehicle information.

The remote assistance display unit303of the remote assistance terminal30displays the future information generated by the future information generation unit302on the display device32. From the future information displayed on the display device32, the operator40can grasp the positional relationship between the vehicle20and the peripheral object at the future time and determine the assistance content for the assistance request.

The operator40inputs the determined assistance content to the input device33. The remote assistance operation unit304of the remote assistance terminal30receives the assistance content input to the input device33.

The command signal transmission unit305of the remote assistance terminal30transmits the assistance content received by the remote assistance operation unit304to the vehicle20together with the future information generated by the future information generation unit302. The assistance content and the future information are received by the command signal reception unit201of the vehicle20.

Based on the future information input from the remote assistance terminal30, the vehicle command change unit202of the vehicle20confirms that the positional relationship between the vehicle20and the peripheral object displayed on the display device32when the assistance content is input is realized.

In response to the confirmation, the autonomous driving system unit203of the vehicle20executes the remote assistance according to the assistance content. In other words, the remote assistance corresponding to the assistance content is not executed in the vehicle20until the above-described confirmation is obtained. By the remote assistance system100executing the above-described processing, the vehicle20can be operated at the timing intended by the operator40.

Next, processing by the remote assistance terminal30and processing by the vehicle20will be described separately.

FIG.13is a flowchart illustrating an example of processing performed by the remote assistance terminal30. The program31cstored in the memory31bof the remote assistance device30causes the processor31ato execute a series of processes shown in this flowchart, and causes the remote assistance device30to function as the operator UI management system.

Step S101is executed in response to reception of the assistance request from the vehicle20. In step S101, it is determined whether or not the calculated future time is valid. If the calculated future time is valid, step S103is executed. If the future time to be calculated is not valid, step S103is executed after execution of step S102. In step S102, the future time is calculated based on the latest delay time.

The reason why steps S101and S102are executed is as follows. For example, it is assumed that a moving average of separately measured communication delays is used to calculate a future time. In this case, if the communication is interrupted more than expected, the interrupted time is included in the moving average value of the communication delays. Therefore, the future time calculated after the reconnection becomes an invalid time different from the future time calculated from the actual delay time. Alternatively, since the communication state is different between before and after the interruption, the delay time before the interruption may not be helpful. Therefore, when the communication is interrupted more than expected, the future time is calculated based on the latest delay time which is not affected by the interruption. Alternatively, the future time is calculated based on the delay time newly calculated after the interruption.

In step S103, future information at a future time is generated based on the vehicle information received from the vehicle20. In step S104, the future information generated in step S103is displayed on the display device32.

In step S105, it is determined whether or not an assistance content has been acquired from the operator40. Step S104and step S105are repeatedly executed until the assistance content is acquired from the operator40. When the assistance content is acquired from the operator40, step S106is executed. In step S106, the acquired assistance content is transmitted to the vehicle20.

FIG.14is a flowchart illustrating an example of processing performed by the vehicle20. The program21cstored in the memory21bof the vehicle20causes the processor21ato execute a series of processes shown in the flowchart ofFIG.14.

In response to the occurrence of the assistance request, the vehicle20executes step S201. The assistance request is programmed to occur in response to the occurrence of a predefined event. In step S201, it is determined whether or not the prediction information transmitted immediately before to the remote assistance terminal30is valid.

An example of the case where the prediction information is invalid is a case where a predetermined time or more has elapsed since the prediction information transmitted last time. Another example of the prediction information being invalid is that the information on the predicted path of an object interfering with the path of the vehicle20has changed significantly from the previously transmitted information. The change in the information on the predicted path includes, for example, a state in which the interfering point approaches by a certain distance or more, a state in which the acceleration/velocity of the interfering object changes by a certain value or more, and a state in which the interfering object closest to the vehicle20changes to an object different from the previous object.

If the prediction information transmitted immediately before is valid, step S203is executed. When the prediction information transmitted immediately before is not valid, step S203is executed after execution of step S202. In step S202, new prediction information is calculated. In step S203, the valid prediction information is transmitted to the remote assistance device30as the vehicle information together with the path plan.

In step S204, it is determined whether or not the assistance content is received from the remote assistance device30. Step S204is repeatedly performed until the assistance content is received. When the assistance content is received, step S205and step S206are performed in parallel. In step S205, the received assistance content and the future information received together with the assistance content are temporarily stored. In step S206, a path that can ensure safety in the current situation of the vehicle20is calculated and temporarily stored.

Next, in step S207, the degree of coincidence between the future information temporarily stored in step S205and peripheral information at the time of execution of the assistance content is calculated. The future information includes a future positional relationship between the vehicle20and the peripheral object displayed on the display device32when the assistance content is input by the operator40. The peripheral information includes the actual positional relationship between the vehicle20and the peripheral object acquired by the external sensor22. More specifically, the degree of coincidence calculated in step S207is the degree of coincidence between the future positional relationship and the actual positional relationship.

In step S208, it is determined whether or not the degree of coincidence calculated in step S207is within an allowable range, that is, whether or not the degree of coincidence is equal to or greater than a threshold value. If the degree of coincidence is equal to or greater than the threshold value, step S209is executed. In step S209, the assistance content temporarily stored in step S205is executed.

When the degree of coincidence is less than the threshold value, execution of the assistance content temporarily stored in step S205is abandoned, and step S210is executed instead. In step S210, it is determined whether the operator40can be re-requested for assistance. If it is possible to re-request the operator40for assistance, the processing returns to the initial process to generate an assistance request. If it is not possible to re-request the operator40for assistance, step S211is executed. In step S211, the path that can ensure the safety temporarily stored in step S206is executed.

FIG.15is a flowchart illustrating another example of the processing executed by the vehicle20. When this example is adopted, the program21cstored in the memory21bof the vehicle20causes the processor21ato execute a series of processes shown in the flowchart ofFIG.15. Hereinafter, description of processing common to the flowchart ofFIG.14will be omitted or simplified, and processing specific to the flowchart ofFIG.15will be mainly described.

According to the flowchart ofFIG.15, step S205and step S206are followed by step S221. In step S221, the time at which the assistance content can be executed in the vehicle20is compared with the future time of the display device32when the assistance content is input. Then, it is determined whether or not the executable time of the assistance content is later than the future time.

When the executable time of the assistance content does not pass the future time, it can be determined that the positional relationship between the vehicle20and the peripheral object displayed on the display device32will be realized or will be realized in the future. Therefore, when the result of the determination in step S221is affirmative, step S209is executed after waiting until the future time in step S222. When the executable time of the assistance content coincides with the future time, the step S209is immediately executed without waiting.

On the other hand, when the time at which the remote assistance can be executed in the vehicle20passes the future time, it can be determined that the positional relationship between the vehicle20and the peripheral object displayed on the display device32is no longer realized. Therefore, when the result of the determination in step S221is negative, step S209is not executed and step S210is executed.

However, when the result of the determination in step S221is negative, step S223may be executed before execution of step S210. In step S223, the degree of coincidence between the future information temporarily stored in step S205and the peripheral information at the time of execution of the assistance content is calculated.

As the result of the determination, if the degree of coincidence is equal to or greater than the threshold value, step S209may be executed. That is, even when the future time has passed, the assistance content temporarily stored in step S205may be executed as long as the positional relationship between the vehicle20and the peripheral object displayed on the display device32is actually realized. If the degree of coincidence is lower than the threshold value, step S209is not executed and step S210is executed.

When the above-described example is adopted as the processing by the vehicle20, it is preferable that the delay time used for the calculation of the future time is set to be longer than the expected actual delay time. By doing so, after receiving the assistance content, the vehicle20can reliably execute the assistance content after waiting until the future time.