Patent Description:
Patent Literature <NUM> discloses a vehicle control device. When a transition from one driving state to another driving state is scheduled, the vehicle control device performs notification of information relating to the transition of the driving state to an occupant and to other vehicles. The driving state includes manual driving, autonomous driving, and remote driving. <CIT> discloses a method for controlling a motor vehicle remotely.

Patent Literature <NUM>: Japanese Laid-Open Patent Application No. <CIT>.

A system for remotely driving a vehicle based on an operation amount input to a remote driving terminal by a remote operator is known. Such a system is sometimes referred to as a remote driving system. The remote driving system starts remote driving when a situation in which the vehicle requires the remote driving occurs. However, at this time, the remote driving system does not immediately start the remote driving. For starting the remote driving, the remote driving system first performs an initial check for checking that the remote driving can be started. The remote driving system starts the remote driving only after obtaining a good result of the initial check. Therefore, in order to smoothly start the remote driving, it is required to finish the initial check early.

An object of the present disclosure is to provide a technique for finishing an initial check early and starting remote driving smoothly in a remote driving system.

The first aspect of the present disclosure relates to a remote driving system that performs remote driving of a vehicle based on an operation amount input to a remote driving terminal. The remote driving system includes at least one processor. The at least one processor detects a first situation in which the remote driving of the vehicle is required. The at least one processor acquires an urgency level of the first situation. The at least one processor performs an initial check for checking that the remote driving can be started at the remote driving terminal when the first situation is detected. The at least one processor omits a part of the initial check according to the urgency level. The initial check at the remote driving terminal includes: a first initial check that checks communication establishment with the vehicle; a second initial check that checks that a brake actuator of the vehicle operates in response to the operation amount of a brake transmitted to the vehicle; a third initial check that checks that a steering actuator of the vehicle operates in response to the operation amount of a steering; and a fourth initial check that performs steering position alignment between the vehicle and the remote driving terminal.

The second aspect of the present disclosure relates to a remote driving terminal that performs remote driving of a vehicle based on an operation amount input by a remote operator. The remote driving terminal includes at least one processor. The at least one processor acquires information indicating that a first situation in which the remote driving of the vehicle is required is detected. The at least one processor acquires an urgency level of the first situation. The at least one processor performs an initial check for checking that the remote driving terminal can start the remote driving when the first situation is detected. The at least one processor omits a part of the initial check according to the urgency level. The initial check at the remote driving terminal includes: a first initial check that checks communication establishment with the vehicle; a second initial check that checks that a brake actuator of the vehicle operates in response to the operation amount of a brake transmitted to the vehicle; a third initial check that checks that a steering actuator of the vehicle operates in response to the operation amount of a steering; and a fourth initial check that performs steering position alignment between the vehicle and the remote driving terminal.

The third aspect of the present disclosure relates to a method for performing remote driving of a vehicle based on an operation amount input into a remote driving terminal. The method includes detecting a first situation in which the remote driving of the vehicle is required, acquiring an urgency level of the first situation, performing an initial check for checking that the remote driving can be started at the remote driving terminal when the first situation is detected, and omitting a part of the initial check according to the urgency level. The initial check at the remote driving terminal includes: a first initial check that checks communication establishment with the vehicle; a second initial check that checks that a brake actuator of the vehicle operates in response to the operation amount of a brake transmitted to the vehicle; a third initial check that checks that a steering actuator of the vehicle operates in response to the operation amount of a steering; and a fourth initial check that performs steering position alignment between the vehicle and the remote driving terminal.

According to the present disclosure, an urgency level of a situation in which remote driving of a vehicle is required is acquired. Then, a part of an initial check is omitted according to the urgency level. By omitting a part of the initial check, it is possible to finish the initial check early and to smoothly start the remote driving. That is, according to the present disclosure, it is possible to smoothly start the remote driving with the urgency level of the situation in which the remote driving of the vehicle is required taken into consideration.

<FIG> is a schematic diagram illustrating a configuration example of a remote driving system <NUM> according to the present embodiment. The remote driving system <NUM> includes a vehicle <NUM>, a remote driving terminal <NUM>, and a management device <NUM>. The vehicle <NUM>, the remote driving terminal <NUM>, and the management device <NUM> can communicate with each other via a communication network.

The vehicle <NUM> is a vehicle to be remotely driven by a remote operator X. The vehicle <NUM> may be an autonomous driving vehicle. The remote driving terminal <NUM> is a terminal device operated by the remote operator X when the remote operator X performs remote driving of the vehicle <NUM>. The remote driving terminal <NUM> may be referred to as a remote cockpit. The management device <NUM> manages the remote driving system <NUM>. Typically, the management device <NUM> is a management server on a cloud. The management device <NUM> may be composed of a plurality of servers which performs distributed processes.

In addition, the remote driving system <NUM> may include an infrastructure sensor <NUM>. The infrastructure sensor <NUM> includes an infrastructure camera. The infrastructure sensor <NUM> may further include a rainfall sensor or the like. The infrastructure sensor <NUM> is installed in an area where the vehicle <NUM> travels. The infrastructure sensor <NUM> and the management device <NUM> can communicate with each other via a communication network. The infrastructure sensor <NUM> may communicate with the vehicle <NUM> and the remote driving terminal <NUM> directly or via the management device <NUM>.

<FIG> are block diagrams illustrating configuration examples of the vehicle <NUM>, the remote driving terminal <NUM>, and the management device <NUM>, respectively.

<FIG> illustrates a configuration example of the vehicle <NUM>. The vehicle <NUM> includes a communication device <NUM>, a sensor group <NUM>, an actuator <NUM>, and a control device <NUM>. In this example, the vehicle <NUM> is equipped with an autonomous driving system and can perform autonomous driving.

The communication device <NUM> communicates with the outside of the vehicle <NUM>. The communication destination of the communication device <NUM> includes the remote driving terminal <NUM> and the management device <NUM>.

The sensor group <NUM> includes a recognition sensor, a vehicle state sensor, a position sensor, and the like. The recognition sensor recognizes (detects) a situation around the vehicle <NUM>. Examples of the recognition sensor include an in-vehicle camera, a laser imaging detection and ranging (LIDAR), and a radar. The vehicle state sensor detects a state of the vehicle <NUM>. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, a brake hydraulic pressure sensor, and the like. The position sensor detects a position and a direction of the vehicle <NUM>. The position sensor includes, for example, a global navigation satellite system (GNSS) sensor. The sensor group <NUM> may further include a rainfall sensor.

The actuator <NUM> includes a steering actuator, a drive actuator, and a brake actuator. The steering actuator steers the wheels. The steering actuator includes, for example, an electric power steering (EPS). The drive actuator is a power source which generates a driving force. Examples of the drive actuator include an engine, an electric motor, and an in-wheel motor. The brake actuator generates a braking force. For example, the brake actuator controls the brake hydraulic pressure to operate the brake.

The control device <NUM> is a computer which controls the vehicle <NUM>. The control device <NUM> includes at least one processor (processing circuitry) <NUM> (hereinafter, simply referred to as a processor <NUM>) and at least one memory <NUM> (hereinafter, simply referred to as a memory <NUM>). The processor <NUM> executes various processes. For example, the processor <NUM> includes a central processing unit (CPU). The memory <NUM> stores various programs and various kinds of information necessary for processing by the processor <NUM>. By the processor <NUM> executing the program stored in the memory <NUM>, the function of the control device <NUM> is realized. Examples of the memory <NUM> include a volatile memory, a non-volatile memory, a hard disk drive (HDD), and a solid state drive (SSD). The control device <NUM> may include at least one electronic control unit (ECU).

The control device <NUM> controls the actuator <NUM> to control traveling of the vehicle <NUM>. The control device <NUM> acquires vehicle information VCL from the sensor group <NUM>. The vehicle information VCL includes recognition sensor information showing a result of recognition by the recognition sensor, vehicle state information acquired from the vehicle state sensor, and position information acquired from the position sensor. The recognition sensor information includes an image captured by the in-vehicle camera. The vehicle state information includes speed information, acceleration information, steering angle information, brake hydraulic pressure information, and the like. They are acquired from the speed sensor, the acceleration sensor, the steering angle sensor, the brake hydraulic pressure sensor, and the like. The vehicle information VCL may further include highly accurate position information acquired by localization. The control device <NUM> can acquire highly accurate position information by performing the localization using map information and the recognition sensor information stored in the memory <NUM>. The vehicle information VCL acquired by the control device <NUM> can be used for the autonomous driving or the remote driving of the vehicle <NUM>.

For example, the control device <NUM> controls the autonomous driving of the vehicle <NUM> based on the vehicle information VCL. More specifically, the control device <NUM> generates a travel plan of the vehicle <NUM> based on the vehicle information VCL. Further, the control device <NUM> generates a target path necessary for the vehicle <NUM> to travel in accordance with the travel plan based on the vehicle information VCL. The target path is a gathering of target positions of the vehicle <NUM>. The target path may be set to extend along the center of the lane. Then, the control device <NUM> controls traveling of the vehicle <NUM> such that the vehicle <NUM> follows the target path.

The control device <NUM> can communicate via the communication device <NUM> and transmit the vehicle information VCL to the remote driving terminal <NUM> and the management device <NUM>. At least while the remote driving of the vehicle <NUM> is performed, the vehicle information VCL is transmitted to the remote driving terminal <NUM>. The vehicle information VCL transmitted to the remote driving terminal <NUM> is referred to by the remote operator X, and the remote driving is performed.

<FIG> illustrates a configuration example of the remote driving terminal <NUM>. The remote driving terminal <NUM> includes a communication device <NUM>, an output device <NUM>, a remote operation member <NUM>, and a control device <NUM>.

The communication device <NUM> communicates with the vehicle <NUM> and the management device <NUM>.

The output device <NUM> outputs various kinds of information and presents the information to the remote operator X. For example, the output device <NUM> includes a display device. The display device displays various kinds of information to the remote operator X. As another example, the output device <NUM> may include a speaker.

The remote operation member <NUM> is a member operated by the remote operator X when the remote operator X remotely drives the vehicle <NUM>. The remote operation member <NUM> includes, for example, a steering operation member, an accelerator pedal, a brake pedal, a direction indicator, and the like. The steering operation member is, for example, a steering wheel. The remote operation member <NUM> may include a touch panel, a button, or the like. An operation amount input by the remote operator X during the remote driving of the vehicle <NUM> is detected by a sensor installed in the remote operation member <NUM>.

The control device <NUM> controls the remote driving terminal <NUM>. The control device <NUM> includes at least one processor (processing circuitry) <NUM> (hereinafter, simply referred to as a processor <NUM>) and at least one memory <NUM> (hereinafter, simply referred to as a memory <NUM>). The processor <NUM> executes various processes. For example, the processor <NUM> includes a CPU. The memory <NUM> stores various programs and various kinds of information necessary for processing by the processor <NUM>. By the processor <NUM> executing the program stored in the memory <NUM>, the function of the control device <NUM> is realized. Examples of the memory <NUM> include a volatile memory, a non-volatile memory, an HDD, and an SSD.

The control device <NUM> communicates with the vehicle <NUM> via the communication device <NUM>. The control device <NUM> receives the vehicle information VCL transmitted from the vehicle <NUM>. The control device <NUM> presents the vehicle information VCL to the remote operator X by displaying the vehicle information VCL including the image on the display device. The remote operator X can recognize the state of the vehicle <NUM>, the situation around the vehicle <NUM>, or the like based on the vehicle information VCL displayed on the display device.

In addition, the control device <NUM> may acquire infrastructure information detected by the infrastructure sensor <NUM> directly or via the management device <NUM>. The infrastructure information acquired by the control device <NUM> may include, for example, an image acquired by the infrastructure camera capturing the vehicle <NUM> and the surroundings thereof. The acquired infrastructure information is displayed on the display device. The remote operator X may recognize the state of the vehicle <NUM>, the situation around the vehicle <NUM>, or the like by referring to the infrastructure information. By presenting the image captured by the infrastructure camera to the remote operator X in addition to the image captured by the in-vehicle camera, accuracy of the remote driving or usability for the remote operator X can be improved.

The control device <NUM> generates remote driving information REM based on the operation amount input by the remote operator X. The remote driving information REM is information for controlling the vehicle <NUM> by the remote driving. The remote driving information REM includes the operation amount of the remote operation member <NUM> input by the remote operator X. The control device <NUM> transmits the remote driving information REM to the vehicle <NUM> as necessary.

<FIG> illustrates a configuration example of the management device <NUM>. The management device <NUM> includes a communication device <NUM> and a control device <NUM>.

The communication device <NUM> communicates with the vehicle <NUM>, the remote driving terminal <NUM>, and the infrastructure sensor <NUM>.

The control device <NUM> controls the management device <NUM>. The control device <NUM> includes at least one processor (processing circuitry) <NUM> (hereinafter, simply referred to as a processor <NUM>) and at least one memory <NUM> (hereinafter, simply referred to as a memory <NUM>). The processor <NUM> executes various processes. For example, the processor <NUM> includes a CPU. The memory <NUM> stores various programs and various kinds of information necessary for processing by the processor <NUM>. By the processor <NUM> executing the program stored in the memory <NUM>, the function of the control device <NUM> is realized. Examples of the memory <NUM> include a volatile memory, a non-volatile memory, an HDD, and an SSD.

The control device <NUM> communicates with the vehicle <NUM> and the remote driving terminal <NUM> via the communication device <NUM>. In addition, the control device <NUM> communicates with the infrastructure sensor <NUM> via the communication device <NUM> as necessary to acquire the infrastructure information. The infrastructure information which the control device <NUM> acquires from the infrastructure sensor <NUM> includes the image captured by the infrastructure camera.

The remote driving system <NUM> performs the remote driving of the vehicle <NUM> based on the operation amount input into the remote driving terminal <NUM> by the remote operator X. When the remote driving system <NUM> detects a "first situation", the remote driving of the vehicle <NUM> is started. The first situation is a situation in which the remote driving of the vehicle <NUM> is required. The first situation may be detected by the vehicle <NUM> itself or may be detected by the management device <NUM>.

The first situation may be, for example, a situation in which a remote driving request (Request for Operation, RFO) is transmitted from the vehicle <NUM> to the management device <NUM>. For example, when the autonomous driving system of the vehicle <NUM> determines that it is difficult to continue the autonomous driving, the remote driving request is transmitted from the vehicle <NUM> to the management device <NUM>. The remote driving request may be transmitted from the vehicle <NUM> while the vehicle <NUM> is traveling or may be transmitted while the vehicle <NUM> is stopped.

Alternatively, the remote driving system <NUM> may detect the first situation based on the infrastructure information acquired from the infrastructure sensor <NUM>. For example, when the rainfall sensor of the infrastructure sensor <NUM> shows that average rainfall in a place where the vehicle <NUM> travels is larger than a threshold value, it is presumed that it is difficult for the vehicle <NUM> to continue the autonomous driving. As another example, when an abnormal state of the vehicle <NUM> is detected from the image captured by the infrastructure camera of the infrastructure sensor <NUM>, it is presumed that the vehicle <NUM> has difficulty in continuing the autonomous driving. For example, the abnormal state of the vehicle <NUM> is a state in which the vehicle <NUM> is meandering. As another example, the abnormal state is a state in which the vehicle <NUM> is stopped although it should be traveling. The remote driving system <NUM> may detect the abnormal state of the vehicle <NUM> like these as the first situation.

Alternatively, the remote driving system <NUM> may detect the first situation based on deviation of the vehicle <NUM> from the target path during the autonomous driving. As shown in <FIG>, there is a situation where a traveling position of the vehicle <NUM> deviates from the target path and a deviation amount becomes larger than a threshold value. The remote driving system <NUM> may detect a situation like this as the first situation. The deviation amount can also be referred to as a distance between the vehicle <NUM> and the target path. The remote driving system <NUM> can calculate the deviation amount by comparing the position of the vehicle <NUM> with the target path. The remote driving system <NUM> may acquire the position of the vehicle <NUM> from the position information of the vehicle <NUM> acquired from the vehicle information VCL or may acquire the position of the vehicle <NUM> from the image captured by the infrastructure camera of the infrastructure sensor <NUM>. Information about the target path to be followed by the vehicle <NUM> can be acquired from the vehicle information VCL.

When the remote driving system <NUM> detects the first situation, the management device <NUM> assigns a certain remote operator X from among a plurality of candidates to the vehicle <NUM>. The management device <NUM> manages assignment relationship between the vehicle <NUM> and the remote operator X and provides information about the assignment relationship to the vehicle <NUM> and the remote driving terminal <NUM>. The vehicle <NUM> and the remote driving terminal <NUM> which have received the information about the assignment relationship establish communication.

While the remote driving is performed, the vehicle <NUM> and the remote driving terminal <NUM> transmit and receive information via the communication network. The communication between the vehicle <NUM> and the remote driving terminal <NUM> may be performed directly or via the management device <NUM>.

The vehicle <NUM> transmits the vehicle information VCL to the remote driving terminal <NUM>. The remote driving terminal <NUM> receives the vehicle information VCL transmitted from the vehicle <NUM> and presents the vehicle information VCL to the remote operator X. For example, the remote driving terminal <NUM> presents the vehicle information VCL by displaying the image on the display device of the output device <NUM>. The remote operator X recognizes the situation around the vehicle <NUM> by viewing the displayed information and performs the remote driving of the vehicle <NUM> by operating the remote operation member <NUM>.

The control device <NUM> generates the remote driving information REM based on the operation amount of the remote operation member <NUM> input by the remote operator X. Then, the control device <NUM> transmits the remote driving information REM to the vehicle <NUM> via the communication device <NUM>.

The vehicle <NUM> receives the remote driving information REM transmitted from the remote driving terminal <NUM>. The vehicle <NUM> controls traveling of the vehicle in accordance with the received remote driving information REM. In this way, the remote driving of the vehicle <NUM> is performed.

As described above, the remote driving of the vehicle <NUM> is started by the remote driving system <NUM> when the first situation is detected. However, the remote driving is not started immediately after the first situation is detected. If the first situation is detected, the remote driving system <NUM> first performs an "initial check" for checking that the remote driving can be started. Then, the remote driving system <NUM> starts the remote driving when it is determined that the remote driving can be started in accordance with a result of the initial check.

<FIG> illustrates a specific example of the initial check. The initial check is performed at the remote driving terminal <NUM>. As illustrated in <FIG>, the initial check includes several steps.

In the first step of the initial check, "communication establishment check" is performed for checking that communication between the vehicle <NUM> and the remote driving terminal <NUM> is normally established. More specifically, in the communication establishment check, it is checked that the vehicle <NUM> and the remote driving terminal <NUM> assigned to this vehicle <NUM> are normally connected to each other and are in a communicable state. For example, the remote driving terminal <NUM> transmits a test signal for the communication establishment check to the vehicle <NUM>. The test signal may be a ping. Then, when the remote driving terminal <NUM> normally receives a response signal transmitted from the vehicle <NUM> in response to the test signal, it is determined that the communication is normally established. When it is determined that the communication is normally established, it means that necessary information can be accurately transmitted and received between the vehicle <NUM> and the remote driving terminal <NUM>.

By the communication establishment check being completed, it is guaranteed that the remote driving terminal <NUM> can normally receive the vehicle information VCL from the vehicle <NUM>. The initial check proceeds to the next step, and "steering position alignment" is performed. The steering position alignment means bringing a steering angle of the steering operation member of the remote driving terminal <NUM> in line with a steering angle of the vehicle <NUM>. The steering angle of the vehicle <NUM> is a steering angle of a steering wheel of the vehicle <NUM>. Alternatively, the steering angle of the vehicle <NUM> may be calculated from a steering angle of the wheels of the vehicle <NUM>. Information about the steering angle of the steering wheel or the steering angle of the wheels of the vehicle <NUM> can be acquired from the vehicle information VCL transmitted from the vehicle <NUM> to the remote driving terminal <NUM>. The steering angle of the steering operation member of the remote driving terminal <NUM> is detected by the sensor installed in the steering operation member.

<FIG> is a diagram illustrating an example of an image displayed on the display device of the remote driving terminal <NUM> when the steering position alignment is performed. A traveling trajectory <NUM>, which is shown by dotted lines, represents a traveling trajectory of the vehicle <NUM> calculated from the current steering angle of the vehicle <NUM>. On the other hand, a traveling trajectory <NUM>, which is shown by solid lines, represents a traveling trajectory of the vehicle <NUM> calculated from the steering angle of the steering operation member of the remote driving terminal <NUM>. The remote driving terminal <NUM> draws the traveling trajectories <NUM> and <NUM> such that they are superimposed on the image included in the vehicle information VCL. Then, the remote driving terminal <NUM> displays the image on which the travel trajectories <NUM> and <NUM> are superimposed on the display device. The steering position alignment means adjusting the steering angle of the steering operation member such that the two traveling trajectories <NUM> and <NUM> come into line with each other. Normally, the steering position alignment is performed by the remote operator X controlling the steering operation member such that the traveling trajectory <NUM> and the traveling trajectory <NUM> come into line with each other. Whether the two trajectories are in line with each other may be automatically determined by the remote driving terminal <NUM> or may be manually determined by the remote operator X. In the case where the determination is made manually, the remote operator X uses a button or the like of the remote operation member <NUM> to input information indicating that the two trajectories come into contact with each other.

By the steering position alignment being completed, it is ready to reflect the operation amount of the steering operation member in steering of the wheels of the vehicle <NUM>. Thereafter, the remote driving information REM can be transmitted from the remote driving terminal <NUM> to the vehicle <NUM>.

Reference is made again to <FIG>. After the steering position alignment is completed and the steering angle of the steering operation member and the steering angle of the wheels correspond to each other, an operation check is performed. The operation check is for checking that the operation amount of the remote operator X transmitted to the vehicle <NUM> as the remote driving information REM is reflected in the operation of the actuator <NUM>, that is, the actuator <NUM> operates in response to the operation amount of the remote operator X. The operation check includes an operation check for the brake and an operation check for the steering. Order of performing the operation check for the brake and performing the operation check for the steering is not limited.

The operation check for the brake is for checking that the operation amount of the brake input into the brake pedal of the remote operation member <NUM> by the remote operator X is reflected in operation of the brake actuator of the vehicle <NUM>. The remote driving terminal <NUM> transmits a test signal to the vehicle <NUM> for requesting the brake actuator to operate. For example, the remote driving terminal <NUM> transmits the remote driving information REM including the operation amount of the brake pedal of the remote operation member <NUM> (test signal) to the vehicle <NUM>. The brake hydraulic pressure is expected to change normally if the brake actuator operates normally in response to this test signal. Therefore, the remote driving terminal <NUM> receives the vehicle information VCL from the vehicle <NUM> and monitors information about the brake hydraulic pressure included in the vehicle information VCL. If the brake hydraulic pressure changes as expected in response to the transmission of the test signal, the remote driving terminal <NUM> determines that the brake actuator operates normally. If the brake actuator operates normally, the operation check for the brake is finished.

The operation check for the steering is performed for checking that the operation amount input into the steering operation member of the remote operation member <NUM> by the remote operator X is reflected in operation of the steering actuator and the wheels of the vehicle <NUM> steers. The remote driving terminal <NUM> transmits a test signal to the vehicle <NUM> for requesting the steering actuator to operate. For example, the remote driving terminal <NUM> transmits the remote driving information REM including the operation amount of the steering operation member of the remote operation member <NUM> (test signal) to the vehicle <NUM>. The steering angle of the vehicle <NUM> is expected to change normally if the steering actuator operates normally in response to this test signal. Therefore, the remote driving terminal <NUM> receives the vehicle information VCL from the vehicle <NUM> and monitors the steering angle information included in the vehicle information VCL. If the steering angle of the vehicle <NUM> changes as expected in response to the transmission of the test signal, the remote driving terminal <NUM> determines that the steering actuator operates normally. If the steering actuator operates normally, the operation check for the steering is finished.

The specific example of the initial check is described above. When detecting the first situation, the remote driving system <NUM> first performs the initial check exemplified in <FIG>. Then, after it is checked that the remote driving can be started as a result of the initial check, the remote driving system <NUM> starts the remote driving. In other words, the remote driving system <NUM> cannot start the remote driving until the initial check is finished. If the initial check takes a long time, the time when the remote driving is started becomes late accordingly.

However, there is a possibility that the remote driving of the vehicle <NUM> is required to start early in a case where the first situation is an urgent situation. For example, in a case where the vehicle <NUM> stops in a dangerous area, it is required to start the remote driving early. In order to start the remote driving early, it is necessary to finish the initial check early.

Therefore, when detecting the first situation, in which the remote driving of the vehicle <NUM> is required, the remote driving system <NUM> acquires an " urgency level," which is an index indicating a degree of urgency of the first situation. Then, the remote driving system <NUM> omits a part of the initial check according to the urgency level. By omitting a part of the initial check, the time necessary for the initial check can be shortened, and the remote driving can be started early. That is, it is possible to smoothly start the remote driving in consideration of the urgency level of the situation in which the remote driving of the vehicle <NUM> is required.

Omitting a part of the initial check means omitting any one or more steps from among the plurality of steps included in the initial check. Which process is omitted is determined based on the urgency level and "priority".

First, the priority is described. The priority of omission is set for each process included in the initial check in accordance with content of each process. <FIG> is a priority map showing the priority of omission. The priority map is stored in any of the memory <NUM> of the vehicle <NUM>, the memory <NUM> of the remote driving terminal <NUM>, and the memory <NUM> of the management device <NUM>. When the initial check is omitted, a process having high priority is preferentially omitted in accordance with a priority map like this.

As shown in the priority map of <FIG>, the priority of omission of the operation check is higher than that of the communication establishment check, and the priority of omission of the steering position alignment is higher than that of the operation check.

The communication establishment check is a process that is most important to be performed in advance when the remote driving of the vehicle <NUM> is started. If the communication between the vehicle <NUM> and the remote driving terminal <NUM> is not normally established, the vehicle information VCL and the remote driving information REM cannot be transmitted and received between the vehicle <NUM> and the remote driving terminal <NUM>. If the vehicle information VCL is not normally transmitted from the vehicle <NUM> to the remote driving terminal <NUM>, the remote operator X cannot obtain information necessary for the remote driving. Further, if the remote driving information REM is not normally transmitted from the remote driving terminal <NUM> to the vehicle <NUM>, the remote driving cannot be performed in the first place. Therefore, the priority of omission of the communication establishment check is set to be the lowest.

Conversely, the steering position alignment is not necessarily required to be performed before the remote driving is started. In a case where the steering position alignment is performed as the initial check, the remote operator X can start to operate the steering operation member with the steering angle of the steering operation member being in line with the steering angle of the wheels, and thus can easily get the feel of the steering. However, even if the steering angle of the steering operation member is not in line with the steering angle of the wheels, the remote operator X can operate the steering operation member to input the operation amount. Therefore, the priority of omission of the steering position alignment is set to be the highest.

In addition, among the operation check, the priority of omission of the operation check for the steering is set to be higher than the priority omission of the operation check for the brake. Between the brake and the steering, the operation check for the brake is more important to be performed in advance than that for the steering. This is because, it is considered that if a situation in which the vehicle <NUM> needs to avoid danger urgently, the remote operator X operates the brake first. Therefore, the priority of omission of the operation check for the steering is set to be higher than that for the brake.

Next, the urgency level will be described. When a part of the initial check is omitted, an amount of omission of the initial check is determined based on the urgency level. The remote driving is required to be started earlier as the urgency level becomes higher. Therefore, the amount of omission of the initial check is made larger as the urgency level becomes higher, and the amount of omission of the initial check is made smaller as the urgency level becomes lower.

The following are examples of a method for calculating the urgency level. For example, in a case where a situation in which the position of the vehicle <NUM> deviates from the target path is detected as the first situation as shown in <FIG>, the urgency level may be calculated based on the deviation amount of the position of the vehicle <NUM> from the target path. In this case, the urgency level is calculated to be higher as the deviation amount becomes larger.

Alternatively, the urgency level may be calculated based on a situation that the vehicle <NUM> is in at the time of detection of the first situation. For example, the urgency level may be calculated as follows. When the vehicle <NUM> is in a dangerous situation, the urgency level is calculated to be the highest. When the vehicle is not in the dangerous situation but in a situation where the vehicle may interfere with other traffic participants, the urgency level is calculated to be the next highest. In the other cases, the urgency level is calculated to be still lower. The dangerous situation is, for example, a situation in which it is raining heavily and the vehicle <NUM> is passing an underpass or is stopped in an underpass. As other examples, the dangerous situation is a situation where an accident such as a fire occurs at a place where the vehicle <NUM> is located or in the vicinity thereof or a situation where the vehicle <NUM> is in a railroad crossing and alarm sound is ringing. Examples of a situation where the vehicle may interfere with other traffic participants includes a situation where an emergency vehicle is approaching the vehicle <NUM>.

The remote driving system <NUM> can determine a situation like these based on information acquired from the infrastructure sensor <NUM> or the sensor group <NUM>. For example, the remote driving system <NUM> can detect a situation where it rains heavily based on information acquired from the rainfall sensor of the sensor group <NUM> or the rainfall sensor of the infrastructure sensor <NUM>. As another example, the remote driving system <NUM> can determine that the vehicle <NUM> is passing or stopping in the underpass or the situation around the vehicle <NUM> including whether the emergency vehicle is approaching based on information acquired from the in-vehicle camera or the infrastructure camera.

As another example of the method for calculating the urgency level, in a case where the vehicle <NUM> is stopped or is expected to stop when the first situation is detected, the urgency level may be calculated based on a stop position of the vehicle <NUM>. For example, the urgency level may be calculated as follows. When the stop position of the vehicle <NUM> is in the dangerous area, the urgency level is calculated to be the highest. When the stop position of the vehicle <NUM> is not in the dangerous area but in an area where the vehicle <NUM> may interfere with other traffic participants, the urgency level is calculated to be the next highest. In the other cases, the urgency level is calculated to be still lower.

The dangerous area mentioned here is a place where the vehicle <NUM> may be in danger if the vehicle <NUM> continues to stop in the area. Examples of the dangerous area include an area on a track of a streetcar, an area in the railroad crossing, and a stopping prohibited area. Examples of the stopping prohibited area include an area in front of a fire station. Examples of the area where the vehicle <NUM> may interfere with other traffic participants include the vicinity of an intersection and a road with a heavy traffic.

The remote driving system <NUM> can determine the stop position of the vehicle <NUM> based on information detected by the infrastructure sensor <NUM> or the sensor group <NUM>. For example, the remote driving system <NUM> can determine that the vehicle <NUM> is stopped in the railroad crossing by acquiring the image captured by the in-vehicle camera or the infrastructure camera. Alternatively, the stop position of the vehicle <NUM> may be acquired by comparing the map information indicating a location of the railroad, the railroad crossing, or the like and the position information acquired from the GNSS sensor of the sensor group <NUM>. The map information is stored in any of the memory <NUM> of the vehicle <NUM>, the memory <NUM> of the remote driving terminal <NUM>, and the memory <NUM> of the management device <NUM>. The remote driving system <NUM> may determine whether the traffic is heavy at the stop position of the vehicle <NUM> based on road traffic information which the management device <NUM> has, road traffic information acquired from an external server, or the like.

Processing executed by the remote driving system <NUM> will be described with reference to <FIG> and <FIG>.

<FIG> is a block diagram illustrating an example of a functional configuration of the remote driving system <NUM>. <FIG> is a flowchart illustrating an example of processing executed by the remote driving system <NUM>. The remote driving system <NUM> includes a first situation detection unit <NUM>, an urgency level calculation unit <NUM>, an initial check unit <NUM>, and a remote driving start determination unit <NUM> as functional blocks.

In Step S110, the first situation detection unit <NUM> detects the first situation. The first situation is a situation in which the remote driving of the vehicle <NUM> is required. A processing entity which realizes the first situation detection unit <NUM> may be the processor <NUM> of the vehicle <NUM> or the processor <NUM> of the management device <NUM>. For example, the processor <NUM> of the vehicle <NUM> may detect the first situation based on the information acquired from the sensor group <NUM>. Alternatively, the processor <NUM> of the management device <NUM> may detect the first situation based on the information acquired from the infrastructure sensor <NUM>. The first situation detection unit <NUM> transmits information indicating that the first situation is detected to the urgency level calculation unit <NUM>, and then the processing proceeds to Step S120.

In Step S120, the urgency level calculation unit <NUM> calculates the urgency level. A processing entity which realizes the urgency level calculation unit <NUM> may be the processor <NUM>, the processor <NUM>, or the processor <NUM>. Alternatively, the urgency level calculation unit <NUM> may be realized by cooperation of these processors.

For example, the processor <NUM> of the vehicle <NUM> acquires information about the stop position of the vehicle <NUM> from the GNSS sensor of the sensor group <NUM>. The processor <NUM> also acquires information indicating a location of the dangerous area from the map information stored in the memory <NUM>. Then, the processor <NUM> calculates the urgency level based on whether the stop position of vehicle <NUM> is included in the dangerous area or not. As another example, the processor <NUM> of the management device <NUM> may detect the traveling position of the vehicle <NUM> based on the infrastructure information acquired from the infrastructure sensor <NUM>. Then, the processor <NUM> or the processor <NUM> may calculate the urgency level by comparing the traveling position of the vehicle <NUM> with the target path. The urgency level calculation unit <NUM> transmits information about the calculated urgency level to the initial check unit <NUM>, and then the processing proceeds to Step S130.

In Step S130, the initial check unit <NUM> performs the initial check. The initial check unit <NUM> is realized by the processor <NUM> of the remote driving terminal <NUM>. When performing the initial check, the initial check unit <NUM> omits a part of the initial check in accordance with the urgency level calculated by the urgency level calculation unit <NUM>. Which step (steps) of the initial check is omitted is determined in accordance with to the urgency level and the priority as described above. When the initial check is finished, the initial check unit <NUM> transmits the result of the initial check to the remote driving start determination unit <NUM>, and then the processing proceeds to Step S140.

In Step S140, the remote driving start determination unit <NUM> determines whether the remote driving can be started or not. A processing entity which realizes the remote driving start determination unit <NUM> may be the processor <NUM>, the processor <NUM>, or the processor <NUM>. Alternatively, the remote driving start determination unit <NUM> may be realized by cooperation of these processors. The remote driving start determination unit <NUM> determines whether the remote driving can be started or not according to the result of the initial check received from the initial check unit <NUM>. If the result of the initial check indicates that the remote driving cannot be started (Step S140; No), the processing ends.

On the other hand, if the result of the initial check indicates that the remote driving can be started (Step S140; Yes), the processing proceeds to Step S150. In Step S150, the remote driving start determination unit <NUM> starts the remote driving of the vehicle <NUM>.

<FIG> is a time chart illustrating an effect of omitting a part of the initial check of the remote driving system <NUM>. An upper chart shows a case in a comparative example, and a lower chart shows a case in an example according to the embodiment realized by the remote driving system <NUM>. At time T1, the first situation is detected. It is the same in both the comparative example and the example according to the embodiment that the initial check is started in response to the detection of the first situation.

In the comparative example, the remote driving is started at time T3 after the initial check is finished. On the other hand, in the example according to the embodiment, since a part of the initial check is omitted, the time required for the initial check is shortened. Therefore, the remote driving is started at time T2, which is the time earlier than time T3.

As described above, according to the remote driving system <NUM> of the present embodiment, a part of the initial check is omitted in accordance with the urgency level, and thus the initial check can be finished early. In this way, the time from when the remote driving is required to when the remote driving is started can be shortened. Further, when the urgency level is particularly high, the time until the remote driving is started can be further shortened by increasing the amount of omission. In addition, which step of the initial check is omitted is determined according to the priority. Since the priority of a particularly important process is set to be low, this process is performed without being omitted, and it is possible to prevent a failure from being found after the remote driving is started. In this way, the remote driving can be started smoothly.

As an example of a scene to which the remote driving system <NUM> according to the present embodiment is applied, a scene of "autonomous transportation in a factory" is considered.

In the autonomous transportation in the factory, the autonomous driving vehicle autonomously travels in a factory ground. For example, the autonomous driving vehicle assembled in an assembly factory travels from the assembly factory to a yard by autonomously traveling along a predetermined route. One or more infrastructure cameras are installed on a road from the assembly factory to the yard. By using the infrastructure camera, the management device <NUM> of the remote driving system <NUM> can remotely monitor the autonomous driving vehicle that is autonomously traveling.

When a situation in which the automatic driving of the autonomous driving vehicle becomes difficult for some reason and the vehicle deviates from the target route is detected, the initial check is performed to start the remote driving. In a case where a route on which the autonomous driving vehicle travels is determined in advance as in the case of the autonomous transportation in the factory, the management device <NUM> can detect such a situation based on the information about the route stored in the memory <NUM> and the image captured by the infrastructure camera. When the initial check is performed, a part of the initial check is omitted according to the deviation amount from the target route. By omitting a part of the initial check, the remote driving can be quickly started. When the situation in which it is difficult that the autonomous driving vehicle continue to autonomously travel occurs, it is also considered to send a staff to the site and start to drive the vehicle by manual driving, but it takes time and labor. Dealing with the situation by the remote driving is more convenient and reduces time and labor.

<FIG> is a diagram for explaining the first modification. In the first modification, the in-vehicle camera includes a front camera capturing a forward image as seen from the vehicle <NUM> and a side camera capturing a lateral image as seen from the vehicle <NUM>. The initial check includes a camera check. The camera check is for checking that the in-vehicle camera operates normally. In other words, the camera check is checking that the image captured by the in-vehicle camera is normally acquired. The camera check further includes a front camera check for the front camera and a side camera check for the side camera.

The priority of omission of the side camera check is set to be higher than the priority of omission of the front camera check. It is considered that the remote operator X acquires more information from the image captured by the front camera than from that by the side camera during the remote driving. Therefore, the priority of omission of the side camera check is set to be higher. Similarly, in a case where the in-vehicle cameras include a camera other than the side camera, the priority of omission of the camera check for the camera other than the front camera is set to be higher than the priority of omission of the front camera check.

As a specific example of an applied scene, a case where the vehicle <NUM> is urgently stopped in the railroad crossing is considered. Since the urgency level of the first situation is high, the camera check for the camera other than the front camera is omitted, and in the initial check, it is checked that at least the front camera is normally operating. The remote operator X can start the vehicle <NUM> traveling by the remote driving referring to at least the image captured by the front camera and can make the vehicle <NUM> exit from the railroad crossing early.

<FIG> is a block diagram illustrating an example of a functional configuration of the remote driving system <NUM> according to the second modification. In the second modification, the remote driving system <NUM> includes a vehicle speed limitation unit <NUM> as a functional block.

The vehicle speed limitation unit <NUM> acquires information about whether the initial check is omitted or not from the initial check unit <NUM>. When a part of the initial check is omitted, the vehicle speed limitation unit <NUM> sets an upper limit to a vehicle speed of the vehicle <NUM>. The upper limit of the vehicle speed set here may be determined unconditionally or may be set in accordance with where the vehicle <NUM> is located. For example, the vehicle speed may be set to be a predetermined amount lower than a speed limit of the road on which the vehicle <NUM> is.

The vehicle speed limitation unit <NUM> may be included in the vehicle <NUM>. In this case, the control device <NUM> controls the vehicle <NUM> such that the vehicle speed does not exceed the upper limit. Alternatively, the vehicle speed limitation unit <NUM> may be included in the remote driving terminal <NUM>. In this case, the vehicle speed limitation unit <NUM> limits the input of the remote operator X such that the vehicle speed does not exceed the upper limit. Alternatively, the vehicle speed limitation unit <NUM> may be included in the management device <NUM>. In this case, information about the vehicle speed set by the management device is transmitted to the vehicle <NUM> or the remote driving terminal <NUM>.

Claim 1:
A remote driving system (<NUM>) that performs remote driving of a vehicle (<NUM>) based on an operation amount input into a remote driving terminal (<NUM>), the remote driving system (<NUM>) comprising at least one processor (<NUM>, <NUM>, <NUM>) configured to:
detect a first situation in which the remote driving of the vehicle (<NUM>) is required; and
perform an initial check for checking that the remote driving can be started at the remote driving terminal (<NUM>) when the first situation is detected, characterised in that
the initial check at the remote driving terminal (<NUM>) includes:
a first initial check that checks communication establishment with the vehicle (<NUM>);
a second initial check that checks that a brake actuator (<NUM>) of the vehicle (<NUM>) operates in response to the operation amount of a brake transmitted to the vehicle;
a third initial check that checks that a steering actuator (<NUM>) of the vehicle (<NUM>) operates in response to the operation amount of a steering; and
a fourth initial check that performs steering position alignment between the vehicle (<NUM>) and the remote driving terminal (<NUM>), and wherein
at least one processor (<NUM>, <NUM>, <NUM>) is further configured to:
acquire an urgency level of the first situation; and
omit a part of the initial check according to the urgency level.