Patent Description:
Patent Literature <NUM> discloses a vehicle control device. When a transition from one driving state to another driving state is scheduled, the vehicle control device performs notification of information relating to the transition of the driving state to an occupant and to other vehicles. The driving state includes manual driving, autonomous driving, and remote driving.

<CIT> discloses a method performed by a first server communicatively connected to a network for supporting tele-operated driving of a vehicle. The method includes receiving a request from a tele-operated driving application client by the client device in the vehicle for tele- operated driving support. The client processes the request and sends a trigger request for teleoperated driving to the first server. This request includes a trigger and information for supporting a tele-operated driving event. The first server processes the request to identify a second server for providing support for the Tele-operated driving event, wherein the processing is based on parameters of the network for providing tele-operated driving support. Finally the first server will set-up a tele-operated driving session between the second server and the application client. The first server may also predict whether a tele-operated driving event is likely in a future time period based on prediction information obtained from one or more of the vehicle, the second server, a network server, and a third party application server.

<CIT> discloses an operator monitoring sensor which is a sensor for monitoring the state of a remote operator that performs remote control of an autonomous driving vehicle.

Patent Literature <NUM>: Japanese Laid-Open Patent Application No. <CIT>.

A system for remotely driving a vehicle based on an operation amount input to a remote driving terminal by a remote operator is known. Such a system is sometimes referred to as a remote driving system. The remote driving system starts remote driving when a situation in which the vehicle requires the remote driving occurs. However, at this time, the remote driving system does not immediately start the remote driving. For starting the remote driving, the remote driving system first performs an initial check for checking that the remote driving can be started. The remote driving system starts the remote driving only after obtaining a good result of the initial check. Therefore, in order to smoothly start the remote driving, it is required to finish the initial check early.

An object of the present disclosure is to provide a technique for finishing an initial check early and starting remote driving smoothly in a remote driving system.

The first aspect of the present disclosure relates to a remote driving system configured to perform remote driving of a vehicle based on an operation amount input to a remote driving terminal of the remote driving system by a remote operator. The remote driving system includes at least one processor A first situation is a situation in which the remote driving of the vehicle is required, and initial check is a process for checking that the remote driving terminal can start the remote driving, the at least one processor is configured to: detect a second situation showing a sign of the first situation; at the remote driving terminal, perform at least a part of the initial check before the first situation is detected in a case where the second situation is detected. The at least one processor starts the remote driving in a case where the first situation is detected wherein the initial check at the remote driving terminal includes at least checking communication establishment with the vehicle and the initial check at the remote driving terminal further includes steering position alignment between the vehicle and the remote driving terminal, and checking that an actuator of the vehicle operates in response to the operation amount transmitted to the vehicle.

The third aspect of the present disclosure relates to a method for performing remote driving of a vehicle based on an operation amount input into a remote driving terminal by a remote operator. A first situation is a situation in which the remote driving of the vehicle is required. An initial check is a process for checking that the remote driving terminal can start the remote driving. The method includes detecting a second situation showing a sign of the first situation at the remote driving terminal, at the remote driving terminal, performing at least a part of the initial check before the first situation is detected in a case where the second situation is detected, and starting the remote driving in a case where the first situation is detected wherein the initial check at the remote driving terminal includes at least checking communication establishment with the vehicle and the initial check at the remote driving terminal further includes steering position alignment between the vehicle and the remote driving terminal, and checking that an actuator of the vehicle operates in response to the operation amount transmitted to the vehicle.

According to the present disclosure, a second situation showing a sign of a first situation in which remote driving of a vehicle is required is detected. When the second situation is detected, at least a part of an initial check is performed before the first situation occurs. By performing at least a part of the initial check in advance, it is possible to shorten the time from detecting the first situation to finishing the initial check and to smoothly start the remote driving.

<FIG> is a schematic diagram illustrating a configuration example of a remote driving system <NUM> according to the present embodiment. The remote driving system <NUM> includes a vehicle <NUM>, a remote driving terminal <NUM>, and a management device <NUM>. The vehicle <NUM>, the remote driving terminal <NUM>, and the management device <NUM> can communicate with each other via a communication network.

The vehicle <NUM> is a vehicle to be remotely driven by a remote operator X. The vehicle <NUM> may be an autonomous driving vehicle. The remote driving terminal <NUM> is a terminal device operated by the remote operator X when the remote operator X performs remote driving of the vehicle <NUM>. The remote driving terminal <NUM> may be referred to as a remote cockpit. The management device <NUM> manages the remote driving system <NUM>. Typically, the management device <NUM> is a management server on a cloud. The management device <NUM> may be composed of a plurality of servers which performs distributed processes.

In addition, the remote driving system <NUM> may include an infrastructure sensor <NUM>. The infrastructure sensor <NUM> includes an infrastructure camera. The infrastructure sensor <NUM> may further include a rainfall sensor or the like. The infrastructure sensor <NUM> is installed in an area where the vehicle <NUM> travels. The infrastructure sensor <NUM> and the management device <NUM> can communicate with each other via a communication network. The infrastructure sensor <NUM> may communicate with the vehicle <NUM> and the remote driving terminal <NUM> directly or via the management device <NUM>.

<FIG> are block diagrams illustrating configuration examples of the vehicle <NUM>, the remote driving terminal <NUM>, and the management device <NUM>, respectively.

<FIG> illustrates a configuration example of the vehicle <NUM>. The vehicle <NUM> includes a communication device <NUM>, a sensor group <NUM>, an actuator <NUM>, and a control device <NUM>. In this example, the vehicle <NUM> is equipped with an autonomous driving system and can perform autonomous driving.

The communication device <NUM> communicates with the outside of the vehicle <NUM>. The communication destination of the communication device <NUM> includes the remote driving terminal <NUM> and the management device <NUM>.

The sensor group <NUM> includes a recognition sensor, a vehicle state sensor, a position sensor, and the like. The recognition sensor recognizes (detects) a situation around the vehicle <NUM>. Examples of the recognition sensor include an in-vehicle camera, a laser imaging detection and ranging (LIDAR), and a radar. The vehicle state sensor detects a state of the vehicle <NUM>. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, a brake hydraulic pressure sensor, and the like. The position sensor detects a position and a direction of the vehicle <NUM>. The position sensor includes, for example, a global navigation satellite system (GNSS) sensor. The sensor group <NUM> may further include a rainfall sensor.

The actuator <NUM> includes a steering actuator, a drive actuator, and a brake actuator. The steering actuator steers the wheels. The steering actuator includes, for example, an electric power steering (EPS). The drive actuator is a power source which generates a driving force. Examples of the drive actuator include an engine, an electric motor, and an in-wheel motor. The brake actuator generates a braking force. For example, the brake actuator controls the brake hydraulic pressure to operate the brake.

The control device <NUM> is a computer which controls the vehicle <NUM>. The control device <NUM> includes at least one processor (processing circuitry) <NUM> (hereinafter, simply referred to as a processor <NUM>) and at least one memory <NUM> (hereinafter, simply referred to as a memory <NUM>). The processor <NUM> executes various processes. For example, the processor <NUM> includes a central processing unit (CPU). The memory <NUM> stores various programs and various kinds of information necessary for processing by the processor <NUM>. By the processor <NUM> executing the program stored in the memory <NUM>, the function of the control device <NUM> is realized. Examples of the memory <NUM> include a volatile memory, a non-volatile memory, a hard disk drive (HDD), and a solid state drive (SSD). The control device <NUM> may include at least one electronic control unit (ECU).

The control device <NUM> controls the actuator <NUM> to control traveling of the vehicle <NUM>. The control device <NUM> acquires vehicle information VCL from the sensor group <NUM>. The vehicle information VCL includes recognition sensor information showing a result of recognition by the recognition sensor, vehicle state information acquired from the vehicle state sensor, and position information acquired from the position sensor. The recognition sensor information includes an image captured by the in-vehicle camera. The vehicle state information includes speed information, acceleration information, steering angle information, brake hydraulic pressure information, and the like. They are acquired from the speed sensor, the acceleration sensor, the steering angle sensor, the brake hydraulic pressure sensor, and the like. The vehicle information VCL may further include highly accurate position information acquired by localization. The control device <NUM> can acquire highly accurate position information by performing the localization using map information and the recognition sensor information stored in the memory <NUM>. The vehicle information VCL acquired by the control device <NUM> can be used for the autonomous driving or the remote driving of the vehicle <NUM>.

For example, the control device <NUM> controls the autonomous driving of the vehicle <NUM> based on the vehicle information VCL. More specifically, the control device <NUM> generates a travel plan of the vehicle <NUM> based on the vehicle information VCL. Further, the control device <NUM> generates a target path necessary for the vehicle <NUM> to travel in accordance with the travel plan based on the vehicle information VCL. The target path is a gathering of target positions of the vehicle <NUM>. The target path may be set to extend along the center of the lane. Then, the control device <NUM> controls traveling of the vehicle <NUM> such that the vehicle <NUM> follows the target path.

The control device <NUM> can communicate via the communication device <NUM> and transmit the vehicle information VCL to the remote driving terminal <NUM> and the management device <NUM>. At least while the remote driving of the vehicle <NUM> is performed, the vehicle information VCL is transmitted to the remote driving terminal <NUM>. The vehicle information VCL transmitted to the remote driving terminal <NUM> is referred to by the remote operator X, and the remote driving is performed.

<FIG> illustrates a configuration example of the remote driving terminal <NUM>. The remote driving terminal <NUM> includes a communication device <NUM>, an output device <NUM>, a remote operation member <NUM>, and a control device <NUM>.

The communication device <NUM> communicates with the vehicle <NUM> and the management device <NUM>.

The output device <NUM> outputs various kinds of information and presents the information to the remote operator X. For example, the output device <NUM> includes a display device. The display device displays various kinds of information to the remote operator X. As another example, the output device <NUM> may include a speaker.

The remote operation member <NUM> is a member operated by the remote operator X when the remote operator X remotely drives the vehicle <NUM>. The remote operation member <NUM> includes, for example, a steering operation member, an accelerator pedal, a brake pedal, a direction indicator, and the like. The steering operation member is, for example, a steering wheel. The remote operation member <NUM> may include a touch panel, a button, or the like. An operation amount input by the remote operator X during the remote driving of the vehicle <NUM> is detected by a sensor installed in the remote operation member <NUM>.

The control device <NUM> controls the remote driving terminal <NUM>. The control device <NUM> includes at least one processor (processing circuitry) <NUM> (hereinafter, simply referred to as a processor <NUM>) and at least one memory <NUM> (hereinafter, simply referred to as a memory <NUM>). The processor <NUM> executes various processes. For example, the processor <NUM> includes a CPU. The memory <NUM> stores various programs and various kinds of information necessary for processing by the processor <NUM>. By the processor <NUM> executing the program stored in the memory <NUM>, the function of the control device <NUM> is realized. Examples of the memory <NUM> include a volatile memory, a non-volatile memory, an HDD, and an SSD.

The control device <NUM> communicates with the vehicle <NUM> via the communication device <NUM>. The control device <NUM> receives the vehicle information VCL transmitted from the vehicle <NUM>. The control device <NUM> presents the vehicle information VCL to the remote operator X by displaying the vehicle information VCL including the image on the display device. The remote operator X can recognize the state of the vehicle <NUM>, the situation around the vehicle <NUM>, or the like based on the vehicle information VCL displayed on the display device.

In addition, the control device <NUM> may acquire infrastructure information detected by the infrastructure sensor <NUM> directly or via the management device <NUM>. The infrastructure information acquired by the control device <NUM> may include, for example, an image acquired by the infrastructure camera capturing the vehicle <NUM> and the surroundings thereof. The acquired infrastructure information is displayed on the display device. The remote operator X may recognize the state of the vehicle <NUM>, the situation around the vehicle <NUM>, or the like by referring to the infrastructure information. By presenting the image captured by the infrastructure camera to the remote operator X in addition to the image captured by the in-vehicle camera, accuracy of the remote driving or usability for the remote operator X can be improved.

The control device <NUM> generates remote driving information REM based on the operation amount input by the remote operator X. The remote driving information REM is information for controlling the vehicle <NUM> by the remote driving. The remote driving information REM includes the operation amount of the remote operation member <NUM> input by the remote operator X. The control device <NUM> transmits the remote driving information REM to the vehicle <NUM> as necessary.

<FIG> illustrates a configuration example of the management device <NUM>. The management device <NUM> includes a communication device <NUM> and a control device <NUM>.

The communication device <NUM> communicates with the vehicle <NUM>, the remote driving terminal <NUM>, and the infrastructure sensor <NUM>.

The control device <NUM> controls the management device <NUM>. The control device <NUM> includes at least one processor (processing circuitry) <NUM> (hereinafter, simply referred to as a processor <NUM>) and at least one memory <NUM> (hereinafter, simply referred to as a memory <NUM>). The processor <NUM> executes various processes. For example, the processor <NUM> includes a CPU. The memory <NUM> stores various programs and various kinds of information necessary for processing by the processor <NUM>. By the processor <NUM> executing the program stored in the memory <NUM>, the function of the control device <NUM> is realized. Examples of the memory <NUM> include a volatile memory, a non-volatile memory, an HDD, and an SSD.

The control device <NUM> communicates with the vehicle <NUM> and the remote driving terminal <NUM> via the communication device <NUM>. In addition, the control device <NUM> communicates with the infrastructure sensor <NUM> via the communication device <NUM> as necessary to acquire the infrastructure information. The infrastructure information which the control device <NUM> acquires from the infrastructure sensor <NUM> includes the image captured by the infrastructure camera.

The control device <NUM> may manage various kinds of information related to the remote driving of the vehicle <NUM>. For example, the control device <NUM> may acquire weather forecast information, traffic information of the area where the vehicle <NUM> travels, or the like and manage these pieces of information. The weather forecast information includes a forecast about future weather in the area where the vehicle <NUM> will travel. The weather forecast information may include information about current weather in the area where the vehicle <NUM> travels.

The remote driving system <NUM> performs the remote driving of the vehicle <NUM> based on the operation amount input into the remote driving terminal <NUM> by the remote operator X. When the remote driving system <NUM> detects a "first situation", the remote driving of the vehicle <NUM> is started. The first situation is a situation in which the remote driving of the vehicle <NUM> is required.

When the remote driving system <NUM> detects the first situation, the management device <NUM> assigns a certain remote operator X from among a plurality of candidates to the vehicle <NUM>. The management device <NUM> manages assignment relationship between the vehicle <NUM> and the remote operator X and provides information about the assignment relationship to the vehicle <NUM> and the remote driving terminal <NUM>. The vehicle <NUM> and the remote driving terminal <NUM> which have received the information about the assignment relationship establish communication.

While the remote driving is performed, the vehicle <NUM> and the remote driving terminal <NUM> transmit and receive information via the communication network. The communication between the vehicle <NUM> and the remote driving terminal <NUM> may be performed directly or via the management device <NUM>.

The vehicle <NUM> transmits the vehicle information VCL to the remote driving terminal <NUM>. The remote driving terminal <NUM> receives the vehicle information VCL transmitted from the vehicle <NUM> and presents the vehicle information VCL to the remote operator X. For example, the remote driving terminal <NUM> presents the vehicle information VCL by displaying the image on the display device of the output device <NUM>. The remote operator X recognizes the situation around the vehicle <NUM> by viewing the displayed information and performs the remote driving of the vehicle <NUM> by operating the remote operation member <NUM>.

The control device <NUM> generates the remote driving information REM based on the operation amount of the remote operation member <NUM> input by the remote operator X. Then, the control device <NUM> transmits the remote driving information REM to the vehicle <NUM> via the communication device <NUM>.

The vehicle <NUM> receives the remote driving information REM transmitted from the remote driving terminal <NUM>. The vehicle <NUM> controls traveling of the vehicle in accordance with the received remote driving information REM. In this way, the remote driving of the vehicle <NUM> is performed.

As described above, the remote driving of the vehicle <NUM> is started by the remote driving system <NUM> when the first situation is detected. However, the remote driving cannot be always started immediately after the first situation is detected. Before the remote driving is started, an "initial check" needs to be performed for checking that the remote driving can be started. The remote driving system <NUM> starts the remote driving after checking that the remote driving can be started in accordance with the result of the initial check.

<FIG> illustrates a specific example of the initial check. The initial check is performed at the remote driving terminal <NUM>. As illustrated in <FIG>, the initial check includes several steps.

In the first step of the initial check, "communication establishment check" is performed for checking that communication between the vehicle <NUM> and the remote driving terminal <NUM> is normally established. More specifically, in the communication establishment check, it is checked that the vehicle <NUM> and the remote driving terminal <NUM> assigned to this vehicle <NUM> are normally connected to each other and are in a communicable state. For example, the remote driving terminal <NUM> transmits a test signal for the communication establishment check to the vehicle <NUM>. The test signal may be a ping. Then, when the remote driving terminal <NUM> normally receives a response signal transmitted from the vehicle <NUM> in response to the test signal, it is determined that the communication is normally established. When it is determined that the communication is normally established, it means that necessary information can be accurately transmitted and received between the vehicle <NUM> and the remote driving terminal <NUM>.

By the communication establishment check being completed, it is guaranteed that the remote driving terminal <NUM> can normally receive the vehicle information VCL from the vehicle <NUM>. The initial check proceeds to the next step, and "steering position alignment" is performed. The steering position alignment means bringing a steering angle of the steering operation member of the remote driving terminal <NUM> in line with a steering angle of the vehicle <NUM>. The steering angle of the vehicle <NUM> is a steering angle of a steering wheel of the vehicle <NUM>. Alternatively, the steering angle of the vehicle <NUM> may be calculated from a steering angle of the wheels of the vehicle <NUM>. Information about the steering angle of the steering wheel or the steering angle of the wheels of the vehicle <NUM> can be acquired from the vehicle information VCL transmitted from the vehicle <NUM> to the remote driving terminal <NUM>. The steering angle of the steering operation member of the remote driving terminal <NUM> is detected by the sensor installed in the steering operation member.

<FIG> is a diagram illustrating an example of an image displayed on the display device of the remote driving terminal <NUM> when the steering position alignment is performed. A traveling trajectory <NUM>, which is shown by dotted lines, represents a traveling trajectory of the vehicle <NUM> calculated from the current steering angle of the vehicle <NUM>. On the other hand, a traveling trajectory <NUM>, which is shown by solid lines, represents a traveling trajectory of the vehicle <NUM> calculated from the steering angle of the steering operation member of the remote driving terminal <NUM>. The remote driving terminal <NUM> draws the traveling trajectories <NUM> and <NUM> such that they are superimposed on the image included in the vehicle information VCL. Then, the remote driving terminal <NUM> displays the image on which the travel trajectories <NUM> and <NUM> are superimposed on the display device. The steering position alignment means adjusting the steering angle of the steering operation member such that the two traveling trajectories <NUM> and <NUM> come into line with each other. Normally, the steering position alignment is performed by the remote operator X controlling the steering operation member such that the traveling trajectory <NUM> and the traveling trajectory <NUM> come into line with each other. Whether the two trajectories are in line with each other may be automatically determined by the remote driving terminal <NUM> or may be manually determined by the remote operator X. In the case where the determination is made manually, the remote operator X uses a button or the like of the remote operation member <NUM> to input information indicating that the two trajectories come into contact with each other.

By the steering position alignment being completed, it is ready to reflect the operation amount of the steering operation member in steering of the wheels of the vehicle <NUM>. Thereafter, the remote driving information REM can be transmitted from the remote driving terminal <NUM> to the vehicle <NUM>.

Reference is made again to <FIG>. After the steering position alignment is completed and the steering angle of the steering operation member and the steering angle of the wheels correspond to each other, an operation check is performed. The operation check is for checking that the operation amount of the remote operator X transmitted to the vehicle <NUM> as the remote driving information REM is reflected in the operation of the actuator <NUM>, that is, the actuator <NUM> operates in response to the operation amount of the remote operator X. The operation check includes an operation check for the brake and an operation check for the steering. Order of performing the operation check for the brake and performing the operation check for the steering is not limited.

The operation check for the brake is for checking that the operation amount of the brake input into the brake pedal of the remote operation member <NUM> by the remote operator X is reflected in operation of the brake actuator of the vehicle <NUM>. The remote driving terminal <NUM> transmits a test signal to the vehicle <NUM> for requesting the brake actuator to operate. For example, the remote driving terminal <NUM> transmits the remote driving information REM including the operation amount of the brake pedal of the remote operation member <NUM> (test signal) to the vehicle <NUM>. The brake hydraulic pressure is expected to change normally if the brake actuator operates normally in response to this test signal. Therefore, the remote driving terminal <NUM> receives the vehicle information VCL from the vehicle <NUM> and monitors information about the brake hydraulic pressure included in the vehicle information VCL. If the brake hydraulic pressure changes as expected in response to the transmission of the test signal, the remote driving terminal <NUM> determines that the brake actuator operates normally. If the brake actuator operates normally, the operation check for the brake is finished.

The operation check for the steering is performed for checking that the operation amount input into the steering operation member of the remote operation member <NUM> by the remote operator X is reflected in operation of the steering actuator and the wheels of the vehicle <NUM> steers. The remote driving terminal <NUM> transmits a test signal to the vehicle <NUM> for requesting the steering actuator to operate. For example, the remote driving terminal <NUM> transmits the remote driving information REM including the operation amount of the steering operation member of the remote operation member <NUM> (test signal) to the vehicle <NUM>. The steering angle of the vehicle <NUM> is expected to change normally if the steering actuator operates normally in response to this test signal. Therefore, the remote driving terminal <NUM> receives the vehicle information VCL from the vehicle <NUM> and monitors the steering angle information included in the vehicle information VCL. If the steering angle of the vehicle <NUM> changes as expected in response to the transmission of the test signal, the remote driving terminal <NUM> determines that the steering actuator operates normally. If the steering actuator operates normally, the operation check for the steering is finished.

The specific example of the initial check is described above. The remote driving system <NUM> performs the initial check exemplified in <FIG> before starting the remote driving. Then, the remote driving system <NUM> starts the remote driving after it is checked that the remote driving can be started as a result of the initial check. In other words, the remote driving system <NUM> cannot start the remote driving until the initial check is completed. Therefore, if it takes a long time from detecting the first situation to finishing the initial check, the time when the remote driving is started becomes late.

However, it is desirable that the time from detecting the first situation to starting the remote driving is short. In particular, in some cases, there is a possibility that the remote driving of the vehicle <NUM> is required to be started rapidly depending on the state of the vehicle <NUM>. For example, if the vehicle <NUM> stops in a railroad crossing or the like, the remote driving is required to be started rapidly. In order to start the remote driving rapidly, it is necessary to complete the initial check rapidly.

Thus, the remote driving system <NUM> according to the present embodiment detects a "second situation". The second situation is a situation showing a sign of the first situation. In other words, the second situation is a situation showing that although the remote driving of the vehicle <NUM> is not required at the present time, it is likely to be required in the future. When the second situation is detected, the remote driving system <NUM> performs at least a part of the initial check before the first situation is detected. By performing at least a part of the initial check before the first situation is detected, it is possible to shorten the time from detecting the first situation to finishing the initial check and to start the remote driving early.

Similarly to the first situation, when the remote driving system <NUM> detects the second situation, the management device <NUM> assigns a certain remote operator X from among a plurality of candidates to the vehicle <NUM>. The management device <NUM> manages assignment relationship between the vehicle <NUM> and the remote operator X and provides information about the assignment relationship to the vehicle <NUM> and the remote driving terminal <NUM>. The vehicle <NUM> and the remote driving terminal <NUM> which have received the information about the assignment relationship establish communication. Thereafter, the remote driving system <NUM> starts at least a part of the initial check.

Specific examples of the first situation and the second situation will be described. The first situation and the second situation may be detected by the vehicle <NUM> or may be detected by the management device <NUM>. Further, the first situation and the second situation may be detected by the same entity, or each may be detected by the different entity.

The first situation may be, for example, a situation in which a remote driving request (Request for Operation, RFO) is transmitted from the vehicle <NUM> to the management device <NUM>. For example, when the autonomous driving system of the vehicle <NUM> determines that it is difficult to continue the autonomous driving, the remote driving request is transmitted from the vehicle <NUM> to the management device <NUM>. The remote driving request may be transmitted from the vehicle <NUM> while the vehicle <NUM> is traveling or may be transmitted while the vehicle <NUM> is stopped.

Alternatively, the remote driving system <NUM> may detect the first situation based on the infrastructure information acquired from the infrastructure sensor <NUM>. For example, when an abnormal state of the vehicle <NUM> is detected from the image captured by the infrastructure camera of the infrastructure sensor <NUM>, it is predicted that the vehicle <NUM> has difficulty in continuing the autonomous driving. For example, the abnormal state of the vehicle <NUM> is a state in which the vehicle <NUM> travels with its body deviating from the lane beyond a threshold value. As another example, the abnormal state is a state in which the vehicle <NUM> is stopped although it should be traveling. The remote driving system <NUM> may detect the abnormal state of the vehicle <NUM> like these as the first situation.

The remote driving system <NUM> may detect the second situation based on the infrastructure information acquired from the infrastructure sensor <NUM>. For example, the second situation may be a situation in which a sign state is detected from the image captured by the infrastructure camera of the infrastructure sensor <NUM>. The sign state is, for example, a state in which the vehicle <NUM> is meandering. As another example, the sign state may be a state in which a vehicle speed of the vehicle <NUM> become extremely low. For example, when the vehicle <NUM> is meandering, a problem may have occurred in the control of the vehicle <NUM> and thus the remote driving of the vehicle <NUM> may be required in the future. Therefore, the second situation may be the state like these.

As another example, the remote driving system <NUM> may detect the first situation and the second situation based on the weather in the area where the vehicle <NUM> travels. This is an example in a case where the vehicle <NUM> is the autonomous driving vehicle.

For example, the first situation may be a situation in which the rainfall in the area where the vehicle <NUM> travels is equal to or greater than a first threshold value. When the rainfall is large, it is difficult for the vehicle <NUM> to continue the autonomous driving and the remote driving is required. Therefore, the first situation may be the situation like this. The remote driving system <NUM> can detect such a situation by the rainfall sensor of the infrastructure sensor <NUM> or the rainfall sensor of the sensor group <NUM>. For example, the remote driving system <NUM> may detect a situation showing that average rainfall in the area where the vehicle <NUM> travels is larger than the first threshold value as the first situation.

Further, in this case, the second situation may be a situation in which the rainfall in the area where the vehicle <NUM> travels is equal to or greater than a second threshold value and less than the first threshold value. The second threshold is smaller than the first threshold value. For example, the remote driving system <NUM> may detect a situation showing that the average rainfall in the area in which the vehicle <NUM> travels is larger than the second threshold value as the second situation.

Alternatively, in this case, the second situation may be a situation in which it is forecasted that it will start raining in the area where the vehicle <NUM> will travel. If such a situation is detected, the rainfall in the area where the vehicle <NUM> will travel is likely to become equal to or larger than the first threshold value in the future. The remote driving system <NUM> can acquire information about where the vehicle <NUM> will travel in the future from the vehicle information VCL. In addition, the forecast about rainfall can be acquired from the weather forecast information. The remote driving system <NUM> refers to these pieces of information and when it is forecasted that it will start raining in the area where the vehicle <NUM> will travel, detects this situation as the second situation.

As still another example, the remote driving system <NUM> may detect the first situation and the second situation based on deviation of the vehicle <NUM> from the target path while the vehicle <NUM> is controlled by the autonomous driving. This is also an example in a case where the vehicle <NUM> is the autonomous driving vehicle. The target path is a path to be followed by the vehicle <NUM> which is controlled by the autonomous driving. As illustrated in <FIG>, there is a situation where the traveling position of the vehicle <NUM> deviates from the target path and a deviation amount becomes larger than a first threshold value. In a situation like this where the vehicle <NUM> greatly deviates from the target path, it is considered that it is difficult to continue the autonomous driving and it is necessary to start the remote driving of the vehicle <NUM>. Therefore, the remote driving system <NUM> may detect such a situation as the first situation.

In this case, the remote driving system <NUM> may detect a situation in which the deviation amount becomes larger than a second threshold value as the second situation. The second threshold value is set to be smaller than the first threshold value. At the time point when the deviation amount becomes larger than the second threshold value, there is a possibility that the autonomous driving system will restore the vehicle <NUM> to the target path thereafter, and it is not unconditionally necessary to make the remote driving system to start at the present time point. However, there is a possibility that the deviation amount will further increase and exceed the first threshold value and the remote driving of the vehicle <NUM> will be required. Therefore, such a situation may be detected as the second situation.

The deviation amount can also be referred to as a distance between the vehicle <NUM> and the target path. The remote driving system <NUM> can calculate the deviation amount by comparing the position of the vehicle <NUM> with the target path. The remote driving system <NUM> may acquire the position of the vehicle <NUM> from the position information of the vehicle <NUM> obtained from the vehicle information VCL or may acquire the position of the vehicle <NUM> from the image captured by the infrastructure camera of the infrastructure sensor <NUM>. Information about the target path to be followed by the vehicle <NUM> can be acquired from the vehicle information VCL.

Processes performed by the remote driving system <NUM> will be described with reference to <FIG> and <FIG>.

<FIG> is a block diagram illustrating an example of a functional configuration of the remote driving system <NUM>. The remote driving system <NUM> includes a first situation detection unit <NUM>, a second situation detection unit <NUM>, an initial check unit <NUM>, and a remote driving start determination unit <NUM> as functional blocks.

The first situation detection unit <NUM> detects the first situation. The first situation is a situation in which the remote driving of the vehicle <NUM> is required. A processing entity which realizes the first situation detection unit <NUM> may be the processor <NUM> of the vehicle <NUM> or may be the processor <NUM> of the management device <NUM>. For example, the processor <NUM> of the vehicle <NUM> may detect the first situation based on the information acquired from the sensor group <NUM>. Alternatively, the processor <NUM> of the management device <NUM> may detect the first situation based on the information acquired from the infrastructure sensor <NUM>. When detecting the first situation, the first situation detection unit <NUM> transmits information showing that the first situation is detected to the initial check unit <NUM>.

The second situation detection unit <NUM> detects a second situation. The processing entity which realizes the second situation detection unit <NUM> may be the processor <NUM> or may be the processor <NUM>. For example, the processor <NUM> of the vehicle <NUM> may detect the second situation based on the weather forecast information acquired from the management device <NUM>. Alternatively, the processor <NUM> of the management device <NUM> may detect the second situation from the information acquired from the infrastructure sensor <NUM>. When detecting the second situation, the second situation detection unit <NUM> transmits information showing that the second situation is detected to the initial check unit <NUM>.

The initial check unit <NUM> performs the initial check. The initial check unit <NUM> is realized by the processor <NUM> of the remote driving terminal <NUM>. The initial check unit <NUM> performs the initial check when the first situation is detected based on the information transmitted from the first situation detection unit <NUM>. In addition, based on the information transmitted from the second situation detection unit <NUM>, the initial check unit <NUM> performs at least a part of the initial check before the first situation is detected in a case where the second situation is detected. When the initial check is finished, the initial check unit <NUM> transmits the result of the initial check to the remote driving start determination unit <NUM>.

The remote driving start determination unit <NUM> determines whether to start the remote driving or not. A processing entity which realizes the remote driving start determination unit <NUM> may be the processor <NUM>, may be the processor <NUM>, or may be the processor <NUM>. Alternatively, the remote driving start determination unit <NUM> may be realized by cooperation of these processors. The remote driving start determination unit <NUM> determines whether to start the remote driving or not based on the result of the initial check received from the initial check unit <NUM> and the information about the first situation received from the first situation detection unit <NUM>. When the first situation is detected and the result of the initial check indicates that the remote driving can be started, the remote driving start determination unit <NUM> starts the remote driving of the vehicle <NUM>. The remote driving start determination unit <NUM> does not start the remote driving if the first situation is not detected. That is, even if the initial check is completed before the first situation is detected, the remote driving is started after the first situation is detected.

<FIG> is a flowchart illustrating an example of processing performed by the remote driving system <NUM> in a case where the second situation is detected.

In Step S110, the remote driving system <NUM> detects the second situation. The process of Step S110 is executed by the second situation detection unit <NUM>. When the second situation is detected, communication between the vehicle <NUM> and the remote driving terminal <NUM> is established. Then, the processing proceeds to Step S120.

In Step S120, the remote driving system <NUM> starts the initial check. By starting the initial check in advance in Step S120, the remote driving system <NUM> can perform at least a part of the initial check before the first situation is detected. The process of Step S120 is executed by the initial check unit <NUM>. After the initial check is started and at least a part of the initial check is performed, the processing proceeds to Step S125.

In Step S125, the remote driving system <NUM> determines whether the second situation is resolved before the first situation is detected or not. In a case where the second situation is resolved before the first situation is detected (Step S125; Yes), the remote driving system <NUM> terminates the initial check without starting the remote driving. In otherwise case (Step S125; No), the processing proceeds to Step S130.

In Step S130, the remote driving system <NUM> determines whether the first situation is detected or not. The process of Step S130 is executed by the first situation detection unit <NUM>. If the first situation is detected (Step S130; Yes), the processing proceeds to Step S140. On the other hand, if the first situation is not detected (Step S130; No), the processing returns to Step S125. The initial check is continued during this process.

In Step S140, if there is a remaining initial check, that is, an initial check which is not finished yet, the remote driving system <NUM> performs it. After the initial check is finished, the processing proceeds to Step S150. The process of Step S140 is executed by the initial check unit <NUM>. Since at least a part of the initial check has already been performed before Step S130, the remote driving system <NUM> should perform only the remaining initial check in Step S140. Therefore, the time until the process of Step S150 is started becomes shorter than that in a case where all the steps of the initial check are performed in Step S140. The initial check may be finished before Step S130. In this case, Step S140 is skipped and the processing proceeds to Step S150. Since Step S140 is skipped, necessary time until the process of Step S150 is started can be further shortened.

In Step S150, the remote driving system <NUM> determines whether to start the remote driving or not. The process of Step S150 is executed by the remote driving start determination unit <NUM>. The remote driving system <NUM> starts the remote driving if the result of the initial check indicates that the remote driving can be started. After the process of Step S150 is performed, the processing ends.

The processing illustrated in the flowchart of <FIG> is an example of processing in a case where the second situation is detected. However, in some cases, the remote driving system <NUM> detects the first situation without detecting the second situation. For example, there is a case where it suddenly starts raining heavily in the area where the vehicle <NUM> travels although it has not forecasted that it rains according to the weather forecast information. In a case like this, the initial check is performed as usual in response to the detection of the first situation. Then, when it is checked that the remote driving can be started by the initial check, the remote driving is started.

<FIG> is a time chart for explaining an effect of the processing performed by the remote driving system <NUM>. An upper chart shows a case in a comparative example. An example <NUM> and an example <NUM> are examples of cases in which the second situation is detected by the remote driving system <NUM>.

In all of the comparative example, the example <NUM>, and the example <NUM>, the first situation is detected at time T3. In the comparative example, the initial check is started in response to the detection of the first situation. Then, in the comparative example, the remote driving is started at time T5, which is the time when the initial check is finished.

On the other hand, in the example <NUM>, the remote driving system <NUM> detects the second situation at time T3 before time T1. Then, the remote driving system <NUM> starts the initial check in response to the detection of the second situation. Therefore, the initial check can be finished before the first situation is detected. Since the initial check is already completed, the remote driving system <NUM> can start the remote driving immediately after the first situation is detected at time T3.

In the example <NUM>, the second situation is detected at time T2, which is the time after time T1 but before time T3. Similarly to the example <NUM>, the remote driving system <NUM> starts the initial check in response to the detection of the second situation. Thus, at time T3, which is the time when the first situation is detected, a part of the initial check has already been performed. Since a part of the initial check is already performed, the time from the detection of the first state to the completion of the remaining initial check becomes shorter than the time until the initial check is completed in the comparative example. In this way, the remote driving system <NUM> can start the remote driving at time T4, which is the time earlier than the time T5.

As described above, according to the remote driving system <NUM> of the present embodiment, in response to detection of the second situation, at least a part of the initial check is performed before the first situation is detected. Thus, the initial check can be finished early, and the time from when the remote driving is required to when the remote driving is started can be shortened. Thus, the remote driving can be smoothly started.

As an example of a scene to which the remote driving system <NUM> according to the present embodiment is applied, a scene of "autonomous transportation in a factory" is considered.

In the autonomous transportation in the factory, the autonomous driving vehicle autonomously travels in a factory ground. For example, the autonomous driving vehicle assembled in an assembly factory travels from the assembly factory to a yard by autonomously traveling along a predetermined route. One or more infrastructure cameras are installed on a road from the assembly factory to the yard. By using the infrastructure camera, the management device <NUM> of the remote driving system <NUM> can remotely monitor the autonomous driving vehicle that is autonomously traveling.

When the vehicle deviates greatly from the target route for some reason and the deviation amount exceeds the first threshold value, it is determined that it is difficult to continue the autonomous driving, and thus the remote driving is started. In a case where a route on which the autonomous driving vehicle travels is determined in advance as in the case of the autonomous transportation in the factory, the management device <NUM> can detect such a situation based on the information about the route stored in the memory <NUM> and the image captured by the infrastructure camera. At this time, at least a part of the initial check required for starting the remote driving has already been performed at the time when the deviation amount exceeds the second threshold value. Therefore, the remote driving system <NUM> can quickly start the remote driving. When the situation in which it is difficult that the autonomous driving vehicle continue to autonomously travel occurs, it is also considered to send a staff to the site and start to drive the vehicle by manual driving, but it takes time and labor. Dealing with the situation by the remote driving is more convenient and reduces time and labor.

A modification will be described. In the modification, a "standby state" is defined as a state of the remote driving terminal <NUM>. Similarly to the above-described embodiment, the remote driving system <NUM> starts the initial check when the second situation is detected. Then, when the initial check is finished before the first situation is detected and its result indicates that the remote driving can be started, the remote driving system <NUM> shifts the remote driving terminal <NUM> to the standby state. The standby state is a state indicating that preparation for the remote driving is completed at the remote driving terminal <NUM>. In other words, the standby state is a state in which the remote driving can be started immediately after the first situation is detected. Further, in the modification, the standby state of the remote driving terminal <NUM> is terminated under a predetermined condition. Specifically, the remote driving system <NUM> terminates the standby state in a case where the second situation is resolved and the first situation is not detected.

<FIG> is a flowchart illustrating an example of processing performed by the remote driving system <NUM> in the modification. In <FIG>, the same processing as that in <FIG> is omitted. Processes before Step S125 is the same as that in the flowchart of <FIG>.

In Step S130, the remote driving system <NUM> determines whether the first situation is detected or not. If the first situation is detected (Step S130; Yes), the processing proceeds to Step S140 similarly to <FIG>. Processes after Step S140 is the same as that in <FIG>. On the other hand, if the first situation is not detected (Step S130; No), the processing proceeds to Step S160.

In Step S160, the remote driving system <NUM> determines whether the remote driving can be started or not based on the result of the initial check. If the result of the initial check indicates that the remote driving can be started (Step S160; Yes), the processing proceeds to Step S170. On the other hand, if the result of the initial check does not indicate that the remote driving can be started or if the initial check is not completed (Step S160; No), the processing returns to Step S125. The initial check is continued during this process.

In Step S170, the remote driving system <NUM> shifts the remote driving terminals <NUM> to the standby state. The standby state is a state indicating that preparation for the remote driving is completed. After the remote driving device <NUM> is shifted to the standby state, the processing proceeds to Step S180.

In Step S180, the remote driving system <NUM> determines whether the first situation is detected or not. If the first situation is detected (Step S180; Yes), the processing proceeds to Step S210. On the other hand, if the first situation is not detected (Step S180; No), the processing proceeds to Step S190.

In Step S210, the remote driving system <NUM> starts the remote driving of the vehicle <NUM>. In Step S210, since the remote driving terminal <NUM> is in the standby state, the remote driving system <NUM> can immediately start the remote driving without loss of time due to the initial check.

In Step S190, the remote driving system <NUM> determines whether the second situation is still detected or not. If the second situation is still detected (Step S190; Yes), the processing returns to Step S180. On the other hand, if the second situation becomes not detected (Step S190; No), the processing proceeds to Step S200.

In Step S200, the remote driving system <NUM> terminates the standby state of the remote driving terminals <NUM>. After the standby state of the remote driving terminal <NUM> is terminated, the processing ends.

Claim 1:
A remote driving system (<NUM>) configured to perform remote driving of a vehicle (<NUM>) based on an operation amount input into a remote driving terminal of the remote driving system by a remote operator (X), the remote driving system (<NUM>) comprising at least one processor (<NUM>, <NUM>, <NUM>) wherein:
a first situation is a situation in which the remote driving of the vehicle (<NUM>) is required,
an initial check is a process for checking that the remote driving terminal can start the remote driving, and
the at least one processor is configured to:
detect a second situation showing a sign of the first situation;
at the remote driving terminal, perform at least a part of the initial check before the first situation is detected in a case where the second situation is detected;
start the remote driving in a case where the first situation is detected, and
wherein the initial check at the remote driving terminal includes at least checking communication establishment with the vehicle (<NUM>) and the initial check at the remote driving terminal further includes steering position alignment between the vehicle (<NUM>) and the remote driving terminal, and checking that an actuator (<NUM>) of the vehicle (<NUM>) operates in response to the operation amount transmitted to the vehicle (<NUM>).