Patent Description:
This document relates to remote vehicle monitoring and control.

Autonomous vehicle navigation is a technology for sensing the position and movement of a vehicle and, based on the sensing, autonomously control the vehicle to navigate towards a destination. Autonomous vehicle navigation can have important applications in transportation of people, goods and services.

<CIT> relates to emergency stopping for autonomous commercial vehicles. A method for controlling a commercial highway vehicle is described. The method includes detecting a failure of a first component based on a first signal from a first sensor. The method also includes classifying, by an automated driving system on the vehicle, a severity of the component failure. The method further includes determining to stop the vehicle if the severity exceeds a threshold severity level. The method also includes determining an emergency stopping distance based on the severity and a current momentum of the vehicle. The method further includes determining a stopping location within the emergency stopping distance. The method also includes stopping the vehicle at the stopping location.

The invention is a method, apparatus and computer-readable storage medium as defined in the appended claim. Disclosed are devices, systems and methods for remote safe driving, which include self-checking systems, emergency handling systems, and remote control systems. In an example, this may be achieved by a remote monitor center controlling some part of the monitoring and emergency handling services on the vehicle, and providing commands to ensure the safety of the vehicle and its passengers in the case of an emergency.

In one aspect, the disclosed technology can be used to provide a method for remote safe driving of a vehicle. This method may be implemented at the vehicle. The method includes detecting an emergency situation, and in response to the detecting the emergency situation, switching operation of the vehicle to a low-power operation mode that comprises shutting down a subset of vehicular components, and periodically transmitting a location of the vehicle to a remote monitoring center.

In another aspect, the disclosed technology can be used to provide another method for remote safe driving of a vehicle. This method may be implemented at a remote data center that is in communication with the vehicle. The method includes selecting at least one of a set of vehicular driving actions, and transmitting, over a secure connection, the at least one of the set of vehicular driving actions to the vehicle, wherein the set of vehicular driving actions is generated based on a classification of a plurality of driver behavior.

In yet another example aspect, a computing apparatus that includes a processor for implementing one of the methods recited herein is disclosed.

In yet another example aspect, a computer program product comprising a computer-readable program medium having code stored thereon is disclosed. The code, when executed by a processor, causes the processor to implement a method as described.

The above and other aspects and features of the disclosed technology are described in greater detail in the drawings, the description and the claims.

Autonomous vehicles use a variety of techniques to detect their surroundings, such as radar, laser light, GPS, odometry and computer vision. Control systems may interpret sensory information to identify appropriate navigation paths, as well as planned and unplanned obstacles and relevant signage along the route. The remote driving of vehicles may further rely on monitoring and classification systems that are capable of analyzing sensory data to distinguish between a variety of factors, e.g. different weather conditions, different cars on the road, and different obstacles.

Another integral feature of autonomous driving should be the safety of the vehicle and its passengers, as well as the safety of neighboring people and property. Thus, autonomous vehicles should be equipped with emergency handling systems to ensure the safe driving of the vehicles, especially when performed remotely. The response to an emergency situation should be rapid and precise, since safety is paramount. One of the main goals to enable widespread use of autonomous vehicles is to achieve and exceed the reliability of human driving behavior, and remote safe driving is integral to this goal. The techniques described in the present document can be incorporated in embodiments of a fully-autonomous vehicle, a semi-autonomous vehicle and/or a control center that controls operation of the autonomous vehicle. In particular, using the disclosed techniques, upon detection of an abnormality, an autonomous vehicle may safely stop further driving and ask for assistance. Similarly, in some embodiments, when a control center becomes aware of an autonomous vehicle's distress condition, the control center may provide the vehicle with further instructions to safely cease driving and wait for further assistance. These, and other, features are further described herein.

<FIG> shows a block diagram of an exemplary system for remote safe driving. As shown therein, the system for safe driving includes a truck <NUM> and a remote center (or remote data center, or remote monitor center) <NUM>, which can communicate with each other through a communication protocol <NUM>. In some embodiments, the truck <NUM> includes an emergency handling system <NUM>, a monitoring system <NUM>, and an electronic control unit (ECU) system <NUM>.

In some embodiments, the emergency handling system defines several emergency status conditions, and corresponding autonomous vehicle actions for each status condition. For example, the emergency status conditions may include:.

In some embodiments, the vehicle may stop in an emergency lane when the "truck (or vehicle) abnormal" status indicator is detected, and may stop in the lane it is currently driving in when any of the other status indicator is detected.

In some embodiments, one or more of the enumerated emergency status conditions may be transmitted to the remote data center over a secure connection as soon as they are detected as part of an emergency signal. In other embodiments, the emergency status conditions may be transmitted as part of periodically transmitted monitoring signals. In yet other embodiments, a semi-persistent approach may be adopted, where periodic monitoring updates are transmitted from the vehicle to the remote data center, but an emergency signal transmission takes precedence and is transmitted as soon as it is generated.

In some embodiments, the remote data center may receive the necessary emergency signals from the vehicle over a dedicated and secure emergency channel. In one example, the status of the vehicle may be derived from the emergency signals received. In another example, the remote data center may receive the status of the vehicle from the vehicle itself, as part of the communication that contained the necessary emergency signals, or in a separate communication. In yet another example, the status condition (which may be a non-emergency or emergency status condition) may be accompanied by a corresponding report providing additional information related to that status condition.

In some embodiments, the emergency signals include a location and a vehicle status message. In an example, the location may be specified in absolute or relative coordinates. The vehicle status message may include a status indicator and specific information elements. In some embodiments, the status indicators may have levels or tiers, as shown in the example table below:.

In some embodiments, each of the enumerated status conditions may take on a value shown in the example table above, and may be transmitted to the remote data center, along with any corresponding information elements that may be required. In some embodiments, the operation of the vehicle, the generation of an emergency status, and the response required to resolve the emergency situation may be implemented as shown in the state diagram in <FIG>.

<FIG> shows an example state diagram <NUM> implementing a procedure for remote safe driving. The state diagram <NUM> is typically, and ideally, operating in a "running" state <NUM>, where the vehicle is operating as intended, with no failures in self-detection tests and no emergency status indications. In addition to operating in the "running" status, the vehicle may periodically (or semi-persistently, or aperiodically) perform the process <NUM>. In an example, "checking process" includes running self-detection tests.

Upon detecting an emergency condition, the exemplary procedure for remote safe driving may implement one of at least two policy strategies. The first policy <NUM> dictates that the vehicle should search for the nearest emergency lane, and safely come to a stop in the emergency lane. The second policy <NUM> dictates that the vehicle stop in the lane it is currently operating in (referred to as the "self-lane"). For example, and under this policy, the vehicle may determine that an immediate stop may be required, and that the driving or emergency conditions may preclude taking the time to search for and move to an emergency lane.

When the vehicle has stopped in the emergency lane <NUM>, the vehicle may take appropriate measures to address the emergency situation, and then restart from the emergency lane <NUM> (and referred to a Process I). Similarly, when the vehicle has stopped in the self-lane <NUM>, it may restart from the self-lane <NUM> (referred to as Process II) after appropriate measures have been taken. Restarting operations from either the emergency lane or the self-lane returns the state of the vehicle to the "running" state <NUM>, as shown in <FIG>.

<FIG> shows an example state diagram <NUM> implementing the "Process I" and "Process II" procedures (as discussed in the context of <FIG>) for remote safe driving. As shown therein, the state diagram <NUM> includes remote data center (or simply, the "data center") <NUM> implementing a sync process <NUM> and a self-checking procedure for each subsystem in the vehicle to ensure that the vehicle may be may be restarted. For example, the sync process <NUM> ensures that the subsystem self-checks are implemented in a coordinated manner. In some embodiments, the vehicle self-checking and self-start process <NUM> includes running the self-check and reporting the results back to the data center, which remotely starts the vehicle.

In some embodiments, the hardware self-checking process <NUM> may include self-checking one or more of the ECU module (e.g., CPU, GPU, memory, mainboard), the sensor module (e.g., camera, radar, IMU, GPS sensor), and the power module (e.g., converter system), and reporting the results of each subsystem self-check to the data center <NUM>.

In some embodiments, the software self-checking process <NUM> may include self-checking the Octopus platform, which is an open-source platform for graph-based program analysis. The software self-checking process may further include self-checking the algorithms modules (e.g., maps and localization, perception, control, motion planning).

In some embodiments, the surrounding checking process <NUM> may ensure other vehicles, objects and/or persons in the vicinity of the vehicle are accounted for prior to restarting.

In some embodiments, the autonomous driving process <NUM> includes bringing the vehicle into a semi- or fully-autonomous driving mode.

<FIG> shows a block diagram of example subcomponents of a vehicle that can support remote safe driving. In some embodiments, the vehicle may include a sensor system <NUM>, a middleware system <NUM>, and an algorithm module <NUM>. In some embodiments, one or more of these subcomponents may be part of an ECU system (e.g. the ECU system <NUM> shown in <FIG>). In other embodiments, the ECU system may control one or more of these systems and modules.

In an example, the sensor system <NUM> may include a CAN bus sensor, a camera, radar capabilities, a GPS unit and/or an IMU, and Lidar capabilities. In another example, the middleware system <NUM> may include the system module, and the algorithm module <NUM> may include a localization module, a perception module, a control module, and a planning module.

In some embodiments, a monitoring system (e.g. the monitoring system <NUM> shown in <FIG>) may periodically (or continuously, or triggered aperiodically) monitor the status of each component of the sensor system <NUM>, middleware system <NUM> and algorithm module <NUM>. For example, the monitoring system receives a location message and a vehicle status message. In some embodiments, the location message may use GPS84 coordinates.

In some embodiments, the vehicle status message may be defined as including the following subfields, one or more of which may be transmitted at each time:.

(<NUM>) vehicle running status as a <NUM>-bit field with a "<NUM>" indicating that the vehicle is running and a "<NUM>" indicating that the vehicle has stopped;
(<NUM>) vehicle self-status using the standard or extended frame formats (as described in CAN <NUM> A and CAN <NUM> B);
(<NUM>) hardware status defined as:.

The message formats shown above are exemplary, and other formats with different lengths for the bitfields, as well as additional bitfields and status indicators, are envisioned as part of the disclosed technology.

Embodiments of the disclosed technology may be advantageously implemented in a modular fashion to support both fully-autonomous as well as semi-autonomous vehicles. For example, a semi-autonomous vehicle that is actively and safely being controlled by a driver may not need to implement autonomous driving (e.g., the autonomous driving process <NUM> in <FIG>), self-checks for the control module (e.g. the control module that is part of the algorithm module <NUM> in <FIG>), or use the control status bitfield as described above. The remote center may configure an implementation of the safe driving system described in this document to suit the needs of the driver and/or passengers of the fully- or semi-autonomous vehicle.

<FIG> shows a flowchart of an example method <NUM>, which may be implemented at the vehicle, for remote safe driving. The method <NUM> includes, at step <NUM>, detecting an emergency situation. In some embodiments, the emergency situation may include an abnormal sensor operation, an abnormal electronic control unit (ECU) operation, an abnormal network operation, a communication failure, a failure of a planning module, a failure of a car control module, a failure of a localization module, or a failure of a perception module. In other embodiments, the emergency situation may include a failure or faulty operation of one or more of the subcomponents shown in the context of <FIG>. In yet other embodiments, the emergency situation may be an event external to the vehicle (e.g. environmental or traffic-related).

The method includes, at step <NUM>, switching, in response to the detecting, operation of the vehicle to a low-power operation mode that comprises shutting down a subset of vehicular components. In some embodiments, the subset of vehicular components include non-essential sensors and subsystems that are non-emergency subsystems. Since recovering from the emergency situation is integral to the safety of the vehicle and its passengers, and to people and property in the vicinity of the vehicle, subsystems that are not required to resolve the emergency situation are turned off in order to ensure enough power is available for critical subsystems. In an example, the non-emergency subsystems may include the vehicle entertainment subsystem, and map and navigation support for retail establishments and points of interest.

In some embodiments, subsystems may correspond to the sensor system, the middleware system and/or the algorithm module (as shown in <FIG>). In other embodiments, a subsystem may correspond to the individual sensors (e.g. radar, camera, LiDAR, etc.) or the individual modules (e.g. localization module, perception module, etc.).

As discussed in the context of the state diagram shown in <FIG>, a response to the emergency situation may include stopping the vehicle in either the self-lane or in an emergency lane. In some embodiments, and once the vehicle has been safely guided to a stop, another set of subcomponents that are critical to autonomous driving are shut down and restarted, and a status check is performed on at least one of the other set of components. This ensures that the vehicle is in a condition to either continue on its route or be diverted to a service station for further inspection. In other embodiments, if it is determined that further operation of the vehicle may not be safe, and other failsafe or precautionary procedures (e.g. calling for a tow truck) may be triggered, either by the emergency handling system or the remote data center.

In some embodiments, performing a status check an restarting (or starting) certain components may be based on the changing environment. For an example, if it starts to get dark while the vehicle remains on the side of the road, a status check may be performed on the hazard lights, which may then be turned on to ensure the visibility of the vehicle. For another example, if rush hour starts and parking restrictions are imposed in the right-most lane in which the vehicle is parked, a status check may be performed on the autonomous driving system (ADS), and the vehicle carefully driven to an alternate safe spot.

The method includes, at step <NUM>, periodically transmitting, in response to the detecting, a location of the vehicle to a remote monitoring center. In some embodiments, as soon as any emergency status condition is detected, the vehicle may periodically transmit its GPS coordinates (or location relative to known mile markers, other landmarks, or Wi-Fi transmitters) to the remote data center. In an example, the period of the transmission of the vehicle location may be much shorter than a period typically used for transmitting monitoring status updates.

In some embodiments, cargo being hauled by the vehicle may be critically important to a customer and is deemed an essential component when an emergency is detected. For example, when the vehicle is transitioning to a low-power operating mode in which non-essential and non-emergency components are shut down, power may be routed to the cargo container to ensure that it is maintained at a predetermined thermal profile. The periodic transmission of the location of the vehicle will advantageously enable the remote center (e.g., <NUM> in <FIG>) to determine how quickly the vehicle can return to operational status (e.g., by tracking a vehicle sent to assist the vehicle having the emergency), and it can explicitly ensure that the thermal profile of the cargo container is maintained per customer requests.

<FIG> shows a flowchart for an example method <NUM>, which may be implemented at a remote data center, for remote safe driving. The method <NUM> includes, at step <NUM>, selecting at least one of a set of vehicular driving actions. In some embodiments, the vehicular driving actions may include parking the vehicle, moving from a predetermined origin to a predetermined destination, and moving to a refueling location. In some embodiments, the enumeration of the vehicular driving actions may be generated based on a classification of driver behavior.

For example, the classification may be a clustering algorithm that uses a data set of driver behavior, which may be used to train the algorithm to identify the aforementioned vehicular driving actions. The clustering algorithm may be a hierarchical clustering algorithm, a centroid-based clustering algorithm, a distribution-based clustering algorithm, or other such supervised learning algorithms.

The method <NUM> includes, at step <NUM>, transmitting, over a secure connection, the at least one of the set of vehicular driving actions to the vehicle. In some embodiments, the secure connection may be the emergency channel (or link) exclusively that is reserved for emergency communications. In other embodiments, the secure connection may be an operational channel (or link) that is typically used for high-speed data communications.

In some embodiments, the operational link may be secured using an Internet layer cryptographic protocol, which enforces authenticity, integrity and secrecy. In an example, the operational link may use the Internet Protocol security (IPsec) protocol, pre-shared keys (PSK) or a public-key cryptosystem, e.g. RSA or Diffie-Hellman key exchange. The emergency channel may be secured using a low-latency Application layer cryptographic protocol, due to the imperative nature of emergency communications. In an example, the Secure Sockets Layer (SSL) protocol or the Transport Layer Security (TLS) protocol may be used for the emergency channel, which due to the time sensitive nature of the messages, does not require explicit client authentication after an initial authentication process.

In some solutions, the other set of vehicular components that are restarted may be selected in response to changes in the external environment. For an example, if the vehicle has come to a stop and dusk is approaching, the hazard lights will be turned on to ensure the visibility of the vehicle by other vehicles.

<FIG> shows an example of a hardware platform <NUM> that can be used to implement some of the techniques described in the present document. For example, the hardware platform <NUM> may implement the methods <NUM> and <NUM> or may implement the various modules described herein. The hardware platform <NUM> may include a processor <NUM> that can execute code to implement a method. The hardware platform <NUM> may include a memory <NUM> that may be used to store processor-executable code and/or store data. The hardware platform <NUM> may further include a communication interface <NUM>. For example, the communication interface <NUM> may implement one or more communication protocols (LTE, Wi-Fi, and so on).

Implementations of the subject matter and the functional operations described in this patent document can be implemented in various systems, digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer program products, e.g., one or more modules of computer program instructions encoded on a tangible and non-transitory computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term "data processing unit" or "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers.

Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices.

Claim 1:
A method for remote safe driving of a vehicle, comprising:
periodically transmitting monitoring signals to a remote monitoring center in communication with the vehicle; and
detecting (<NUM>) an emergency situation;
the method being characterized by the following steps:
in response to the detecting, switching (<NUM>) operation of the vehicle to a low-power operation mode that comprises shutting down a subset of vehicular components; and
periodically transmitting (<NUM>), to the remote monitoring center, a location of the vehicle where the emergency situation has been detected,
wherein a period of the transmitting of the location of the vehicle is shorter than a period of the transmitting of the monitoring signals.