Patent Description:
As shown in <FIG>, in a common method for a person <NUM> to hail a vehicle such as a taxicab or a ride-sharing vehicle, the person enters a request by interacting with a mobile app. The request is processed by the mobile app and then forwarded through the cellular telephone network or the Internet to a central server <NUM> where it is processed and then forwarded through the cellular telephone network to a driver of the vehicle <NUM>.

<CIT> describes a method for recognizing a person hailing a public passenger vehicle. The method is implemented by a system including an image capturing device, a processor, and a notification unit. In the method, the image capturing device successively captures images in front of the public passenger vehicle at a fixed rate. The processor determines whether any one of the images has a human body. Afterwards the processor determines whether the human body remains in a predetermined posture for a predetermined duration. When it is determined that the human body remains in the predetermined posture for longer than the predetermined duration the notification unit informs a driver of the public passenger vehicle.

<CIT> relates to the interaction between an autonomous vehicle and an external environment of the autonomous vehicle. Such interaction can occur in various ways. For example, a non-verbal human gesture in the external environment can be detected. The non-verbal human gesture can be identified. A future driving maneuver can be determined based on the identified non-verbal human gesture. The autonomous vehicle can be caused to implement the determined future driving maneuver. As another example, the external environment of the autonomous vehicle can be detected to identify a person (e.g. a human pedestrian, a human bicyclist, a human driver or occupant of another vehicle, etc.) therein.

The identified person can be located. It can be determined whether the person is potentially related to a future driving maneuver of the autonomous vehicle. The autonomous vehicle can be caused to send a directional message to the person.

<CIT> describes a method comprising responding to user input provided by a pedestrian at a wearable user input device, by causing motion of a vehicle to change. A corresponding computer program code and apparatus are also described.

Here we describe a system and techniques for a person to use in hailing a vehicle, such as an autonomous vehicle. As shown in <FIG>, in some implementations of the system and techniques that we describe here, a user <NUM> hails a vehicle <NUM> by communication that is achieved directly with the vehicle using one or more wireless signaling modes <NUM>. In some cases, a server <NUM> can also or alternatively be in direct communication with the vehicle.

We sometimes refer to the system and techniques broadly as a "hailing system. " We use the term "hailing" or "hail" broadly to include, for example, any action by any person in any context (for instance a pedestrian) to signal, call, flag down, alarm, or in any other way to attract the attention of a vehicle for the purpose of indicating the person's wish to ride in (alone or with others), hire for service, cause a delivery to be made by, or otherwise make use of the vehicle (such as an autonomous vehicle).

Hailing can be done by, through the medium of, or with the assistance of a hailing device. We use the phrase "hailing device" broadly to include, for example, any apparatus, equipment, instrument, or other device associated with hailing, for example, a phone or other handheld device, a smartwatch or bracelet or other wearable device, or a mounted or otherwise stationary device located on or near a sidewalk or road edge, to name only a few. We use the phrase "potential rider" broadly to include, for example, any person in any context who wishes to ride in (alone or with others), hire for service, cause a delivery to be made by, or otherwise make use of a vehicle (such as an autonomous vehicle). We use the term "hire a vehicle for service" broadly to include, for example, to obtain access in any way and in any context to a vehicle for the purpose of, for example, riding in the vehicle, causing a delivery to be made by the vehicle, or otherwise making use of the vehicle. Although we sometimes refer in our discussion to autonomous vehicles, the hailing system could also be used for semi-autonomous vehicles and in some contexts for other kinds of vehicles driven in other ways. We sometimes use the term "autonomous vehicle" to refer broadly to any such vehicle. An autonomous vehicle can be hailed when it is passing along a roadway nearby to a potential rider, or when it is stopped, or in other contexts.

An autonomous vehicle can drive safely without human intervention during part of a journey or an entire journey. As shown in <FIG> and <FIG>, an autonomous vehicle includes sensors, actuators, computers, and communication devices to enable automated generation and following of routes through the environment. Autonomous vehicles typically contain one or more video sensors, at least one of which is typically oriented in the forward direction in order to capture data related to the scene in front of the vehicle. Additionally, autonomous vehicles may contain one or more LIDAR sensors. Autonomous vehicles typically are equipped with a wireless two-way communication capability to access data and information stored on servers in a cloud service. As also shown in <FIG> and <FIG>, the autonomous vehicle can communicate wirelessly with a server at a central location.

These and other capabilities can enable the autonomous vehicle to react to detected visual or other physical features in the environment, including particular shapes, colors, textures, human gestures, and other visual stimuli. Autonomous vehicles may also be equipped with auditory sensors to capture and react to sounds in the environment, and horns, speakers, bells, or other noisemakers to emit sound. We will refer to these visual and auditory capabilities and devices and a broad range of other detection capabilities and devices by the general phrase "stimulus detectors.

Thus, as shown in <FIG>, the stimulus detectors <NUM> of an autonomous vehicle <NUM> can include video <NUM>, audio <NUM>, image <NUM>, light <NUM>, RF <NUM>, for example.

The autonomous vehicles used with the hailing system that we describe here may also be equipped with lights, display screens, light-emitting diodes, or other light emitting devices <NUM> mounted, for example, on the exterior of the vehicle such that they can be viewed from outside the vehicle. Autonomous vehicles may have displays <NUM>, lights, sirens, klaxons, or other sound emitters <NUM>, and other signaling devices mounted, for example, external to the vehicle. We will refer to these signaling devices and techniques and to a broad range of other signaling capabilities and equipment as "signaling devices. " Often the signaling devices operate in modes that are perceptible by and understood by people.

Conventionally, vehicles (such as taxicabs and vehicles being driven by people for ride-hailing services) can be hired for service by a potential rider using a ride hailing software process <NUM> that may run on a processor unit <NUM> embedded within a smartphone, laptop, tablet, or other computing device <NUM>. The potential rider <NUM> typically inputs <NUM> to the ride hailing software process a desired pick-up location and may also input a desired drop-off (i.e., a goal) location (e.g., a destination address or street intersection).

In known systems, the desired pick-up location is then communicated wirelessly to a centralized server tasked with assigning movement directions to one or more and potentially a large fleet (e.g., hundreds or thousands) of vehicles. The centralized server then runs a routing software process that provides by wireless communication to a particular vehicle the potential rider's pick-up location. The particular vehicle then is driven to the desired pick-up location to meet the potential rider. Once the potential rider enters the vehicle and completes an identification process (not necessarily in that order), the vehicle is considered hired for service and begins the journey to the desired drop-off location.

In some typical use cases of autonomous vehicles, the desired pick-up or goal location may be specified by an algorithm (which may be located on a centralized server in the cloud and tasked with optimizing the locations of a fleet of autonomous vehicles with a goal of minimizing rider wait times when hailing a vehicle), or may be provided by another process (e.g., an emergency process that identifies the nearest hospital as the goal position due to a detected medical emergency on board the vehicle).

In implementations of the hailing system that we describe here, a potential rider of a vehicle (e.g., an autonomous vehicle) may wish to hire a vehicle for service without, for example, inputting a desired pick-up location into a smartphone, laptop, tablet, or other computing device, but rather to simply hail the vehicle (such as an unoccupied autonomous vehicle passing on a roadway). The hailing system that we describe here can also apply to other scenarios, such as hailing an autonomous vehicle that is parked at a particular location.

As shown in <FIG>, in implementations of our hailing system, hailing of an autonomous vehicle <NUM> can be accomplished by sending information from or on behalf of or in association with the potential rider <NUM> to the autonomous vehicle using one or a combination of two or more of a variety of signaling modes including those discussed below. We use the term "signaling mechanism" broadly to include, for example, any device, equipment, apparatus, or system by which information about the hailing can be sent to the autonomous vehicle. We use the term "signaling mode" broadly to include, for example, any form in which information can be sent by wireless signals or other methods, including non-electronic methods. Among others, the signaling mode can include displaying an image or sequence of images, emitting a sequence of light signals, emitting a sequence of sound signals, emitting a wireless communication signal, or engaging in gestures or facial expressions, to name a few.

In most cases, the information signaling mechanism and mode are designed to increase the incidence rate of true detections (i.e., scenarios in which the autonomous vehicle correctly identifies that a hailing request has been directed to the vehicle) and to reduce the incidence rate of false detections (i.e., scenarios in which an autonomous vehicle incorrectly identifies that a hailing request has been directed to the vehicle when none has been sent or one has been sent to another vehicle, for example ).

As shown in <FIG>, the potential rider <NUM> can signal the autonomous vehicle using a hailing device <NUM> such as a mobile device (or a wide variety of other devices).

In some instances, the hailing may be performed by a potential rider (or someone else, such as a companion) who is carrying a hailing device (e.g., a smartphone). In some cases (see <FIG>), the hailing may be performed using a hailing device that is mounted or installed in a fixed location, such as on a structure at which vehicle hailing may commonly occur (e.g., outside a busy hotel). In some cases, the hailing device may be typical human capabilities performed by a potential rider (or someone on her behalf) who is not carrying any external hailing device, but rather uses other signaling mechanisms, such as a hand, or facial, or other gesture or whistling, yelling, or making another sound to transmit information. In some implementations, the hailing may be performed by interacting with a device attached to the autonomous vehicle itself (e.g., a smart card reader that can read the user's smart card or a smart device that can communicate with the user's smartphone.

Hailing using the hailing system can be done in a way that complies with a predefined and commonly accepted hailing protocol that could cover the manner in which the signaling mechanism is to operate, the information that is to be transmitted, its format, and a wide variety of other aspects. A wide variety of entities that operate individual autonomous vehicles and fleets of them to be hired for service and entities that produce software, hardware, and other equipment for use in autonomous vehicles and hailing devices can subscribe to such a hailing protocol.

A desirable aspect of such a hailing protocol is that it may enable a potential rider to hail a vehicle more quickly (because vehicles of more than one fleet operator comply with the protocol) than would be achievable by hailing an autonomous vehicle that is operated by one specific fleet operator under a proprietary hailing method. Interoperability among fleet operators could also require a common central cloud or system that could broadcast the hailing requests to all fleet operators.

Another desirable aspect of the hailing system that we describe here is that the potential rider may be able to hail a vehicle more quickly than if the rider were required to input a desired pick-up location (which is not required in implementations of the hailing system), wait for the system to assign a vehicle to her, and then wait for that vehicle to arrive at the pickup location. Another desirable aspect of some implementations of the hailing system is that the potential rider may be able to hail a vehicle anonymously, e.g., without using a ride hailing software process that may be tied to a user account that includes personally identifying information. In other words, the potential rider need not say where she is headed or expose any of her personal information in the process of using the hailing system.

On the other hand, depending on the signaling mechanism, certain kinds of information associated with the hailing request can be transmitted to the autonomous vehicle by the hailing system. This information may include, but not be limited to, information about the number of passengers, the desired class or cost of service (which may be used to exclude vehicles of a certain size or class from responding to the hailing request), information about a priority or emergency hailing request, or personal identifying information about the rider, or a combination of any two of more of those, and of other information. Transmitted information may employ any of a number of possible encryption techniques.

The hailing device can include, for example, one or more of the following (and wide variety of others):.

Exemplary methods for hailing an autonomous vehicle include but are not limited to the following methods, which may be employed either independently or in combination.

As shown in <FIG>, hailing of an autonomous vehicle may be achieved by displaying a particular image or sequence of images <NUM>, <NUM> (for example, displayed alternating at a fixed repeat rate) on a hailing device <NUM> (<NUM> in <FIG>) in a manner (e.g., presenting the device in an orientation such that the image or images are displayed toward the roadway at shoulder height or above) that is likely to lie within a visual line of sight of, for example, video sensors <NUM> mounted on a passing autonomous vehicles. For example, the hailing device can be located at a sidewalk or road edge, or at an intersection where the potential rider is standing.

The image or images to be displayed may exhibit one or a combination of any two or more of the following properties:.

Information related to a hailing request (e.g., the desired class or cost of service, or the presence of a priority or emergency hailing request) may be encoded in the image by the encoding and light modulation processes <NUM> and <NUM> using various methods, including but not limited to the following individual methods, or combinations of these methods:.

Hailing of an autonomous vehicle may be achieved by emitting light from a hailing device in a manner (e.g., presenting the device in an orientation such that the emitted light is directed toward the roadway at shoulder height or above) that is likely to lie within a visual line of sight of video sensors mounted on an autonomous vehicle (e.g., from a sidewalk or road edge, or by a potential rider who is standing at an intersection).

The emitted light to be displayed may exhibit one or a combination of two or more of the following properties based on execution of encoding and light modulation processes <NUM> and <NUM> on the hailing device <NUM>:.

Information related to the hailing request (e.g., the desired class or cost of service, or the presence of a priority or emergency hailing request) may be encoded in the emitted light by the encoding processes <NUM> using various methods, including but not limited to the following individual methods, or combinations of these methods:.

As shown in <FIG>, hailing of a passing autonomous vehicle may be achieved by performing a (e.g., uncommon) gesture or sequence of gestures <NUM> at a position and in an orientation that is likely to lie within visual line of sight of video or LIDAR sensors mounted on an autonomous vehicle (e.g., from a sidewalk or road edge, or when a potential rider is standing at an intersection). A software process <NUM> (<FIG>) running on a processor <NUM> mounted on the autonomous vehicle would then analyze captured data from the video or LIDAR sensors to detect the presence of the gesture(s) that are intended to represent a hailing request.

The gesture(s) to be displayed may exhibit the following properties:.

Information related to the hailing request (e.g., the desired class or cost of service, or the presence of a priority or emergency hailing request) may be encoded in the gesture(s) through various methods, including but not limited to the following individual methods, or combinations of these methods:.

Hailing of a passing autonomous vehicle may be achieved by emitting from a hailing device (here including a smartphone or similar device, a whistle or similar device, or the potential rider using her mouth) an uncommon sound or sequence of sounds of sufficient volume that they can be detected by sensors mounted on a passing autonomous vehicle and not produce incidences of false detection. The emitting of the sound and the features and characteristics of the emitted sound can be controlled and modulated by one of the encoding or sound modulation processes <NUM>, <NUM> running on a hailing device.

The emitted sound or sounds to be emitted may exhibit the following properties:.

Information related to the hailing request (e.g., the desired class or cost of service, or the presence of a priority or emergency hailing request) may be encoded in the emitted sound by the encoding and sound modulation processes <NUM>, <NUM> using various methods, including but not limited to the following individual methods, or combinations of these methods:.

An autonomous vehicle may be hailed using a hailing device that directly communicates wirelessly with the autonomous vehicle without the hailing request passing through the central server from the potential rider to the autonomous vehicle. The hailing device could be a mobile device or could be fixed at a particular location. The hailing device and a communication device on the autonomous vehicle both would need to support an appropriate communication technology, for example, they would both need to include transmitters, receivers, and communication software. Many existing communication technologies such as Radio Frequency (RF), Infrared (IR), WiFi, WiMax, and Bluetooth may be used or adapted for this purpose. The hailing device could broadcast the request to nearby vehicles, and once a particular autonomous vehicle has responded it can then communicate exclusively with that vehicle. In the case of a hailing device, communication process <NUM> can handle the communications and compliance with the appropriate protocols.

Direct wireless communication can achieved, in some examples, by use of vehicle to infrastructure (V2I) compatible protocols. In such examples, a hailing device can send a hailing request to infrastructure (using the V2I compatible protocol). The infrastructure then relays the hailing request to vehicles in the vicinity (say, within a range of <NUM> meters). Autonomous vehicles pick up the relayed hailing requests using their V2I-compatible receivers. This mode of hailing is direct in that no central server is involved. The infrastructure could be a wide range of devices such as WiFi access points mounted on, say, streetlights.

Each of the various signaling modes that we have discussed can be used in combination with one or more of the other modes, and with additional modes not mentioned explicitly. Using modes in combination can reduce the incidence rate of false detections. For example, a hailing method may rely on executing a particular gesture (e.g., waving one's arm above one's head) while displaying a hailing device that is emitting a particular temporal sequence of colors. A wide range of other combinations of signaling modes would also be possible.

As mentioned above, an autonomous vehicle may also be hailed by a hailing device, such as a hailing device forming part of a kiosk that is fixed at a location. This is particularly relevant, but not limited, to areas where autonomous vehicles are commonly hailed, or where there is usually a good chance of an autonomous vehicle being present nearby, or where there is high pedestrian throughput, or at autonomous vehicle stations. A hailing device in a fixed location could provide a user interface in a variety of modes, from a simple button to something more complex that includes a screen, speakers, microphone, touch-screen, or a combination of these and similar input/output components. The user interface could enable the potential rider (or someone on his behalf) to indicate a destination location, a number of riders, a class of service, a time to arrive, and a variety of other pieces of information relevant to the hailing. The hailing device at the fixed location could broadcast the hailing request to one or more nearby vehicles either by attempting to communicate directly with the one or more nearby vehicles or by communication through a central system, using appropriate wireless communication devices including receivers and transmitters for this purpose.

Once a hailing request receiver on the autonomous vehicle (e.g. one of the communication devices <NUM> shown in <FIG>) detects a hailing request communicated using one of the signaling modes and devices described above or other similar signaling modes or devices, a processor on the autonomous vehicle processes the information contained in the hailing request and, among other things, decides whether to accept the request. This decision may be made by the process running on the autonomous vehicle or by a central system to which the vehicle is connected or by a combination of the two. The decision can be based on a combination of a number of factors. These factors include, but are not limited to: the vehicle's current location, the vehicle's current speed, current traffic conditions in the surrounding area, and the vehicle's determination of its ability to come to a safe stop and pick up the passenger. Nearby autonomous vehicles may also communicate with each other using vehicle-to-vehicle (V2V) communication to ensure that multiple vehicles do not respond to the same hailing request. Communication between vehicles between for this purpose could also be achieved using a centralized system.

Although we have described various methods for hailing an autonomous vehicle by sending a hailing request from (or on behalf of) the potential rider or from a hailing device, thereby yielding one-way communication between the potential rider and the autonomous vehicle, in some cases it may be desirable to enable two-way communication between the potential rider (or an associated hailing device) and the autonomous vehicle, for the purpose of confirming a hailing request.

To confirm a hailing request, upon detection of, for example, an image-based, light-based, auditory, gestural, or other hailing request from a potential rider, the autonomous vehicle may transmit an image-based, light-based, or auditory receipt confirmation signal, for example, with the intent that this receipt confirmation would be uniquely identifiable as a response signal by a sensor or sensors mounted on the hailing device. Upon receipt of this receipt confirmation by the hailing device, the hailing device may transmit a hailing confirmation signal to the autonomous vehicle that is different from the original hailing request. Upon receipt of this hailing confirmation by the autonomous vehicle, the autonomous vehicle would initiate a pickup sequence including stopping near the potential rider for the purpose of picking up the rider.

Exemplary signaling modes for sending a receipt confirmation include those for sending a hailing request. More specifically, such methods include but are not limited to:.

The image or images displayed, light or lights displayed, and sound or sounds emitted by the autonomous vehicle may exhibit exemplary properties that include, for example, those described above for image or images displayed, light or lights displayed, and sound or sounds emitted by the hailing device. Information may also be encoded in the image or images displayed, light or lights displayed, and sound or sounds emitted by the autonomous vehicle using, for example, the methods described above for the image or images displayed, light or lights displayed, and sound or sounds emitted by the hailing device.

Once a receipt confirmation has been received by the hailing device, the hailing device may transmit a hailing confirmation to the autonomous vehicle. Signaling modes for sending for the hailing confirmation can include those that can be employed for transmitting a hailing request. However, it may not be necessary to encode within the hailing confirmation signal information related to the hailing request (e.g., the desired class or cost of service, or the presence of a priority or emergency hailing request). As a result, the hailing confirmation signal could be simpler than the hailing request.

Communication may also be verbal through speech output (utterances) by one or more speakers mounted on the exterior of the vehicle and verbal responses (utterances) received from the potential rider through one or more microphones mounted on the autonomous vehicle. In order to do so, a processor on the autonomous vehicle may execute a speech synthesis program or dialog system contained on the vehicle, play back recorded speech, or broadcast speech received from a human remote operator connected to the vehicle via a wireless link. The volume may be chosen to be appropriate for the distance of the potential rider to the autonomous vehicle. Once a verbal response has been received by the autonomous vehicle, a speech recognizer or decoding program on the vehicle may determine whether to interpret the received voice signal as a hailing confirmation of the hailing request.

Claim 1:
A method performed by an autonomous vehicle that has a processor (<NUM>) and video or LIDAR sensors mounted thereon, wherein the processor is configured to analyse captured data from the video or LIDAR sensors, the method comprising
receiving directly at the video or LIDAR sensors of the autonomous vehicle a hailing request carried by a signaling mode that comprises a human gesture and, optionally a facial expression, the hailing request corresponding to an indication of an intention of a potential rider to use an autonomous vehicle, wherein the signaling mode comprises a common human gesture in combination with a secondary, visually distinctive stimulus in order to reduce false detections, and wherein the secondary, visually distinctive stimulus is the potential rider looking at or tracking the autonomous vehicle by her head pose while executing the common human gesture, and
processing the hailing request at the processor of the autonomous vehicle to determine whether to respond to the hailing request.