Patent Description:
A vehicle, such as an autonomous vehicle, may traverse a portion of a vehicle transportation network. In traversing the vehicle transportation network, the vehicle may provide a service, such as a taxi service, a shuttle, a delivery service, or the like. A service recipient (such as a customer, a passenger, or the like) of the vehicle may inadvertently leave an item in vehicle and/or not fully finish an in-vehicle activity when the service completes An example is given by <CIT>.

Disclosed herein are aspects, features, elements, implementations, and embodiments of interactive external vehicle-user communication.

Variations in these and other aspects, features, elements, implementations, and embodiments of the methods, apparatus, procedures, and algorithms disclosed herein are described in further detail hereafter.

A vehicle, such as an autonomous vehicle, or a semi-autonomous vehicle, may traverse a portion of a vehicle transportation network, to provide a service. In an example, the vehicle, may operate as a taxi. That is, the service can be a taxi service.

The taxi (i.e., the vehicle) may roam (autonomously) the vehicle transportation network to pick up customers. The terms "customer" and "passenger" may be used interchangeably herein. The vehicle may be hailed by a customer by way of a physical gesture. Upon entry into the vehicle, the customer may provide a destination address. The customer can provide the destination address verbally (e.g., for example, in response to a prompt by a human-machine interface of the vehicle). The customer can provide the destination address electronically, such as by typing the address into a console of the vehicle. Other ways may be available for the customer to provide the destination address. The destination may be provided by the customer in the form of a physical address, a cross road location, a landmark name, GPS coordinates, or some other form.

In another example, the customer may hail the vehicle via a computer application, such as an application operating on a device, such as a handheld device (e.g., a mobile device, a tablet, a phablet, or the like). The customer may send a request, via the computer application, to a dispatching location, which may be a computer-based (such a cloud-based) application that receives a request, which can include the location of the customer. The computer-based application may electronically dispatch a vehicle to the customer location. The request may also include a destination of the customer.

In yet other examples, the service can be a shuttle service. As used herein, a shuttle service is similar to a taxi service except that the vehicle can be simultaneously used (e.g., occupied) by more than one customer.

Efficiently providing the service may include providing information to a customer. In some examples, customer information that can be used to contact the customer may be available for providing information to the customer. For example, information may be provided via a phone number, an email address, some other customer information, or a combination thereof.

For example, information regarding items that a customer may be leaving behind (e.g., leaving inside the vehicle) may need to be provided (e.g., output, sent, communicated, etc.) to the customer when the customer leaves the vehicle, such as at the end of the service. The end of the service can mean that the vehicle has arrived at the destination provided by the customer.

For example, information regarding activities related to the service may need to be provided to the customer. For example, information regarding un-finished activities related to the service may need to be provided. Examples of un-finished activities can include that the customer did not close a compartment (e.g., a trunk, a door, etc.) of the vehicle upon retrieving an item of the customer; an incomplete payment (such as due to a declined credit card, an insufficient balance in a debit account, or the like) by the customer for the service.

Providing information (e.g., a message) to the customer may include determining various parameters for determining a notification modality for outputting the message and outputting the message using the notification modality, which may improve the quality, or at least the perceived quality, of the service by the customer. For example, the customer may use the message to retrieve a left-behind item, to close a compartment of the vehicle, to complete an un-finished activity, or a combination thereof.

The various parameters for determining the notification modality include a distance of the customer from the vehicle, and may include a determined value of a left-behind item, the compartment that is left open, the nature of the un-finished activity, more, fewer, other parameters, or a combination thereof.

Outputting the message may include controlling one or more notification modalities to output the message.

The notification modalities available for the vehicle to output the message include haptic feedback, and may include one or more interior (e.g., inside the vehicle) luminous presentation units, one or more external luminous presentation units, one or more internal aural presentation units, one or more external aural presentation units, one or more electronic communication network control units, one or more velocity, kinetic state, or motion control units, or a combination thereof. Fewer, more, or other modalities can also be available.

Outputting the message includes operating one or more touch and/or non-touch haptic mechanisms, as vibrating a part of the vehicle. The haptic feedback may be provided via touch and/or non-touch feedback mechanisms or technologies.

Outputting the message may include controlling an external and/or an internal luminous presentation unit, or a combination of external and/or internal luminous presentation units, of the vehicle to output at least a portion of the message.

An internal luminous presentation unit may be, for example, one or more interior lights, individual lights, groups of lights, such as light bars, light strips, light emitting diode (LED) strips, color-changing light strips, in-vehicle graphic displays, image projectors, augmented reality devices, text displays, or any other unit or device capable of outputting any portion of the message through the control or manipulation of light.

An external luminous presentation unit may be, for example, one or more headlights, taillights, turn signals, individual lights, groups of lights, such as light bars, light strips, light emitting diode (LED) strips, color-changing light strips, on-vehicle graphic displays, image projectors, augmented reality devices, text displays, or any other unit or device capable of outputting any portion of the message of the vehicle control information through the control or manipulation of light.

For example, controlling the external luminous presentation unit may include controlling a brightness or luminance of the external luminous presentation unit, controlling a color or combination of colors of the external luminous presentation unit, controlling a direction of output of the external luminous presentation unit, controlling an output frequency, rate, timing, or period of the external luminous presentation unit, controlling motion of the luminous output or the external luminous presentation unit, controlling a pattern of the luminous output, controlling an animation of the luminous output, or otherwise modulate the output of the external luminous presentation unit, controlling the external luminous presentation unit to display, or otherwise output, text, graphics, or a combination thereof, controlling the external luminous presentation unit to animate the output of the external luminous presentation unit, or any combination thereof.

Outputting the message may include controlling an external and/or an internal aural presentation unit, or a combination of external and/or internal aural presentation units, of the vehicle to output at least a portion of the message. An aural presentation unit (e.g., an internal or an external aural presentation unit) may be, for example, a horn, a speaker, or any other unit or device capable of outputting any portion of the message of the vehicle control information through the control or manipulation of sound. For example, controlling the external (internal) aural presentation unit may include controlling a frequency, tone, phase, or amplitude of the external aural presentation unit, which may include controlling the external (internal) aural presentation unit to output a sequence or series of sounds.

Outputting the message may include controlling an electronic communication network control unit of the vehicle to output information indicating at least a portion of the message, such as by outputting, sending, broadcasting, or transmitting an electronic communication signal via an electronic communication medium to an external device. For example, the external device may be a mobile device of the customer; and outputting the message can include sending a text message, an SMS, or the like to the customer. For example, the electronic communication signal can include sending an email to the customer.

Outputting the message may include controlling a velocity control unit, which may include controlling a directional state control unit of the vehicle, a kinetic state control unit of the vehicle, or both, to output at least a portion of the message of the vehicle control information.

Controlling a directional state control unit, such as a steering unit, of the vehicle to output at least a portion of the message of the vehicle control information may include controlling the directional state control unit to control, or alter, a directional state of the vehicle, such as by turning or steering the vehicle, such that at least a portion of the message of the vehicle control information is output. Controlling the directional state may include maintaining a current directional state such that at least a portion of the message of the vehicle control information is output.

Controlling a kinetic state control unit, such as an accelerator or a braking unit, of the vehicle to output at least a portion of the message of the vehicle control information may include controlling the kinetic state control unit to control, or alter, a kinetic state of the vehicle, such as by accelerating or decelerating the vehicle, such that at least a portion of the message of the vehicle control information is output. Controlling the kinetic state may include maintaining a current kinetic state such that at least a portion of the message of the vehicle control information is output.

Implementations according to this disclosure provide an external notification system designed, for example, for robo-service (e.g., autonomous shuttle, autonomous taxi, autonomous delivery, etc.). Situations whereby a customer leaves behind an item in the vehicle can be resolved by implementations of this disclosure. Some such situations may be referred to as lost-and found situations. A system and/or a process of the vehicle can notify a customer when a left-over (i.e., left-behind) item in the vehicle is detected upon the customer leaving or is away from the vehicle. In an example, notification messages can be triggered (e.g., sent, delivered, output, read out, conveyed, or triggered in any way possible) via different modalities based at least on one of a category of the left-behind item, a distance of the customer from the vehicle, a value of the item, a location of the item within the vehicle, more, fewer, other criteria, or a combination thereof.

Additional situations that can be resolved by implementations according to this disclosure can include calling back a customer because of an unfinished in-vehicle activity. Examples of unfinished in-vehicle activity can include unfinished/uncompleted in-vehicle payment and the customer leaving a door or a trunk of vehicle open. In such situations and based, for example, on the importance of the unfinished activity, the customer distance from the vehicle, other criteria, or a combination thereof, different notification messages can be triggered via different modalities.

Implementations according to this disclosure can provide a control logic for combining object and/or event attributes and customer distance to trigger, for example, the external notifications system and the solution (as further described below).

Although described herein with reference to an autonomous vehicle, the methods and apparatus described herein may be implemented in any vehicle capable of autonomous or semi-autonomous operation. Although described with reference to a vehicle transportation network, the method and apparatus described herein may include the autonomous vehicle operating in any area navigable by the vehicle.

<FIG> is a diagram of an example of a vehicle in which the aspects, features, and elements disclosed herein may be implemented. As shown, a vehicle <NUM> includes a chassis <NUM>, a powertrain <NUM>, a controller <NUM>, wheels <NUM>, and may include any other element or combination of elements of a vehicle. Although the vehicle <NUM> is shown as including four wheels <NUM> for simplicity, any other propulsion device or devices, such as a propeller or tread, may be used. In <FIG>, the lines interconnecting elements, such as the powertrain <NUM>, the controller <NUM>, and the wheels <NUM>, indicate that information, such as data or control signals, power, such as electrical power or torque, or both information and power, may be communicated between the respective elements. For example, the controller <NUM> may receive power from the powertrain <NUM> and may communicate with the powertrain <NUM>, the wheels <NUM>, or both, to control the vehicle <NUM>, which may include accelerating, decelerating, steering, or otherwise controlling the vehicle <NUM>.

The powertrain <NUM> may include a power source <NUM>, a transmission <NUM>, a steering unit <NUM>, an actuator <NUM>, or any other element or combination of elements of a powertrain, such as a suspension, a drive shaft, axles, or an exhaust system. Although shown separately, the wheels <NUM> may be included in the powertrain <NUM>.

The power source <NUM> may include an engine, a battery, or a combination thereof. The power source <NUM> may be any device or combination of devices operative to provide energy, such as electrical energy, thermal energy, or kinetic energy. For example, the power source <NUM> may include an engine, such as an internal combustion engine, an electric motor, or a combination of an internal combustion engine and an electric motor, and may be operative to provide kinetic energy as a motive force to one or more of the wheels <NUM>. The power source <NUM> may include a potential energy unit, such as one or more dry cell batteries, such as nickelcadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion); solar cells; fuel cells; or any other device capable of providing energy.

The transmission <NUM> may receive energy, such as kinetic energy, from the power source <NUM>, and may transmit the energy to the wheels <NUM> to provide a motive force. The transmission <NUM> may be controlled by the controller <NUM> the actuator <NUM> or both. The steering unit <NUM> may be controlled by the controller <NUM> the actuator <NUM> or both and may control the wheels <NUM> to steer the vehicle. The actuator <NUM> may receive signals from the controller <NUM> and may actuate or control the power source <NUM>, the transmission <NUM>, the steering unit <NUM>, or any combination thereof to operate the vehicle <NUM>.

The controller <NUM> may include a location unit <NUM>, an electronic communication unit <NUM>, a processor <NUM>, a memory <NUM>, a user interface <NUM>, a sensor <NUM>, an electronic communication interface <NUM>, or any combination thereof. Although shown as a single unit, any one or more elements of the controller <NUM> may be integrated into any number of separate physical units. For example, the user interface <NUM> and processor <NUM> may be integrated in a first physical unit and the memory <NUM> may be integrated in a second physical unit. Although not shown in <FIG>, the controller <NUM> may include a power source, such as a battery. Although shown as separate elements, the location unit <NUM>, the electronic communication unit <NUM>, the processor <NUM>, the memory <NUM>, the user interface <NUM>, the sensor <NUM>, the electronic communication interface <NUM>, or any combination thereof may be integrated in one or more electronic units, circuits, or chips.

The processor <NUM> may include any device or combination of devices capable of manipulating or processing a signal or other information now-existing or hereafter developed, including optical processors, quantum processors, molecular processors, or a combination thereof. For example, the processor <NUM> may include one or more special purpose processors, one or more digital signal processors, one or more microprocessors, one or more controllers, one or more microcontrollers, one or more integrated circuits, one or more Application Specific Integrated Circuits, one or more Field Programmable Gate Array, one or more programmable logic arrays, one or more programmable logic controllers, one or more state machines, or any combination thereof. The processor <NUM> may be operatively coupled with the location unit <NUM>, the memory <NUM>, the electronic communication interface <NUM>, the electronic communication unit <NUM>, the user interface <NUM>, the sensor <NUM>, the powertrain <NUM>, or any combination thereof. For example, the processor may be operatively coupled with the memory <NUM> via a communication bus <NUM>.

The memory <NUM> may include any tangible non-transitory computer-usable or computer-readable medium, capable of, for example, containing, storing, communicating, or transporting machine readable instructions, or any information associated therewith, for use by or in connection with the processor <NUM>. The memory <NUM> may be, for example, one or more solid state drives, one or more memory cards, one or more removable media, one or more read-only memories, one or more random access memories, one or more disks, including a hard disk, a floppy disk, an optical disk, a magnetic or optical card, or any type of non-transitory media suitable for storing electronic information, or any combination thereof.

The electronic communication interface <NUM> may be a wireless antenna, as shown, a wired communication port, an optical communication port, or any other wired or wireless unit capable of interfacing with a wired or wireless electronic communication medium <NUM>. Although <FIG> shows the electronic communication interface <NUM> communicating via a single communication link, a communication interface may be configured to communicate via multiple communication links. Although <FIG> shows a single communication interface <NUM>, a vehicle may include any number of communication interfaces.

The communication unit <NUM> may be configured to transmit or receive signals via a wired or wireless electronic communication medium <NUM>, such as via the electronic communication interface <NUM>. Although not explicitly shown in <FIG>, the communication unit <NUM> may be configured to transmit, receive, or both via any wired or wireless communication medium, such as radio frequency (RF), ultraviolet (UV), visible light, fiber optic, wireline, or a combination thereof. Although <FIG> shows a single communication unit <NUM> and a single communication interface <NUM>, any number of communication units and any number of communication interfaces may be used. The communication unit <NUM> may include a dedicated short-range communications (DSRC) unit, an on-board unit (OBU), or a combination thereof.

The location unit <NUM> may determine geolocation information, such as longitude, latitude, elevation, direction of travel, or speed, of the vehicle <NUM>. For example, the location unit may include a global positioning system (GPS) unit, such as a Wide Area Augmentation System (WAAS) enabled National Marine -Electronics Association (NMEA) unit, a radio triangulation unit, or a combination thereof. The location unit <NUM> may be used to obtain information that represents, for example, a current heading of the vehicle <NUM>, a current position of the vehicle <NUM> in two or three dimensions, a current angular orientation of the vehicle <NUM>, or a combination thereof.

The user interface <NUM> may include any unit capable of interfacing with a person, such as a virtual or physical keypad, a touchpad, a display, a touch display, a heads-up display, a virtual display, an augmented reality display, a haptic display, a feature tracking device, such as an eye-tracking device, a speaker, a microphone, a video camera, a sensor, a printer, or any combination thereof. The user interface <NUM> may be operatively coupled with the processor <NUM>, as shown, or with any other element of the controller <NUM>. Although shown as a single unit, the user interface <NUM> may include one or more physical units. For example, the user interface <NUM> may include an audio interface for performing audio communication with a person, and a touch display for performing visual and touch-based communication with the person. The user interface <NUM> may include multiple displays, such as multiple physically separate units, multiple defined portions within a single physical unit, or a combination thereof.

The sensor <NUM> may include one or more sensors, such as an array of sensors, which may be operable to provide information that may be used to control the vehicle. The sensors <NUM> may provide information regarding current operating characteristics of the vehicle. The sensors <NUM> may include, for example, a speed sensor, acceleration sensors, a steering angle sensor, traction-related sensors, braking-related sensors, steering wheel position sensors, eye tracking sensors, seating position sensors, or any sensor, or combination of sensors, that is operable to report information regarding some aspect of the current dynamic situation of the vehicle <NUM>.

The sensors <NUM> may include sensors that are operable to obtain information regarding the physical environment surrounding the vehicle <NUM>. For example, one or more sensors may detect road geometry and obstacles, such as fixed obstacles, vehicles, and pedestrians. The sensors <NUM> may be or include one or more video cameras, laser-sensing systems, infrared-sensing systems, acoustic-sensing systems, or any other suitable type of on-vehicle environmental sensing device, or combination of devices, now known or later developed. In some embodiments, the sensors <NUM> and the location unit <NUM> may be combined.

Although not shown separately, in some embodiments, the vehicle <NUM> may include a trajectory controller. For example, the controller <NUM> may include the trajectory controller. The trajectory controller may be operable to obtain information describing a current state of the vehicle <NUM> and a route planned for the vehicle <NUM>, and, based on this information, to determine and optimize a trajectory for the vehicle <NUM>. The trajectory controller may output signals operable to control the vehicle <NUM> such that the vehicle <NUM> follows the trajectory that is determined by the trajectory controller. For example, the output of the trajectory controller may be an optimized trajectory that may be supplied to the powertrain <NUM>, the wheels <NUM>, or both. In some embodiments, the optimized trajectory may be control inputs such as a set of steering angles, with each steering angle corresponding to a point in time or a position. In some embodiments, the optimized trajectory may be one or more paths, lines, curves, or a combination thereof.

One or more of the wheels <NUM> may be a steered wheel, which may be pivoted to a steering angle under control of the steering unit <NUM>, a propelled wheel, which may be torqued to propel the vehicle <NUM> under control of the transmission <NUM>, or a steered and propelled wheel that may steer and propel the vehicle <NUM>.

Although not shown in <FIG>, a vehicle may include units, or elements not shown in <FIG>, such as an enclosure, a Bluetooth® module, a frequency modulated (FM) radio unit, a Near Field Communication (NFC) module, a liquid crystal display (LCD) display unit, an organic light-emitting diode (OLED) display unit, a speaker, or any combination thereof.

The vehicle <NUM> is an autonomous vehicle. An autonomous vehicle may be controlled autonomously, without direct human intervention, to traverse a portion of a vehicle transportation network. Although not shown separately in <FIG>, in some implementations, an autonomous vehicle may include an autonomous vehicle control unit, which may perform autonomous vehicle routing, navigation, and control. In some implementations, the autonomous vehicle control unit may be integrated with another unit of the vehicle. For example, the controller <NUM> may include the autonomous vehicle control unit.

In some implementations, the autonomous vehicle control unit may control or operate the vehicle <NUM> to traverse the vehicle transportation network, or a portion thereof, in accordance with current vehicle operation parameters. In another example, the autonomous vehicle control unit may control or operate the vehicle <NUM> to perform a defined operation or maneuver, such as parking the vehicle. In another example, autonomous vehicle control unit may generate a route of travel from an origin, such as a current location of the vehicle <NUM>, to a destination based on vehicle information, environment information, vehicle transportation network information representing the vehicle transportation network, or a combination thereof, and may control or operate the vehicle <NUM> to traverse the vehicle transportation network in accordance with the route. For example, the autonomous vehicle control unit may output the route of travel to a trajectory controller that may operate the vehicle <NUM> to travel from the origin to the destination using the generated route.

<FIG> is a diagram of an example of a portion of a vehicle transportation and communication system in which the aspects, features, and elements disclosed herein may be implemented. The vehicle transportation and communication system <NUM> may include one or more vehicles <NUM>/<NUM>, such as the vehicle <NUM> shown in <FIG>, which may travel via one or more portions of one or more vehicle transportation networks <NUM>, and may communicate via one or more electronic communication networks <NUM>. Although not explicitly shown in <FIG>, a vehicle may traverse an area that is not expressly or completely included in a vehicle transportation network, such as an off-road area.

The electronic communication network <NUM> may be, for example, a multiple access system and may provide for communication, such as voice communication, data communication, video communication, messaging communication, or a combination thereof, between the vehicle <NUM>/<NUM> and one or more communication devices <NUM>. For example, a vehicle <NUM>/<NUM> may receive information, such as information representing the vehicle transportation network <NUM>, from a communication device <NUM> via the electronic communication network <NUM>.

A vehicle <NUM>/<NUM> may communicate via a wired communication link (not shown), a wireless communication link <NUM>/<NUM>/<NUM>, or a combination of any number of wired or wireless communication links. For example, as shown, a vehicle <NUM>/<NUM> may communicate via a terrestrial wireless communication link <NUM>, via a non-terrestrial wireless communication link <NUM>, or via a combination thereof. In some implementations, a terrestrial wireless communication link <NUM> may include an Ethernet link, a serial link, a Bluetooth link, an infrared (IR) link, an ultraviolet (UV) link, or any link capable of providing for electronic communication.

A vehicle <NUM>/<NUM> may communicate with another vehicle <NUM>/<NUM>. For example, a host, or subject, vehicle <NUM> may receive one or more automated inter-vehicle messages, such as a basic safety message (BSM), from a remote, or target, remote vehicle <NUM>, via a direct communication link <NUM>, or via an electronic communication network <NUM>. For example, the remote vehicle <NUM> may broadcast the message to host vehicles within a defined broadcast range, such as <NUM> meters. In some embodiments, the vehicle <NUM> may receive a message via a third party, such as a signal repeater (not shown) or another remote vehicle (not shown). In some embodiments, a vehicle <NUM>/<NUM> may transmit one or more automated inter-vehicle messages periodically, based on, for example, a defined interval, such as <NUM> milliseconds.

Automated inter-vehicle messages may include vehicle identification information, geospatial state information, such as longitude, latitude, or elevation information, geospatial location accuracy information, kinematic state information, such as vehicle acceleration information, yaw rate information, speed information, vehicle heading information, braking system status information, throttle information, steering wheel angle information, or vehicle routing information, or vehicle operating state information, such as vehicle size information, headlight state information, turn signal information, wiper status information, transmission information, or any other information, or combination of information, relevant to the transmitting vehicle state. For example, transmission state information may indicate whether the transmission of the transmitting vehicle is in a neutral state, a parked state, a forward state, or a reverse state.

The vehicle <NUM> may communicate with the electronic communication network <NUM> via an access point <NUM>. An access point <NUM>, which may include a computing device, may be configured to communicate with a vehicle <NUM>, with an electronic communication network <NUM>, with one or more communication devices <NUM>, or with a combination thereof via wired or wireless communication links <NUM>/<NUM>. For example, an access point <NUM> may be a base station, a base transceiver station (BTS), a Node-B, an enhanced Node-B (eNode-B), a Home Node-B (HNode-B), a wireless router, a wired router, a hub, a relay, a switch, or any similar wired or wireless device. Although shown as a single unit, an access point may include any number of interconnected elements.

The vehicle <NUM> may communicate with the electronic communication network <NUM> via a satellite <NUM>, or other non-terrestrial communication device. A satellite <NUM>, which may include a computing device, may be configured to communicate with a vehicle <NUM>, with an electronic communication network <NUM>, with one or more communication devices <NUM>, or with a combination thereof via one or more communication links <NUM>/<NUM>. Although shown as a single unit, a satellite may include any number of interconnected elements.

An electronic communication network <NUM> may be any type of network configured to provide voice, data, or any other type of electronic communication. For example, the electronic communication network <NUM> may include a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a mobile or cellular telephone network, the Internet, or any other electronic communication system. The electronic communication network <NUM> may use a communication protocol, such as the transmission control protocol (TCP), the user datagram protocol (UDP), the internet protocol (IP), the real-time transport protocol (RTP) the HyperText Transport Protocol (HTTP), or a combination thereof. Although shown as a single unit, an electronic communication network may include any number of interconnected elements.

A vehicle <NUM> may identify a portion or condition of the vehicle transportation network <NUM>. For example, the vehicle may include one or more on-vehicle sensors <NUM>, such as sensor <NUM> shown in <FIG>, which may include a speed sensor, a wheel speed sensor, a camera, a gyroscope, an optical sensor, a laser sensor, a radar sensor, a sonic sensor, or any other sensor or device or combination thereof capable of determining or identifying a portion or condition of the vehicle transportation network <NUM>.

A vehicle <NUM> may traverse a portion or portions of one or more vehicle transportation networks <NUM> using information communicated via the electronic communication network <NUM>, such as information representing the vehicle transportation network <NUM>, information identified by one or more on-vehicle sensors <NUM>, or a combination thereof.

Although, for simplicity, <FIG> shows one vehicle <NUM>, one vehicle transportation network <NUM>, one electronic communication network <NUM>, and one communication device <NUM>, any number of vehicles, networks, or computing devices may be used. In some embodiments, the vehicle transportation and communication system <NUM> may include devices, units, or elements not shown in <FIG>. Although the vehicle <NUM> is shown as a single unit, a vehicle may include any number of interconnected elements.

Although the vehicle <NUM> is shown communicating with the communication device <NUM> via the electronic communication network <NUM>, the vehicle <NUM> may communicate with the communication device <NUM> via any number of direct or indirect communication links. For example, the vehicle <NUM> may communicate with the communication device <NUM> via a direct communication link, such as a Bluetooth communication link.

A vehicle <NUM>/<NUM> may be associated with an entity <NUM>/<NUM>, such as a driver, operator, or owner of the vehicle. An entity <NUM>/<NUM> associated with a vehicle <NUM>/<NUM> may be associated with one or more personal electronic devices <NUM>/<NUM>/<NUM>/<NUM>, such as a smartphone <NUM>/<NUM> or a computer <NUM>/<NUM>. A personal electronic device <NUM>/<NUM>/<NUM>/<NUM> may communicate with a corresponding vehicle <NUM>/<NUM> via a direct or indirect communication link. Although one entity <NUM>/<NUM> is shown as associated with one vehicle <NUM>/<NUM> in <FIG>, any number of vehicles may be associated with an entity and any number of entities may be associated with a vehicle.

<FIG> is a diagram of a portion of a vehicle transportation network in accordance with this disclosure. A vehicle transportation network <NUM> may include one or more unnavigable areas <NUM>, such as a building, one or more partially navigable areas, such as parking area <NUM>, one or more navigable areas, such as roads <NUM>/<NUM>, or a combination thereof. An autonomous vehicle, such as the vehicle <NUM> shown in <FIG>, one of the vehicles <NUM>/<NUM> shown in <FIG>, a semi-autonomous vehicle, or any other vehicle implementing autonomous driving, may traverse a portion or portions of the vehicle transportation network <NUM>.

The vehicle transportation network may include one or more interchanges <NUM> between one or more navigable, or partially navigable, areas <NUM>/<NUM>/<NUM>. For example, the portion of the vehicle transportation network shown in <FIG> includes an interchange <NUM> between the parking area <NUM> and road <NUM>. In some embodiments, the parking area <NUM> may include parking slots <NUM>.

A portion of the vehicle transportation network, such as a road <NUM>/<NUM>, may include one or more lanes <NUM>/<NUM>/<NUM>/<NUM>/<NUM> and may be associated with one or more directions of travel, which are indicated by arrows in <FIG>.

A vehicle transportation network, or a portion thereof, such as the portion of the vehicle transportation network shown in <FIG>, may be represented as vehicle transportation network information. For example, vehicle transportation network information may be expressed as a hierarchy of elements, such as markup language elements, which may be stored in a database or file. For simplicity, the figures herein depict vehicle transportation network information representing portions of a vehicle transportation network as diagrams or maps; however, vehicle transportation network information may be expressed in any computer-usable form capable of representing a vehicle transportation network, or a portion thereof. The vehicle transportation network information may include vehicle transportation network control information, such as direction of travel information, speed limit information, toll information, grade information, such as inclination or angle information, surface material information, aesthetic information or a combination thereof.

A portion, or a combination of portions, of the vehicle transportation network may be identified as a point of interest or a destination. For example, the vehicle transportation network information may identify a building, such as the unnavigable area <NUM>, and the adjacent partially navigable parking area <NUM> as a point of interest, an autonomous vehicle may identify the point of interest as a destination, and the autonomous vehicle may travel from an origin to the destination by traversing the vehicle transportation network. Although the parking area <NUM> associated with the unnavigable area <NUM> is shown as adjacent to the unnavigable area <NUM> in <FIG>, a destination may include, for example, a building and a parking area that is physically or geospatially non-adjacent to the building.

Identifying a destination may include identifying a location for the destination, which may be a discrete uniquely identifiable geolocation. For example, the vehicle transportation network may include a defined location, such as a street address, a postal address, a vehicle transportation network address, a GPS address, or a combination thereof for the destination.

A destination may be associated with one or more entrances, such as the entrance <NUM> shown in <FIG>. The vehicle transportation network information may include defined entrance location information, such as information identifying a geolocation of an entrance associated with a destination. Predicted entrance location information may be determined as described herein.

The vehicle transportation network may be associated with, or may include, a pedestrian transportation network. For example, <FIG> includes a portion <NUM> of a pedestrian transportation network, which may be a pedestrian walkway. A pedestrian transportation network, or a portion thereof, such as the portion <NUM> of the pedestrian transportation network shown in <FIG>, may be represented as pedestrian transportation network information. The vehicle transportation network information may include pedestrian transportation network information. A pedestrian transportation network may include pedestrian navigable areas. A pedestrian navigable area, such as a pedestrian walkway or a sidewalk, may correspond with a non-navigable area of a vehicle transportation network. Although not shown separately in <FIG>, a pedestrian navigable area, such as a pedestrian crosswalk, may correspond with a navigable area, or a partially navigable area, of a vehicle transportation network.

A destination may be associated with one or more docking locations, such as the docking location <NUM> shown in <FIG>. A docking location <NUM> may be a designated or undesignated location or area in proximity to a destination at which an autonomous vehicle may stop, stand, or park such that docking operations, such as passenger loading or unloading, may be performed.

The vehicle transportation network information may include docking location information, such as information identifying a geolocation of one or more docking locations <NUM> associated with a destination. The docking location information may be defined docking location information, which may be docking location information manually included in the vehicle transportation network information. For example, defined docking location information may be included in the vehicle transportation network information based on user input. The docking location information may be automatically generated docking location information as described herein. Although not shown separately in <FIG>, docking location information may identify a type of docking operation associated with a docking location <NUM>. For example, a destination may be associated with a first docking location for passenger loading and a second docking location for passenger unloading. Although an autonomous vehicle may park at a docking location, a docking location associated with a destination may be independent and distinct from a parking area associated with the destination.

In an example, an autonomous vehicle may identify a point of interest, which may include the unnavigable area <NUM>, the parking area <NUM>, and the entrance <NUM>, as a destination. The autonomous vehicle may identify the unnavigable area <NUM>, or the entrance <NUM>, as a primary destination for the point of interest, and may identify the parking area <NUM> as a secondary destination. The autonomous vehicle may identify the docking location <NUM> as a docking location for the primary destination. The autonomous vehicle may generate a route from an origin (not shown) to the docking location <NUM>. The autonomous vehicle may traverse the vehicle transportation network from the origin to the docking location <NUM> using the route. The autonomous vehicle may stop or park at the docking location <NUM> such that passenger loading or unloading may be performed. The autonomous vehicle may generate a subsequent route from the docking location <NUM> to the parking area <NUM>, may traverse the vehicle transportation network from the docking location <NUM> to the parking area <NUM> using the subsequent route, and may park in the parking area <NUM>.

As mentioned above, the various parameters for outputting the message include the distance of the customer from the vehicle, and may include the determined value of a left-behind item, the compartment that is left open, the nature of the un-finished activity, more, fewer, other parameters, or a combination thereof.

In an example, the notification modality for outputting the message is based on the location of the customer. That is, the modality is based, at least partially, on the distance and/or proximity of the customer to the vehicle.

<FIG> is a diagram <NUM> of different locations of a customer with respect to a vehicle in accordance with the invention. The diagram <NUM> includes a vehicle <NUM> and a customer <NUM>. The vehicle <NUM> is an autonomous vehicle (AV), such as the vehicle <NUM> shown in <FIG> or one of the vehicles <NUM>/<NUM> shown in <FIG>. The AV may be a semi-autonomous vehicle, or any other vehicle implementing autonomous driving. The AV is configured to traverse a portion of a vehicle transportation network. The customer <NUM> can be a service recipient of a service provided by the vehicle <NUM>. For example, the customer <NUM> can be a taxi passenger or a shuttle passenger of the vehicle <NUM>.

At a first distance <NUM> away from the vehicle <NUM>, the customer <NUM> is shown as leaving the vehicle <NUM>. While the customer <NUM> is shown as being outside the vehicle <NUM>, "leaving" the vehicle can include, according to a non claimed embodiment, that the customer <NUM> is still inside the vehicle <NUM> but with a door of the vehicle <NUM> open indicating an intention of the customer <NUM> of exiting the vehicle <NUM>; according to the invention, "leaving" the vehicle includes that a the customer <NUM> is outside the vehicle with the customer's hand on a door (such as the handle) of the vehicle. If the customer <NUM> is at or within a first distance from the vehicle <NUM>, a first notification is used to output a message to the customer <NUM>. If the customer's hand is touching the door while the door is open and the customer is outside the vehicle, then the door panel, the door handle, or other part of the door are vibrated, to provide a haptic notification. The first notification modality can, additionally, include an internal audio signal, an external audio signal having a first volume level, an external visual reminder, other notification modes, or a combination thereof.

At a second distance <NUM> away from the vehicle <NUM>, the customer <NUM> is shown as within the second distance away from the vehicle <NUM>. In an example, the second distance can be a distance between <NUM> and <NUM> meters away from the vehicle <NUM>. If the customer <NUM> is within the second distance from the vehicle <NUM>, then a second notification modality is used to output the message to the customer <NUM>. The second notification modality includes an external audio signal that may have having a second volume level. The second notification modality may further include the external visual reminder, other visual reminders, other notification modes, or a combination thereof. The second volume level can be higher than the first volume level.

At a third distance <NUM> away from the vehicle <NUM>, the customer <NUM> is shown as being within the third distance (i.e., between the second distance <NUM> and the third distance <NUM>) away from the vehicle <NUM>. In an example, the third distance can be a distance between <NUM> and <NUM> meters away from the vehicle <NUM>. If the customer <NUM> is within the third distance from the vehicle <NUM>, then a third notification modality is used to output the message to the customer <NUM>. The third notification modality includes an external audio signal that may have a third volume level. The third notification modality may further include a voice message, a text message, other notification modes, or a combination thereof. The third volume level can be higher than the second volume level.

If the customer <NUM> is beyond the third distance <NUM> away from the vehicle <NUM>, then a fourth notification modality is used to output the message to the customer <NUM>. The fourth notification modality includes a voice call and/or a text message, and may further include other notification modes, or a combination thereof.

While three distinct distances and four notification modalities are described with respect to the diagram <NUM>, the disclosure herein is not so limited. Any number of distances can be used, and any number and types of respective modalities can be used, provided they remain under the scope of the claims. Similarly, different messages or similar messages can be output based on the distance, the value of the item, the nature of the unfinished in-vehicle activity, and so on, provided they remain under the scope of the claims.

<FIG> shows process <NUM> for communicating a message to a customer (i.e., a recent occupant) based on a distance of the customer from a vehicle in accordance with the invention. The process <NUM> is implemented in or by an autonomous vehicle (AV), such as the vehicle <NUM> shown in <FIG> or one of the vehicles <NUM>/<NUM> shown in <FIG>. The AV may be a semi-autonomous vehicle, or any other vehicle implementing autonomous driving. The process <NUM> is executed by a controller, such as the controller <NUM> of <FIG>. The process <NUM> is stored, as processor-executable instructions, in a memory, such as the memory <NUM> of <FIG>. The processor-executable instructions are executed by the processor <NUM> of <FIG>.

The process <NUM> communicates the message to the customer in response to detecting that the customer may be leaving an item (e.g., an item that belongs to the customer) in the vehicle. The process <NUM> uses different modalities to communicate the message to the customer based on a distance of the customer from the vehicle. In some example, the message may not be an explicit (e.g., verbal or textual) message; rather, the message may be intended as a reminder to the customer that the customer may have forgotten/left an item in the AV.

The process <NUM> can be executed in response to the vehicle being in a service mode. For example, if the vehicle is in a service mode (i.e., performing/executing a service, such as a taxi or a shuttle service), then any occupant of the vehicle can be considered to be a temporary occupant. As such, any item that the customer may leave inside the vehicle at the end of the service (e.g., when the vehicle arrives at the customer's destination) can be considered a forgotten or lost item. As such, the customer most likely wishes to be notified of lost or forgotten items.

In an example, the process <NUM> may not be executed in situations where the vehicle is not in a service mode. Any items left inside the vehicle may be considered to belong to an occupant who may be returning to the vehicle. As such, the item may be considered to be intentionally left in the vehicle. In another example, the process <NUM> is executed whether the vehicle is or is not in a service mode.

At <NUM>, the process <NUM> detects whether the customer is leaving the vehicle. The process <NUM> can detect that the customer is leaving the vehicle in response to the vehicle arriving at a destination of the customer, in response to a door of the vehicle being opened, in response to the customer being outside the vehicle, in response to other criteria, or a combination thereof.

If the customer is leaving the vehicle, the process proceeds to <NUM>; otherwise the process <NUM> returns to execute <NUM>. The process <NUM> can be continuously executed, such as every several milliseconds.

At <NUM>, the process <NUM> detects whether an item is detected within the vehicle. If an item is detected, then the process <NUM> proceeds to <NUM>. If an item is not detected, the process <NUM> proceeds back to <NUM>. In an example, the process <NUM> terminates if an item is not detected at <NUM>.

The process <NUM> can detect an item within a vehicle in any number of ways.

For example, the vehicle can include internal sensors, which can be used to detect an item inside the vehicle. The internal sensors can include one or more internal cameras (inside the cabin of the vehicle and/or in storage compartments, such as a trunk), one or more weight differential sensors, other sensors, or a combination thereof. For example, before entry of the customer into the vehicle to start the service, one or more internal cameras of the vehicle can be used to capture first images of the interior of the vehicle and upon detecting that the customer is leaving the vehicle, the one or more internal cameras can be used to capture second images. Using differences between the first images and the second images, the process <NUM> can detect that one or more items are left behind. That is, the mere fact that differences between the first images and the second images exist can be indicative of left behind items. In an example, image recognition can be used to identify the item(s) left behind.

For example, the process <NUM> can maintain trip-based data. The trip-based data can include (e.g., maintain) for a customer, the item(s) left by the customer in the vehicle. As such, the process <NUM> does not send messages, as further described below, to a subsequent customer regarding an item that is left in the vehicle by a previous customer.

For example, the process <NUM> can detect an item within a vehicle based on the fact that a compartment (e.g., a trunk) was opened by the customer at the beginning of the service but not at the end of the service. As such, the process <NUM> can infer that the customer is leaving an item in the compartment.

Examples of items that may be left behind by the customer include a hat, sunglasses, a jacket, a bag, a back pack, a suitcase, a cup of coffee, a water bottle, and the like. In some cases, items may fall inadvertently from pockets of the customer, such as a wallet, a cell phone, coins, monetary bills, a paper or notebook that includes valuable information, and the like. In some cases, the items left behind may be intentionally left behind, such as an empty cup of coffee, an empty water bottle, or other items that can be considered trash.

At <NUM>, the process <NUM> determines whether the customer is by the AV. The customer is by the AV if the customer is outside the AV and is touching the door (for example, to close the door). If the customer is by the AV, the process <NUM> proceeds to <NUM>; otherwise the process <NUM> proceeds to <NUM>. At <NUM>, the process <NUM> selects a first notification modality for sending the message to the customer. The message can be intended to be a reminder or a memory tickler to the customer that the customer may have left the item inside the AV. The message can be explicit regarding the item, or, as mentioned above, an unfinished in-vehicle activity, and an action to be taken by the customer.

In an example, the first notification modality can include one or more modalities. According to the invention, the first notification modality includes haptic feedback (by vibrating the door, a door panel, or other part of the vehicle). The first notification modality can further include an external audio signal. For example, in a case that the vehicle is equipped with an external speaker, the external audio can include a message that states essentially "You may have left an item in the car. " In a case that the item is identified using object detection or classification, the message can include a description of the item. For example, the message can essentially state "You may have left a wallet in the car. " In an example, the external audio signal can include sounding the horn. In an example, the horn can be sounded at a first volume level that is audible by the customer, using a first pattern, or a combination thereof. The first notification modality can further include an external visual reminder. The visual reminder can include, for example, the flashing of lights, the projecting of an image of the left-behind item in the vicinity of the customer, the projecting of a textual message that may be similar to the above described external audible signal. The first volume can be <NUM>%, <NUM>%, or other percent, of a normal sound emitted by the vehicle.

At <NUM>, if the customer is within a first distance of the vehicle, the process <NUM> proceeds to <NUM>; otherwise the process <NUM> proceeds to <NUM>. In an example, the first distance can be <NUM> to <NUM> meters away from the vehicle. The distance of the customer to the vehicle can be determined (e.g., measured, estimated, etc.) using sensor data. In an illustrative, non-limiting example, a point cloud received from a light detection and ranging (LiDAR) sensor (i.e., a sensor of the sensor <NUM>) may be converted into an object, which can be determined to be the customer. The LiDAR data can be used to determine the distance of the customer to the vehicle. Similarly, sensor data from several sensors may be fused together to determine the distance of the customer.

At <NUM>, the process <NUM> selects a second notification modality for sending the message to the customer. As mentioned above, the message can be intended to be a reminder or a memory tickler, and/or an explicit message, to the customer that the customer may have left the item inside the AV.

In an example, the second notification modality can include one or more modalities. According to the invention, the second notification modality includes an external audio signal. The external audio signal can be similar to the external audio signal described with respect to the first notification modality. For example, a second volume of the external audio signal can be <NUM>%, <NUM>%, or other percent, of the normal sound emitted by the vehicle. In another example, the second volume can be higher than the first volume. For example, the second volume can be an additional <NUM>% higher than the first volume. The second notification modality can further an external visual reminder, which can be similar to the external visual reminder described above with respect to the first notification modality.

At <NUM>, if the customer is within a second distance of the vehicle, the process <NUM> proceeds to <NUM>; otherwise the process <NUM> proceeds to <NUM>. In an example, the second distance can be <NUM> to <NUM> meters. The distance of the customer to the vehicle can be determined (e.g., measured, estimated, etc.) using sensor data, as described above with respect to the first distance.

At <NUM>, the process <NUM> selects a third notification modality for sending the message to the customer. In an example, the third notification modality can include one or more modalities. According to the invention, the third notification modality includes an external audio signal. The external audio signal can be similar to the external audio signal described with respect to the second notification modality but can have a third volume that is higher than the second volume. The second notification modality can further include a textual message that is sent to the customer. For example, the contact information of the user can be used to send a text message, an email, or other text-based message to the customer. For example, the contact information can be used to place a voice call the customer. For example, the voice call can include delivering an automated message to the customer to the effect that the item was left in the vehicle. In an example, the vehicle itself can send the text message and/or place the voice call to the customer. In another example, the vehicle sends the text message and/or places the voice call by causing a communication device, such as the communication device <NUM> of <FIG>, to send the text message and/or place the voice call. The text message can be sent to the user via (e.g., to) an application that is specific to the service and/or via an application that is specific to the service provider. For example, the application can be available (e.g., executing, running, in a state capable of receiving notifications, etc.) on a device (such as a portable device, a wearable device, etc.) of the user. In an example, the application can be an application via which the user requested the service.

At <NUM>, the process <NUM> selects a fourth notification modality for sending the message to the customer. According to the invention, the fourth notification modality includes a text message and/or a voice call, as described above with respect to the third notification modality.

In some implementations, the message is not sent using a subsequent modality in the case that the item is no longer in the vehicle. For example, if after sending the message using the first notification modality, at <NUM>, and before sending the message using the second notification modality, at <NUM>, the item is not detected within the vehicle, then the message is not sent using the second notification modality or any subsequent modality; similarly, if after sending the message using the second notification modality, at <NUM>, and before sending the message using the third notification modality, at <NUM>, the item is not detected within the vehicle, then the message is not sent using the third notification modality or any subsequent modality; and so on. That is, after sending the message using one modality, the process <NUM> can terminate when the process detects that the item is no longer in vehicle, such as if the customer removed the item from the vehicle.

In some implementations, the message is not sent using a subsequent modality in the case that the customer acknowledges the message. For example, if after sending the message using the first notification modality, at <NUM>, and before sending the message using the second notification modality, at <NUM>, the process <NUM> receives an acknowledgement from the customer, then the message is not sent using the second notification modality or any subsequent modality; similarly, if after sending the message using the second notification modality, at <NUM>, and before sending the message using the third notification modality, at <NUM>, receives an acknowledgement from the customer, then the message is not sent using the third notification modality or any subsequent modality; and so on. That is, after sending the message using one modality, the process <NUM> can terminate when the process receives an acknowledgement from the customer. In an example, the acknowledgement can be an explicit acknowledgement. In an example, the acknowledgement can be an implicit acknowledgement.

In an example, an explicit acknowledgment can include receiving a text message response from the customer to the text message of the third notification modality. In an example, an explicit acknowledgment can include the customer completely and/or partially listening to the voice call.

An implicit acknowledgment can include a gesture by the customer that the process <NUM> can construe as dismissing the message or acknowledging the message. For example, a module (such as a human-machine interface (HMI) module of the vehicle, which may be executed by a controller, such as the controller <NUM> of <FIG>) may use one or more methodologies, techniques, or technologies of motion detection and confirmation for translating one or more gestures of the customer into an acknowledgement. For example, sensor (e.g., the sensor <NUM> of <FIG>) data (e.g., images, LiDAR data, etc.) may be compared to templates wherein a template correlates to an acknowledgement. For example, the vehicle may include one or more classifiers trained to recognize gestures, movements, and/or body positions and determine an acknowledgement. For example, a gesture recognition classifier may be used to determine whether a received gesture signal matches a gesture template to identify the gesture. For example, a backward swipe of a hand may be construed as the customer dismissing the message. For example, a thumbs-up gesture, or an OK gesture, can be construed as an acknowledgement of the message. The HMI module can be trained to recognize other or additional gestures as the customer acknowledging and/or dismissing the message.

In an implementation, the process <NUM>, may not detect the customer at one distance, but may detect the customer at another distance. As such, the process <NUM> sends a message when the customer is detected at the other distance. For example, the customer may not be detected at <NUM> but is detected at <NUM>. As such, the process does not perform the operation <NUM>, but does perform the operation <NUM>.

In an implementation, not falling under the scope of the claims and not specifically shown in <FIG>, the process <NUM> can send only one notification. For example, if the process <NUM> performs one of <NUM>, <NUM>, or <NUM>, then any other subsequent message is not sent. In an implementation, the process <NUM> always performs the operation <NUM>, unless the customer retrieved the item.

The process <NUM> describes selecting a modality for sending a message (or messages) to the customer based on the distance of the customer to the vehicle. In some implementations, the modality can be selected based, additionally, on the value of the item. For example, the message(s) may be sent using more modalities and at different distance intervals for a more valuable item than a less valuable item. For example, a wallet may be considered more valuable than a cup of coffee. For example, money (e.g., coins and/or paper bills) may be considered more valuable than a crumbled piece of paper. In some examples, the location of the item in the vehicle can also be used to select the number of modalities to use.

<FIG> is an example of a process <NUM> for communicating a message to a customer based on a value of an item left in a vehicle in accordance with implementations of this disclosure. The process <NUM> is implemented in or by an autonomous vehicle (AV), such as the vehicle <NUM> shown in <FIG> or one of the vehicles <NUM>/<NUM> shown in <FIG>. The AV may be a semi-autonomous vehicle, or any other vehicle implementing autonomous driving. The process <NUM> can be executed by a controller, such as the controller <NUM> of <FIG>. The process <NUM> can be stored, as processor-executable instructions, in a memory, such as the memory <NUM> of <FIG>. The processor-executable instructions can be executed by the processor <NUM> of <FIG>.

The process <NUM> communicates the message to the customer in response to detecting that the customer may be leaving an item (e.g., an item that belongs to the customer) in the vehicle. The process <NUM> uses different modalities to communicate the message(s) to the customer based on a value of the item and a distance of the customer from the vehicle. In some example, the message(s) may not be explicit (e.g., verbal or textual) message(s); rather, the message(s) may be intended to reminder the customer that the customer may have forgotten/left an item in the AV.

The process <NUM> can be executed in response to the vehicle being in a service mode. For example, if the vehicle is in a service mode (i.e., performing/executing a service, such as a taxi or shuttle service), then any occupant of the vehicle can be considered a temporary occupant (i.e., a customer). As such, any item that the customer may leave inside the vehicle at the end of the service (e.g., when the vehicle arrives at the customer's destination) can be considered a forgotten or lost item. As such, the customer most likely wishes to be notified of lost or forgotten items.

At <NUM>, the process <NUM> detects an item within the AV. The process <NUM> can detect the item as described with respect to <NUM> of <FIG>. Using, for example, object recognition and classification, the process <NUM> can determine a value for the item. For simplicity of explanation, the process <NUM> is described with respect to three item values; namely, trash, high value, and not high value. A trash item is an item that is of zero or negligible value, such as a crumbled piece of paper or an empty water bottle. A high value item may be, for example, a wallet, a ring, a mobile phone, etc. Examples of non-high value items include sunglasses, a hat, etc..

In an example, a threshold dollar value can be used to determine whether the item is or is not of high value. For example, if the item is likely to be valued at greater than $<NUM> (or some other threshold dollar value), then the item can be considered to be a high value item; otherwise the item can be considered a non-high value item.

The value of the item can also be determined based on an intangible value of the item. For example, a ring, a necklace, and other items that may have emotional value to the owner can be considered high-value items, regardless of their monetary values.

At <NUM>, the process <NUM> determines whether the item is a valued as a trash item. If the item is valued as trash, the process <NUM> proceeds to <NUM>; otherwise the process <NUM> proceeds to <NUM>.

At <NUM>, the process <NUM> can, at predetermined distances of the customer from the vehicle, send a text message to the customer indicating that the customer has left an item in the vehicle. As such, the selected modality is the sending of a text message.

The predetermined distances can be as described with respect to the process <NUM>. That is, the process <NUM> can send a first text message to the customer when the customer is within the first distance away from vehicle, a second text message when the customer is within the second distance away from the vehicle, and so on. In an example, the same text message is sent at every distance, or at some of the distances. In another example, a different message can be sent at every distance. In an example, the text message can request that the customer remove the item from them vehicle. In an example, when the customer is beyond the second distance, the text message can indicate that the item will be trashed (i.e., disposed of).

In an example, the text message can provide an incentive to the customer to remove the item from the vehicle. Such a message can be sent, for example, when the customer is within the second distance away from the vehicle. In an example, the incentive can be a discount on a cost (e.g., ride fare) related to the just-completed service. As such, upon the process <NUM> detecting that the customer has removed then item, the process <NUM> can cause the payment system to credit the customer according to the incentive. In another example, the incentive can be a discount on a subsequent service. As such, upon the process <NUM> detecting that the customer has removed the item, the process <NUM> can send, or cause to be sent, a discount voucher, an incentive code, or the like, to the customer.

At <NUM>, the process <NUM> determines whether the item is a high-value item. If the item is not a high-value item, the process <NUM> proceeds to one of <NUM>, <NUM>, <NUM>, or <NUM>, depending on the distance of the customer to the vehicle; otherwise, the process <NUM> proceeds to block <NUM>. While the operations <NUM>-<NUM> are arranged differently than the operations of the process <NUM>, a person skilled in the art can appreciate that the operations <NUM>-<NUM> can be as described with respect to the process <NUM>.

At <NUM>, if the customer is by the vehicle, the process <NUM> proceeds to <NUM>. The process <NUM> can detect that the customer is by the vehicle as described with respect to <NUM> of <FIG>. At <NUM>, the process <NUM> selects a first solution for dealing with the item. In an example, the first solution can include selecting a first notification modality for sending a message (e.g., a reminder) to the customer that the customer may be forgetting the item in the vehicle and sending the message. The first solution is as described with respect to notifying via a first notification modality at <NUM>.

At <NUM>, if the customer is within a first distance of the vehicle, the process <NUM> proceeds to <NUM>. The firts distance is as described with respect to the first distance of <FIG>. As such, the process <NUM> detects that the customer is within the first distance of the vehicle as described with respect to <NUM> of <FIG>. At <NUM>, the process <NUM> selects a second solution for dealing with the item. In an example, the second solution can include selecting a second notification modality for sending a message to the customer and sending the message. According to the invention, the second solution is as described with respect to notifying via a second notification modality, at <NUM> of <FIG>.

At <NUM>, if the customer is within a second distance of the vehicle, the process <NUM> proceeds to <NUM>. According to the invention, the second distance is as described with respect to the second distance of <FIG>. As such, the process <NUM> detects that the customer is within the second distance of the vehicle as described with respect to <NUM> of <FIG>. At <NUM>, the process <NUM> selects a third solution for dealing with the item. The third solution includes selecting a third notification modality for sending a message to the customer and sending the message. According to the invention, the third solution is as described with respect to notifying via a third notification modality, at <NUM> of <FIG>.

At <NUM>, if the customer is beyond the second distance from the vehicle, the process <NUM> proceeds to <NUM>. The second distance is as described with respect to the second distance of <FIG>. As such, the process <NUM> detects that the customer is beyond the second distance from the vehicle as described with respect to <NUM> of <FIG>. At <NUM>, the process <NUM> selects a fourth solution for dealing with the item. The fourth solution includes selecting a fourth notification modality for sending a message to the customer and sending the message. According to the invention, the fourth third solution is as described with respect to notifying via a fourth notification modality, at <NUM> of <FIG>.

Reference is now again made to <NUM> to provide example solutions that can be selected by the process <NUM> in a case that, at <NUM>, the item is determined to be trash. In an example, when the process <NUM> determines that the customer is not likely to return to retrieve the item, such as, for example, when the customer is beyond the second distance from the vehicle, a solution can be selected based on the trash level. In an example, the process <NUM> can simply cause the vehicle to be operated to reroute itself to a location where the item can be removed. In an example, a remote operator can, using the internal cameras of the vehicle, determine the level of trash in the vehicle and, accordingly, route or not route the vehicle to a servicing (e.g., cleaning) facility. Similarly, if the customer is beyond the second distance of the vehicle, at <NUM> of <FIG>, the process <NUM> can select a notification modality to notify a remote operate that an item is left in the vehicle. The remote operator may initiate a voice call to the customer. The remote operator may determine whether to route the vehicle to a servicing facility.

Returning to <FIG>, at the block <NUM>, the process <NUM> can select one or more solutions in response to a high-value item being left in the vehicle by a customer. In the block <NUM>, the process <NUM> selects different solutions depending on the distance of the customer from the vehicle. Determining the distance(s) of the customer from the vehicle is(are) illustrated by a decision <NUM> (and ellipsis <NUM>). The selected solution(s) is(are) illustrated by a solution <NUM> (and ellipsis <NUM>). According to the invention, the decision <NUM> is as described with respect to any of <NUM>, <NUM>, and <NUM> of <FIG> or any of <NUM>, <NUM>, <NUM>, and <NUM> of <FIG>. The solution <NUM> is as described with respect to any of <NUM>, <NUM>, <NUM>, and <NUM> of <FIG> or any of <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> of <FIG>. In an example, the decision <NUM> can determine that the customer is beyond a certain distance of the vehicle and the solution <NUM> can be that the vehicle is operated to follow behind the customer, stay stationary, or similar solution such that the vehicle remains proximal to the to customer. In another example, the vehicle can be caused to navigate to a servicing location so that the item can be retrieved and deposited in lost-and-found. The customer can then be sent a notification of the address of the lost-and-found location so that the customer can claim the item.

Examples use cases illustrating decisions, solutions and/or notification modalities, as described with respect to <FIG>, are now provided. In the scenarios below, a system, such as the controller <NUM> of the vehicle <NUM> of <FIG>, is executing a process that is consistent with the ones described with respect to <FIG>.

In a first illustrative use case, a customer is detected (e.g., at <NUM>) as leaving the car and door close is detected. A left-over item is detected and identified as sunglasses (e.g., at <NUM> or <NUM>). If the customer's hand is still on the door (e.g., at <NUM> or at <NUM>), the system sends (e.g., at <NUM> or <NUM>) haptic feedback and, optionally, external audio and/or external visual reminders (e.g., the first notification modality of <FIG> or the first solution of <FIG>) reminding the customer to confirm that sunglasses are left in the vehicle and to take them, if they belong to the customer. If the customer is within <NUM> meter of the vehicle (e.g., the first distance at <NUM> or the first distance at <NUM>), the system sends external audio with higher volume and optionally external visual reminders (e.g., the second notification modality of <FIG> or the second solution of <FIG>). If the customer is within <NUM> to <NUM> meters of the vehicle (e.g., second distance at <NUM> or the second distance at <NUM>), the system sends external audio with highest volume and optionally call/voice messages and/or text message reminders (e.g., the third notification modality of <FIG> or the third solution of <FIG>). If the customer is beyond <NUM> meters of the vehicle, the system sends call/voice message and/or text message reminders (e.g., the fourth notification modality of <FIG> or the fourth solution of <FIG>).

In a second illustrative use case, a customer is detected (e.g., at <NUM>) as leaving the car and door close is detected. A left-over item is detected and identified as coffee cup (e.g., at <NUM> or <NUM>). If the customer's hand is still on the door (e.g., at <NUM> or at <NUM>), the system sends (e.g., at <NUM> or at <NUM>) haptic feedback and, optionally, external audio and/or external visual reminders (e.g., the first notification modality of <FIG>) reminding the customer to confirm that coffee cup is left in the vehicle and to take it, if it belongs to the customer. If the customer is within <NUM> meter of the vehicle (e.g., the first distance at <NUM> or at <NUM>), the system sends an external audio with higher volume and optionally external visual reminders (e.g., the second notification modality of <FIG> or at <NUM>). If the customer is beyond <NUM> meter of the vehicle, the system sends a text message with a statement that the coffee cup will be cleaned up/trashed (e.g., the fourth notification modality of <FIG> or at <NUM>). Note that in this scenario, because the item is determined to be a trash item, the system does not check whether the customer is within a second distance (e.g., <NUM>-<NUM> meters) of the vehicle.

In a third illustrative use case, a customer is detected (e.g., at <NUM>) as leaving the car and door close is detected. A left-over item is detected and identified as coins (e.g., at <NUM> or <NUM>). As such, the item is identified at <NUM> of <FIG> as a non-high value item. If the customer's hand is still on the door (e.g., at <NUM> or at <NUM>), the system sends (e.g., at <NUM> or <NUM>) haptic feedback and, optionally, external audio and/or external visual reminders (e.g., the first notification modality of <FIG> or the first solution of <FIG>) reminding the customer to confirm that coins are left in the vehicle and to take them, if they belong to the customer. If the customer is within <NUM> meter of the vehicle (e.g., the first distance at <NUM> or the first distance at <NUM>), the system sends external audio with higher volume and optionally external visual reminders (e.g., the second notification modality of <FIG> or the second solution of <FIG>). If the customer is within <NUM> to <NUM> meters of the vehicle (e.g., second distance at <NUM> or the second distance at <NUM>), the system sends external audio with highest volume and optionally text message reminders (e.g., the third notification modality of <FIG> or the third solution of <FIG>). If the customer is beyond <NUM> meters of the vehicle, the system sends a text message reminder (e.g., the fourth notification modality of <FIG> or the fourth solution of <FIG>). In an example, the text message can include the options "I will retrieve" or "donate. " If the system does not receive a response to the message (for example, within a predefined period of time, which may be stated in the message), the system can assume that the customer chose "donate. " If the customer responds by selecting the "I will retrieve" option, the system can cause the vehicle to wait for the customer to return to retrieve the item.

<FIG> is an example of a process <NUM> for providing a service by an autonomous vehicle (AV) in accordance with implementations of this disclosure. The process <NUM> is implemented in or by an autonomous vehicle (AV), such as the vehicle <NUM> shown in <FIG> or one of the vehicles <NUM>/<NUM> shown in <FIG>. The AV may be a semi-autonomous vehicle, or any other vehicle implementing autonomous driving. The process <NUM> can be executed by a controller, such as the controller <NUM> of <FIG>. The process <NUM> is stored, as processor-executable instructions, in a memory, such as the memory <NUM> of <FIG>. The processor-executable instructions are executed by the processor <NUM> of <FIG>.

At <NUM>, the process <NUM> detects, based on sensor information, an object within the AV. The object (i.e., item) can be detected as described with respect to <NUM> of <FIG> or <NUM> of <FIG>. At <NUM>, the process <NUM> determines that the object belongs to a recent occupant of the AV. As described above, in an example, the process <NUM> can determine that the object belongs to the recent occupant (i.e., a customer) based on image differences between one or more images that are captured before the start of a service that is provided to the recent occupant using the AV and one or more images that are captured at the end of the service. In another example, and as described above, trip-based data can be used to determine that the object belongs to the recent occupant, as described above.

In response to determining that the object belongs to the recent occupant of the AV, the process <NUM>, at <NUM>, selects a notification modality for sending a message to the recent occupant regarding the object; and, at <NUM>, sends the message using the notification modality. The process <NUM> selects the notification modality based on a proximity of the recent occupant to the AV. The notification modality is selected consistent with the foregoing descriptions of <FIG> and the illustrative use cases.

In a case that the recent occupant is within the AV, the notification modality can include displaying, on a display of the AV, an image of the object. In a case that the recent occupant is not within the AV, the notification modality can include an external audible notification. A volume of the external audible notification can be based on a distance between the recent occupant and the AV. In a case that the recent occupant is not within the AV, the notification modality can include an external visible notification. In a case that the recent occupant is beyond a threshold distance of the AV, the notification modality can include sending an electronic notification to the recent occupant. The electronic notification can be or can include one or more of a text message, an email, other electronic notification, or a combination thereof.

In an example, the process <NUM> can include determining a value of the object and selecting the notification modality can be further based on the value of the object.

<FIG> is an example of a process <NUM> for providing a service by an autonomous vehicle (AV) in accordance with implementations of this disclosure, which do not fall under the scope of the claims but are useful to understahd the invention. The process <NUM> is implemented in or by an autonomous vehicle (AV), such as the vehicle <NUM> shown in <FIG> or one of the vehicles <NUM>/<NUM> shown in <FIG>. The AV may be a semi-autonomous vehicle, or any other vehicle implementing autonomous driving. The process <NUM> is executed by a controller, such as the controller <NUM> of <FIG>. The process <NUM> is stored, as processor-executable instructions, in a memory, such as the memory <NUM> of <FIG>. The processor-executable instructions are executed by the processor <NUM> of <FIG>.

At <NUM>, the process <NUM> detects, using sensor information, an object within the AV. The object (i.e., item) can be detected as described with respect to <NUM> of <FIG>. At <NUM>, the process <NUM> selects, based on a first distance of a recent occupant to the AV, a first notification modality for sending a first message to the recent occupant, the first message indicating that the object is left by the recent occupant in the AV. At <NUM>, the process <NUM> sends, using the first notification modality, the first message to the recent occupant. At <NUM>, the process <NUM> selects, based on a second distance of the recent occupant to the AV, a second notification modality for sending a second message to the recent occupant. At <NUM>, the process <NUM> sends, using the second notification modality, the second message to the recent occupant.

In an example, the second notification modality can include sending a text message to the recent occupant and the process <NUM> can further include receiving, from the recent occupant, a response to the text message. The first notification modality includes at least haptic feedback, and may include an audio signal, or a visual signal. In an example, the second notification modality can include at least one of an audio signal or a visual signal.

In an example, the process <NUM> can further include selecting the second notification modality and sending the second message in response to detecting that the object within the AV after sending, using the first notification modality, the message. In an example, the process <NUM> can further include selecting, based on a third distance of the recent occupant to the AV, a third notification modality for sending a third message to the recent occupant; and sending, using the third notification modality, the third message to the recent occupant.

As mentioned above, in addition to sending messages to a recent occupant regrading an item left in a vehicle, the teachings herein can be used to send the recent occupant messages regarding un-finished activities related to the service may need to be provided using the vehicle.

Accordingly, a process that can be executed by the vehicle, such as by a controller of the vehicle, can include detecting that a recent occupant of the AV has not completed an in-vehicle activity; and, in response to the detecting that the recent occupant of the AV has not completed an in-vehicle activity, selecting, based on a proximity of the recent occupant to the AV, a notification modality for sending a message to the recent occupant regarding the in-vehicle activity, and sending the message using the notification modality. The in-vehicle activity can include that at least one of a door of the vehicle or a trunk of the vehicle is open. The in-vehicle activity can include an unfinished electronic payment of the recent occupant.

As used herein, the terminology "computer" or "computing device" includes any unit, or combination of units, capable of performing any method, or any portion or portions thereof, disclosed herein.

As used herein, the terminology "processor" indicates one or more processors, such as one or more special purpose processors, one or more digital signal processors, one or more microprocessors, one or more controllers, one or more microcontrollers, one or more application processors, one or more Application Specific Integrated Circuits, one or more Application Specific Standard Products; one or more Field Programmable Gate Arrays, any other type or combination of integrated circuits, one or more state machines, or any combination thereof.

As used herein, the terminology "memory" indicates any computer-usable or computer-readable medium or device that may tangibly contain, store, communicate, or transport any signal or information that may be used by or in connection with any processor. For example, a memory may be one or more read only memories (ROM), one or more random access memories (RAM), one or more registers, low power double data rate (LPDDR) memories, one or more cache memories, one or more semiconductor memory devices, one or more magnetic media, one or more optical media, one or more magneto-optical media, or any combination thereof.

As used herein, the terminology "instructions" may include directions or expressions for performing any method, or any portion or portions thereof, disclosed herein, and may be realized in hardware, software, or any combination thereof. For example, instructions may be implemented as information, such as a computer program, stored on memory that may be executed by a processor to perform any of the respective methods, algorithms, aspects, or combinations thereof, as described herein. Instructions, or a portion thereof, may be implemented as a special purpose processor, or circuitry, that may include specialized hardware for carrying out any of the methods, algorithms, aspects, or combinations thereof, as described herein. In some implementations, portions of the instructions may be distributed across multiple processors on a single device, on multiple devices, which may communicate directly or across a network such as a local area network, a wide area network, the Internet, or a combination thereof.

As used herein, the terminology "example," "embodiment," "implementation," "aspect," "feature," or "element" indicates serving as an example, instance, or illustration. Unless expressly indicated, any example, embodiment, implementation, aspect, feature, or element is independent of each other example, embodiment, implementation, aspect, feature, or element and may be used in combination with any other example, embodiment, implementation, aspect, feature, or element, provided they fall under the scope of the claims.

As used herein, the terminology "determine" and "identify," or any variations thereof, includes selecting, ascertaining, computing, looking up, receiving, determining, establishing, obtaining, or otherwise identifying or determining in any manner whatsoever using one or more of the devices shown and described herein.

As used herein, the terminology "or" is intended to mean an inclusive "or" rather than an exclusive "or. " That is, unless specified otherwise, or clear from context, "X includes A or B" is intended to indicate any of the natural inclusive permutations. That is, if X includes A; X includes B; or X includes both A and B, then "X includes A or B" is satisfied under any of the foregoing instances. In addition, the articles "a" and "an" as used in this application and the appended claims should generally be construed to mean "one or more" unless specified otherwise or clear from context to be directed to a singular form.

Further, for simplicity of explanation, although the figures and descriptions herein may include sequences or series of steps or stages, elements of the processes disclosed herein may occur in various orders or concurrently. Additionally, elements of the processes disclosed herein may occur with other elements not explicitly presented and described herein. Furthermore, not all elements of the processes described herein may be required to implement a method in accordance with this disclosure. Although aspects, features, and elements are described herein in particular combinations, each aspect, feature, or element may be used independently or in various combinations with or without other aspects, features, and elements, provided they fall under the scope of the claims.

Claim 1:
An autonomous vehicle (<NUM>, <NUM>, <NUM>), in the following referred to as AV, comprising:
an internal sensor configured to detect an object within the AV; and
a sensor configured to determine a distance of a recent occupant to the AV; and
a processor (<NUM>) configured to execute instructions stored on a non-transitory computer readable medium to:
detect, based on sensor information, the object within the AV (<NUM>);
in response to determining that the recent occupant to the AV is outside the AV and is touching a door of the AV and that the object is within the AV, vibrating a part of the AV (<NUM>);
select, based on the recent occupant being outside of the AV and the recent occupant being not beyond a threshold distance from the AV, a first notification modality, including an external audio signal, for sending a message to the recent occupant, the message indicating that the object is left by the recent occupant in the AV;
send, using the first notification modality, the message to the recent occupant (<NUM>,<NUM>);
select, based on the recent occupant being outside of the AV and the recent occupant being beyond the threshold distance from the AV, a second notification modality, including a text message and/or a voice call, for sending the message to the recent occupant; and
send, using the second notification modality, the message to the recent occupant (<NUM>).