METHODS AND SYSTEMS FOR UNIDIRECTIONAL AND BIDIRECTIONAL COMMUNICATIONS

Methods and systems are provided for notifying a user. In one embodiment, a method includes: receiving perception data from a sensing device; determining a presence of an agent based on the perception data. In response to the determined presence, determining at least one of a type and a location of the agent based on the perception data; and selectively communicating directly to the agent based on at least one of the type and the location of the agent.

TECHNICAL FIELD

The technical field generally relates to communications between a robotic device and a human or other object, and more particularly to methods and systems for managing unidirectional and bidirectional communications between a robotic device and a human or other object.

BACKGROUND

Various driving scenarios require communication or confirmation between two individuals. For example, when a vehicle is approaching a cross walk and an individual is about to or is walking across the cross walk, the individual typically looks to the individual driving the vehicle for acknowledgement of their presence and confirmation that they intend to stop. In another example, when a vehicle is waiting for a right-of-way at a non-signalized intersection, the driver of one vehicle looks to the driver of another vehicle to wave them on. In each of these examples, humans communicate informally and navigate the vehicle based on the informal communication.

An autonomous vehicle is, for example, a driverless vehicle that is automatically controlled to carry passengers from one location to another. Autonomous vehicles do not have the benefit of the presence of a human to communicate to other humans outside of the vehicle. Other autonomous robotic devices are similarly unable to communicate. Accordingly, it is desirable to provide methods and systems to manage communications from a robotic device such as an autonomous vehicle. It is further desirable to provide methods and systems to manage unidirectional and bidirectional communications between a robotic device and a human or other object. Furthermore, other desirable features and characteristics of the present invention will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and the foregoing technical field and background.

SUMMARY

Methods and systems are provided for notifying a user. In one embodiment, a method includes: receiving perception data from a sensing device; determining a presence of an agent based on the perception data. In response to the determined presence, determining at least one of a type and a location of the agent based on the perception data; and selectively communicating directly to the agent based on at least one of the type and the location of the agent.

In one embodiment, a system includes a non-transitory computer readable medium. The non-transitory computer readable medium includes a first module that, by a processor, receives perception data from a sensing device, and that determines a presence of an agent based on the perception data. The non-transitory computer readable medium further includes a second module that, in response to the determined presence, determines, by a processor, at least one of a type and a location of the agent based on the perception data. The non-transitory computer readable medium further includes a third module that, by a processor, selectively communicates directly to the agent based on at least one of the type and the location of the agent.

DETAILED DESCRIPTION

The following detailed description is merely exemplary in nature and is not intended to limit the application and uses. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the preceding technical field, background, brief summary or the following detailed description. It should be understood that throughout the drawings, corresponding reference numerals indicate like or corresponding parts and features. As used herein, the term module refers to any hardware, software, firmware, electronic control component, processing logic, and/or processor device, individually or in any combination, including without limitation: application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and memory that executes one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality.

Embodiments may be described herein in terms of functional and/or logical block components and various processing steps. It should be appreciated that such block components may be realized by any number of hardware, software, and/or firmware components configured to perform the specified functions. For example, an embodiment may employ various integrated circuit components, e.g., memory elements, digital signal processing elements, logic elements, look-up tables, or the like, which may carry out a variety of functions under the control of one or more microprocessors or other control devices. In addition, those skilled in the art will appreciate that embodiments may be practiced in conjunction with any number of control systems, and that the system described herein is merely one example embodiment.

With reference now toFIG. 1, an exemplary communication system10is shown to be associated with a vehicle12. As can be appreciated, the vehicle12may be any vehicle type such as, but not limited to a road vehicle, an off-road vehicle, an aircraft, a watercraft, a train, etc. As can further be appreciated, the communication system10may be associated with devices other than a vehicle12, such as, but not limited to robotic devices, and is not limited to the present vehicle example. For exemplary purposes, the disclosure will be discussed in the context of the communication system10being associated with a vehicle12.

Although the figures shown herein depict an example with certain arrangements of elements, additional intervening elements, devices, features, or components may be present in actual embodiments. It should also be understood thatFIG. 1is merely illustrative and may not be drawn to scale.

In various embodiments, the vehicle12is an autonomous vehicle. The autonomous vehicle12is, for example, a driverless vehicle that is automatically controlled to carry passengers from one location to another. For example, components of the autonomous vehicle12may include: a sensor system13, an actuator system14, a data storage device16, and at least one control module18. The sensor system13includes one or more sensing devices13a-13nthat sense observable conditions of the exterior environment and/or the interior environment of the vehicle12. The sensing devices13a-13ncan include, but are not limited to, radars, lidars, and cameras. The actuator system14includes one or more actuator devices14a-14nthat control one or more vehicle components (not shown). In various embodiments, the vehicle components are associated with vehicle operation and can include, but are not limited to, a throttle, brakes, and a steering system. In various embodiments, the vehicle components are associated with interior and/or exterior vehicle features and can include, but are not limited to, doors, a trunk, and cabin features such as air, music, lighting, etc.

The data storage device16stores data for use in automatically controlling the vehicle12. In various embodiments, the data storage device16stores defined maps of the navigable environment. In various embodiments, the defined maps may be predefined by and obtained from a remote system20. For example, the defined maps may be assembled by the remote system20and communicated to the vehicle12(wirelessly and/or in a wired manner) and stored by the control module18in the data storage device16. As can be appreciated, the data storage device16may be part of the control module18, separate from the control module18, or part of the control module18and part of a separate system.

The control module18includes at least one processor22and memory24. The processor22can be any custom made or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the control module18, a semiconductor based microprocessor (in the form of a microchip or chip set), a macroprocessor, or generally any device for executing instructions. The memory24may be one or a combination of storage elements that store data and/or instructions that can be performed by the processor22. The instructions may include one or more separate programs, each of which comprises an ordered listing of executable instructions for implementing logical functions.

The instructions, when executed by the processor22, receive and process signals from the sensor system13, perform logic, calculations, methods and/or algorithms for automatically controlling the components of the vehicle12, and generate control signals to the actuator system14to automatically control the components of the vehicle12based on the logic, calculations, methods, and/or algorithms. Although only one control module18is shown inFIG. 1, embodiments of the vehicle12can include any number of control modules18that communicate over any suitable communication medium or a combination of communication mediums and that cooperate to process the sensor signals, perform logic, calculations, methods, and/or algorithms, and generate control signals to automatically control features of the vehicle12.

In various embodiments, the communication system10generally includes one or more instructions that are embodied within the control module18(as shown as the communication instructions100. These instruction100, when executed by the processor22, generally detect the presence of an individual or object outside of the vehicle12, and manage unidirectional and bidirectional communications between the detected individual or object outside of the vehicle12. In various embodiments, the detected individual can be a pedestrian, a biker, a traffic conductor such as a policeman or construction worker, or other human in proximity to the vehicle12. In various other embodiments, the detected object can be another autonomous vehicle, an emergency vehicle, infrastructure, or other object in proximity to the vehicle12. For ease of the discussion, the disclosure will commonly refer to an individual and an object as an agent.

The communication system10detects the presence of the agent by way of at least one perception detection device26. In various embodiments, the perception detection device26can include at least one sensing device such as, but not limited to, a camera, a radar, a lidar, or other sensing device that is disposed at one or more locations around the vehicle12. As can be appreciated, the perception detection device26can be one or more of the sensing devices13a-13nof the sensor system13discussed above for controlling the autonomy of the vehicle12and/or can be another sensing device dedicated to the communication system10. The sensing device senses the environment around the outside of the vehicle12and generates sensor signals based thereon.

In various embodiments, the instructions100of the control module18receive the sensor signals from the perception detection device26and processes the sensor signals to detect whether an agent is in proximity to the vehicle12, and generates data indicating the presence of an agent in proximity to the vehicle12. For example, the instructions, when executed by the processor22, detect an agent in the scene captured by the sensing device, determine a location of the agent (e.g., a location relative to the vehicle12, or other coordinate system), determine a type of the agent (e.g., pedestrian, driver, biker, traffic conductor, infrastructure, emergency vehicle, other autonomous vehicle, personal device, etc.), and/or determines a gesture made by the agent (e.g., a head nod, a wave of a hand, stopping movement of the legs, etc.) and generates the data indicating the presence of the agent based on the location, type and/or the gesture.

In various embodiments, the instructions of the control module18process the data indicating the presence of the agent to determine whether the agent requires a communication, and if the agent requires a communication, what type of communication to communicate to the agent, where to make the communication such that it is directed to the agent, and for how long to communicate to the agent. In various embodiments, the instructions of the control module18process the data indicating the presence of the agent to determine whether the agent has confirmed receipt of the communication, for example, by way of a gesture (e.g., a head nod, a wave of the hand, stopping movement of the legs, etc.).

The communication system10communicates with the agent by way of a signaling system28. The signaling system includes a plurality of signaling devices28a-28ndisposed at locations around the vehicle12. A signaling device28ais selected from the plurality of signaling devices28a-28nfor the communication based on the signaling device's location on the vehicle12and the agent's location relative to the vehicle12. For example, a signaling device28alocated on the vehicle12in the direct line of site of the agent can be selected to make the communication to the agent.

In various embodiments, the signaling devices28a-28ncan include one or more visual devices, aural devices, and/or haptic devices. For example, the visual devices communicate an acknowledgement of the detection of the agent and/or gesture by, for example, displaying a particular light, a color of a light, a message, a predefined image, and/or a captured image of the agent. In another example, the aural devices communicate acknowledgment of the detection of the agent and/or gesture by, for example, playing a particular sound or a phrase. In still another example, the haptic devices communicate an acknowledgment of the detection of the agent or gesture by, activating a vibration.

Referring now toFIG. 2and with continued reference toFIG. 1, a dataflow diagram illustrates sub-modules of the control module18in more detail in accordance with various exemplary embodiments. As can be appreciated, various exemplary embodiments of the control module18, according to the present disclosure, may include any number of modules and/or sub-modules. In various exemplary embodiments, the modules and sub-modules shown inFIG. 2may be combined and/or further partitioned to similarly manage communications to and from an agent. In various embodiments, the control module18receives inputs from the perception detection device26, from one or more of the sensors13a-13nof the vehicle12, from other modules (not shown) within the vehicle12, and/or from other modules within the control module18. In various embodiments, the control module18includes a presence detection module30, a signaling device selection module32, and a communication module34.

The presence detection module30receives as input perception data36from the perception detection device26. The presence detection module30processes the perception data36to determine whether an agent is in proximity to the vehicle12. For example, a scene is constructed from the perception data36and elements within the scene are identified and classified into a type38using identification and classification techniques generally known in the art. If an element of the scene is classified as a type that is an agent (e.g., an individual or object), a location40of the element relative to the vehicle12is determined from the perception data36. For example, the element can be determined to be located at a left front of the vehicle12, a left back of the vehicle12, a right front of the vehicle12, a right back of the vehicle12, a center front of the vehicle12, a center back of the vehicle12, a left side of the vehicle12, a right side of the vehicle1, etc. If an element of the scene is classified as an agent, then a gesture41of the agent is determined. For example, a position or posture of the agent is compared to a previous position or posture to determine the gesture41.

The signaling device selection module32receives as input the type38of the agent, the location40of the agent, and vehicle data42. The vehicle data42indicates a current operational status of the vehicle12such as, but not limited to, a braking status, steering status, a vehicle speed, etc. The signaling device selection module32determines if a communication should be made to the agent based on the type38of the agent, the location40of the agent, and the vehicle data42. If it is determined that a communication should be made, the signaling device selection module32determines what type of communication should be made.

For example, the signaling device selection module32includes a plurality of scenarios. Each scenario is associated with one or more locations of agents and/or one or more types of agents. Each scenario includes one or more conditions of the vehicle12and associated communication types. The signaling device selection module32selects a scenario based on the type38of the agent and the location40of the agent, and evaluates the vehicle data42based on the selected scenario. If the vehicle data42indicates that conditions of the vehicle12under the scenario are met, then an associated communication type44is selected.

The communication module34receives as input the communication type44, the location40of the agent, and the gesture41of the agent. The communication module34selects a signaling device based on the communication type44and the location40of the agent. For example, the communication module selects a signaling device located on the vehicle relative to a line of sight of the location of the agent. In another example, the communication module selects a signaling device28afrom the plurality of signaling devices28a-28nthat is best suited for the communication type44. The communication module34generates communication signals46to communicate directly to the agent based on the selected signaling device28a.In various embodiments, the communication module34ends the communication of the communication signals46when the agent is no longer present and/or when the gesture41of the agent indicates that the agent has confirmed the communication.

With reference now toFIG. 3, and with continued reference toFIGS. 1 and 2, a flowchart illustrates a method200for managing unidirectional and bidirectional communications between a vehicle and an agent. The method200can be implemented in connection with the vehicle ofFIG. 1and can be performed by the control module18ofFIG. 2in accordance with various exemplary embodiments. As can be appreciated in light of the disclosure, the order of operation within the method200is not limited to the sequential execution as illustrated inFIG. 3, but may be performed in one or more varying orders as applicable and in accordance with the present disclosure. As can further be appreciated, the method200ofFIG. 3may be enabled to run continuously, may be scheduled to run at predetermined time intervals during operation of the control module18, and/or may be scheduled to run based on predetermined events.

In various embodiments, the method200may begin at205. The perception data36is received from the perception detection device26at210and processed. It is determined whether an agent is present at220. If an agent is not present, and a communication has not been previously sent to an agent at230, the method may end at240. If an agent is not present, and a communication was previously sent to the agent at230, the communication is ended at250, and the method may end at240.

If, at220, an agent is determined to be present, the perception data36is further processed to determine the location40and the type38of the agent at260. Vehicle data42is received at270. A scenario is selected based on the location40and/or the type38of the agent at280. The vehicle data42is evaluated based on the selected scenario to select a signaling device28ato make the communication, and to select the type of communication at290. Communication signals46are then generated to the selected signaling device28abased on the type of communication at300. The signaling device28areceives the communication signals46and communicates directly to the agent visually, aurally, and/or haptically at310.

Optionally, a confirmation of the communication between the agent and the vehicle12can be made at320-340. For example, additional perception data36is received at320and processed. It is determined whether the agent made a confirmation gesture at330. If it is determined that the agent made a confirmation gesture at330, the communication is ended at250and the method may end at240. If it is determined that the agent did not make a confirmation gesture at330, and it is desirable to communicate to the agent again at340, communication signals46are then generated to the selected signaling device28abased on the type of communication at300. The signaling device receives the communication signals and communicates to the agent visually, aurally, and/or haptically at310.

As can be appreciated, the perception data36can be evaluated for a confirmation gesture any number of times before proceeding to step250and ending the communication when the agent is no longer present.