Visually indicating vehicle caution regions

An example system includes a vehicle, a light projector connected to the vehicle, and a control system. The control system is configured to determine a planned operating region for the vehicle within an environment. The control system is also configured to determine that the planned operating region is within a threshold distance of an object within the environment and, in response, determine a caution region to illuminate with the light projector near the object. The control system is further configured to cause the light projector to project an indication of the caution region near the object. The projected indication remains fixed in relation to the object as the vehicle moves toward the planned operating region.

BACKGROUND

A warehouse may be used for storage of goods by a variety of different types of commercial entities, including manufacturers, wholesalers, and transport businesses. Example stored goods may include raw materials, parts or components, packing materials, and finished products. In some cases, the warehouse may be equipped with loading docks to allow goods to be loaded onto and unloaded from delivery trucks or other types of vehicles. The warehouse may also use rows of pallet racks to allow for storage of pallets, which are flat transport structures that contain stacks of boxes or other objects thereon. Additionally, the warehouse may use machines or vehicles for lifting and moving goods or pallets of goods, such as cranes, forklifts, and pallet jacks. Human operators may be employed to operate machines, vehicles, and other equipment. In some cases, one or more of the machines or vehicles may be robotic devices guided by computer control systems.

SUMMARY

In an example embodiment, visual indications may be projected onto an environment in which a robotic device is operating alongside human workers to inform the human workers of potentially dangerous areas and locations. The potentially dangerous areas, or caution regions, may arise when the robotic device moves close to an object in the environment, such as a wall, pallet rack, or pallet, and thus creates a pinch zone between the vehicle and the object. Expected caution regions may be determined before they arise by analyzing the proximity between objects in the environment and areas to which the robotic device is planned to move. A projector on the vehicle may be used to illuminate surfaces of the environment in and around the caution regions to visually indicate areas that human workers should avoid. The visual indications may remain fixed with respect to the object and as the robotic device moves around, thus consistently highlighting the caution region regardless of current vehicle position. The visual indications may be used to replace physically painted or taped lines in a warehouse environment.

In a first embodiment, a system is provided that includes a vehicle, a light projector connected to the vehicle, and a control system. The control system is configured to determine a planned operating region for the vehicle within an environment. The control system is also configured to determine that the planned operating region is within a threshold distance of an object within the environment and, in response, determine a caution region to illuminate with the light projector near the object. The control system is further configured to cause the light projector to project an indication of the caution region near the object. The projected indication remains fixed in relation to the object as the vehicle moves toward the planned operating region.

In a second embodiment, a method is provided that includes determining a planned operating region for a vehicle within an environment. The method also includes determining that the planned operating region is within a threshold distance of an object within the environment and, in response, determining a caution region to illuminate with a light projector near the object, where the light projector is connected to the vehicle. The method further includes causing the light projector to project an indication of the caution region near the object, where the projected indication remains fixed in relation to the object as the vehicle moves toward the planned operating region.

In a third embodiment, a non-transitory computer readable storage medium is provided having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations. The operations include determining a planned operating region for a vehicle within an environment. The operations also include determining that the planned operating region is within a threshold distance of an object within the environment and, in response, determining a caution region to illuminate with a light projector near the object, where the light projector is connected to the vehicle. The operations further include causing the light projector to project an indication of the caution region near the object, where the projected indication remains fixed in relation to the object as the vehicle moves toward the planned operating region

In a fourth embodiment, a system is provided that includes means for determining a planned operating region for a vehicle within an environment. The system also includes means for determining that the planned operating region is within a threshold distance of an object within the environment. The system additionally includes means for, in response to determining that the planned operating region is within the threshold distance of the object, determining a caution region to illuminate with a light projector near the object, where the light projector is connected to the vehicle. The system yet further includes means for causing the light projector to project an indication of the caution region near the object, where the projected indication remains fixed in relation to the object as the vehicle moves toward the planned operating region.

DETAILED DESCRIPTION

Throughout this description, the articles “a” or “an” are used to introduce elements of the example embodiments. Any reference to “a” or “an” refers to “at least one,” and any reference to “the” refers to “the at least one,” unless otherwise specified, or unless the context clearly dictates otherwise. The intent of using the conjunction “or” within a described list of at least two terms is to indicate any of the listed terms or any combination of the listed terms.

The use of ordinal numbers such as “first,” “second,” “third” and so on is to distinguish respective elements rather than to denote a particular order of those elements. For purpose of this description, the terms “multiple” and “a plurality of” refer to “two or more” or “more than one.”

Further, unless context suggests otherwise, the features illustrated in each of the figures may be used in combination with one another. Thus, the figures should be generally viewed as component aspects of one or more overall embodiments, with the understanding that not all illustrated features are necessary for each embodiment. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. Further, unless otherwise noted, figures are not drawn to scale and are used for illustrative purposes only. Moreover, the figures are representational only and not all components are shown. For example, additional structural or restraining components might not be shown.

Autonomous robotic devices may be used to carry out tasks and actions in an environment such as a warehouse without the need for constant human control. In some instances, the robotic devices may operate alongside human workers. Such a shared environment may arise, for example, when a warehouse utilizing human-operated vehicles transitions to using autonomously operated vehicles to handle its inventory. Measures may be taken to ensure that the human workers and the robotic devices safely cooperate with one another in the shared environment.

One such safety measure may involve classifying areas of the environment into safe and unsafe regions and projecting onto these regions corresponding visual indications to inform the human occupants of where they should position themselves to avoid the unsafe areas. The classification of the areas may be dynamic, that is, it may change over time based on, for example, a task that a vehicle is performing or is planned to perform, or an area that the vehicle is occupying or is planned to occupy, among other factors. This is in contrast to the conventional practice of painting or taping fixed lines on the floor of the warehouse to delineate the different areas therein, which does not allow for the shapes and sizes of the areas to quickly change depending on what is going on within the environment at the time.

The classifications may include areas within object safety thresholds, planned operating regions, caution regions, and safety regions, among other possibilities. An object safety threshold may be a distance away from a perimeter of an object within the environment that delineates a buffer region around the object. The object safety threshold may be defined by a warehouse safety standard and would ordinarily be indicated with fixed lines painted or taped on the floor around the object.

A planned operating region may include an area within the environment planned to be occupied by the vehicle within a future period of time (e.g., in the next 30 seconds). The planned operating region may be determined based on a plurality of vehicle footprints representing areas within the environment planned to be occupied by the vehicle as it moves along a path. The path may be determined for the vehicle by a control system on the vehicle (i.e., a same control system could both plan the path and control projection of the visual indications), or a centralized control system remote from the vehicle and configured to synchronize operations of multiple vehicles within the environment.

A caution region may arise when the planned operating region intersects with an area within the object safety threshold. That is, the caution region may indicate areas within the environment where the vehicle is planned to come within the object safety threshold of an object, thus creating a low-escape or pinch zone. The caution region may be larger than the intersection between the planned operating region and the area within the object safety threshold, extending, for example, from the intersection to the perimeter of the object. Determining caution regions may allow vehicles to more safely operate in tight areas and maneuver closely to objects because the visual indications ensure that such areas are likely to be free of occupants.

For example, in a warehouse setting, caution regions may arise in deep aisles of the warehouse, as well as when a vehicle is loading or unloading objects from the bottom level of racks. In deep aisles and bottom rack levels, escape routes might not be available or might be difficult to identify for human occupants of the warehouse. The approach herein described may provide a way to reduce the risk of a human occupant inadvertently going to an area without an escape route by illuminating any caution regions before the vehicle moves into or near the caution regions. Thus, the approach herein described may allow vehicles to meet levels of safety regulations or safety standards that the vehicle might not otherwise meet.

Safety regions may include any areas of the environment that are not planned to be occupied by vehicles within the future period of time and are not within the object safety threshold of an object. That is, safety regions may be occupied by human workers without significant risk of a vehicle coming near them (e.g., within 50 centimeters). Whether a region is determined as safe may be based on any safety standard for the application in which the vehicle is used or on values or parameters defined by a user or operator of the vehicle. The caution regions and safety regions may change over time as the vehicle performs different tasks in different regions of the environment.

The vehicle may be equipped with projectors to project, onto the environment, visual indications of the classified regions. Projectors may be mounted on the top, front, back, sides of, and/or underneath the vehicle, and may be moveable or fixed with respect to the vehicle. The visual indications, when projected onto the environment, may delineate or span an entirety of the classified areas. That is, a visual indication delineating a caution region may illuminate an area within the environment that is at least as large as and includes therein the caution region. Alternatively, visual indication delineating the caution region may illuminate a border within the environment that encircles the caution region. However, the area within the environment illuminated by the projector may also be smaller than the caution region and might not overlap with the caution region. Rather, the projector may illuminate an area in the direction and general vicinity of the caution region.

As the vehicle moves through the environment, positions and orientations of the projectors may be adjusted to maintain the different indicated regions in a fixed position in relation to objects within the environment. That is, although the vehicle may move through the environment, the control system may operate to keep the projected visual indications stationary or fixed in the environment. For example, the control system may control the projector to project a visual indication of a caution region around a stationary pallet onto a fixed portion of a floor of the warehouse around the pallet regardless of movements of the vehicle. In some instances, however, the actual position of the projected visual indications may vary from the planned or expected position of the visual indications due to, for example, delay in physically reorienting the projector, limitations in capabilities of the projector, and/or occlusions along the path of projected light. The control system may operate to minimize an error or difference between the planned position of the visual indications and the actual position thereof.

Further, as the vehicle moves and the relative position and orientation between the projectors and the environment changes, the control system may determine keystone corrections to apply to the visual indications such that their appearance on surfaces in the environment does not vary as the vehicle moves. Additionally, the projections may be updated as the different regions and their positions in the environment change over time. For example, a caution region may be illuminated before it is occupied by the vehicle, but may cease to be illuminated after the vehicle clears the caution region.

Successful projection of the visual indications onto the environment may be verified by capturing and analyzing images of the environment. The images and/or other sensor data may be used to determine whether any human workers currently occupy or are expected to occupy any of the caution regions. Additional warnings may be provided to compel human workers to exit or avoid any caution regions. For example, an escape route for the human worker may be determined and projected onto the environment, allowing the human worker to easily avoid the vehicle and the caution region. The vehicle may be configured to stop and wait for any human workers to move to a safe location before entering any caution regions.

II. EXAMPLE ROBOTIC SYSTEMS

Referring now to the Figures,FIG. 1illustrates an example configuration of a robotic system that may be used in connection with the embodiments described herein. Robotic system100may be configured to operate autonomously, semi-autonomously, or using directions provided by user(s). Robotic system100may be a vehicle implemented in various forms, such as forklifts, pallet jacks, cars, trucks, motorcycles, buses, boats, airplanes, helicopters, lawn mowers, earth movers, boats, snowmobiles, aircraft, recreational vehicles, amusement park vehicles, farm equipment, construction equipment, warehouse equipment, trams, golf carts, trains, and trolleys. Other forms are possible as well. Furthermore, robotic system100may also be referred to as a robot, robotic device, mobile robot, or robotic vehicle, among other designations.

As shown inFIG. 1, robotic system100may include processor(s)102, data storage104, and controller(s)108, which together may be part of a control system118. Robotic system100may also include sensor(s)112, power source(s)114, mechanical components110, electrical components116, communication link(s)120, and light projector(s)122. Nonetheless, robotic system100is shown for illustrative purposes, and may include more or fewer components. The various components of robotic system100may be connected in any manner, including wired or wireless connections. Further, in some examples, components of robotic system100may be distributed among multiple physical entities rather than a single physical entity. Other example illustrations of robotic system100may exist as well.

Processor(s)102may operate as one or more general-purpose hardware processors or special purpose hardware processors (e.g., digital signal processors, application specific integrated circuits, etc.). Processor(s)102may be configured to execute computer-readable program instructions106, and manipulate data107, both of which are stored in data storage104. Processor(s)102may also directly or indirectly interact with other components of robotic system100, such as sensor(s)112, power source(s)114, mechanical components110, electrical components116, communication link(s)120, or light projector(s)122.

Data storage104may be one or more types of hardware memory. For example, data storage104may include or take the form of one or more computer-readable storage media that can be read or accessed by processor(s)102. The one or more computer-readable storage media can include volatile and/or non-volatile storage components, such as optical, magnetic, organic, or another type of memory or storage, which can be integrated in whole or in part with processor(s)102. In some embodiments, data storage104can be a single physical device. In other embodiments, data storage104can be implemented using two or more physical devices, which may communicate with one another via wired or wireless communication. As noted previously, data storage104may include computer-readable program instructions106and data107. Data107may be any type of data, such as configuration data, sensor data, and/or diagnostic data, among other possibilities.

Controller108may include one or more electrical circuits, units of digital logic, computer chips, and/or microprocessors that are configured to (perhaps among other tasks) interface between any combination of mechanical components110, sensor(s)112, power source(s)114, electrical components116, control system118, communication link(s)120, light projector(s)122, users of robotic system100. In some embodiments, controller108may be a purpose-built embedded device for performing specific operations with one or more subsystems of robotic system100.

Control system118may monitor and physically change the operating conditions of robotic system100. In doing so, control system118may serve as a link between portions of robotic system100, such as between mechanical components110or electrical components116. In some instances, control system118may serve as an interface between robotic system100and another computing device. Further, control system118may serve as an interface between robotic system100and a user. For instance, control system118may include various components for communicating with robotic system100, including a joystick, buttons, and/or ports, etc. The example interfaces and communications noted above may be implemented via a wired or wireless connection, or both. Control system118may perform other operations for robotic system100as well.

In some implementations, control system118of robotic system100may also include communication link(s)120configured to send and receive information. Communication link(s)120may transmit data indicating the state of the various components of robotic system100. For example, information read by sensor(s)112may be transmitted via communication link(s)120to a separate device. Other diagnostic information indicating the integrity or health of power source(s)114, mechanical components110, electrical components116, processor(s)102, data storage104, light projector(s)122, or controller108may be transmitted via communication link(s)120to an external communication device.

In some implementations, robotic system100may receive information at communication link(s)120that is then processed by processor(s)102. The received information may indicate data that is accessible by processor(s)102during execution of program instructions106. Further, the received information may change aspects of controller(s)108that may affect the behavior of mechanical components110or electrical components116. In some cases, the received information may indicate a query requesting a piece of information (e.g. the operational state of one or more of the components of robotic system100). Processor(s)102may subsequently transmit the piece of information back out via communication link(s)120.

In some cases, communication link(s)120may include a wired connection. Robotic system100may include one or more ports to interface communication link(s)120to an external device. Communication link(s)120may include, in addition to or alternatively to the wired connection, a wireless connection. Some example wireless connections may utilize a cellular connection, such as CDMA, EVDO, GSM/GPRS, or 4G telecommunication, such as WiMAX or LTE. Alternatively or in addition, the wireless connection may utilize a Wi-Fi connection to transmit data to a wireless local area network (WLAN). In some implementations, the wireless connection may also communicate over an infrared link, Bluetooth, or a near-field communication (NFC) device.

During operation, control system118may communicate with other systems of robotic system100via wired or wireless connections, and may further be configured to communicate with one or more users or operators of the vehicle. As one possible illustration, control system118may receive an input (e.g., from a user or from another vehicle) indicating an instruction to move a pallet from a first location of a warehouse to a second location of the warehouse. The input to control system118may be received via communication link(s)120.

Based on this input, control system118may perform operations to cause robotic system100to use sensors112to analyze the environment of the warehouse to locate the pallet and subsequently use mechanical components110to pick up and move the pallet.

Operations of control system118may be carried out by processor(s)102. Alternatively, these operations may be carried out by controller108, or a combination of processor(s)102and controller108. In some embodiments, control system118may partially or wholly reside on a device other than robotic system100, and therefore may at least in part control robotic system100remotely. Communication link(s)120may be used at least in part to carry out the remote communication.

Mechanical components110represent hardware of robotic system100that may enable robotic system100to perform physical operations. As a few examples, robotic system100may include physical members such robotic arm(s), wheel(s), track(s), linkage(s), and/or end effector(s). The physical members or other parts of robotic system100may further include motors and actuators arranged to move the physical members in relation to one another. Robotic system100may also include one or more structured bodies for housing control system118or other components, and may further include other types of mechanical components. Mechanical components110used in a given robot may vary based on the design of the robot, and may also be based on the operations and/or tasks the robot may be configured to perform.

In some examples, mechanical components110may include one or more removable components. Robotic system100may be configured to add and/or remove such removable components, which may involve assistance from a user and/or another robot. For example, robotic system100may be configured with removable arms, linkages, and/or end effectors so that these members can be replaced or changed as needed or desired based on a task robotic system100is expected or planned to perform. In some embodiments, robotic system100may include one or more removable and/or replaceable battery units or sensors. Other types of removable components may be included within some embodiments.

Robotic system100may include sensor(s)112arranged to sense aspects of robotic system100. Sensor(s)112may include one or more force sensors, torque sensors, velocity sensors, acceleration sensors, position sensors, proximity sensors, motion sensors, location sensors, load sensors, temperature sensors, touch sensors, depth sensors, ultrasonic range sensors, infrared sensors, object sensors, and/or cameras (e.g., a depth camera and/or a stereo camera), among other possibilities. Within some examples, robotic system100may be configured to receive sensor data from sensors that are physically separated from the robot (e.g., sensors that are positioned on other robots or located within the environment in which the robot is operating).

Sensor(s)112may provide sensor data to processor(s)102(perhaps by way of data107) to allow for interaction of robotic system100with its environment, as well as monitoring of the operation of robotic system100. The sensor data may be used in evaluation of various factors for activation, movement, and deactivation of mechanical components110and electrical components116by control system118. For example, sensor(s)112may capture data corresponding to the terrain of the environment, location and/or identity of nearby objects (e.g., pallets, environmental landmarks), which may assist with environment recognition and navigation. In an example configuration, sensor(s)112may include RADAR (e.g., for long-range object detection, distance determination, and/or speed determination), LIDAR (e.g., for short-range object detection, distance determination, and/or speed determination), SONAR (e.g., for underwater object detection, distance determination, and/or speed determination), VICON® (e.g., for motion capture), one or more cameras (e.g., stereoscopic cameras for three-dimensional (3D) vision), a global positioning system (GPS) transceiver, and/or other sensors for capturing information of the environment in which robotic system100is operating. Sensor(s)112may monitor the environment in real time, and detect obstacles, elements of the terrain, weather conditions, temperature, and/or other aspects of the environment.

Further, robotic system100may include sensor(s)112configured to receive information indicative of the state of robotic system100, including sensor(s)112that may monitor the state of the various components of robotic system100. Sensor(s)112may measure activity of systems of robotic system100and receive information based on the operation of the various features of robotic system100, such the operation of wheels, linkages, actuators, end effectors, and/or other mechanical and/or electrical features of robotic system100. The data provided by sensor(s)112may enable control system118to determine errors in operation as well as monitor overall operation of components of robotic system100.

As an example, robotic system100may use a time-of-flight (ToF) camera to scan portions of the environment to detect obstacles along a planned path of the vehicle, identify environmental landmarks within an environment of robotic system100, and locate objects of interest, such as pallets or boxes. The ToF camera may have a limited field of view. Mechanical components110and electrical components116may work in coordination to move the ToF camera along a trajectory to direct a field of view of the ToF camera at different portions of the environment.

As another example, sensor(s)112may include one or more velocity and/or acceleration sensors. Sensor(s)112may measure both linear and angular velocity and/or acceleration. For instance, sensor(s)112may include an inertial measurement unit (IMU) having a 3-axis accelerometer, a 3-axis gyroscope, and a 3-axis magnetometer. The IMU may sense velocity and acceleration in the world frame, with respect to the gravity vector. The velocity and acceleration sensed by the IMU may then be translated to that of robotic system100based on the location of the IMU in robotic system100and the kinematics of robotic system100.

Robotic system100may include other types of sensors not explicated discussed herein. Additionally or alternatively, robotic system100may use particular sensors for purposes not enumerated herein.

Robotic system100may also include one or more power source(s)114configured to supply power to various components of robotic system100. Among other possible power systems, robotic system100may include a hydraulic system, electrical system, batteries, and/or other types of power systems. As an example illustration, robotic system100may include one or more batteries configured to provide charge to components of robotic system100. Some of mechanical components110or electrical components116may each connect to a different power source, may be powered by the same power source, or be powered by multiple power sources.

Any type of power source may be used to power robotic system100, such as electrical power or a gasoline engine. Additionally or alternatively, robotic system100may include a hydraulic system configured to provide power to mechanical components110using fluid power. Components of robotic system100may operate based on hydraulic fluid being transmitted throughout the hydraulic system to various hydraulic motors and hydraulic cylinders, for example. In one example, the hydraulic fluid may be used to actuate the forks of a forklift, fork truck, and/or pallet jack. The hydraulic system may transfer hydraulic power by way of pressurized hydraulic fluid through tubes, flexible hoses, or other links between components of robotic system100. Power source(s)114may charge using various types of charging, such as wired connections to an outside power source, wireless charging, combustion, or other examples.

Electrical components116may include various mechanisms capable of processing, transferring, and/or providing electrical charge or electric signals. Among possible examples, electrical components116may include electrical wires, circuitry, and/or wireless communication transmitters and receivers to enable operations of robotic system100. Electrical components116may interwork with mechanical components110to enable robotic system100to perform various operations. Electrical components116may be configured to provide power from power source(s)114to mechanical components110, for example. Further, robotic system100may include electric motors. Other examples of electrical components116may exist as well.

Although not shown inFIG. 1, robotic system100may include a chassis and/or an operator cabin, which may connect to or house components of robotic system100. The structure of the chassis and/or cabin may vary within examples and may further depend on operations that a given vehicle may have been designed to perform. For example, a vehicle developed to carry large, heavy loads may have a wide, rigid chassis that enables placement of the load. Similarly, a vehicle designed to carry light loads at high speeds may have a narrow, small chassis that does not have substantial weight. Further, the chassis, cabin, and/or the other components may be developed using various types of materials, such as metals or plastics. Within other examples, a vehicle may have a chassis with a different structure or made of various types of materials.

The chassis, cabin, and/or the other components may include or carry sensor(s)112. These sensors may be positioned in various locations on robotic system100, such as on top of the chassis to provide a high vantage point for sensor(s)112.

Robotic system100may carry a load, such as a type of cargo that is to be transported. The load may also represent external batteries or other types of power sources (e.g., solar panels) that robotic system100may utilize. Carrying the load represents one example use for which robotic system100may be configured, but robotic system100may be configured to perform other operations as well.

Robotic system100may further include light projectors(s)122arranged to project light onto portions of the environment around robotic system100. Light projector(s)122may include photo or video projectors (e.g., Digital Light Processing (DLP) projectors, Liquid Crystal Display (LCD) projectors, Liquid Crystal on Silicon (LCoS) projectors, Light-Emitting Diode (LED) projectors, or laser projectors), a holographic projector (accomplished by varying uses of lasers (e.g., plasma lasers), interference, diffraction, light intensity, and/or other parameters known or not yet known in the field of holography), parabolic aluminized reflector (PAR) lights, strip lights, spotlights, LCD's, LED's, or combinations thereof. Light projector(s)122may be connected to the chassis or cabin of robotic system100, and may be configured to move with respect thereto in one or more degrees of freedom. For example, light projector(s)122may be configured to translate with respect to the chassis in one or more degrees of freedom (e.g., along the x-axis, y-axis, and/or z-axis) and to rotate with respect to the chassis in at least one degree of freedom (e.g., yaw, pitch, and/or roll). Movement of light projector(s)122with respect to the vehicle may allow different portions of the environment around the vehicle to be selectively illuminated by light projector(s)122.

FIG. 2is a block diagram of system200, in accordance with example embodiments. System200includes planning system210and robotic device220. Planning system210can include offboard planner212that can coordinate motion of one or more robotic devices operating in an environment. Offboard planner212can include roadmap planner214. Offboard planner212and/or roadmap planner214can generate one or more asynchronous paths216for a robotic device, such as robotic device220, to follow while operating in an environment.

A roadmap graph, prototype graph, or other roadmap representing an environment can be received, determined, or otherwise provided to planning system210, offboard planner212and/or roadmap planner214. Asynchronous paths216can be one or more paths based on the roadmap graph, prototype graph, or other roadmap. For example, if the roadmap graph, prototype graph, or other roadmap has a plurality of edges that connect a plurality of intersections, asynchronous paths216can be specified in terms of the plurality of edges and/or the plurality of intersections.

Robotic device220can include onboard software230and/or hardware250. Robotic device220may represent robotic system100, onboard software230may represent program instructions106and data107, and hardware250may represent mechanical components110and electrical components116. Onboard software230can include one or more of: localization subsystem232, obstacle detection subsystem234, odometry subsystem236, path-following subsystem238, and trajectory-following subsystem242. Localization subsystem232can be used to localize a robotic device, that is, determine a location of the robotic device within an environment. Localization subsystem232can generate position estimates of the robotic device and/or other objects that can be used to localize the robotic device, assist the robotic device in following a path, such as asynchronous paths216, and/or assist the robotic device in following a trajectory, such as trajectories240. Once the position estimates are generated, localization subsystem232can provide the position estimates to path-following subsystem238.

An asynchronous path, or path for short, can be a time-invariant plan or other information indicating how robotic device220can travel from a starting point SP to an ending point EP; i.e., an (asynchronous) path does not take time into account. In contrast, a trajectory can include values of a steering angle and of traction motor velocity that robotic device220can follow for a planning time interval.

The planning time interval can be a duration of time that a robotic device is guided, or planned to follow a path, route, and/or travel. In some embodiments, the planning time can be a predetermined amount of time; e.g., five seconds, one second, 0.2 seconds, or 0.1 seconds. For example, a predetermined planning time interval can be determined based on a user input that specifies a value for the planning time interval. In other embodiments, the planning time interval can be determined based on one or more other values; e.g., a stitch time, a time associated with a uniform edge (or path) cost, an estimated time to travel along a trajectory. Other techniques for determining the planning time interval and values for the planning time interval are possible as well.

Then, one or more trajectories can be used to describe how robotic device220can travel from starting point SP to an ending point EP in a time-variant manner. In some embodiments, a trajectory can also provide information about values of other variables than a steering angle and a traction motor velocity over the planning time interval, such as, but not limited to, other kinematic variables (e.g., velocity and acceleration) of robotic device220, and actuator positions of robotic device220.

As an example, a path to drive a car from a location “home” to a location “work” may include an ordered listing of streets that a control entity, such as a person or control device of an autonomous vehicle, can use to drive the car from home to work. In this example, a trajectory from home to work can involve one or more instructions specifying velocity and/or acceleration that the control entity can use to drive the car from home to work. In some examples, the trajectory can take traffic, obstacles, weather, and other time-sensitive conditions into account; e.g., the trajectory to go from home to work can indicate that the control entity “turn right for 10 seconds at 20 MPH or less”, “accelerate to 55 MPH and drive straight for 3 minutes”, “slow to 20 MPH within 30 seconds”, “turn left for 10 seconds at 20 MPH or less”, etc. In some embodiments, the trajectory can be changed along the way; e.g., to account for obstacles, changes in path, etc.

Obstacle detection subsystem234can determine whether one or more obstacles are blocking a path and/or a trajectory of robotic device220. Examples of these obstacles can include, but are not limited to, pallets, objects that may have fallen off a pallet, robotic devices, and human operators working in the environment. If an obstacle is detected, obstacle detection subsystem234can provide one or more communications indicating obstacle detection to path-following subsystem238. The one or more communications indicating obstacle detection can include location information about one or more positions of one or more obstacles detected by obstacle detection subsystem234and/or identification information about the one or more obstacles detected by obstacle detection subsystem234. Odometry subsystem236can use data, such as data from servo drives252, to estimate one or more changes in position of robotic device220over time.

Path-following subsystem238and/or trajectory-following subsystem242can act as a planner aboard robotic device220. This onboard planner can follow one or more paths, such as asynchronous paths216, based on position estimates provided by localization subsystem232.

Path-following subsystem238can receive asynchronous paths216, position estimate inputs from localization subsystem232, location information about one or more positions of one or more obstacles from obstacle detection subsystem234, and/or information about one or more changes in position from odometry subsystem236, and generate one or more trajectories240as outputs.

Hardware250can include servo drives252and/or motors254. Servo drives252can include one or more servo drives. Servo drives252can include an electronic amplifier used to power one or more servomechanisms and/or can monitor feedback signals from the servomechanism(s). Servo drives252can receive control signals, such as trajectories244, from onboard software230, and can provide electric current to the servomechanism(s) to produce motion proportional to the control signals. In some embodiments, servo drives252can compare status information received from the servomechanism(s) with an expected status as commanded by trajectories244. Then, servo drives252can adjust a voltage frequency or pulse width of the provided electric current to correct for deviations between received status information and an expected status. In other embodiments, servo drives252can provide information, such as the feedback signals and/or location-related information, to onboard software230.

One or more motors254can be part or all of the servomechanism(s) powered by servo drives252. For example, motors254can use the electric current provided by servo drives252to generate mechanical force to drive part or all of robotic device220; e.g., motors254can provide force to propel robotic device220and/or drive one or more effectors of robotic device220.

Path planning of robotic devices within an environment, such as an environment that includes indoor settings, such as a warehouse, office building, or home, and/or outdoor settings, such as a park, parking lot, or yard, can be performed with respect to a roadmap graph, which is a connected graph of paths that agents, such as robotic devices, may follow. Using roadmap graphs to plan agent routing within the environment rather than taking a free-space approach can reduce a total planning state space and so making large-scale multi agent coordination tractable. Further, the use of roadmap graphs can enable operators to intuitively control areas in which robotic devices are allowed to navigate.

Roadmap graph generation can first involve generation of a prototype graph, which indicates the rough position of lanes and directions of travel. In some examples, a prototype graph can be a directed graph that indicates lanes and directions of travel of robotic devices. In other examples, a prototype graph can be generated manually based on a map or drawing of the environment.

III. EXAMPLE VEHICLES

FIGS. 3A and 3Billustrate example vehicles that may embody robotic system100and/or robotic device220. Other vehicles which vary in form from those illustrated here as well as other types of robotic devices may also be included.

FIG. 3Ashows an autonomous guided vehicle (AGV), according to an example embodiment. More specifically, AGV340may be a relatively small, mobile robotic device that is capable of transporting individual boxes or cases. AGV340may include wheels342to allow for locomotion within a warehouse environment. Additionally, top surface344of AGV340may be used to places boxes or other objects for transport. In some examples, top surface344may include rotating conveyors to move objects to or from AGV340.

AGV340may include sensors connected to AGV340and configured to move with respect to AGV340to observe different portions of the environment (e.g., rotate and/or translate with respect to AGV340). AGV340may also include connected thereto projector346configured to move with respect to AGV340much like the sensors. In some example implementations, projector346may be connected to a different portion of AGV340than shown inFIG. 3A. AGV340may also include fixed and/or movable projectors (not shown) mounted to the front, back, and sides thereof.

In additional examples, AGV340may be powered by one or more batteries that can be quickly recharged at a battery charging station and/or exchanged for fresh batteries at a battery exchange station. In further examples, AGV340may additionally include other components not specifically identified here, such as sensors for navigation. AGVs with different shapes and sizes also may be included within a robotic warehouse fleet, possibly depending on the types of packages handled by a warehouse.

FIG. 3Bshows an autonomous fork truck, according to an example embodiment. More specifically, autonomous fork truck360may include forklift362for lifting and/or moving pallets of boxes or other larger materials. In some examples, forklift362may be elevated to reach different racks of a storage rack or other fixed storage structure within a warehouse. Autonomous fork truck360may additionally include wheels364for locomotion to transport pallets within the warehouse. In additional examples, the autonomous fork truck may include a motor and power supply as well as a sensing system. Autonomous fork truck360may also vary in size or shape from the one illustrated inFIG. 3B.

Further, autonomous fork truck360may include projector366configured to move (e.g., rotate and/or translate) with respect to autonomous fork truck360to project visual indications onto different portions of the environment. In some embodiments, projector366may be connected to a different portion of autonomous fork truck360than shown inFIG. 3B. Autonomous fork truck360may also include projector368mounted to a side thereof, which may be fixed or movable, as well as similar projector mounted to the front, rear, and other side of autonomous fork truck360.

Any of the robotic devices described herein may include one or more sensor(s) such as force sensors, proximity sensors, load sensors, position sensors, touch sensors, depth sensors, ultrasonic range sensors, infrared sensors, Global Positioning System (GPS) receivers, sonar, optical sensors, biosensors, Radio Frequency identification (RFID) sensors, Near Field Communication (NFC) sensors, wireless sensors, compasses, smoke sensors, light sensors, audio sensors, microphones, speakers, radar, cameras (e.g., color cameras, grayscale cameras, and/or infrared cameras), depth sensors (e.g., Red Green Blue plus Depth (RGB-D), lasers, a light detection and ranging (LIDAR) device, a structured-light scanner, and/or a time-of-flight camera), a stereo camera, motion sensors (e.g., gyroscope, accelerometer, inertial measurement unit (IMU), and/or foot step or wheel odometry), and/or range sensors (e.g., ultrasonic and/or infrared), among others. The sensor(s) may provide sensor data to a processor(s) to allow for appropriate interaction of a robotic device with the environment. Additionally, a robotic device may also include one or more power source(s) configured to supply power to various components of the robotic device. Any type of power source may be used such as, for example, a gasoline engine or a battery.

IV. CAUTION REGION DETECTION AND PROJECTION

FIGS. 4A-4Hindicate example operations for identifying and projecting visual indications of caution regions in an environment.FIG. 4Aillustrates a top-down view of vehicle400and path401determined for vehicle400to follow through an environment. The environment includes therein pallet422and a pallet rack434onto which pallets424,426,428,430, and432are stacked.

Vehicle400may be a pallet jack or fork truck having tines402and404. Tines402and404may allow vehicle400to interact with pallets422-432, or other storage structures, by placing the tines into slots within the pallet, thereby enabling pick-up, transportation, and drop-off of the pallet. Vehicle400may also include sensor406for scanning the environment, allowing for vehicle localization and obstacle detection within the environment, among other operations. Vehicle400may further include projector408for projecting visual indications onto the environment.

Path401may run from (or may form part of a path running from) a starting position within the environment to target position within the environment. Path401may be planned around known or fixed obstacles (e.g., walls) in the environment based on kinematic and dynamic properties of vehicle400. Path401may be made up of a plurality of discrete ordered positions410,412,414,416,418, and420(i.e., positions410-420). Positions410-420may be target positions for vehicle400to follow in sequence to move along path401. Before or while causing vehicle400to move along path401, the control system may analyze path401to determine whether any caution regions will arise as a result of vehicle400moving therealong, and may cause projector408to project indications of any of these caution regions onto the environment.

In order to determine the caution regions, the control system may first determine a planned operating region for vehicle400. The planned operating region may be, for example, an area within the environment planned to be occupied by vehicle400during a period of time (e.g., next 30 seconds). Thus, the control system may determine, for each respective position of positions410-420along path401, a vehicle footprint indicating an area within the environment planned to be occupied by vehicle400at the respective position.

FIG. 4Billustrates a plurality of vehicle footprints413,415,417,419, and421(i.e., footprints413-421) projected along path401at corresponding positions412,414,416,418, and420, respectively. Footprints413-421are shown with alternating line patterns for clarity. Each of footprints413-421indicates the area within the environment, or within the map representing the environment, that is planned to be occupied by vehicle400when vehicle400(e.g., the centroid of vehicle400) is at the respective position along path401. Footprint413, for example, represents a first area within the environment planned to be occupied by vehicle400when the centroid of vehicle400reaches position412, while footprint415represents a second area within the environment planned to be occupied by vehicle400when the centroid of vehicle400reaches position414, and so on.

In some embodiments, the map may be represented as an occupancy grid that includes a number of cells that represent corresponding areas in the environment. Each cell may be assigned a state that indicates the status of the area represented by the cell. Particularly, a cell may be assigned as having an obstacle, free space, or unknown. Cells with obstacles may represent physical features within the environment, including fixed, movable, and mobile objects. Cells with free space may be traversable by the vehicle without striking objects in the environment. Unknown cells may require additional sensor data to determine whether the area includes an obstacle or not (i.e., has free space). The control system (e.g., local or remote) may periodically update and adjust the occupancy grid based on new measurements of the environment from sensors coupled to one or more vehicles navigating the environment.

The pose (i.e., position and orientation) of footprints413-421may be determined based on the physical size of vehicle400(e.g., mass and volume), as well as the steering angles and velocities planned to be commanded to vehicle400to cause vehicle400to follow path401. In some embodiments, each footprint, in addition to representing the area in the environment expected to be occupied by the vehicle, may include a buffer region around the area. For example, each footprint might be 10% larger than an actual physical size of vehicle400to account for errors in sensing, vehicle positioning, and simulation, among others. In some embodiments, the density of positions410-420, and thus the density of footprints413-421, may be greater or smaller than that shown inFIGS. 4A-4H.

Planned operating region436may be determined by fitting a boundary to a union of footprints413-421, as illustrated inFIG. 4C. Planned operating region436may be defined by an area enclosed by the boundary and may represent regions likely to be occupied by vehicle400within a future time period. A caution region may be detected by determining whether planned operating region436intersects with threshold areas around any obstacles within the environment.

In some implementations, footprints413-421and planned operating region436may be determined after determining path401(or a longer path of which path401is a part) but before vehicle400is caused to move along path401. Alternatively, footprints413-421and planned operating region436may be determined after determining path401and while vehicle400moves along path401. Similarly, caution regions may be detected before or while vehicle400moves along path401. For example, caution regions arising due to fixed objects within the environment may be determined before vehicle400is caused to follow path401, while caution regions arising due to non-fixed (e.g., moveable or mobile) objects may be determined while vehicle400moves along path401(i.e., when the non-fixed objects come into view of a sensor on vehicle400).

FIG. 4Dillustrates threshold area438around pallet rack434and threshold area442around pallet422. Threshold area438may be defined as an area located within threshold distance440of a perimeter of pallet rack434. Similarly, threshold area442may be defined as an area located within threshold distance444of a perimeter of pallet422. Threshold distances440and444may be referred to as object safety thresholds. In some embodiments, threshold distances440and444may be defined by a safety standard for a workspace in which robotic devices work alongside humans. In some implementations, threshold distances greater than those defined by the safety standard may be used.

The threshold distances may be fixed, or may be dynamically sized based on a classification or type of object (e.g., whether the object is fixed, movable, or moving), a size of the object, a type of vehicle, or a speed with which the vehicle is traveling, among other factors. For example, threshold distance444around pallet422may be greater than threshold distance440around pallet rack434because pallet422is movable and pallet rack434is fixed. In another example, threshold distance440may be increased in proportion to increases in speed of vehicle400. In some embodiments, a maximum speed of vehicle400may be limited when vehicle400enters into threshold areas438or442.

FIG. 4Eillustrates intersection448between threshold area438and planned operating region436, as well as intersection446between threshold area442and planned operating region436. Intersections446and448indicate regions of the environment where vehicle400is planned to cross into threshold areas438and442, respectively, while following path401. Intersection446may arise because vehicle400may be planned to pick up pallet422. An additional intersection may arise between another planned operating region and threshold area438as vehicle400loads pallet422onto pallet rack434. A caution region may be determined based on each of intersections446and448.

FIG. 4Fillustrates caution regions452and456determined based on intersections446and448, respectively. Caution region452may be determined by extending intersection448horizontally towards pallet rack434. Caution region456may be determined by extending intersection446vertically towards pallet422.

In some embodiments, caution regions452and456may be smaller, larger, and/or may have a different shape than shown inFIG. 4F. For example, caution regions452and456may span the same area as intersections448and446. In another example, caution regions452and456may include circles circumscribed around intersections448and446. In some embodiments, the shape, size, and other aspects of the caution regions may be defined by the safety standard. For example, when no good escape area/route exists for an occupant due to a vehicle's position in the environment, the caution region might have a different size or shape than when there is an area or route for the occupant to avoid the vehicle. Caution regions452and456may be indicated in the occupancy grid by assigning to corresponding cells of the occupancy grid a “caution” state.

Projector408of vehicle400may project visual indications of caution regions452and456, as indicated by light beams450and454. A pose (i.e., orientation and position) of projector408relative to vehicle400to direct beams450and454onto caution regions452and456, respectively, may be determined and projector408may be positioned accordingly, as shown inFIG. 4F. The visual indications may contain patterns, text, and/or images that communicate to occupants of the environment that vehicle400is planned to come within threshold distance444of pallet422and threshold distance440of pallet rack434. Thus, the visual indications may operate to compel occupants of the environment to avoid caution regions452and456. In some embodiments, caution regions452and456may be indicated on one or more maps of the environment before visual indications thereof are projected by projector408(e.g., before vehicle400moves along path401).

As vehicle400moves through the environment, the pose of projector408may be adjusted to keep the projected visual indications of caution regions452and456fixed in relation to pallet rack434and pallet422, respectively. That is, motion of projector408may operate to keep visual indications consistently projected onto caution regions452and456, regardless of the position of vehicle400along path401.

FIG. 4G, for example, illustrates vehicle400after it has moved along path401to approximately position414(i.e., footprint415). The pose of projector408has been adjusted by turning projector408in a counterclockwise direction relative to the pose, in a reference frame of vehicle400, shown inFIG. 4F. The adjusted pose thus ensures that light beams450and454continue to be directed onto caution regions452and456, respectively. Similarly,FIG. 4Hillustrates vehicle400after it has moved along path401to approximately position418(i.e., footprint419). The pose of projector408has been further adjusted by turning projector408in a clockwise direction relative to the pose, in the reference frame of vehicle400, shown inFIG. 4G. The indication of caution region452is no longer projected since vehicle400has already entered and exited caution region452while passing position416. However, the further adjusted pose ensures that light beam454continues to be directed onto caution region456which is yet to be occupied by vehicle400.

In some implementations, however, an actual position of the visual indications projected onto caution regions456and/or452to vary from a planned or expected position of these visual indications. Thus, although the planned position of the visual indications may be fixed in relation to the environment, the actual position may vary from the planned position by, for example, several millimeters, centimeters, or meters, depending on the circumstances. In one example, such variation may be caused by projector408not being able to be reoriented with respect to vehicle400quickly enough to compensate for changes in position of vehicle400relative to the environment. In another example, image processing capabilities of projector408or of the control system (e.g., refresh rate) might cause a delay in updating the actual visual projection to match the planned or expected visual projections. In a further example, occlusions along the paths of light beams454or450might also cause differences in position and appearance between the actual and planned visual indications. Nevertheless, the control system may operate to minimize an error or difference between the planned position and planned appearance of the visual indications and the actual position and actual appearance thereof.

In some embodiments, the visual indications may be projected onto surfaces within the environment that are not perpendicular to projector408. Contents of the visual indications may therefore become distorted due to the keystone effect. For example, text contained within the visual indications may appear smaller on surfaces close to projector408and larger on surfaces farther away from projector408. The projections of the visual indication may be adjusted to correct for the keystone effect.

A keystone correction may be determined for each of the visual indications based on a relative position and a relative angle between projector408, vehicle400, and the surfaces within the environment onto which the visual indications are to be projected. The keystone correction for a particular projector pose and vehicle position may be determined before projector408and vehicle400are moved to the particular projector pose and vehicle position. Thus, caution regions452and456may be indicated without any apparent visual distortions resulting from repositioning of vehicle400and projector408. Keeping the projected visual indications of caution regions452and456fixed in relation to pallet rack434and pallet422may therefore further involve modulating the projected light beams450and454according to the keystone correction to account for any distortion of the visual indications resulting from repositioning of vehicle400and projector408along path401.

In some embodiments, sensor406may be a camera and may be used to capture images of caution regions452and456. The control system may, based on the images, verify that the visual indications are being projected onto caution regions452and456as planned. If the visual indications are not being projected as planned, the control system may indicate a fault with projector408to an operator of vehicle400. In some embodiments, the control system may also stop vehicle400until proper functionality of projector408is restored.

When the visual indication are being projected, the control system may, based on the images, determine additional keystone corrections to apply to the visual indications to remove therefrom any keystone distortion detected based on the images. Still further, the control system may determine, based on the images and/or other sensor data, whether any unexpected occupants are present or are predicted to be present in the caution regions while vehicle400in operating nearby. When unexpected occupants are present, the control system may stop the vehicle, change the appearance of the visual projections (e.g., flash alternating red and yellow warnings), and/or cause a speaker on the vehicle to emit a warning sound, among other possibilities. In some embodiments, the control system may determine an escape path for the unexpected occupants to follow to move out of or avoid the caution region. The escape path may be projected onto the environment to allow the occupants to move in synchrony with the vehicle to avoid the vehicle as well as the caution region.

In addition to projecting visual indications of caution regions before they are occupied by the vehicle (i.e., future caution regions), the control system may also determine and project visual indications of caution regions currently occupied by the vehicle (i.e., current caution regions).FIG. 5illustrates vehicle400simultaneously projecting a first visual indication of future caution region504ahead of vehicle400and a second visual indication of current caution region500adjacent to vehicle400, as indicated by light beams506and502, respectively. Vehicle400may additionally include side projectors458and460. As vehicle400moves along path508, current and future footprints of vehicle400may overlap with threshold area438around pallet rack434. Thus, to indicate to occupants of the environment that caution regions500and504are dangerous and should not be occupied, projectors460and408may project corresponding visual indications of these caution regions.

In some embodiments, projectors458and460may be fixedly connected to a chassis of vehicle400, and may be used to project visual indications in a fixed direction relative to vehicle400. That is, projectors458and460may be capable of projecting visual indications towards the left and right of vehicle400, respectively. Accordingly, projectors458and460may be used to indicate current caution regions on the sides of vehicle400, as well as to indicate the direction, relative to vehicle400, of any future caution regions. For example, to indicate a caution region expected to arise on the left of vehicle400at a future time, projector458may illuminate an area near the left side of vehicle400. The visual indication may include an arrow pointing towards the caution region, but might not be projected onto the caution region. Alternatively, projectors458and460may be movable with respect to vehicle400and may thus be used to indicate both current and future caution regions by projecting the visual indication onto these regions.

In either case, projectors460and458may be connected to a different portion of vehicle400than projector408and may therefore be able to project visual indications onto portions of the environment that might not be within a field-of-view range of projector408. For example, by being connected under a body of vehicle400, projector460can project a visual indication of caution region500very close to vehicle400, while projector408, positioned on top of the body of vehicle400, might be unable to project that closely near vehicle400. In some examples, rather than being connected to vehicle400, projectors408,458,460, or other additional projectors may be placed throughout the environment, and may thus project visual indications on behalf of one or more vehicles located in a vicinity of the projectors.

Notably, the visual projections herein described may be used as an alternative to physical indications such as painted lines, taped lines, traffic cones, or other physical delineations of potential caution regions around objects within the environment. In a warehouse, for example, safety standards may indicate that areas on the floor that could be occupied by a vehicle and located within, for example, 50 centimeters of a pallet rack are to be permanently painted, delineated, or otherwise marked as potential caution regions. Such marking of the floor of the warehouse may be expensive, may wear off with time and use, and is not dynamically adjustable based on planned operations of a vehicle. The visual projections, on the other hand, do not experience any wear and allow the caution regions to be dynamically marked without any physical changes to the warehouse. Additionally, the visual projections may be used to indicate caution regions around fixed objects (e.g., pallet rack434, walls, etc.), as well as movable objects (e.g., pallet422) and mobile objects (e.g., other vehicles). That is, the visual projections may be used to indicate caution regions around moveable and moving objects that would be too impractical to indicate using conventional methods.

V. SAFETY REGION DETECTION AND PROJECTION

FIGS. 6A and 6Billustrate safety regions600and602within the environment that are planned to be unoccupied by vehicle400within a predetermined future period of time. Safety regions600and602may be determined based on planned operating region436. In one example, any area of the environment that is not within threshold distance604of planned operating region436may be a safety region. In another example, any area of the environment that is not within threshold distance604of planned operating region436and is also not within threshold areas438and442may be a safety region, as illustrated inFIG. 6A. Safety regions600and602may thus indicate to occupants of the environment where the occupants should positions themselves to avoid vehicle400.

Light projector408may be caused to project additional indications of safety regions602and600, as illustrated by light beams606and608inFIG. 6B. Visual indications of safety regions602and600, as well as visual indications of caution regions452and456, may be projected simultaneously by projector408. To this end, in some embodiments, projector408may project a single beam wide enough to include all of regions600,602,452, and456. Alternatively, projector408may include multiple independently controllable sub-units that individually project indications onto each of regions600,602,452, and456.

In some embodiments, projector408may also be used to project onto the environment visual indications of planned operating region436, path401, positions410-420, and footprints413-421, among other information conveying intent or planned operations of vehicle400. The visual indications of caution regions, safety regions, and planned operations may be visually distinct from one another, allowing occupants to easily distinguish the different areas and thus avoid occupying areas that are unsafe.

VI. EXAMPLE VISUAL INDICATIONS OF CAUTION REGIONS

FIGS. 7A, 7B, and 7Cillustrate example structures and contents of visual indications of a caution region. A caution region may be predicted or expected to arise around pallet700when a vehicle comes or plans to come within a threshold distance of pallet700to, for example, pick up and move pallet700. AsFIG. 7Ashows, visual indication702may therefore be projected onto a surface (e.g., ground surface) within the environment around pallet700to inform occupants of the caution region.

In some embodiments, visual indication702may be larger than the caution region around pallet700. For example the caution region of pallet700may be similar to caution region456in front of pallet422, as shown inFIGS. 4F-4H. However, visual indication702may be projected onto the area of the caution region as well as areas on the side of pallet700, as shown inFIG. 7A. Visual indication702may be larger than the caution region to provide, out of an abundance of caution, a safety factor beyond the caution region. Visual indication702may also be larger than the caution region to account for the possibility that pallet700may be picked up and moved by the vehicle, and may thus be inadvertently pushed back or moved sideways during pick-up. In general, regardless of the motivation, the visual indications may be larger, equal in size to, or smaller than the caution region.

Regardless of size, visual indication702may include therein colors, patterns, images, or text that informs occupants of the caution region. The colors, patterns, images, or text may be defined by the safety standard, or may be custom-tailored to particular environments. For example, as shown inFIG. 7A, visual indication702may include text that reads “WARNING: PINCH REGION” to indicate to occupants that objects within the extent of space illuminated by visual indication702may get pinched between pallet700and the vehicle. In another example, the text may read “WARNING: LOW ESCAPE REGION” to indicate to occupants that occupants might not be able to easily leave the caution region once the vehicle moves into it. In a further example, the text may be more general, such as “CAUTION” or “KEEP OUT.”

In some embodiments, additional visual indications704and706may be provided adjacent to visual indication702, as shown inFIG. 7B. Visual indications702,704, and706may be visually distinct from one another, including different colors, patterns, images, or text. Such visual differences may be used to convey information about different levels of danger associated with occupying areas corresponding to or underlying each visual indication, or an amount of time remaining until the vehicle occupies the areas onto which the visual indications are projected. For example, visual indication702may include therein text that reads “WARNING: PINCH REGION IN . . .3,” visual indication704may include therein text that reads “ . . .2,” and visual indication706may include therein text that reads “ . . .1.” Thus, visual indications702,704, and706may inform occupants that the vehicle will occupy the area under indication706in one second, the area under indication704in two seconds, and the area under indication702in three seconds.

The text may be updated as the vehicle approaches the caution region. For example, indication702may be updated to read “WARNING: PINCH REGION IN . . .2,” indication704may be updated to read “ . . .1,” and indication706might no longer be displayed as the vehicle enters the underlying area.

FIG. 7Cshow that, in addition to indicating caution regions, the visual projections may also indicate the threshold distances or threshold areas around objects which may give rise to potential caution regions. That is, in addition to visual indication710of a caution region, the projector may also project visual indication708delineating an extent of the threshold area around pallet rack434which gives rise to the caution region. Visual indication708may therefore provide information ordinarily provided by lines taped or painted on the floor of a warehouse.

FIG. 7Dillustrates a 3D view of vehicle400projecting visual indication702around pallet700using light projector408. Visual indication702is projected onto the floor of the environment and surrounds pallet700. In some embodiments, visual indication702may additionally or alternatively be projected onto payload712loaded onto pallet700to provide a yet further indication of the caution region.

In some embodiments, the visual indication may take the form of a holographic 3D indication of the caution regions. Such an indication may be an actual-scale hologram, projected so as to “fill” the airspace at least where the vehicle will occupy the caution region in the future. Additionally or alternatively, the indication may be projected as a smaller-scale hologram. The 3D hologram may also depict the planned operating region as well as the safety regions. The 3D hologram may be projected or otherwise displayed as an open quadric surface, a closed quadric surface, convex hull, isosurface, or other complex or non-complex surface/volume.

In some embodiments, vehicle400may be configured to use a warning field and/or a protective field to facilitate safe navigation through the environment. The warning field may be a first virtual (i.e., defined in software) region defined around a portion of vehicle400. For example, the warning field may be a region extending 1 meter in front of vehicle400. Vehicle400may be configured to use a sensor to detect objects within the environment. When an object is detected within the warning field, vehicle400may be caused to travel under a threshold speed (i.e., the maximum speed of vehicle400may be limited). This may allow vehicle400to slow down in anticipation of a potential collision with the object, thus making operation of vehicle400safer.

The protective field may be a subset of the warning field. For example, the protective field may be a region extending half a meter in front of vehicle400. When an object is detected within the protective field, vehicle400may be caused to come to a stop to avoid a potential collision with the object. In some examples, the size and positioning of the warning and/or protective fields with respect to vehicle400may vary, for example, based on speed or a task performed by vehicle400.

In addition to projecting visual indications of caution regions, projector408may also be used to project a visual indication of the warning fields and/or the protective fields of vehicle400. This may allow human occupants of the environment to see and understand why vehicle400is behaving in a particular way, as well as better predict the future behavior of vehicle400. In some instances, however, the protective and/or warning fields may be disabled. For example, when vehicle400moves pallets in deep aisles or loads pallets onto pallet racks (e.g., onto the low level of a pallet rack), the warning and protective fields may be disabled so as to allow vehicle400to come within a distance of the pallet racks needed to transport, load, or unload pallets, but that would otherwise cause vehicle400to slow down or stop. In such cases, the projected visual indications of the disabled warning and/or protective fields may be used to warn occupants of the environment of the fact that the warning and/or protective fields are disabled, and vehicle400thus might not stop for nearby obstacles. When the warning and/or protective fields are disabled, the visual indications thereof may be visually different from the visual indications projected when the warning and/or protective fields are enabled in order to prompt occupants of the environment to be more cautious when the vehicle's warning and/or protective fields are disabled. Additionally, vehicle400may provide audible signals to indicate that the warning and/or protective fields are disabled.

VII. ADDITIONAL EXAMPLE OPERATIONS

FIG. 8illustrates flowchart800of example operations related to detection of caution regions and projecting of visual indications of the detected caution regions by a robotic system (e.g., robotic system100, robotic device220, AGV340, or fork truck360). These operations may be executed by control system118of robotic system100, onboard software230of robotic device220, or planning system210, for example.

Block802may involve determining, by a control system, a planned operating region for a vehicle within an environment.

Block804may involve determining, by the control system, that the planned operating region is within a threshold distance of an object within the environment.

Block806may involve in response to determining that the planned operating region is within the threshold distance of the object, determining, by the control system, a caution region to illuminate with a light projector near the object, wherein the light projector is connected to the vehicle.

Block808may involve causing, by the control system, the light projector to project an indication of the caution region near the object, wherein the projected indication remains fixed in relation to the object as the vehicle moves toward the planned operating region.

Keeping the projected indication fixed in relation to the object may involve the control system adjusting a position and orientation of the light projector to reduce or minimize a difference between a planned position of the visual indications within the environment and an actual position thereof. In some circumstances the actual position of the projected visual indications may vary from the planned position of the visual indications due to, for example, delay in physically reorienting the projector, limitations in capabilities of the projector, and/or occlusions along the path of projected light, among other causes.

In some embodiments, the caution region may include a buffer area near at least part of a perimeter of the object.

In some embodiments, determining the caution region may involve determining an intersection between the planned operating region and a first area, where the first area (i) surrounds the perimeter of the object and (ii) is within the threshold distance of the perimeter of the object. Determining the caution region may also involve determining a second area between the intersection and the perimeter of the object and combining the intersection and the second area to form the caution region.

In some embodiments, causing the light projector to project the indication of the caution region near the object may involve projecting the indication of the buffer area onto a ground surface of the environment near the object.

In some embodiments, the light projector may be repositionable with respect to the vehicle in two or more degrees of freedom. The vehicle may be caused to move towards the planned operating region and, while the vehicle moves towards the planned operating region, the light projector may be repositioned with respect to the vehicle to project the caution region in a fixed position in relation to the object.

In some embodiments, a keystone correction to apply to the indication of the caution region may be determined based on (i) a positioning of the light projector with respect to the vehicle and (ii) a positioning of the vehicle with respect to the caution region. The light projector may be caused to project the indication of the caution region modified according to the determined keystone correction.

In some embodiments, the light projector may be caused to project the indication of the caution region near the object before the planned operating region is occupied by the vehicle.

In some embodiments, the light projector may be caused to project the indication of the caution region near the object while the planned operating region is occupied by the vehicle.

In some embodiments, the planned operating region includes an area to be occupied by the vehicle while picking up the object, and the caution region includes an area between the planned operating region and the object.

In some embodiments, the threshold distance may be a first threshold distance. Based on the planned operating region, a safety region may be determined within the environment that is planned to be unoccupied by the vehicle, where the planned operating region is beyond a second threshold distance of the safety region. The light projector may be caused to project another indication of the safety region.

In some embodiments, based on a position of the planned operating region and a position of the object within the environment, a distance between the planned operating region and the object may be determined. Based on the distance between the planned operating region and the object, a size of the caution region may be determined. The light projector may be caused to project an indication of the caution region having the determined size.

In some embodiments, a size of the caution region may be determined based on a speed planned for the vehicle within the planned operating region. The light projector may be caused to project the indication of the caution region having the determined size.

In some embodiments, the light projector may include at least a first light source124connected to a first side of the vehicle and at least a second light source126connected to a second side of the vehicle. An orientation of the vehicle with respect to the caution region may be determined. Based on the determined orientation, at least one of the first light source or the second light source may be selected to illuminate the caution region. The selected light source may be caused to project the indication of the caution region.

In some embodiments, the first light source may be connected underneath a body of the vehicle on the first side of the body. The second light source may be connected underneath the body of the vehicle on the second side of the body.

In some embodiments, the caution region may include an area within the threshold distance of the object.