AUTONOMOUS AND TELEOPERATED SENSOR POINTING ON A MOBILE ROBOT

A computer-implemented method executed by data processing hardware of a robot causes the data processing hardware to perform operations. The operations include receiving a sensor pointing command that commands the robot to use a sensor to capture sensor data of a location in an environment of the robot. The sensor is disposed on the robot. The operations include determining, based on an orientation of the sensor relative to the location, a direction for pointing the sensor toward the location, and an alignment pose of the robot to cause the sensor to point in the direction toward the location. The operations include commanding the robot to move from a current pose to the alignment pose. After the robot moves to the alignment pose and the sensor is pointing in the direction toward the location, the operations include commanding the sensor to capture the sensor data of the location in the environment.

TECHNICAL FIELD

This disclosure relates to sensor pointing for mobile robots.

BACKGROUND

Robotic devices are used to autonomously or semi-autonomously perform tasks such as navigating to a specified location and capturing sensor data with one or more sensors. For example, the robot may be tasked to navigate to a point of interest and, using a camera, capture image data of the point of interest without user input or supervision. In these scenarios, the user expects the sensor data capture to be accurate and repeatable.

SUMMARY

An aspect of the present disclosure provides a computer-implemented method that when executed by data processing hardware of a robot causes the data processing hardware to perform operations. The operations include receiving a sensor pointing command that commands the robot to use a sensor to capture sensor data of a target location in an environment of the robot. The sensor is disposed on the robot. The operations include determining, based on an orientation of the sensor relative to the target location, a target direction for pointing the sensor toward the target location, and an alignment pose of the robot to cause the sensor to point in the target direction toward the target location. The operations further include commanding the robot to move from a current pose to the alignment pose. After the robot moves to the alignment pose and the sensor is pointing in the target direction toward the target location, the operations include commanding the sensor to capture the sensor data of the target location in the environment.

In some implementations, the operations further include, in response to receiving the sensor pointing command, commanding the robot to navigate to a target point of interest in the environment. In those implementations, determining the target direction and the alignment pose includes determining the target direction and the alignment pose after the robot navigates to the target point of interest.

In some embodiments, the sensor includes a camera.

In some examples, the sensor includes a pan-tilt-zoom (PTZ) sensor. In further examples, after determining the target direction for pointing the PTZ sensor toward the target location, the operations further include determining that a center of a field of sensing of the PTZ sensor is not aligned with the target direction. In those further examples, the operations also further include determining PTZ alignment parameters for aligning the center of the field of sensing of the PTZ sensor with the target direction. In those further examples, the operations also further include commanding, using the PTZ alignment parameters, the PTZ sensor to adjust the center of the field of sensing of the PTZ sensor to align with the target direction. In even further examples, after commanding the PTZ sensor to adjust the center of the field of sensing of the PTZ sensor, the operations further include receiving, from the PTZ sensor, alignment feedback data indicating an error between the adjusted center of the field of sensing of the PTZ sensor and the target direction. In those even further examples, determining the alignment pose of the robot is further based on the received alignment feedback data. In additionally further examples, the error indicates that a difference between the adjusted center of the field of sensing of the PTZ sensor and the target direction is greater than a threshold difference.

In some implementations, receiving the sensor pointing command includes receiving a user input indication indicating selection of a ray or a point relative to a known coordinate frame of the robot.

In some embodiments, the sensor pointing command includes a model of an object and, before commanding the sensor to capture the sensor data, the operations further include capturing image data using a camera in a requested direction. In those embodiments, the operations also include determining whether the object, using the model of the object, is present in the captured data and, when the object is present in the captured image data, determining the target direction to center the object within a field of sensing of the sensor.

In some examples, the sensor pointing command includes an object classification of an object from an output of an object detector and, before commanding the sensor to capture the sensor data, the operations further include capturing image data using a camera in a requested direction. In those examples, the operations also include determining whether the classified object, using the output of the object detector is present in the captured image data and, when the classified object is present in the captured image data, determining the target direction to center the classified object within a field of sensing of the sensor.

In some implementations, the sensor is fixed to the robot. In some embodiments, determining the alignment pose of the robot that causes the sensor to point in the target direction includes determining inverse kinematics of the robot. In some examples, determining the alignment pose of the robot that causes the sensor to point in the target direction includes processing image data captured by a second sensor that is different from the sensor.

Another aspect of the present disclosure provides a system. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform operations. The operations include receiving a sensor pointing command that commands a robot to use a sensor to capture sensor data of a target location in an environment of the robot. The sensor is disposed on the robot. The operations include determining, based on an orientation of the sensor relative to the target location, a target direction for pointing the sensor toward the target location, and an alignment pose of the robot to cause the sensor to point in the target direction toward the target location. The operations further include commanding the robot to move from a current pose to the alignment pose, and commanding the sensor to capture the sensor data of the target location in the environment.

In some implementations, the operations further include, in response to receiving the sensor pointing command, commanding the robot to navigate to a target point of interest in the environment. In those implementations, determining the target direction and the alignment pose includes determining the target direction and the alignment pose after the robot navigates to the target point of interest.

In some embodiments, the sensor includes a camera.

In some examples, the sensor includes a pan-tilt-zoom (PTZ) sensor. In further examples, after determining the target direction for pointing the sensor toward the target location, the operations further include determining that a center of a field of sensing of the PTZ sensor is not aligned with the target direction. In those further examples, the operations also further include determining PTZ alignment parameters for aligning the center of the field of sensing of the PTZ sensor with the target direction. In those further examples, the operations also further include commanding, using the PTZ alignment parameters, the PTZ sensor to adjust the center of the field of sensing of the PTZ sensor to align with the target direction. In even further examples, after commanding the PTZ sensor to adjust the center of the field of sensing of the PTZ sensor, the operations further include receiving, from the PTZ sensor, alignment feedback data indicating an error between the adjusted center of the field of sensing of the PTZ sensor and the target direction. In those even further examples, determining the alignment pose of the robot is further based on the received alignment feedback data. In additionally further examples, the error indicates that a difference between the adjusted center of the field of sensing of the PTZ sensor and the target direction is greater than a threshold difference.

In some implementations, receiving the sensor pointing command includes receiving a user input indication indicating selection of a ray or a point relative to a known coordinate frame of the robot.

In some embodiments, the sensor pointing command includes a model of an object and, before commanding the sensor to capture the sensor data, the operations further include capturing image data using a camera in a requested direction. In those embodiments, the operations also include determining whether the object, using the model of the object, is present in the captured data and, when the object is present in the captured image data, determining the target direction to center the object within a field of sensing of the sensor.

In some examples, the sensor pointing command includes an object classification of an object from an output of an object detector and, before commanding the sensor to capture the sensor data, the operations further include capturing image data using a camera in a requested direction. In those examples, the operations also include determining whether the classified object, using the output of the object detector is present in the captured image data and, when the classified object is present in the captured image data, determining the target direction to center the classified object within a field of sensing of the sensor.

In some implementations, the sensor is fixed to the robot. In some embodiments, determining the alignment pose of the robot that causes the sensor to point in the target direction includes determining inverse kinematics of the robot. In some examples, determining the alignment pose of the robot that causes the sensor to point in the target direction includes processing image data captured by a second sensor that is different from the sensor.

Yet another aspect of the present disclosure provides a computer-implemented method that when executed by data processing hardware of a robot causes the data processing hardware to perform operations. The operations include receiving a sensor pointing command that commands the robot to use a sensor to capture sensor data of a target location in an environment of the robot. The sensor is disposed on the robot. The operations include determining, based on an orientation of the sensor relative to the target location, a target direction for pointing the sensor toward the target location. The operations include with the sensor pointing in the target direction toward the target location, commanding the sensor to capture the sensor data of the target location in the environment.

In some implementations, the operations further include determining, based on the orientation of the sensor relative to the target location, an alignment pose of the robot to cause the sensor to point in the target direction toward the target location, and commanding the robot to move from a current pose to the alignment pose. In further implementations, commanding the sensor to capture the sensor data occurs after the robot moves to the alignment pose and the sensor is pointing in the target direction toward the target location.

An additional aspect of the present disclosure provides a system. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform operations. The operations include receiving a sensor pointing command that commands a robot to use a sensor to capture sensor data of a target location in an environment of the robot. The sensor is disposed on the robot. The operations include determining, based on an orientation of the sensor relative to the target location, a target direction for pointing the sensor toward the target location. The operations include with the sensor pointing in the target direction toward the target location, commanding the sensor to capture the sensor data of the target location in the environment.

In some implementations, the operations further include determining, based on the orientation of the sensor relative to the target location, an alignment pose of the robot to cause the sensor to point in the target direction toward the target location, and commanding the robot to move from a current pose to the alignment pose. In further implementations, commanding the sensor to capture the sensor data occurs after the robot moves to the alignment pose and the sensor is pointing in the target direction toward the target location.

In another aspect, a method of autonomous and teleoperated sensor pointing is provided. The method includes detecting an orientation of a sensor of a robot relative to a target location using a computing system of the robot, determining, based on the detected orientation of the sensor, a target direction for pointing the sensor and an alignment pose of the robot using the computing system, controlling one or more joints of the robot to move the robot from a current pose to the alignment pose and to point the sensor in the target direction, and once the robot has moved to the alignment pose and the sensor is pointing in the target direction, capturing sensor data using the sensor.

In another aspect, a robot is provided. The robot includes a sensor, a computing system configured to detect an orientation of the sensor relative to a target location, and to determine a target direction for pointing the sensor and an alignment pose of the robot based on the detected orientation, a plurality of joints, wherein the computing system is further configured to control one or more of the plurality of joints to move the robot from a current pose to the alignment pose and to point the sensor in the target direction, wherein the sensor is configured to capture sensor data with the robot positioned in the alignment pose and the sensor pointing in the target direction.

DETAILED DESCRIPTION

Modern autonomous and semi-autonomous robots are equipped with complex mapping, localization, and navigation systems. Additionally, these robots are often equipped with one or more sensors for capturing data related to the environment in which the robot is travelling. Autonomous and semi-autonomous robots can be used in autonomous inspection applications, where the robot is instructed to travel to a point of interest (POI) or is brought to the POI by a user and is instructed to capture data using one or more of its sensors in a manner specified by a command provided to the robot.

Embodiments herein are directed toward systems and methods for implementing a command to capture sensor data of a target environment of a robot with accuracy and/or ability to conform the capture technique relative to the target environment. The robot utilizes a navigation system to determine a travel path to the target environment and uses a sensor pointing system to determine an orientation of the sensor at the robot relative to the target environment. Based on instructions received by the robot, the sensor pointing system commands the robot to move in a way that aligns the sensor relative to the target environment in a desired way. For example, the movement can include changing from a current pose of the robot to an alignment pose that causes the sensor to point in a target direction.

By implementing the sensor pointing system in this manner, sensor data is accurately captured from the target environment and/or from a target object in a target orientation of the sensor relative to the target environment. Moreover the sensor data can be captured by the robot executing instructions without intervention and/or supervision by a user.

Referring toFIGS.1A and1B, in some implementations, a robot100includes a body110with one or more locomotion based structures such as legs120a-dcoupled to the body110and that enable the robot100to move within the environment30. In some examples, each leg120is an articulable structure such that one or more joints J permit members122of the leg120to move. For instance, each leg120includes a hip joint JH (for example, JHb and JHd ofFIG.1A) coupling an upper member122,122Uof the leg120to the body110and a knee joint JK (for example, JKa, JKb, JKc, and JKdofFIG.1A) coupling the upper member122Uof the leg120to a lower member122Lof the leg120. AlthoughFIG.1Adepicts a quadruped robot with four legs120a-d, the robot100may include any number of legs or locomotive based structures (e.g., a biped or humanoid robot with two legs, or other arrangements of one or more legs) that provide a means to traverse the terrain within the environment30.

In order to traverse the terrain, each leg120has a distal end124(for example, distal ends124a,124b,124c, and124dofFIG.1A) that contacts a surface of the terrain (i.e., a traction surface). In other words, the distal end124of each leg120is the end of the leg120used by the robot100to pivot, plant, or generally provide traction during movement of the robot100. For example, the distal end124of a leg120corresponds to a foot of the robot100. In some examples, though not shown, the distal end124of the leg120includes an ankle joint such that the distal end124is articulable with respect to the lower member122Lof the leg120.

In the examples shown, the robot100includes an arm126that functions as a robotic manipulator. The arm126may be configured to move about multiple degrees of freedom in order to engage elements of the environment30(e.g., objects within the environment30). In some examples, the arm126includes one or more members128, where the members128are coupled by joints J such that the arm126may pivot or rotate about the joint(s) J. For instance, with more than one member128, the arm126may be configured to extend or to retract. To illustrate an example,FIG.1Adepicts the arm126with three members128corresponding to a lower member128L, an upper member128U, and a hand member128H(e.g., also referred to as an end-effector150). Here, the lower member128Lmay rotate or pivot about a first arm joint JA1located adjacent to the body110(e.g., where the arm126connects to the body110of the robot100). The lower member128Lis also coupled to the upper member128Uat a second arm joint JA2, while the upper member128Uis coupled to the hand member128Hat a third arm joint JA3.

In some examples, such as inFIG.1A, the hand member128Hor end-effector150is a mechanical gripper that includes a moveable jaw and a fixed jaw configured to perform different types of grasping of elements within the environment30. In the example shown, the end-effector128Hincludes a fixed first jaw and a moveable second jaw that grasps objects by clamping the object between the jaws. The moveable jaw is configured to move relative to the fixed jaw to move between an open position for the gripper and a closed position for the gripper (e.g., closed around an object).

In some implementations, the arm126additionally includes a fourth joint JA4. The fourth joint JA4may be located near the coupling of the lower member128Lto the upper member128Uand functions to allow the upper member128Uto twist or rotate relative to the lower member128L. In other words, the fourth joint JA4may function as a twist joint similarly to the third joint JA3or wrist joint of the arm126adjacent the hand member128H. For instance, as a twist joint, one member coupled at the joint J may move or rotate relative to another member coupled at the joint J (e.g., a first member coupled at the twist joint is fixed while the second member coupled at the twist joint rotates). In some implementations, the arm126connects to the robot100at a socket on the body110of the robot100. In some configurations, the socket is configured as a connector such that the arm126attaches or detaches from the robot100depending on whether the arm126is needed for operation.

The robot100has a vertical gravitational axis (e.g., shown as a Z-direction axis AZ) along a direction of gravity, and a center of mass CM, which is a position that corresponds to an average position of all parts of the robot100where the parts are weighted according to their masses (i.e., a point where the weighted relative position of the distributed mass of the robot100sums to zero). The robot100further has a pose P based on the CM relative to the vertical gravitational axis AZ(i.e., the fixed reference frame with respect to gravity) to define a particular attitude or stance assumed by the robot100. The attitude of the robot100can be defined by an orientation or an angular position of the robot100in space. Movement by the legs120relative to the body110alters the pose P of the robot100(i.e., the combination of the position of the CM of the robot and the attitude or orientation of the robot100). Here, a height generally refers to a distance along the z-direction (e.g., along the z-direction axis AZ). The sagittal plane of the robot100corresponds to the Y-Z plane extending in directions of a y-direction axis AYand the z-direction axis AZ. In other words, the sagittal plane bisects the robot100into a left and a right side. Generally perpendicular to the sagittal plane, a ground plane (also referred to as a transverse plane) spans the X-Y plane by extending in directions of the x-direction axis AXand the y-direction axis AY. The ground plane refers to a ground surface14where distal ends124of the legs120of the robot100may generate traction to help the robot100move within the environment30. Another anatomical plane of the robot100is the frontal plane that extends across the body110of the robot100(e.g., from a left side of the robot100with a first leg120ato a right side of the robot100with a second leg120b). The frontal plane spans the X-Z plane by extending in directions of the x-direction axis AXand the z-direction axis Az.

In order to maneuver about the environment30or to perform tasks using the arm126, the robot100includes a sensor system130with one or more sensors132,132. For example,FIG.1Aillustrates a first sensor132,132amounted at a front of the robot100(i.e., near a front portion of the robot100adjacent the front legs120a—b), a second sensor132,132bmounted near the hip of the second leg120bof the robot100, a third sensor132,132ccorresponding to one of the sensors132mounted on a side of the body110of the robot100, a fourth sensor132,132dmounted near the hip of the fourth leg120dof the robot100, and a fifth sensor132,132emounted at or near the end-effector128Hof the arm126of the robot100. The sensors132may include vision/image sensors, inertial sensors (e.g., an inertial measurement unit (IMU)), force sensors, and/or kinematic sensors. Some examples of sensors132include a camera such as a stereo camera a visual red-green-blue (RGB) camera, or a thermal camera, a time-of-flight (TOF) sensor, a scanning light-detection and ranging (LIDAR) sensor, or a scanning325laser-detection and ranging (LADAR) sensor. Other examples of sensors132include microphones, radiation sensors, and chemical or gas sensors.

In some examples, the sensor132has a corresponding field(s) of view FVdefining a sensing range or region corresponding to the sensor132. For instance,FIG.1Adepicts a field of a view FVfor the robot100. Each sensor132may be pivotable and/or rotatable such that the sensor132, for example, changes the field of view FVabout one or more axis (e.g., an x-axis, a y-axis, or a z-axis in relation to a ground plane). In some examples, multiple sensors132may be clustered together (e.g., similar to the first sensor132a) to stitch a larger field of view FVthan any single sensor132. With sensors132placed about the robot100, the sensor system130may have a 360 degree view or a nearly 360 degree view (with respect to the X-Y or transverse plane) of the surroundings of the robot100.

When surveying a field of view FVwith a sensor132, the sensor system130generates sensor data134(e.g., image data) corresponding to the field of view FV. The sensor system130may generate the field of view FVwith a sensor132mounted on or near the body110of the robot100(e.g., sensor(s)132a,132b). The sensor system may additionally and/or alternatively generate the field of view FVwith a sensor132mounted at or near the end-effector150of the arm126(e.g., sensor(s)132e). The one or more sensors132capture the sensor data134that defines the three-dimensional point cloud for the area within the environment30of the robot100. In some examples, the sensor data134is image data that corresponds to a three-dimensional volumetric point cloud generated by a three-dimensional volumetric image sensor132. Additionally or alternatively, when the robot100is maneuvering within the environment30, the sensor system130gathers pose data for the robot100that includes inertial measurement data (e.g., measured by an IMU). In some examples, the pose data includes kinematic data and/or orientation data about the robot100, for instance, kinematic data and/or orientation data about joints J or other portions of a leg120or arm126of the robot100. With the sensor data134, various systems of the robot100may use the sensor data134to define a current state of the robot100(e.g., of the kinematics of the robot100) and/or a current state of the environment30about the robot100. In other words, the sensor system130may communicate the sensor data134from one or more sensors132to any other system of the robot100in order to assist the functionality of that system.

In some implementations, the sensor system130includes sensor(s)132coupled to a joint J. Moreover, these sensors132may couple to a motor M that operates a joint J of the robot100(e.g., sensors132,132b-d). Here, these sensors132generate joint dynamics in the form of joint-based sensor data134. Joint dynamics collected as joint-based sensor data134may include joint angles (e.g., an upper member122Urelative to a lower member122Lor hand member128Hrelative to another member of the arm126or robot100), joint speed, joint angular velocity, joint angular acceleration, and/or forces experienced at a joint J (also referred to as joint forces). Joint-based sensor data generated by one or more sensors132may be raw sensor data, data that is further processed to form different types of joint dynamics, or some combination of both. For instance, a sensor132measures joint position (or a position of member(s)122coupled at a joint J) and systems of the robot100perform further processing to derive velocity and/or acceleration from the positional data. In other examples, a sensor132is configured to measure velocity and/or acceleration directly.

As the sensor system130gathers sensor data134, a computing system140stores, processes, and/or to communicates the sensor data134to various systems of the robot100(e.g., the control system170, a sensor pointing system200, a navigation system300, and/or remote controller10, etc.). In order to perform computing tasks related to the sensor data134, the computing system140of the robot100(which is schematically depicted inFIG.1Aand can be implemented in any suitable location(s), including internal to the robot100) includes data processing hardware142and memory hardware144. The data processing hardware142is configured to execute instructions stored in the memory hardware144to perform computing tasks related to activities (e.g., movement and/or movement based activities) for the robot100. Generally speaking, the computing system140refers to one or more locations of data processing hardware142and/or memory hardware144.

In some examples, the computing system140is a local system located on the robot100. When located on the robot100, the computing system140may be centralized (e.g., in a single location/area on the robot100, for example, the body110of the robot100), decentralized (e.g., located at various locations about the robot100), or a hybrid combination of both (e.g., including a majority of centralized hardware and a minority of decentralized hardware). To illustrate some differences, a decentralized computing system140may allow processing to occur at an activity location (e.g., at motor that moves a joint of a leg120) while a centralized computing system140may allow for a central processing hub that communicates to systems located at various positions on the robot100(e.g., communicate to the motor that moves the joint of the leg120).

Additionally or alternatively, the computing system140can utilize computing resources that are located remote from the robot100. For instance, the computing system140communicates via a network180with a remote system160(e.g., a remote server or a cloud-based environment). Much like the computing system140, the remote system160includes remote computing resources such as remote data processing hardware162and remote memory hardware164. Here, sensor data134or other processed data (e.g., data processing locally by the computing system140) may be stored in the remote system160and may be accessible to the computing system140. In additional examples, the computing system140is configured to utilize the remote resources162,164as extensions of the computing resources142,144such that resources of the computing system140reside on resources of the remote system160.

In some implementations, as shown inFIGS.1A and1B, the robot100includes a control system170. The control system170may be configured to communicate with systems of the robot100, such as the at least one sensor system130, the navigation system300(e.g., with navigation commands302), and/or the sensor pointing system200(e.g., with body pose commands230). The control system170may perform operations and other functions using the computing system140. The control system170includes at least one controller172that is configured to control the robot100. For example, the controller172controls movement of the robot100to traverse about the environment30based on input or feedback from the systems of the robot100(e.g., the sensor system130and/or the control system170). In additional examples, the controller172controls movement between poses and/or behaviors of the robot100. At least one the controller172may be responsible for controlling movement of the arm126of the robot100in order for the arm126to perform various tasks using the end-effector150. For instance, at least one controller172controls the end-effector150(e.g., a gripper) to manipulate an object or element in the environment30. For example, the controller172actuates the movable jaw in a direction towards the fixed jaw to close the gripper. In other examples, the controller172actuates the movable jaw in a direction away from the fixed jaw to close the gripper.

A given controller172may control the robot100by controlling movement about one or more joints J of the robot100. In some configurations, the given controller172is software or firmware with programming logic that controls at least one joint J and/or a motor M which operates, or is coupled to, a joint J. A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an “application,” an “app,” or a “program.” For instance, the controller172controls an amount of force that is applied to a joint J (e.g., torque at a joint J). As programmable controllers172, the number of joints J that a controller172controls is scalable and/or customizable for a particular control purpose. A controller172may control a single joint J (e.g., control a torque at a single joint J), multiple joints J, or actuation of one or more members128(e.g., actuation of the hand member128H) of the robot100. By controlling one or more joints J, actuators or motors M, the controller172may coordinate movement for all different parts of the robot100(e.g., the body110, one or more legs120, the arm126). For example, to perform a behavior with some movements, a controller172may be configured to control movement of multiple parts of the robot100such as, for example, two legs120a-b, four legs120a-d, or two legs120a-bcombined with the arm126. In some examples, a controller172is configured as an object-based controller that is setup to perform a particular behavior or set of behaviors for interacting with an interactable object.

With continued reference toFIG.1B, an operator12(also referred to herein as a user or a client) may interact with the robot100via the remote controller10that communicates with the robot100to perform actions. For example, the operator12transmits commands174to the robot100(executed via the control system170) via a wireless communication network16. Additionally, the robot100may communicate with the remote controller10to display an image on a user interface190(e.g., UI190) of the remote controller10. For example, the UI190is configured to display the image that corresponds to three-dimensional field of view Fv of the one or more sensors132. The image displayed on the UI190of the remote controller10is a two-dimensional image that corresponds to the three-dimensional point cloud of sensor data134(e.g., field of view Fv) for the area within the environment30of the robot100. That is, the image displayed on the UI190may be a two-dimensional image representation that corresponds to the three-dimensional field of view Fv of the one or more sensors132.

In some implementations, as shown inFIG.2, the robot100is located in the environment30and is equipped with the sensor system130that includes the sensor132(disposed on the body110, in this example) on the robot100and having a field of view Fv that includes at least a portion of the environment30surrounding the robot100. The computing system140of the robot100is equipped with data processing hardware and memory hardware with the memory hardware including instructions to be executed by the data processing hardware. The computing system140is configured to operate the navigation system300and the sensor pointing system200(for instance, in autonomous inspection applications) to navigate the robot100to a POI and uses a sensor132to capture sensor data134at the POI in a particular way all without user input or supervision.

In the illustrated embodiment, the computing system140includes the navigation system300that generates or receives a navigation map222from map data210obtained by the computing system140. The navigation system300generates a route path212that plots a path around large and/or static obstacles from a start location (e.g., the current location of the robot100) to a destination. The navigation system300is in communication with the sensor pointing system200. The sensor pointing system200may receive the route path212or other data from the navigation system300in addition to sensor data134from the sensor system130.

The sensor pointing system200receives a sensor pointing command220(e.g., from the user12) that directs the robot100to capture sensor data134of a target location250(e.g., a specific area or a specific object in a specific area) and/or in a target direction TD. The sensor pointing command220may include one or more of the target location250, the target direction TD, an identification of a sensor132(or multiple sensors) to capture sensor data134with, etc. When the robot is proximate the target location, the sensor pointing system200generates one or more body pose commands230(e.g., to the control system170) to position the sensor132such that the target location250and/or the target direction TDare within the field of sensing of the sensor132. For example, the sensor pointing system200determines necessary movements of the sensor132and/or of the robot100(i.e., adjust a position or orientation or pose P of the robot) to align the field of sensing of the sensor132with the target location250and/or target direction TD.

In some examples, and as discussed in more detail below, the sensor pointing system200directs the pose P of the robot100to compensate for a sensed error in sensor132configuration or orientation. For example, the robot100may alter its current pose P to accommodate a limited range of motion of the field of view FVof the sensor, avoid occluding the captured sensor data, or match a desired perspective of the target location250. Thus, in some implementations, the sensor pointing system200, based on an orientation of the sensor132relative to the target location250, determines the target direction TDto point the sensor132toward the target location250.

Alternatively or additionally, the sensor pointing system determines an alignment pose PAof the robot to cause the sensor132to point in the target direction TDtoward the target location250. The sensor pointing system200may command the robot100to move to the alignment pose PAto cause the sensor132to point in the target direction TD. After the robot100moves to the alignment pose PA, and with the sensor132pointing in the target direction TDtoward the target location250, the sensor pointing system200may command the sensor132to capture sensor data134of the target location250in the environment30.

In other words, the computing system140is configured to receive the sensor pointing command220(e.g., from the user12) that, when implemented, commands the robot100to capture sensor data134using the sensor132(or multiple sensors) disposed on the robot100. Based on the orientation of the sensor132relative to the target location250, the sensor pointing system200determines the target direction TDand the alignment pose P of the robot100. The determined target direction TDpoints the sensor132toward the target location250and the determined alignment pose PAof the robot100causes the sensor132to point in the target direction TDtoward the target location250. The sensor pointing system200may command the robot100to move from a current pose P of the robot100to the alignment pose PAof the robot. After the robot100moves to the alignment pose PAand with the sensor132pointing in the target direction TDtoward the target location250, the sensor pointing system200commands the sensor132to capture sensor data134of the target location250in the environment30.

As will become apparent from this disclosure, the sensor pointing system200, along with other features and elements of the methods and systems disclosed herein, make the data capture of target locations250in environments30repeatable and accurate as the robot100is sensitive to sensed and unsensed error in the robot's position, orientation, and sensor configuration. The sensor pointing system200allows the robot100to overcome odometry and sensor error when capturing sensor data134relative to the target location250at least in part by determining the target direction TDfor pointing the sensor132at the target location250and the alignment pose PAfor achieving the target direction TDbased on the orientation of the sensor132relative to the target location250.

In some examples, in response to receiving the sensor pointing command220, the sensor pointing system200commands the robot100to navigate to a target point of interest (POI)240within the environment30. In such examples, the sensor pointing system200determines the target direction TDand the alignment pose PAof the robot100after the robot100navigates to the target POI240.

Referring now toFIG.3, in some examples, the navigation system300(e.g., based on map data210, sensor data134, etc.) generates a series of route waypoints310on the graph map222for the navigation route212that plots a path around large and/or static obstacles from a start location (e.g., the current location of the robot100) to a destination (e.g., the target POI240). Route edges312connect corresponding pairs of adjacent route waypoints310. The robot100, when navigating the environment30, travels from route waypoint310to route waypoint310by traversing along the route edges312. In some examples, the target POI240is a route waypoint310on the navigation map222. In the example shown, the robot100travels along the navigation route212until reaching the target POI240(i.e., a specified route waypoint310). In some examples, the target POI240is the final route waypoint310along the navigation route212, while in other examples, the navigation route212continues on with additional route waypoints310and route edges320for the robot100to continue along after capturing the sensor data134at the target POI240and the navigation route212may include any number of target POIs240for capturing sensor data134at various locations along the navigation route212.

Thus, based on guidance provided by the navigation system300, the robot100arrives at a route waypoint310defined by the target POI240. After arrival at the waypoint, the sensor pointing system200may determine an orientation of the sensor132relative to the target location250. Based on the orientation of the sensor132relative to the target location250, the sensor pointing system200determines the target direction TDfor pointing the sensor132toward the target location250.

Although examples herein (e.g.,FIG.2) illustrate the sensor132integrated into the body of the robot100at a front portion of the robot100with a field of view FVprimarily forward of the robot, the sensor132(or sensors) may be disposed in any suitable manner on the robot100. The sensor132may include any number of different types of sensors such as a camera, LIDAR, and/or microphone. For example, the sensor132may be built into the body110of the robot100or attached as a payload. In some examples, the sensor132is disposed on the articulated arm126. Additionally, the sensor132may be permanently fixed to the robot100as part of its original manufacture or alternatively disposed or mounted at the robot100(e.g., client hardware) and connected to the sensor pointing system200via client software (FIG.4). The sensor132may have any fixed or pivotable (e.g., a pan-tilt-zoom (PTZ) sensor such as a PTZ camera) field of view/field of sensing. Because the orientation of the sensor132is based at least in part on the pose P of the robot100, movement of the robot100, such as to the alignment pose PA, changes the field of view of the sensor132.

The target direction TD, in some examples, is parameterized by the sensor pointing command220. In other words, the sensor pointing command220may include instructions as to how the sensor data134of the target location250should be captured, such as from a certain direction, angle, zoom, focus, and/or distance relative to the target location250or with the target location250framed a certain way in the field of view FVof the sensor. Thus, the sensor pointing command220may include parameters for capturing sensor data134of the target location250, such as angle, height, proximity, and direction of the sensor relative to the target location, and parameters related to placement of the target location250within the captured sensor data134. The parameters may also include configuration for the sensor132while capturing the sensor data134(e.g., zoom, focus, exposure, control of illumination sources, etc.). The sensor pointing system200may determine the target direction TDbased on the parameters of the sensor pointing command220. Alternatively, the target direction TDmay be provided by the sensor pointing command220. Based on the parameters of the sensor pointing command220and/or the target direction TD, the sensor pointing system200commands the robot100(e.g., to the alignment pose PA) and/or sensor to move to orient the sensor132toward the target location250.

Referring now toFIGS.1B and4, the sensor pointing system200may determine the target direction TDand the alignment pose PAin response to receiving the sensor pointing command220. The sensor pointing command220may originate from an onboard autonomous mission manager402(i.e., generated from mission data or parameters, robot configuration, etc.) and/or from client software410that includes robot command software412. Thus, the user12may communicate sensor pointing commands220to the robot (e.g., wirelessly via the controller10) or robot100may generate the sensor pointing command(s) within the context of an autonomous mission. Here, the sensor pointing system200includes a sensor pointing service200. The sensor pointing commands220are communicated to the sensor pointing service200(e.g., by the autonomous mission manager402or the robot command software412) to determine the target direction TDand sensor configurations for capturing the sensor data134.

In some implementations, the client software410(in communication with the computing system140of the robot100) includes object detectors and scene alignment processors414that process the sensor data134captured by the sensor132. For example, the object detectors detect objects present in captured image data. In other implementations, the sensor pointing system200includes the object detectors and/or scene alignment processors and processes the sensor data134automatically. The client software410may execute locally at the robot100or may execute remote from the robot100(e.g., at the controller10, the remote system160, or at any other server exterior the robot100and in communication with the computing system140of the robot100).

The sensor pointing system200may also be in communication with the mechanical systems of the robot100. For example, as shown inFIG.4, the sensor pointing system200may communicate the body pose commands230to a robot command service404of the computing system140, and various sensors132disposed at or in communication with the robot100, such as base robot sensor hardware406, advanced plug-in sensors408, and client hardware420. For example, the client hardware420, includes advanced sensor hardware422, fixed sensor hardware424, and a pan-tilt-zoom (PTZ) sensor payload426at the robot100. In certain implementations, the robot100can be instructed to move in multiple command ways, including both map navigation and robot commands.

In some implementations, the PTZ payload hardware426(e.g., a sensor132) communicates with PTZ plug-in services409at the robot which is operable to, for example, receive sensor data134from the PTZ payload hardware426and communicate PTZ commands430to the PTZ payload hardware426. The PTZ plug-in service409may be sensor specific (i.e., a hardware interface) and thus likely to execute client-side (i.e., external to the robot100). In some examples, the PTZ plug-in services409execute within the sensor132. In some implementations, the PTZ payload hardware426is a sensor132(e.g., a PTZ camera) temporarily mounted to or connected with the robot100. The sensor pointing system200may delegate reconfiguration of the PTZ payload hardware426to the PTZ plug-in409.

When the robot includes a PTZ sensor132, and after the system obtains or determines the target direction TDfor pointing the PTZ sensor132toward the target location250, the sensor pointing system200may sense or detect and correct any existing error (i.e., discrepancy) between the current direction of the PTZ sensor132(e.g., a vector along the center of the field of sensing of the PTZ sensor132) and the target direction TD. The center of the field of sensing refers to a vector that originates at the PTZ sensor132and extends away from the PTZ sensor132such that the sensor's field of sensing to the left and to the right of the vector are of equivalent size and the sensor's field of sensing above and below the vector are of equivalent size.

In such implementations, the sensor pointing system200determines whether the center of a field of sensing of the PTZ sensor132(or other sensor) is aligned with the target direction TDand, if the center of field of sensing, (i.e., the “aim”) of the PTZ sensor132is not aligned with the target direction TD, the sensor pointing system200determines PTZ alignment parameters for aligning the center of the field of sensing of the PTZ sensor132with the target direction TD. Furthermore, the sensor pointing system200may command the PTZ sensor132, e.g., using the PTZ alignment parameters, to adjust the center of the field of sensing of the PTZ sensor132(e.g., commanding the PTZ sensor132to pan, tilt, and/or zoom) to align with the target direction TD. Thus, the target direction TDmay be parameterized, at least in part, by PTZ alignment parameters.

In some implementations, after commanding the PTZ sensor132to adjust the center of the field of sensing of the PTZ sensor, the sensor pointing system200receives, from the PTZ sensor132(e.g., via the PTZ plug-in services409), alignment feedback data440. The alignment feedback data440indicates the current PTZ parameters of the PTZ sensor132. That is, the alignment feedback data440indicates the current orientation of the PTZ sensor132relative to the pose P of the robot100. In some examples, the sensor pointing system200determines a difference, based on the alignment feedback data440, between the current alignment510(FIG.5) of the center of the field of sensing of the PTZ sensor132and the target direction TD. When there is a difference (e.g., above a threshold difference), the sensor pointing system200determines, based on the difference between the current alignment of the center of the field of sensing of the PTZ sensor132and the target direction TD, the alignment pose PAthat will correct the difference between the pointing direction of the PTZ sensor132and the target direction TD. Thus, in these examples, determining the alignment pose PAof the robot100is based on the received alignment feedback data440from the PTZ sensor132. In other examples, such as when the sensor132is fixed, alignment of the sensor132relies entirely on the alignment post PAof the robot100.

Referring now toFIG.5, a schematic view500includes a three dimensional (3D) representation of a portion of the robot100and the target location250. Here, the sensor pointing system200has commanded the sensor132(e.g., a PTZ sensor132) to align with the target direction TD, however, the alignment feedback data440from the PTZ sensor132indicates a difference or error between a current alignment510of the sensor132(i.e., and adjusted center of the field of sensing) and the target direction TD, where the target direction TDand the current alignment510are represented by respective vectors to the target location250and originating at the sensor132. For example, the alignment error can arise from a variety of sources, for instance, the PTZ sensor132encountered a failure or the commanded orientation would require the PTZ sensor132to move beyond its capabilities (i.e., insufficient range of motion) or the field of sensing may be occluded by a portion of the sensor132itself or a portion of the robot100. Based on the error or difference or discrepancy between the current alignment510of the PTZ sensor132and the target direction TD, the sensor pointing system200determines an alignment pose PAfor the robot100that will adjust the orientation of the PTZ sensor132such that the sensor will point at the target location250in the target direction TD. Thus, the initial alignment of the PTZ sensor132and alignment feedback data440from the PTZ sensor132indicating the error between the current alignment510and the target direction TD, in this example, results in the sensor pointing system200determining the alignment pose PAfor pointing the PTZ sensor132at the target location250in the target direction TD. The sensor pointing system200may command the robot100to move the current pose P to the determined alignment pose PAprior to capturing sensor data134with the sensor132.

The sensor pointing command220may parameterize the target direction TDin other manners to capture the desired sensor data134of the target location250. For example, the sensor pointing command220may parameterize the target direction TDas a selected direction, a selected ray or vector (e.g., that originates from the robot100or the sensor132), or based on a point relative to one of any known coordinate frames of the robot100. In such implementations, the sensor pointing command220may include a user input indication indicating selection of a ray or a point relative to a known coordinate frame of the robot100. The user input indication may constitute the target location250and the sensor pointing system may determine the target direction TDand/or the alignment pose PAof the robot100to point or aim the sensor132in such a way that the user input indication is at the center of the field of view of the sensor132.

In some implementations, the sensor pointing command220parameterizes the target direction TDbased on object detection capabilities of the robot100, such as enabled by an object data base or world object service407(FIG.4) of the computing system140of the robot100. The object database407stores objects that the robot100has previously detected or may receive detection information from object detectors414of the client software410. Object detection may be performed by any system remote or local to the robot100. For example, the computing system140may receive object detection indications from the client software410or may receive object detections or perform object detection at the world object service portion407of the computing system140. Additionally, the sensor pointing system200may determine the target direction TDbased on the orientation of the sensor132relative to the target location250based at least in part on detecting an object that is at, near, or defines the target location250. In these examples, the target direction TDmay be based on an aspect or feature of the detected object.

In some examples, the sensor pointing command220may parameterize the target direction TDvia a two dimensional (2D) or 3D model of an object in an object database or world object service407of the computing system140of the robot100(FIG.4). In this scenario, the model of the object defines the target location250or is at or near the target location250in the environment. For example, the sensor pointing command220includes a model of an object that is to be detected and the target direction TDis intended to point the sensor132at a particular portion of the detected object or to point the sensor132in a particular orientation relative to the detected object.

In some examples, before aligning the sensor132in the target direction TDto capture sensor data134of the target location250, the sensor pointing system200captures image data134of the environment30using a camera132. Using the provided model of the object, the sensor pointing system200determines whether the object is present in the captured image data134. When the object is present in the captured image data134, the sensor pointing system200may determine the target direction TDrelative to the detected object. For example, the determined target direction TDmay center the object within a field of view FVof the sensor132. In some examples, the sensor132is controllable to move relative to the body of the robot such that the sensor132adjusts to align the sensor132in the target direction TDin combination with, or without, changing the pose of the robot100.

In some examples, the sensor pointing command220parameterizes the target direction TDvia an object classification of an object to be detected. In such examples, the sensor pointing command220includes an object classification. The object classification is an output of the object detector that matches detected objects to corresponding classifications. Thus, the sensor pointing command220may include a category or classification of object(s) to be detected, and the sensor pointing command220is parameterized relative to the indicated classification.

In certain implementations, before aligning the sensor132in the target direction TDto capture sensor data134of the target location250, the sensor pointing system200captures image data134of the environment30using the camera132. The sensor pointing system200processes the captured image data134to detect an object and determines a classification of the detected object. Using the output of the object detector, the sensor pointing system200determines whether the classified object is present in the captured image data134. When the classified object is present in the captured image data134, the sensor pointing system200determines the target direction TDof the sensor132relative to the classified object. For example, the sensor pointing system200determines the target direction TDto center the classified object within a field of view FVof the sensor132.

Thus, when the robot100is in the environment30, the sensor pointing system200may perform object detection to determine whether an object is present in the environment30and whether the detected object matches a model of an object or a classification of an object provided to the system. The sensor pointing system200may scan a portion of the environment (e.g., based on parameters in the sensor pointing command220such as a requested direction) in an attempt to acquire the location of the object. When the modeled object or classified object is present, the sensor pointing system200determines the target direction TDrelative to the object and the necessary PTZ commands430and/or alignment pose PAto align the field of sensing of the sensor132with the target direction TD. The determined target direction TDand corresponding alignment pose P may be relative to a feature or aspect of an object detected using the model of the object or that satisfies the provided object classification.

The sensor pointing command220, in some implementations, parameterizes the target direction TDbased on scene alignment, where determining the alignment pose PAof the robot100that aligns the sensor132in the target direction TDinvolves processing image data134captured by a second sensor132different from a primary sensor132pointed in the target direction TDand capturing sensor data of the target location250. For example, a camera132disposed at the robot100captures image data134of the environment30and the sensor pointing system200uses the captured image data134to confirm or correct alignment of the primary sensor132(e.g., a LIDAR sensor, a directional microphone, etc.) with the target direction TD. The sensor pointing system200may also use a reference image of the target location250and compare the reference image to captured image data134. The sensor pointing system200then derives a transformation from the comparison and determines a target direction TDand alignment pose P to achieve the transformation from the captured image data134to the reference image.

Referring back toFIG.4, once the target direction TDis determined based on, for example, the orientation of the sensor132relative to the target location250, the sensor pointing system200may determine the alignment pose PAof the robot100to cause the sensor132to point in the target direction TDtoward the target location250. The sensor pointing system200produces body pose commands450to command the robot100from its current pose P to the alignment pose PA. For example, the sensor pointing system200determines the alignment pose PAand body pose commands450from inverse kinematics of the robot100. The body pose commands450may be communicated to a robot command service404of the computing system140of the robot100, which controls the mechanical systems of the robot100responsive to the body pose commands450to achieve the alignment pose P.

Referring now toFIG.6, a flow chart600discloses one embodiment of a process for implementing a sensor pointing command220. Here, the target direction TDis parameterized via object detection and/or scene alignment, where captured image data134is processed to at least guide the sensor pointing command220to align the sensor132and the target direction TD. The process600includes a feedback loop607for iterative adjustment of the sensor132and body adjustment. To begin, the user12may specify a mission for the robot100and at step602, a command commands the robot100to navigate to a target POI240(i.e., waypoint A) in the environment30. The command may be at least in part responsive to receiving the sensor pointing command220or a separate navigation command302. The navigation command302is communicated and implemented by the navigation system300of the robot100. At step604, the sensor pointing command220is communicated to the sensor pointing system200which determines the target direction TDand alignment pose PAof the robot100. Before capturing the sensor data134, the appropriate object detector or scene alignment processor414is triggered at step606and a command C606to capture and process image data is communicated to the image sensor132and integrated image processors414. Sensor and body pose adjustments are calculated at step608with an iterative feedback loop607providing repeated sensor and body adjustments based on the captured image data, as necessary. In other words, if needed, a command C608may be communicated to the sensor pointing system200at step608, based on the iterative feedback loop607, to adjust the sensor132and/or body pose P. At step610, with the robot100in the alignment pose PAand the sensor132pointed in the target direction TDtoward the target location250, the sensor132captures sensor data of the target location250.

FIG.7is a flowchart of one embodiment of a method700for autonomous and teleoperated sensor pointing on a mobile robot. The method700, at step702, includes receiving a sensor pointing command220that commands the robot100to use a sensor132to capture sensor data134of a target location250in an environment30of the robot. The sensor132is disposed on or at the robot100. At step704, the method700includes determining, based on the orientation of the sensor132relative to the target location250, the target direction TDfor pointing the sensor132toward the target location250. At step706, the method700includes determining, based on the orientation of the sensor132relative to the target location250, the alignment pose PAof the robot100to cause the sensor132to point in the target direction TDtoward the target location250. At step708, the method700includes commanding the robot100to move from its current pose P to the alignment pose PA. The method700, at step710, includes, after the robot100moves to the alignment pose PAand with the sensor132pointing at the target location250in the target direction TD, commanding the sensor132to capture the sensor data134of the target location250in the environment30.

Thus, the present system is configured to receive a sensor pointing command that commands the robot to use a sensor to capture sensor data of a target location and determine, based on an orientation of the sensor relative to the target location, a target direction for pointing the sensor toward the target location, and/or an alignment pose for achieving the target direction. The system can command the robot to move to the alignment pose and/or capture the sensor data of the target location with the sensor pointing in the target direction. The systems and methods described herein thus allow for accurate and repeatable sensor data capture based on orientation of the sensor relative to the target location. Furthermore, some implementations further enhance accuracy and/or repeatability by using feedback from the sensor itself, image data captured by a second sensor at the robot, and/or object detection capabilities based on object classification and/or based on a provided model of an object.

The computing device800includes a processor810, memory820, a storage device830, a high-speed interface/controller840connecting to the memory820and high-speed expansion ports850, and a low speed interface/controller860connecting to a low speed bus870and a storage device830. Each of the components810,820,830,840,850, and860, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor810can process instructions for execution within the computing device800, including instructions stored in the memory820or on the storage device830to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display880coupled to high speed interface840. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices800may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The storage device830is capable of providing mass storage for the computing device800. In some implementations, the storage device830is a computer-readable medium. In various different implementations, the storage device830may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory820, the storage device830, or memory on processor810.

The high speed controller840manages bandwidth-intensive operations for the computing device800, while the low speed controller860manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In some implementations, the high-speed controller840is coupled to the memory820, the display880(e.g., through a graphics processor or accelerator), and to the high-speed expansion ports850, which may accept various expansion cards (not shown). In some implementations, the low-speed controller860is coupled to the storage device830and a low-speed expansion port890. The low-speed expansion port890, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device800may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server800aor multiple times in a group of such servers800a, as a laptop computer800b, or as part of a rack server system800c.

FIG.9is a schematic view of a robot100with a sensor pointing system130according to another embodiment. The embodiment ofFIG.9is similar to the embodiment ofFIG.2, except the robot100includes a specific configuration of a PTZ senor132shown inFIG.9.

A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. For example, while processes or blocks are presented in a given order, alternative embodiments may perform routines having steps, or employ systems having blocks, in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and/or modified. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed in parallel, or may be performed at different times. Furthermore, the elements and acts of the various embodiments described above can be combined to provide further embodiments. Indeed, the methods and systems described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the methods and systems described herein may be made without departing from the spirit of the disclosure. Accordingly, other implementations are within the scope of the following claims.