Patent Description:
Certain aspects of the present disclosure generally relate to machine learning and, more particularly, to improving systems and methods of motion planning.

Mobile agents (e.g., robots) are often faced with unidentified objects and obstacles while moving around an environment. When choosing actions to take in an environment, agents can choose actions that decrease their distance to a goal state, increase their information about the environment, and avoid obstacle collisions, or some combination thereof.

Gathering information about the environment is an important objective when attempting to take actions with limited resources or in unknown environments. One type of environmental information that can be useful in oft traversed domains is the shape and extent of objects in the domain. For the purposes of navigation, the rough extent of an object on the order of the size of the agent itself is useful. Information about the shape and extent of an object at this scale is often gathered through depth-based sensors such as structured light, laser and radar types of sensing technologies or through depth calculations made by co-registering features in many camera views (either from more than one camera or one camera at different points in time). However, most techniques for estimating the shape and extent of objects simply take advantage of movements by the agent, but do not prioritize movements that would be advantageous to the estimation of the shape and extent of an object.

Patent application <CIT> relates to a system and method for identifying objects using a robotic system. For example, a first image of at least one object is captured with an image capture device that is moveable with respect to the object, the first captured image is processed to determine a first pose of at least one feature the object, a first hypothesis is determined that predicts a predicted pose of the identified feature based upon the determined first pose, the image capture device is moved to capture a second image of the object, the captured second image is processed to identify a second pose of the feature, and the second pose of the object is compared with the predicted pose of the object.

Patent application <CIT> relates to a method for estimating the visibility of features on surfaces of object instances in multi-object scenes, to a method for perception planning in multi-object scenes, to an active perception framework and to an autonomous service robot.

Patent application <CIT> relates to a system and method for 3D measurement and surface reconstruction of an image reconfigurable vision.

Patent application <CIT> relates to a technique for determining the next best view during the automated acquisition of the complete surface geometry of objects using range cameras.

Additional features and advantages of the disclosure will be described below. It should be appreciated by those skilled in the art that this disclosure may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. It should also be realized by those skilled in the art that such equivalent constructions do not depart from the teachings of the disclosure as set forth in the appended claims. The novel features, which are believed to be characteristic of the disclosure, both as to its organization and method of operation, together with further objects and advantages, will be better understood from the following description when considered in connection with the accompanying figures. It is to be expressly understood, however, that each of the figures is provided for the purpose of illustration and description only and is not intended as a definition of the limits of the present disclosure.

Based on the teachings, one skilled in the art should appreciate that the scope of the disclosure is intended to cover any aspect of the disclosure, whether implemented independently of or combined with any other aspect of the disclosure. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects set forth. In addition, the scope of the disclosure is intended to cover such an apparatus or method practiced using other structure, functionality, or structure and functionality in addition to or other than the various aspects of the disclosure set forth. It should be understood that any aspect of the disclosure disclosed may be embodied by one or more elements of a claim.

The detailed description and drawings are merely illustrative of the disclosure rather than limiting, the scope of the disclosure being defined by the appended claims.

Aspects of the present disclosure are directed to motion planning and, more particularly, to improved efficiency in selection of movements in a sequence of movements by an agent (e.g., a robot, a drone, or a motor vehicle) equipped with a single camera. One issue addressed in the present disclosure is how to choose movements of the agent to best estimate the shape and extent of an object or obstacle in an unknown environment. For example, if a drone is deployed and observes an object in a scene, a control input may be determined so as to move the drone to determine the shape and extent of the object using visual sensors (e.g., a single camera) rather than depth sensors. In doing so, estimation of the object shape and extent may be performed faster and/or more accurately than conventional methods (e.g., employing random or unrelated movements).

In accordance with aspects of the present disclosure, a current location of the agent and a visual camera view of the environment may be provided as inputs. In turn, systems and methods of the present disclosure may output an action command. The action command may be in the form of velocity commands to a set of actuators or a preprogrammed motion primitive that specifies a trajectory through space over a finite time window, for example.

In some aspects, a camera frame may be processed to determine one or more bounding boxes around likely objects in an environment. Because the distance to the object may be unknown, each bounding box (e.g., a two-dimensional bounding box) may define a rectangular pyramid whose tip is centered at the camera's focal point and extends through the rectangle at the image plane. The base of the pyramid may be constrained to exist some distance away from the camera image plane. The distance may, in some aspects, be set based on the known resolution of the camera or scale of the environment. For example, a mobile device camera (e.g., cell phone camera) operating indoors may have a smaller maximum extent of the pyramid than a professional single-lens reflex (SLR) camera operating outdoors. As such, a rectangular pyramid for the mobile device camera may comprise an estimate of the object's shape and extent based on a single frame.

A movement may then be selected based on this current estimate of the object shape and extent. The agent (e.g., robot) may take the selected movement. A second camera frame may be processed to determine additional bounding boxes around the likely objects, and a new estimate for each object's shape and location may be determined. This second frame also produces a rectangular pyramid, however, because there are two pyramid estimates, confidence that the object lies within the intersection of these two rectangular pyramids, which is a smaller area than the initial estimate, may be increased.

This process may be repeated over time as actions are taken. In this way, new estimates of the object's location may be generated and the shape and extent of the object can be determined. In one exemplary aspect, the next action may be chosen such that the expected intersection area after the next camera measurement is minimized, subject to the constraint that the entire bounding box remains visible in the camera view. Accordingly, movements may be selected that would be more likely to reduce (or even minimize) the intersection area over time compared with an arbitrary set of movements.

<FIG> illustrates an example implementation of the aforementioned motion planning using a system-on-a-chip (SOC) <NUM>, which may include a general-purpose processor (CPU) or multi-core general-purpose processors (CPUs) <NUM> in accordance with certain aspects of the present disclosure. Variables (e.g., neural signals and synaptic weights), system parameters associated with a computational device (e.g., neural network with weights), delays, frequency bin information, and task information may be stored in a memory block associated with a neural processing unit (NPU) <NUM>, in a memory block associated with a CPU <NUM>, in a memory block associated with a graphics processing unit (GPU) <NUM>, in a memory block associated with a digital signal processor (DSP) <NUM>, in a dedicated memory block <NUM>, or may be distributed across multiple blocks. Instructions executed at the general-purpose processor <NUM> may be loaded from a program memory associated with the CPU <NUM> or may be loaded from a dedicated memory block <NUM>.

The SOC <NUM> may also include additional processing blocks tailored to specific functions, such as a GPU <NUM>, a DSP <NUM>, a connectivity block <NUM>, which may include fourth generation long term evolution (<NUM> LTE) connectivity, unlicensed Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, and the like, and a multimedia processor <NUM> that may, for example, detect and recognize gestures. In one implementation, the NPU is implemented in the CPU, DSP, and/or GPU. The SOC <NUM> may also include a sensor processor <NUM>, image signal processors (ISPs), and/or navigation <NUM>, which may include a global positioning system.

The SOC <NUM> may be based on an ARM instruction set. In an aspect of the present disclosure, the instructions loaded into the general-purpose processor <NUM> may comprise code for observing an object from a first pose of an agent having a controllable camera. The instructions loaded into the general-purpose processor <NUM> may also comprise code for determining at least one subsequent control input to move the agent and the camera to observe the object from a subsequent pose, to reduce an expected enclosing measure of an object based on visual data collected from the camera. The instructions loaded into the general-purpose processor <NUM> may also comprise code for controlling the agent and the camera based on the subsequent control input.

<FIG> illustrates an example implementation of a system <NUM> in accordance with certain aspects of the present disclosure. As illustrated in <FIG>, the system <NUM> may have multiple local processing units <NUM> that may perform various operations of methods described herein. Each local processing unit <NUM> may comprise a local state memory <NUM> and a local parameter memory <NUM> that may store parameters of a neural network. In addition, the local processing unit <NUM> may have a local (neuron) model program (LMP) memory <NUM> for storing a local model program, a local learning program (LLP) memory <NUM> for storing a local learning program, and a local connection memory <NUM>. Furthermore, as illustrated in <FIG>, each local processing unit <NUM> may interface with a configuration processor unit <NUM> for providing configurations for local memories of the local processing unit, and with a routing connection processing unit <NUM> that provides routing between the local processing units <NUM>.

In one configuration, a machine learning model is configured for observing an object from a first pose of an agent having a controllable camera. The model is also configured for determining a subsequent control input to move the agent and the camera to observe the object from a subsequent pose, to minimize an expected enclosing measure of an object based on visual data collected from the camera. The model is further configured for controlling the agent and the camera based on the subsequent control input(s). The model includes observing means, determining means, and/or controlling means. In one aspect, the observing means, determining means, and/or controlling means may be the general-purpose processor <NUM>, program memory associated with the general-purpose processor <NUM>, memory block <NUM>, local processing units <NUM>, and or the routing connection processing units <NUM> configured to perform the functions recited. In another configuration, the aforementioned means may be any module or any apparatus configured to perform the functions recited by the aforementioned means.

According to certain aspects of the present disclosure, each local processing unit <NUM> may be configured to determine parameters of the model based upon desired one or more functional features of the model, and develop the one or more functional features towards the desired functional features as the determined parameters are further adapted, tuned and updated.

<FIG> is a diagram illustrating an exemplary technique for estimating the shape of an object. Referring to <FIG>, a single object (e.g., table <NUM> shown with shadow <NUM>) is shown in an image (e.g., a red, green, blue (RGB) image). Of course, this is merely exemplary for ease of illustration and understanding, and additional objects may be included in the image. Using an object localization process, a two-dimensional (2D) silhouette of the object or bounding box <NUM> may be generated. A color or greyscale solution can be used for bounding box detection. The 2D silhouette <NUM> may be represented by sk(xi, yi) ∈ B, where B = {<NUM>,<NUM>} is the binary space. For example, if sk(xi, yi) = <NUM>, the object is visible in pixel (xi, yi) of image Ik. Otherwise, the object is not visible in pixel (xi, yi). One goal is to determine the 3D silhouette of the object or silhouette image <NUM> (e.g., three-dimensional bounding box) using the 2D image.

Given a sequence of RGB images I<NUM>:t = {I<NUM>,. , It} and associated camera poses p<NUM>:t = {p<NUM>,. , pt}, a sequence of object silhouettes s<NUM>:t = {s<NUM>,. , st} can be calculated.

At each time step k, camera intrinsics M, camera pose pk, and silhouette image sk can be used to calculate an inverse-projective cone Ck(xw, yw, zw) ∈ B or the visual hull of the object. In one example, (xi, yi) may be the projection of point (xw, yw, zw) onto the camera image frame at the current timestep. In that case, Ck(xw, yw, zw) = sk(xi, yi). In other words, if Ck(xw, yw, zw) = <NUM>, then silhouette sk has indicated that the 3D object could potentially contain point (xw, yw, zw). Otherwise, the object certainly does not contain point (xw, yw, zw). Accordingly, the visual hull measurement model may be expressed as Ck = h(pk, sk, M).

The camera may take multiple snapshots or photographs of an object from multiple different poses p<NUM>:t obtaining corresponding silhouettes s<NUM>:t. The camera intrinsics M, silhouettes and/or camera poses may in turn be used to calculate corresponding visual hulls C<NUM>:t at each time step. A joint visual hull Vt may then be calculated as the intersection of the visual hulls given by: <MAT> The joint visual hull provides an approximation of the shape and location in 3D space.

In some aspects, the visual hulls is used to determine a subsequent control input for moving the camera such that a measure m on the joint visual hull for the object may be reduced or minimized. The subsequent control input to produce this so-called active shape-from-silhouette (ASfS) reconstruction may be given by: <MAT> where u is the control input for movement of the camera and/or agent from a set of possible movements U, and h and f are functions of the camera dynamics and observation models. The control input u can be a vector including direction and speed. The variable Ck is the cone or visual hull, pk is the pose (e.g., position and/or orientation of the camera in 3D space), sk is the silhouette (or 2D bounding box), k is the time step or number of cones). The variable t is time, and M is the camera properties (e.g., type of lens). The variable m is the measure (e.g., volume) being reduced. The measure m can also be a surface area, height, or width, for example, if moving in a single direction and the goal is to avoid the object.

Using a minimized measure (e.g., volume) of the joint visual hull, a subsequent control input is determined to move the agent and/or camera to a subsequent position from which to observe the object. A new joint visual hull may be determined using the visual data at the subsequent position. The new joint visual hull may similarly be minimized to determine a subsequent control input. In this way, the shape and extent of an object may be efficiently determined. In one configuration, the information is stored, creating a map for later re-use.

The subsequent control input defined in Equation <NUM> isdetermined using batch processing or in a streaming or sequential manner. In one example, when batch processing is employed, all of the control inputs u<NUM>:t may be determined offline. In another example, when the subsequent control input is determined in a streaming or sequential manner, the subsequent control input ut+<NUM> may be determined incrementally online, given the history of inputs u<NUM>:t and the observed visual hulls C<NUM>:t.

<FIG> are diagrams illustrating a visual hull projection from an image plane according to a camera <NUM> with a pose. Referring to <FIG>, a crescent shaped object <NUM> is observed within the field of view <NUM> of a camera <NUM> with pose pk in a real world environment. The pose pk of the camera <NUM> produces a projection of the crescent shaped object's image plane silhouette <NUM>. <FIG> shows a visual hull <NUM> of the object <NUM>. In some aspects, the visual hull <NUM> indicates the potential location of the object. The visual hull <NUM> may be determined based on camera intrinsics (e.g., a type of camera lens (e.g., fish eye lens)), the pose (e.g., position and/or orientation) of the camera <NUM> and/or the object's silhouette <NUM>. The visual hull is in 3D, whereas the silhouette is in 2D.

<FIG> illustrate an exemplary joint visual hull in accordance with aspects of the present disclosure. As shown in <FIG>, a crescent shaped object <NUM> is observed in a first view of a camera (e.g., camera <NUM>) at a first pose producing an object silhouette <NUM>. The crescent shaped object is also observed in a second view of the camera at second pose producing a second object silhouette <NUM>. The object silhouettes <NUM> and <NUM>, camera intrinsics and/or corresponding camera poses may be used to respectively determine visual hulls <NUM> and <NUM>, as shown in <FIG>.

In <FIG>, a joint visual hull <NUM> is determined as the intersection of the visual hulls <NUM> and <NUM>. The joint visual hull <NUM> provides an approximation of the object shape and location in 3D space using the 2D images produced by the camera. Furthermore, in accordance with aspects of the present disclosure, visual hulls <NUM> and <NUM> may be used to determine a subsequent control input to move the camera (and/or agent) to a position to capture an image of the object such that a measure m on the joint visual hull <NUM> may be minimized.

<FIG> illustrates a method <NUM> for motion planning in accordance with aspects of the present disclosure. In block <NUM>, the process observes an object from a first pose of an agent having a controllable camera.

In block <NUM>, the process determines a subsequent control input to move the agent and/or the camera to observe the object from a subsequent pose, to reduce or even minimize an expected enclosing measure (e.g., volume) of an object (e.g., visual hull) based on visual data collected from the camera. In some aspects, the subsequent control input may be determined sequentially or using sequential processing (e.g., processing one control input at a time) or by using batch processing of potential subsequent control inputs. The batch processing may use techniques such as receding horizon control (e.g., forecast <NUM> steps into future, and then perform the next analysis) or other analysis techniques. In addition, the subsequent control input may be determined to minimize or reduce a cost to minimize the expected enclosing volume (e.g., joint visual hull). In some aspects, the cost may comprise effort, time, work, and/or energy expended in moving the agent or camera to determine the shape and extent of the object.

Furthermore, in block <NUM>, the process controls the agent and the camera based on the subsequent control input. In some aspects, the agent and the camera may be controlled to move about the object using a minimum number of control inputs.

<FIG> is a block diagram illustrating a method <NUM> of motion planning in accordance with aspects of the present disclosure. In block <NUM>, the process observes an object from a first pose of an agent having a controllable camera to produce a first 2D object silhouette or silhouette image. In block <NUM>, the process calculates a first visual hull. The first visual hull is calculated based on camera intrinsics such as the camera lens type, the pose of the camera, the first object silhouette or a combination thereof. The first visual hull is three-dimensional and may comprise a volume in which the object may be located.

In block <NUM>, the object is observed from a second pose of the agent or camera. In some aspects, the second pose may be randomly selected. A second 2D object silhouette is produced. In block <NUM>, the process calculates a second visual hull. Similar to the first visual hull, the second visual hull may be calculated based on the camera intrinsics, the second object silhouette or a combination thereof.

In block <NUM>, the process computes a joint visual hull based on the first visual hull and the second visual hull. The joint visual hull iscomputed as the intersection of the first visual hull and the second visual hull. As such, the joint visual hull may provide greater confidence that the object is located within the space defined by the intersection. Furthermore, the joint visual hull may also provide an approximation of the object shape in a 3D space.

In block <NUM>, the process determines a subsequent control input to move the agent and/or camera to a next pose so as to minimize the joint visual hull. That is, rather than moving based on a control input determined based on a random selection process or based on an incremental step process, the subsequent control input isselected so as to minimize the joint visual hull and thereby more efficiently determine the shape and extent of the object of interest.

In block <NUM>, the process evaluates the type of processing for determining the control input. If the type of processing is sequential processing, in block <NUM>, the process controls the agent and or camera to move based on the determined subsequent control input, in block <NUM>.

On the other hand, if batch processing is indicated, in block <NUM>, the process evaluates whether the desired batch size has been reached. The batch size may be arbitrarily determined according to design preference. If the desired batch size (e.g., <NUM> subsequent control inputs) has not been reached, the process returns to block <NUM> to determine a next subsequent control input. In this scenario, the camera is not actually moved to the next location at the next time step. Rather, in some aspects, a projected visual hull of the object is determined for the next pose of the camera and used to determine the next subsequent control input.

If the desired batch size has been reached, in block <NUM>, the process controls the agent and or camera to move based on the determined subsequent control input, in block <NUM>. In this scenario, the agent is moved based on the last determined control input in the batch.

In some aspects, methods <NUM> and <NUM> may be performed by the SOC <NUM> (<FIG>) or the system <NUM> (<FIG>). That is, each of the elements of methods <NUM> and <NUM> may, for example, but without limitation, be performed by the SOC <NUM> or the system <NUM> or one or more processors (e.g., CPU <NUM> and local processing unit <NUM>) and/or other components included therein.

The means may include various hardware and/or software component(s) and/or module(s), including, but not limited to, a circuit, an application specific integrated circuit (ASIC), or processor. Generally, where there are operations illustrated in the figures, those operations may have corresponding counterpart means-plus-function components with similar numbering.

Additionally, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like. Furthermore, "determining" may include resolving, selecting, choosing, establishing and the like.

The various illustrative logical blocks, modules and circuits described in connection with the present disclosure may be implemented or performed with a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller or state machine.

The steps of a method or algorithm described in connection with the present disclosure may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in any form of storage medium that is known in the art. Some examples of storage media that may be used include random access memory (RAM), read only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable disk, a CD-ROM and so forth. A storage medium may be coupled to a processor such that the processor can read information from, and write information to, the storage medium.

The functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in hardware, an example hardware configuration may comprise a processing system in a device. The network adapter may be used to implement signal processing functions. For certain aspects, a user interface (e.g., keypad, display, mouse, joystick, etc.) may also be connected to the bus.

The processor may be responsible for managing the bus and general processing, including the execution of software stored on the machine-readable media. Machine-readable media may include, by way of example, random access memory (RAM), flash memory, read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable Read-only memory (EEPROM), registers, magnetic disks, optical disks, hard drives, or any other suitable storage medium, or any combination thereof. The computer-program product may comprise packaging materials.

In a hardware implementation, the machine-readable media may be part of the processing system separate from the processor. However, as those skilled in the art will readily appreciate, the machine-readable media, or any portion thereof, may be external to the processing system. By way of example, the machine-readable media may include a transmission line, a carrier wave modulated by data, and/or a computer product separate from the device, all which may be accessed by the processor through the bus interface. Although the various components discussed may be described as having a specific location, such as a local component, they may also be configured in various ways, such as certain components being configured as part of a distributed computing system.

The processing system may be configured as a general-purpose processing system with one or more microprocessors providing the processor functionality and external memory providing at least a portion of the machine-readable media, all linked together with other supporting circuitry through an external bus architecture. Alternatively, the processing system may comprise one or more neuromorphic processors for implementing the neuron models and models of neural systems described herein. As another alternative, the processing system may be implemented with an application specific integrated circuit (ASIC) with the processor, the bus interface, the user interface, supporting circuitry, and at least a portion of the machine-readable media integrated into a single chip, or with one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic, discrete hardware components, or any other suitable circuitry, or any combination of circuits that can perform the various functionality described throughout this disclosure.

The machine-readable media may comprise a number of software modules. The software modules include instructions that, when executed by the processor, cause the processing system to perform various functions. Furthermore, it should be appreciated that aspects of the present disclosure result in improvements to the functioning of the processor, computer, machine, or other system implementing such aspects.

A storage medium may be any available medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Additionally, any connection is properly termed a computer-readable medium.

Claim 1:
A method of motion planning, comprising:
observing an object with a camera of an agent from a plurality of different poses of the camera, (<NUM>);
characterized by:
determining an expected volume of the object based on a joint visual hull, wherein the joint visual hull is calculated as an intersection of a plurality of three-dimensional, 3D, hulls determined from silhouette images of the object observed with the camera at the plurality of different poses;
determining at least one subsequent control input of a plurality of potential subsequent control inputs to move the agent and the camera to observe the object from a subsequent pose (<NUM>), using sequential or batch processing of the plurality of potential subsequent control inputs, wherein the subsequent input for movement of the camera and agent is determined from a set of possible movements by minimizing the expected volume of the object in dependence on the determined 3D visual hulls; and
controlling the agent and the camera to move to a location corresponding to at least one subsequent pose based on the determined at least one subsequent control input (<NUM>).