Patent Description:
A computer can model an environment by processing data characterizing the environment using digital image processing methods. Data characterizing the environment can be, for example, image data depicting the environment. The computer can use an environment model as a basis for interacting with the environment. For example, a computer can interact with an environment using robotic actuators.

The document <CIT> relates to a method for processing photos. The method is applied to a mobile terminal and includes the steps of memorizing a new photo photographed under a certain photographing mode in a root folder in the photographing mode; matching the new photo with a template in a template library of the photographing mode; and moving the new photo into a folder corresponding to the template matched with the new photo if matching succeeds. Via the method, the new photo photographed can be automatically classified and memorized in the folder corresponding to the template in the photographing mode during photographing process. The invention further discloses a system for processing the photos.

The document <CIT> describes systems and methods provided for detecting target object(s) within image(s) based on selective template matching. More specifically, the systems and methods relate to template generation, selection and matching based on the identification of regions of interest within image(s). Training images showing target object(s) can be obtained and regions of interest that are deemed more likely to contain part(s) of the target object can be identified based on the training images. Subsequent to the identification of regions of interest, templates for target object detection can be generated based thereon. Templates can be applied on testing images. Based on the test application of templates, a subgroup of templates can be selected to serve as a basis for target object detection in subsequent images.

The document <CIT>, in order to reduce the number of templates and effectively combine the templates used for recognizing an object, discloses a template creation device including an acquisition unit configured to acquire a plurality of templates from a plurality of images of different poses of a single object, or a plurality of images for a plurality of objects; a clustering unit configured to divide the plurality of templates into a plurality of groups on the basis of a similarity score; and an integration unit configured to combine the templates in a group into an integrated template, and to create a new template set from the plurality of integrated templates corresponding to each group in the plurality of groups.

The document: <NPL>; relates to discriminatively trained templates for 3D object detection.

The invention is defined in independent claims <NUM>, <NUM>, and <NUM>.

This specification describes a system implemented as computer programs on one or more computers in one or more locations that learns a template representation library.

According to a first embodiment there is provided a method according to claim <NUM>.

In some implementations, determining whether a position of the given object in the environment can be inferred based on the template representation library using template matching techniques includes, for each of multiple template representations from the template representation library, determining whether a similarity measure between the template representation and a region of the first image exceeds a threshold.

In some implementations, generating a reconstruction of the environment from multiple images of the environment includes applying stereo reconstruction techniques to the images of the environment.

In some implementations, the reconstruction of the environment includes multiple coordinates defining a three-dimensional reconstruction of the environment.

In some implementations, determining the estimated position of the given object using the reconstruction of the environment includes determining a segmentation of the environment into multiple segmented regions based on the reconstruction of the environment. A segmented region is identified as the given object. The estimated position of the given object is determined based on the segmented region identified as the given object.

In some implementations, determining a segmentation of the environment into multiple regions based on the reconstruction of the environment includes determining a watershed transformation of an image representing the reconstruction of the environment.

In some implementations, generating a new template representation of the given object using the estimated position of the given object includes cropping a region of an image of the environment defined by the estimated position of the given object.

In some implementations, the method includes physically interacting with the environment based on the estimated position of the given object determined using the reconstruction of the environment. An interaction success condition is determined. New template representations of the given object are not determined using the estimated position of the given object if the interaction does not satisfy the interaction success condition.

In some implementations, physically interacting with the environment based on the estimated position of the given object determined using the reconstruction of the environment includes attempting to manipulate the given object using a robotic actuator based on the estimated position of the given object determined using the reconstruction of the environment.

According to a second embodiment there is provided a system according to claim <NUM>.

According to a third embodiment there is provided a computer-readable storage medium according to claim <NUM>.

The system described in this specification instructs, with little to no human intervention, a template representation library that can be used by an agent (e.g., a mechanical agent) to interact with an environment. More specifically, whenever the system determines the current template representation library is inadequate to infer the position of an object of interest in the environment using template matching techniques, the system automatically generates new template representations representing the object of interest and augment the template representation library with the new template representations. In contrast, some conventional systems lack a mechanism to automatically recover when the conventional system determines the template representation library is inadequate to infer the position of an object of interest in the environment using template matching techniques. In some of these conventional systems, manual human intervention is required to generate new template representations to augment the template representation library. The system described in this specification obviates the need for manual human intervention in constructing template representation libraries. Any of the advantages above constitute an improvement in the field of computer vision technology.

This specification describes a system for building a template representation library which an agent (e.g., a robotic agent) can use to interact with a physical environment. More specifically, the agent can use the template representation library to infer the locations of objects of interest in the environment using template matching techniques, and subsequently interact with these objects of interest (e.g., by picking them up using a mechanical gripping device).

When the current template representation library is inadequate to infer the position of an object of interest in the environment, the system automatically (i.e., with little to no manual human intervention) augments the template representation library with new template representations representing the object of interest. To generate the new template representations, the system captures multiple images of the environment, generates a reconstruction of the environment (e.g., a three-dimensional (3D) reconstruction), and determines the position of the object of interest from the reconstruction of the environment. After determining the position of the object of interest from the reconstruction of the environment, the system can determine the new template representations by cropping regions which depict the object of interest from the captured images of the environment.

These features and other features are described in more detail below.

<FIG> shows an example template learning system <NUM>. The template learning system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The template learning system <NUM> is configured to learn a template representation library <NUM>. A template representation library <NUM> is a collection of image data, and the image data may be of different types, depending on the implementation used. In some implementations, the template representation library <NUM> is a collection of images which each depict a respective physical object (e.g., a computer hardware component, a machine part, or a manufacturing tool). In some implementations, the template representation library <NUM> is a collection of feature representations which are each derived from an image depicting a respective physical object. A feature representation derived from an image may include, for example, data indicating the positions of points of interest and line segments in the image. In some implementations, the template representation library <NUM> defines one or more models of common features that span multiple images depicting a respective physical object. The components of the template representation library <NUM> (which can be images, feature representations, and the like) are referred to in this specification as template representations. That is, a template representation refers to image data (e.g., an image or a feature representation derived from an image) that characterizes the appearance (e.g., shape, texture, color, reflectivity, and the like) of an object.

In some cases, each template representation in the template representation library <NUM> represents a same type of object, while in other cases, different template representations in the template representation library <NUM> represent different types of objects. Each template representation may represent a respective object while excluding a representation of the background of the object (e.g., the background depicting an area surrounding the respective object). The template representations in the template representation library <NUM> may differ from one to another based on factors including: (i) the lighting when the image used to generate the template representation was captured, (ii) the perspective (e.g., location and angle) of the camera which captured the image used to generate the template representation, (iii) the resolution of the image used to generate the template representation, (iv) the color balance of the image used to generate the template representation, and (v) the object depicted in the image used to generate the template representation, amongst others.

An agent <NUM> can use the template representation library <NUM> to determine the position of a physical object of interest in a physical environment <NUM>. Determining the position of the object of interest in the environment <NUM> can allow the agent to physically interact with the object of interest. For example, the environment <NUM> may be a warehouse environment and the agent <NUM> may be a robotic agent interacting with the environment by picking up objects of interest and moving them to different locations in the environment <NUM>. As another example, the environment <NUM> may be a manufacturing environment and the agent <NUM> may be a robotic agent interacting with the environment by assembling objects of interest into manufactured products on an assembly line.

To determine the position of the object of interest, the system <NUM> captures one or more images <NUM> of the environment <NUM>. The system <NUM> sequentially captures the images <NUM> of the environment <NUM> by repeatedly changing the perspective of the camera capturing the images <NUM>. For example, the system <NUM> may sequentially capture the images <NUM> of the environment <NUM> by moving the camera capturing the images <NUM> in a predetermined sequence of different perspectives (e.g., in a spiral shape).

For each image <NUM> of the environment <NUM>, the system processes the image <NUM> using a template matching engine <NUM> to determine whether the position of the object of interest in the image <NUM> can be inferred based on the template representation library <NUM> using template matching techniques. That is, the template matching engine <NUM> uses template matching techniques to determine whether any of the template representations in the template representation library <NUM> "match" any of the regions of any of the images <NUM>. More specifically, the template matching engine <NUM> determines whether a similarity measure (which can be represented as a numerical value) between any of the template representations from the template representation library <NUM> and any of the regions of the images <NUM> exceeds a given threshold. As an illustrative example, the template matching engine <NUM> may determine that the example template representation image <NUM> (which depicts a rectangular object) matches the region <NUM> of the image <NUM> (which also depicts a rectangular object). As will be described in more detail with reference to <FIG>, the template matching engine <NUM> can use any appropriate template matching technique, for example, an interest point template matching technique, a cross correlation template matching technique, a sum-of-absolute-differences template matching technique, or a combination thereof.

When the template matching engine <NUM> determines that the position of the object of interest in an image <NUM> can be inferred based on the template representation library <NUM> using template matching techniques, the template matching engine <NUM> outputs data defining the position <NUM> of the object of interest in the image <NUM>. An advantage of successfully inferring the position of an objection based on the template representation library <NUM> using template matching techniques is that the procedure to determine a matching template is more efficient than the procedure of determining the position through a reconstruction of the environment as will be explained later. In that sense, the template representation library <NUM> can be understood as a library of observations, possibly previously encountered, that enables the determination the position of an object efficiently without reconstruction of the environment. For example, the data defining the position <NUM> of the object of interest in an image <NUM> may include coordinates of the center of the object of interest, coordinates defining a long- and short- axis of the object of interest, or both. The agent <NUM> can use the determined position <NUM> of the object of interest to physically interact with the object of interest. For example, the agent <NUM> can manipulate the object of interest (e.g., by picking it up) using a robotic actuator (e.g., a mechanical gripping device).

In some cases, the system <NUM> determines that the current template representation library <NUM> is inadequate to infer the position <NUM> of the object of interest using template matching techniques. That is, the template matching engine <NUM> determines that none of the template representations in the template representation library <NUM> match any of the regions of any of the captured images <NUM> of the environment. The current template representation library <NUM> may be inadequate because, for example, the object of interest includes features (e.g., shape, color, ports, accessories, and the like) that are not included in the objects represented by any of the current template representations (i.e., template representations currently included in the template representation library <NUM>). As another example, the current template representation library <NUM> may be inadequate because the images <NUM> of the environment depict the object of interest from perspectives which are different from those of the images used to generate the current template representations.

When the system <NUM> determines that the current template representation library <NUM> is inadequate to infer the position <NUM> of the object of interest using template matching techniques, the system <NUM> automatically augments the template representation library <NUM> with new template representations <NUM> while requiring little manual human intervention, or even no manual human intervention. The new template representations <NUM> enhance the template representation library <NUM> since they characterize the object of interest differently than the current template representations. For example, the new template representations <NUM> may be images which depict the object of the interest from different perspectives than the current template representations. As another example, the object of interest represented by the new template representations <NUM> may be a type of object which is not represented by any of the current template representations. After augmenting the template representation library <NUM> with the new template representations <NUM>, the template matching engine <NUM> can use the augmented template representation library <NUM> to infer the positions of other objects of interest in the environment <NUM> in the future.

To determine the new template representations <NUM>, the system <NUM> provides the images <NUM> of the environment <NUM> to a reconstruction engine <NUM>. The reconstruction engine <NUM> is configured to process the images <NUM> to generate a reconstruction <NUM> of the environment <NUM> which characterizes a geometry of the environment <NUM>. For example, the reconstruction <NUM> of the environment <NUM> may characterize a three-dimensional (3D) structure of the environment <NUM> by multiple 3D coordinates (e.g., coordinates with x, y, and z components) defining various positions on surfaces in the environment <NUM>. The reconstruction engine <NUM> may generate the reconstruction <NUM> of the environment <NUM> using any appropriate reconstruction technique, for example, stereo reconstruction techniques.

The system <NUM> provides the reconstruction <NUM> of the environment <NUM> to a localization engine <NUM> which is configured to process the reconstruction <NUM> to determine an estimated position <NUM> of the object of interest in the environment <NUM>. For example, as will be described in more detail with reference to <FIG>, the localization engine <NUM> determines a segmentation of the environment <NUM> into multiple different segmented regions, where each segmented region represents a respective object or a background area. After determining the segmentation of the environment <NUM>, the localization engine <NUM> identifies one of the segmented regions to be the object of interest using, for example, prior knowledge about the expected shape and the expected position of the object of interest. For example, the expected shape of the object of interest may be approximately rectangular and the expected position of the object of interest may be approximately in the "center" of the environment (e.g., in some frame of reference of the environment). In this example, the localization engine <NUM> identifies the segmented region which most closely conforms with the expected shape of the object of interest, and alternatively but not covered by the present invention, the expected position of the object of interest, or both, as the object of interest.

After identifying a segmented region of the environment <NUM> as the object of interest, the localization engine determines the estimated position <NUM> of the object of interest using the segmented region (as will be described in more detail with reference to <FIG>). The estimated position <NUM> of the object of interest determined by the localization engine <NUM> may be represented in any appropriate numerical format, and may be expressed with reference to any predetermined frame of reference of the environment. For example, the estimated position <NUM> of the object of interest may be defined by, for example, coordinates of the center of the object of interest, coordinates defining a long- and short- axis of the object of interest, or both.

After determining the estimated position <NUM><NUM> of the object of interest from the reconstruction <NUM> of the environment, the system <NUM> generates the new template representations <NUM> using the estimated position <NUM> of the object of interest. More specifically, the system <NUM> can generate the new template representations <NUM> by determining respective regions of the images <NUM> of the environment which, according to the estimated position <NUM>, depict the object of interest. The system <NUM> can crop these respective regions in the images <NUM> of the environment which depict the object of interest and determine new template representations <NUM> from the cropped image regions. For example, the system <NUM> can determine the new template representations <NUM> to be the cropped image regions. As another example, the system <NUM> can determine the new template representations <NUM> to be feature representations derived from the cropped image regions.

Optionally, prior to generating the new template representations <NUM>, the agent <NUM> can attempt to physically interact with the environment <NUM> based on the estimated position <NUM> of the object of interest determined by the localization engine <NUM> from the reconstruction <NUM> of the environment <NUM>. For example, the agent <NUM> can attempt to manipulate the object of interest (e.g., by picking it up) using a robotic actuator (e.g., a mechanical gripping device) based on the estimated position <NUM> of the object of interest. The system <NUM> may evaluate the success of the attempted interaction (e.g., by determining whether the agent successfully picked up the object of interest), and may refrain from generating the new template representations <NUM> using the estimated position <NUM> if the interaction is determined to be unsuccessful. In this manner, the system <NUM> can avoid augmenting the template representation library <NUM> with erroneous new template representations when the system <NUM> is unable to accurately estimate the position of the object of interest from the reconstruction <NUM> of the environment <NUM>.

By repeatedly augmenting the template representation library <NUM> with new template representations <NUM>, the system <NUM> can progressively construct a comprehensive template representation library <NUM>. In some cases, the template representation library <NUM> may initially be empty, in which case each template representation eventually included in the template representation library <NUM> was at one point a new template representation <NUM> generated by the system <NUM>. In other cases, the template representation library <NUM> may be initialized with a set of multiple default template representations (e.g., manually acquired images).

<FIG> is a flow diagram of an example process for augmenting a template representation library with a new template representation. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a template learning system, e.g., the template learning system <NUM> of <FIG>, appropriately programmed in accordance with this specification, can perform the process <NUM>.

The system obtains one or more images depicting the physical environment (<NUM>). The environment may be, for example, a warehouse environment or a manufacturing environment. The physical environment includes a physical object of interest (e.g., a computer hardware component, a machine part, or a manufacturing tool). The images may be represented in any appropriate format, for example, as grayscale images or color images (e.g., red-green-blue (RGB) images). The system may sequentially capture the images of the environment by repeatedly changing the perspective of the camera capturing the images. For example, the system may sequentially capture the images of the environment by moving the camera capturing the images in a predetermined sequence of different perspectives (e.g., in a spiral shape).

For each of the images of the environment, the system determines whether the position of the object of interest in the image can be inferred based on the template representation library using template matching techniques (i.e., whether a template matching condition is satisfied) (<NUM>). For a given image of the environment, the system determines that the position of the object of interest in the image can be inferred based on the template representation library if any of the template representations in the template representation library match any of the regions of the image of the environment. More specifically, the system determines that the position of the object of interest in an image of the environment can be inferred based on the template representation library if a similarity measure between any of the template representations and any of the regions of the image of the environment exceeds a given threshold.

The system can use any appropriate template matching technique, for example, an interest point template matching technique, a cross correlation template matching technique, a sum-of-absolute-differences template matching technique, or a combination thereof. In a particular example, the template representations may be images and the system can apply an interest point template matching technique. In this example, the system processes a template representation and an image of the environment to determine respective interest points in each image. An interest point refers to a coordinate defining a location of a corner, a blob, or any other distinctive image feature. The system can determine that the template representation matches a given region of the image of the environment if applying an affine transformation to the interest points of the template representation cause them to align (either approximately or exactly) with at least a threshold number of interest points of the image in the given region.

If the system determines the position of the object of interest can be inferred from the images of the environment based on the template representation library using template matching techniques, the system maintains the current template representation library (<NUM>). The system can provide the position of the object of interest to the agent, which can subsequently manipulate the object of interest (e.g., by picking it up) using a robotic actuator (e.g., a mechanical gripping device). The system can represent the position of the object of interest by, for example, the coordinates of the center of the object of interest, coordinates defining a long- and short- axis of the object of interest, or both. Optionally, rather than maintaining the current template representation library in response to determining that the position of the object of interest can be inferred from the images of the environment based on the template representation library, the system can augment the template representation library with new template representations determined from the obtained images of the environment. In this manner, the system can enhance the robustness of the template representation library even when the template representation library is adequate to infer the position of the object of interest.

If the system determines the position of the object of interest cannot be inferred from the images of the environment based on the template representation library using template matching techniques, the system generates a reconstruction of the environment which characterizes a geometry of the environment (<NUM>). For example, the reconstruction of the environment may characterize a 3D structure of the environment by a plurality of 3D coordinates (e.g., coordinates with x, y, and z components) defining various positions on surfaces in the environment. The system may generate the reconstruction of the environment using any appropriate reconstruction technique. In a particular example, the system may generate the reconstruction of the environment using stereo reconstruction techniques. Stereo reconstruction techniques can process a pair of images of the environment taken from different viewpoints and "triangulate" respective coordinates defining various positions on surfaces in the environment.

The system determines an estimated position of the object of interest using the reconstruction of the environment (<NUM>). As will be described in more detail with reference to <FIG>, the system determines the position of the object of interest by determining a segmentation of the environment into multiple different segmented regions. The system identifies one of the segmented regions as the object of interest using prior knowledge about the expected shape and, alternatively and not covered by the present invention, expected position of the object of interest in the environment. After identifying a segmented region as the object of interest, the system can determine the estimated position of the object of interest from the segmented region.

Optionally, the agent can attempt to physically interact with the environment based on the estimated position of the object of interest (e.g., as determined in <NUM>) (<NUM>). For example, the agent can attempt to manipulate the object of interest (e.g., by picking it up) using a robotic actuator (e.g., a mechanical gripping device) based on the estimated position of the object of interest.

The system may evaluate the success of the attempted interaction by determining whether an interaction success condition is satisfied (<NUM>). For example, the system may determine the interaction success condition is satisfied if the agent successfully picks up the object of the interest (e.g., using a mechanical gripping device). In response to determining the interaction success condition is satisfied, the system infers that the estimated position of the object (e.g., as determined in <NUM>) accurately localizes the object, and thereafter generate new template representations representing the object of interest using the estimated position of the object of interest (<NUM>).

To generate a new template representation representing the object of interest using the estimated position of the object of interest, the system determines e a region of an image of the environment which, according to the estimated position of the object of interest, depicts the object of interest. After determining a region of an image of the environment which depicts the object of interest using the estimated position of the object of interest, the system can generate a new template representation by cropping the determined region from the image of the environment. The system can thereafter generate the new template representation from the cropped image region. For example, the system can generate the new template representation by determining the new template representation to be the cropped image region. As another example, the system can generate the new template representation by deriving a feature representation of the cropped image region (e.g., including data defining points of interest and line segments in the cropped image region). After generating the new template representations, the system can augment the template representation library with the new template representations (<NUM>).

In response to determining the interaction success condition is not satisfied (e.g., because the agent failed to pick up the object of interest), the system determines that the estimated position of the object of interest (e.g., as determined in <NUM>) may be inaccurate. The system can return to step <NUM> and repeat the preceding steps to obtain a different (and ideally, more accurate) estimate of the position of the object of interest.

<FIG> is a flow diagram of an example process for determining an estimated position of an object of interest from a reconstruction of the environment. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a template learning system, e.g., the template learning system <NUM> of <FIG>, appropriately programmed in accordance with this specification, can perform the process <NUM>.

The system uses the reconstruction of the environment (e.g., as determined in <NUM>) to determine a segmentation of the environment into multiple different regions (<NUM>). Each segmented region represents a respective object or a background area of the environment. For example, if the environment is a manufacturing environment including an assembly line conveyor belt, then the segmented regions may correspond to portions of the conveyor belt and various objects sitting on the conveyor belt. The system may determine the segmentation of the environment using any appropriate segmentation technique. For example, the system may apply a watershed transformation to an image representing the reconstruction of the environment. Applying a watershed transformation to an image generates a segmentation of the image into different regions which are separated by edges (i.e., areas of high image intensity gradient magnitude).

The system identifies a segmented region of the environment as the object of interest (<NUM>). To identify a segmented region as the object of interest, the system relies on prior knowledge about the expected shape and, alternatively and not covered by the present invention, the expected position of the object of interest in the environment. For example, the expected shape of the object of interest may be approximately rectangular and the expected position of the object of interest may be approximately in the "center" of the environment (e.g., based on some frame of reference of the environment). In this example, the system may identify a segmented region which most closely confirms with the expected shape of the object of interest, the expected position of the object of interest, or both, as the object of interest.

The system determines the estimated position of the object of interest based on the segmented region of the environment identified as the object of interest (<NUM>). For example, if the estimated position of the object of interest is defined by the coordinates of the center of the object of interest, the system can determine the estimated position of the object of interest by computing the center of mass of the segmented region identified as the object of interest. As another example, if the estimated position of the object of interest is additionally defined by coordinates defining a long- and short- axis of the object of interest, the system can determine the estimated position of the object of interest by computing the long- and short- axes of the segmented region identified as the object of interest.

The subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

The subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components.

While operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results.

Claim 1:
A method (<NUM>) implemented by a data processing apparatus, the method comprising:
obtaining (<NUM>) a plurality of images depicting a physical environment, wherein the environment comprises a given physical object, wherein the plurality of images are sequentially captured images of the physical environment from a repeatedly changed perspective of the camera during capturing;
determining (<NUM>) whether any of the template representations in a template representation library matches any of the regions of any of the plurality of images and a position of the given object in the plurality of images can be inferred from a region determined by template matching, wherein the template representation library comprises a plurality of template representations of respective objects;
in response to determining that the position of the given object in the environment cannot be inferred based on the template representation library using template matching techniques:
generating (<NUM>) a 3D reconstruction of the environment from the plurality of images of the environment;
determining (<NUM>) an estimated position of the given object by using the 3D reconstruction of the environment;
generating (<NUM>) a new template representation of the given object using the estimated position of the given object; and
augmenting (<NUM>) the template representation library with the new template representation;
wherein determining the estimated position of the given object using the 3D reconstruction of the environment comprises:
determining (<NUM>) a segmentation of the environment into a plurality of segmented regions based on the 3D reconstruction of the environment, wherein each segmented region represents a respective object or a background;
identifying (<NUM>) a segmented region as the given object using knowledge about an expected shape of the given object in the environment; and
determining (<NUM>) the estimated position of the given object based on the segmented region identified as the given object.