A method implemented by a data processing apparatus, the method comprising: obtaining a plurality of images depicting a physical environment, wherein the environment comprises a given physical object, wherein the plurality of images are sequentially captured images of the physical environment from a repeatedly changed perspective of the camera during capturing; determining whether any of the template representations in a template representation library matches any of the regions of any of the plurality of images and a position of the given object in the plurality of images can be inferred from a region determined by template matching, wherein the template representation library comprises a plurality of template representations of respective objects; in response to determining that the position of the given object in the environment cannot be inferred based on the template representation library using template matching techniques: generating a 3D reconstruction of the environment from the plurality of images of the environment; determining an estimated position of the given object by using the 3D reconstruction of the environment; generating a new template representation of the given object using the estimated position of the given object; and augmenting the template representation library with the new template representation; wherein determining the estimated position of the given object using the 3D reconstruction of the environment comprises: determining a segmentation of the environment into a plurality of segmented regions based on the 3D reconstruction of the environment, wherein each segmented region represents a respective object or a background; identifying a segmented region as the given object using knowledge about an expected shape of the given object in the environment; and determining the estimated position of the given object based on the segmented region identified as the given object. obtaining a plurality of images depicting a physical environment, wherein the environment comprises a given physical object, wherein the plurality of images are sequentially captured images of the physical environment from a repeatedly changed perspective of the camera during capturing; determining whether any of the template representations in a template representation library matches any of the regions of any of the plurality of images and a position of the given object in the plurality of images can be inferred from a region determined by template matching, wherein the template representation library comprises a plurality of template representations of respective objects; in response to determining that the position of the given object in the environment cannot be inferred based on the template representation library using template matching techniques: generating a 3D reconstruction of the environment from the plurality of images of the environment; determining an estimated position of the given object by using the 3D reconstruction of the environment; generating a new template representation of the given object using the estimated position of the given object; and augmenting the template representation library with the new template representation; wherein determining the estimated position of the given object using the 3D reconstruction of the environment comprises: determining a segmentation of the environment into a plurality of segmented regions based on the 3D reconstruction of the environment, wherein each segmented region represents a respective object or a background; identifying a segmented region as the given object using knowledge about an expected shape of the given object in the environment; and determining the estimated position of the given object based on the segmented region identified as the given object. generating a 3D reconstruction of the environment from the plurality of images of the environment; determining an estimated position of the given object by using the 3D reconstruction of the environment; generating a new template representation of the given object using the estimated position of the given object; and augmenting the template representation library with the new template representation; wherein determining the estimated position of the given object using the 3D reconstruction of the environment comprises: determining a segmentation of the environment into a plurality of segmented regions based on the 3D reconstruction of the environment, wherein each segmented region represents a respective object or a background; identifying a segmented region as the given object using knowledge about an expected shape of the given object in the environment; and determining the estimated position of the given object based on the segmented region identified as the given object. determining a segmentation of the environment into a plurality of segmented regions based on the 3D reconstruction of the environment, wherein each segmented region represents a respective object or a background; identifying a segmented region as the given object using knowledge about an expected shape of the given object in the environment; and determining the estimated position of the given object based on the segmented region identified as the given object.