Patent Description:
Unsupervised object size estimation using single camera view and no reference object is a very complex and uncommon task for computer vision systems. Despite the complexity of the problem, the information provided by the object sizing is useful in a variety of domains, such as video surveillance systems, medical images, and other kind of systems. Object profiling, object identification or object state recognition are examples of higher-level tasks that may rely on the object size information. There is a number of problems arising from the decision of using a single camera-based approach. However, it also presents a number of advantages.

In real world solutions, it is not always possible to have reference objects with known size.

Supervised methods require continuous interaction between the user and the system. In contrast, the advantage of an unsupervised method is to provide output without user intervention. Thus, there is a need for systems that provide an accurate and non-reference-based object sizing information without the use of a reference object.

Common state of the art solutions size objects by means of some external sensors or the interaction of multiple cameras. Current solutions that provide object size measurement are sometimes inaccurate or require a guide object with a known size to be precise.

Additionally, object size measurement is a complex task, and there are some issues that generally appear, for instance, the irregular shape of some objects, the occlusion that can suffer when the objects of interest are in realistic environments, lack of knowledge of the object's scale or the distance of the camera sensor to the object.

One approach is described in <CIT> which is a case of a single camera, no extra sensor and no reference algorithm. In this method, movement of the camera is essential to obtain the object distance which is a drawback because there are many scenarios where movement of the camera is not possible.

Another approach based on a single camera without external sensor and no reference object is described in <CIT>. This method introduces a focus swept so as to keep away from the required movement of the camera to acquire two pictures to compute the size, but selecting two points of the image by the user is needed to measure the distance. Supervised sizing systems are generally at a disadvantage as compared to unsupervised systems.

<CIT> introduces an approximation that perform object detection combined with view detection and 3D reconstruction, however this proposal uses two cameras.

D1 <CIT> discloses a measuring method, comprising performing image collection on a measured object, and determining the size of a collected objective image of the measured object; and determining, according to the size of the objective image of the measured object, a position of the objective image and a distance between the measured object and an image collection device, a first actual length and/or a second actual length of the measured object.

D2 <CIT> discloses a method for determining an object's size based on calibration data is disclosed. The calibration data is measured by capturing images with an image sensor and a lens module, having at least one objective of the capsule camera at a plurality of object distances and/or back focal distances and deriving from the images characterizing a focus of each objective for at least one-color plane. Images of lumen walls of gastrointestinal (GI) tract are captured using the capsule camera. Object distance for at least one region in the current image is estimated based on the camera calibration data and relative sharpness of the current image in at least two-color planes. The size of the object is estimated based on the object distance estimated for one or more regions overlapping with an object image of the object and the size of the object image.

D3 <CIT> discloses a three-dimensional shape measuring method comprises projecting slit light on a measuring object 10D and receiving the reflection light; repeating operation acquiring two-dimensional images in the measuring object 10D in prescribed times by changing a focal distance; calculating the contrast of an image face in a plurality of areas A-G; extracting a high contrast area exceeding a prescribed threshold by the contrast of the image face at each two-dimensional image acquired by each focal distance; acquiring distance information of each area by performing triangulation in the high contrast area; and performing matching adjustment of a measurement range 20E so as to be included in the measurement range 20E having prescribed measurement depth(a) by each area based on the distance information.

An unsupervised method is presented for determining the size of an object captured in an image. The method includes: capturing a set of images with a camera, where each image in the set of images is captured with the camera set at a different f-number; identifying a given image in the set of images; calculating a hyperfocal distance using parameters of the camera when taking the given image; and quantifying one or more dimensions of the object as a function of the hyperfocal distance.

The method further includes: capturing a triggering image of a scene and identifying presence of an object in the triggering image. The presence of an object in the image may be identified using a background subtraction method.

In one embodiment, for each image in the set of images, a quality metric for the image is computed, where the quality metric quantifies the focus of an image based on the contrast.

The given image in the set of images can be identified by identifying a subset of images from the set of images, where each image in the subset of images has a contrast metric with a value greater than a threshold, and selecting the given image from the subset of images, where the given image has the lowest f-number amongst the images in the subset of images.

In some embodiments, the one or more dimensions of the object may be quantified using the triangle similarity theorem and the contrast metric is further defined as fuzzy entropy.

In other embodiments, the method further includes: identifying the object in the triggering image using a neural network; extracting a region of interest from each image in the set of images, where the region of interest contains the identified object; and for each image in the set of images, computing the contrast metric for the image using only the region of interest extracted from the image.

<FIG> depicts a general computer vision system <NUM> which employs a standard architecture for designing computer vision applications. The main goal of the system is to provide a complete object characterization. Also, the defined architecture facilitates the integration of the multiple algorithms required to perform object profiling in the same architecture.

Generality object detector <NUM>, from now on "generality detector, view classifier <NUM> and 3D reconstruction <NUM> are common to all the tasks and are described in detail later in the document. Depending on the application, the design follows the Categorizer branch <NUM> or the Identifier branch <NUM>. The final step in the Categorizer branch is determining the object state, for example if the object is broken or not, as well as if it has missing parts or not. The methods set forth in this disclosure lie in the Identifier branch. The Identifier branch has itself two possible subbranches: sizing related applications <NUM> and features description applications <NUM>. More specifically, this disclosure pertains to object sizing <NUM>; whereas, the features branch <NUM> returns object attributes like the color, the material, etc..

<FIG> depicts a system <NUM> for determining the size of an object captured in an image. The system <NUM> is comprised generally of: a camera <NUM>, an image preprocessor <NUM>, an object detector <NUM> and an object size calculator <NUM>. The image preprocessor <NUM> is interfaced with the camera <NUM> and interacts with the camera <NUM> to capture a set of images <NUM>, where each image in the set of images <NUM> is captured with the camera set at a different f-number. The object detector <NUM> is configured to receive the set of images <NUM>. For each image in the set of images, the object detector <NUM> processes a given image with the highest quality metric, i.e. the highest contrast / f-number ratio, amongst the images in the set of images. The object size calculator <NUM> in turn receives the given image from the object detector <NUM>. The object size calculator <NUM> uses the hyperfocal distance, calculated using parameters of the camera, and quantifies one or more dimensions of the object as a function of the hyperfocal distance.

<FIG> further illustrates an example embodiment for determining the size of an object captured in an image. At step <NUM>, a camera with a convex lens captures a frame. A frame is defined as an image of the observed scene. In this embodiment, objects to be sized appears in the frame.

At step <NUM>, the presence of objects in the frame is evaluated. A background subtraction algorithm can be used to detect foreground objects in the scene. In one embodiment, the selected algorithm is Background Subtraction using Local Singular Valued Decomposition <NUM> Binary Pattern (BSLSVDBP) combined with objectness algorithm based on Binarized Normed Gradient as described by <NPL>). This BSLSVDBP algorithm is single-parameter dependent. The parameter is the learning rate. It is selected depending on the scenario. It is possible to use other background segmentation methods or region proposal methods to detect an object presence in the scene. This is a key step for the embodiment, as it addresses the unsupervision challenge.

If objects are detected in step <NUM>, a focus swept is performed at step <NUM>. The focus is defined as the camera aperture value to change the hyperfocal distance with a focal length and circle of confusion fixed values. The camera focus range is swept. For each focus value, a frame-focus value pair is saved. The number of saved frames is set as the maximun value between a minimum value (<NUM>) and the total discrete values in the camera focus range. The resulting frame set is forwarded at <NUM> to the next step. Object identification is performed at step <NUM>. The analysis is performed only in one of the frames contained in the set. The analysis output may be: known object together with its associated category, or unknown object. In one embodiment, object identification is carried out by a neural network algorithm developed particularly for this duty. For example, a ResNet-<NUM> neural network pre-trained in Image-Net may be used as described by <NPL>) although other neural networks designed to carry out object recognition can be used in place. The model has been trained with a specific dataset composed of pre-defined (known) objects, where the objects are selected depending of the scenario where the system is implemented. For example, in an airport scenario, the selected objects are handbags, backpacks, trolleys, etc. The set of known objects may be changed by defining a new dataset to train.

Depending upon whether the object was identified, the method proceeds along one of two different paths. In the case the object is unknown, a region of interest (ROI) is extracted at <NUM> from each image in the frame set, where the region of interest contains the unknown object. For example, the region of interest is defined by the object bounding box and its mask, for example output in the step <NUM>. Note that the same area is isolated in all the frames of the frame set. The resulting set of isolated areas is used as an input for step <NUM>.

Next, the method identifies a given image from the set of isolated areas using a quality metric define as the ratio: focus metric / f - number, at step <NUM>. In one embodiment, the set of isolated areas is scanned in search of the subset of images that surpass a particular focus metric threshold. The threshold is determined empirically. The quality is analyzed in terms of focus, specifically using a contrast metric. A contrast metric is computed for each image. In one embodiment, the contrast metric is computed with the fuzzy entropy at a pixel level on its neighbor window of size w x w pixels: By default, w = <NUM>. This value can be modified depending on the size of the regions of interest. The full region of interest contrast is computed as the sum of all pixel contrast measurements. From the regions of interest with a contrast value that exceeds the threshold, we will select the one with the lowest aperture value (i.e., f-number). This selected image, along with its associated focus value, is forwarded to the next steps.

A hyperfocal distance (H) is calculated at step <NUM> using parameters of the camera when taking the image. In one embodiment, the hyperfocal distance is computed with the following equation: <MAT> where f is the focal distance, N is the F-number (i.e., f/D where D is the diameter of the aperture) and c is the circle of confusion limit. f and c are parameters associated to the camera, and N is also received from the camera when the image was captured.

One or more dimensions of the object are quantified at <NUM> as a function of the hyperfocal distance. In the example, the proper object sizing (real object size - ROS) is computed using the triangle similarity theorem. For example, the object size is computed as follow: <MAT> where SOS is the object size in the sensor, H is the hyperfocal distance and f is the focal distance.

Lastly, the size of the object is presented at step <NUM>, for example on a display of the system. For unknown objects, two dimensions are determined from the image: height of the object and width of the object.

In the case the object is known, the pixels that contain the object are isolated at step <NUM>. The process is performed in the complete set of frames. In the example embodiment, the segmentation algorithm is based on a neural network that segment the object from the background, and the output is the area of pixels that contains the object. For example, the segmentation is performed using a U-Net method as described by <NPL>am, <NUM> which is incorporated in its entirety herein. The U-Net method is pre-trained with the ImageNet dataset and refined with the known object dataset. It is applied over that frame to obtain the segmentation, however others algorithms that perform segmentation can be applied. The U-Net returns masks with the segmentation of frames. The masks are applied over the frames and the object of interest is isolated in all the frames of the frame set.

Next, the method identifies a given image, from the set of isolated areas, having the lowest f-number of those images considered "in focus" based on the focus metric threshold, at step <NUM>. This step is the same as described in relation to step <NUM>. Likewise, a hyperfocal distance (H) is calculated at step <NUM> using the parameters of the camera in the same manner as described in relation to step <NUM>.

In the example embodiment, view classification of the object is executed at step <NUM>. A neural network designed specifically for this task performs the object view classification. This network presents a VGG16 architecture with similar parameters to those described by <NPL>) although other algorithms designed for the same task or for object pose estimation can be used. The neural network was trained with a specific dataset composed of <NUM> views of each of the known objects. Although the number of views can vary, this number of views allows to sample the object every <NUM> degrees in the superior, central and inferior position (see <FIG>) covering the object completely.

A three-dimensional reconstruction of the object is carried out in step <NUM>. In one embodiment, a 3D reconstruction of the object is carried out by a neural network ResNet-<NUM> refined to identify the 3D reconstruction (see <FIG>) of the object and produce the spatial information based on the pose information obtained in the previous step. The neural network is trained with a dataset including <NUM> views for all the known objects and the 3D reconstruction models for each object. Once the 3D model is retrieved (meaning a 3D model of a known object has been identified), the 3D reconstruction is adjusted to the view of the object that has been isolated in the image selected by creating a correspondence of the points of the original object view contained in the dataset and the isolated object. For example, if the isolated area is classified as a middle-frontal view of a Rubik's Cube (see <FIG>), through this information the algorithm retrieves the respective 3D model of the Rubik's Cube, and adjust the dimensions to that of the object that has been isolated. In another example, the isolated area is classified as a middle-lateral view of a watering can (see <FIG>), the associated 3D model of the watering can is identified, and the water can dimensions are computed by adjusting them to that of the isolated object.

One or more dimensions of the object are then quantified at <NUM>. In one embodiment, the object sizing is calculated in a similar way as in step <NUM>. If some part of the object is occluded, the dimensions are predicted with the model and the known dimensions of the visible parts. For example, if the identified object is a Rubik's Cube and the view is a middle-frontal view, the depth dimension is not visible as can been seen in <FIG>. In that case, the depth is inferred by the model and the known visible dimensions. In the case of the middle-lateral watering can view, the occluded dimension is the width, the algorithm infers it. In the case of a known object, three dimensions of the object are quantified: height, width and depth of the object.

The unsupervised method for determining the size of an object is suitable for different applications. For example, the method can be used to automate the hand-luggage size control in an airport, where a camera in an unsupervised manner can automatically determine the size of hand luggage. In this case, the disclosure would facilitate a reduction of time in the boarding process.

Another application is to automatically measure the size of trucks to authorize or not the access of large vehicles to restricted roads that large vehicles cannot use.

A third example of application is the measurement of postal, UPS or Fedex packages to calculate automatically shipping costs.

The techniques described herein may be implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions that are stored on a non-transitory tangible computer readable medium. The computer programs may also include stored data. Non-limiting examples of the non-transitory tangible computer readable medium are nonvolatile memory, magnetic storage, and optical storage.

Certain aspects of the described techniques include process steps and instructions described herein in the form of an algorithm. It should be noted that the described process steps and instructions could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by real time network operating systems.

Claim 1:
An unsupervised method for determining size or dimension of an object captured in an image, comprising:
capturing, by a camera (<NUM>), a triggering image of a scene,
identifying, by a computer processor, presence of an object in the triggering image,
capturing, by the camera (<NUM>), a set of images (<NUM>), each image in the set of images (<NUM>) is captured with the camera (<NUM>) set at a different f-number and the set of images (<NUM>) is captured in response to the identification of presence of the object in the image,
wherein the method comprising furthermore
identifying, by the computer processor, a given image in the set of images (<NUM>) by identifying a subset of images from the set of images (<NUM>), where each image in the subset of images has a contrast metric with a value greater than a threshold, and selecting the given image from the subset of images, where the given image has lowest f-number amongst the images in the subset of images,
calculating, by the computer processor, a hyperfocal distance using parameters of the camera (<NUM>) when taking the given image,
identifying, by the computer processor, the object in the triggering image,
retrieving, by the computer processor, a three-dimensional model for the identified object, and
quantifying, by the computer processor, three dimensions of the object using a function of the hyperfocal distance and the retrieved three-dimensional model.