Patent Description:
In computer vision and computer graphics, 3D reconstruction generally refers to the process of capturing the shape and appearance of real objects and representing the thus captured objects in a 3D model. The quality of 3D reconstruction using existing algorithms, such as structure from motion (SfM), commonly relies on properties of captured image such as field of view, scene coverage, captured context, etc. For a data set of captured images to be useful, the images should represent scenes having been captured with content overlapping from one image to the next. One reason for this is that a comparatively large overlap between individual images guarantees that they can be properly registered and stitched. The field of view should thus be wide for object mapping, as used as part of 3D reconstruction, to be successful, and hence a camera lens with comparatively short focal length should be used.

In order to build a 3D model with annotated objects in the scene, objects need to be identified in the images. This process is commonly referred to as visual object detection. Visual object detection generally refers to detection instances of semantic objects of a certain class (such as humans, buildings, or cars) in digital images and videos. For Machine Learning approaches, it becomes necessary to first define features, then using a technique, such as support vector machine (SVM), to perform the classification. Deep learning techniques are able to perform end-to-end object detection without specifically defining features, and are typically based on convolutional neural networks (CNN). CNN based object detectors are accurate but require significant amount of computational and memory resources, which is a challenge for many implementations. Visual object detection commonly requires a close view to the object that is to be detected in order to capture as many detailed features of the object as possible, which improves detection and classification of the object. The field of view should thus be narrow for visual object detection to be successful, and hence a camera lens with comparatively long focal length should be used.

Hence, object mapping and visual object detection have contradicting requirements for being successful. There is therefore a need for improved joint visual object detection and object mapping. Patent document <CIT> and the scientific paper of <NPL>, disclose related technologies.

An object of embodiments herein is to provide efficient joint visual object detection and object mapping not suffering from the issues noted above, or at least where the above issues have been mitigated or reduced.

According to a first aspect there is presented a method for joint visual object detection and object mapping to a 3D model. The method is performed by an image processing device. The method comprises obtaining a first sequence of digital images of a scene as captured by a first image capturing unit, and obtaining a second sequence of digital images of the scene as captured by a second image capturing unit. The second sequence of digital images is time-wise synchronized with the first sequence of digital images by being captured time-wise in parallel with the first sequence of digital images. The first image capturing unit has a narrower field of view than the field of view of the second image capturing unit. The first image capturing unit and the second image capturing unit have a known spatial relation. The method comprises performing joint visual object detection and object mapping to the 3D model. The joint visual object detection and object mapping to the 3D model comprises extracting a set of objects from at least the first sequences of digital images and the second sequence of digital images by performing visual object detection on both the first sequence of digital images and the second sequence of digital images. The joint visual object detection and object mapping to the 3D model comprises mapping the extracted set of objects to the 3D model in accordance with the second sequence of digital images and the known spatial relation, and thereby registering the scene to the 3D model. The joint visual object detection and object mapping to the 3D model comprises providing, as a result of how many objects are detected either in the first sequence of digital images or in the second sequence of digital images, an indication to move the image capturing units closer towards, or farther from, the scene.

According to a second aspect there is presented an image processing device for joint visual object detection and object mapping to a 3D model. The image processing device comprises processing circuitry. The processing circuitry is configured to cause the image processing device to obtain a first sequence of digital images of a scene as captured by a first image capturing unit, and to obtain a second sequence of digital images of the scene as captured by a second image capturing unit. The second sequence of digital images is time-wise synchronized with the first sequence of digital images by being captured time-wise in parallel with the first sequence of digital images. The first image capturing unit has a narrower field of view than the field of view of the second image capturing unit. The first image capturing unit and the second image capturing unit have a known spatial relation. The processing circuitry is configured to cause the image processing device to perform joint visual object detection and object mapping to the 3D model. The joint visual object detection and object mapping to the 3D model comprises extracting a set of objects from both the first sequences of digital images and the second sequence of digital images by performing visual object detection on both the first sequence of digital images and the second sequence of digital images. The joint visual object detection and object mapping to the 3D model comprises mapping the extracted set of objects to the 3D model in accordance with the second sequence of digital images and the known spatial relation, and thereby registering the scene to the 3D model. The joint visual object detection and object mapping to the 3D model comprises providing, as a result of how many objects are detected either in the first sequence of digital images or in the second sequence of digital images, an indication to move the image capturing units closer towards, or farther from, the scene.

According to a third aspect there is presented a computer program for joint visual object detection and object mapping to a 3D model, the computer program comprising computer program code which, when run on an image processing device, causes the image processing device to perform a method according to the first aspect.

Advantageously, these aspects provide efficient joint visual object detection and object mapping.

Advantageously, the provided joint visual object detection and object mapping does not suffer from the issues noted above.

These aspects increase the accuracy of both visual object detection and 3D modeling. Advantageously, these aspects reduce the user's effort and the time required for an optimal capture of the scene.

This inventive concept may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept which is defined by the appended claims to those skilled in the art.

The embodiments disclosed herein relate to mechanisms for joint visual object detection and object mapping to a 3D model. In order to obtain such mechanisms, there is provided an image processing device <NUM>, a method performed by the image processing device <NUM>, a computer program product comprising code, for example in the form of a computer program, that when run on an image processing device <NUM>, causes the image processing device <NUM> to perform the method.

As noted above, there is a need for improved joint visual object detection and object mapping.

In more detail, in joint visual object detection and object mapping applications, where objects are automatically detected and registered onto 3D model, might provide some benefits. However, existing mechanism for such joint detection and mapping cannot perform optimally when operating on one and the same sequence of digital images due to the contradicting requirements described above. The scene that is to be subjected to the joint visual object detection and object mapping has to be captured with a narrow field of view (and hence a camera lens with comparatively long focal length should be used) for successful object detection and with a wide field of view (and hence a camera lens with comparatively short focal length should be used) for successful image registration.

According to a non-limiting illustrative example of a technician inspecting an installation, this would require the technician to first get close to the objects of interest and then step away in an attempt to properly register detected object. Such approach is not only time consuming, but is also inaccurate, as for example multiple instances of the same object class could be detected in a close view, but fail to register to the model, due to loss of the surrounding context.

At least some of the herein disclosed embodiments are therefore based on the image processing device having direct access to simultaneously captured data from two image capturing units, for example as provided in a dual camera. Having two image capturing units with different fields of view solve the contradicting requirements described above. Visual data in terms of sequences of images, as simultaneously captured from two image capturing units with different fields of view might thus be used for joint visual object detection and object mapping. Access to simultaneously captured high level of details and wider background allows for improved detector performance without loss in registration of the detected objects.

<FIG> is a schematic diagram illustrating an image processing device <NUM> according to an embodiment. The image processing device <NUM> comprises a first image capturing unit 120a. The first image capturing unit 120a has a first field of view 130a. The image processing device <NUM> further comprises a second image capturing unit 120b. The second image capturing unit 120b has a second field of view 130b. The first image capturing unit 120a and the second image capturing unit 120b might be part of a dual camera <NUM>. The image capturing units 120a, 120b are configured to capture respective sequences of digital images.

It is assumed that the first image capturing unit 120a has a narrower field of view 130a than the field of view 130b of the second image capturing unit 120b. Hence, the first image capturing unit 120a is equipped with a camera lens with longer focal length than the focal length of the camera lens that the second image capturing unit 120b is equipped with. The first image capturing unit 120a and the second image capturing unit 120b have a known spatial relation <NUM>. In some aspects, the known spatial relation <NUM> causes a first sequence of digital images as captured by the first image capturing unit 120a and a second sequence of digital images as captured by the second image capturing unit 120b to, for scenes captured in the far field, be centered at the same point. In this respect, <FIG> illustrates only the first field of view 130a and the second field of view 130b as appearing in the near field. Here, the near field is defined as the parallax between a first digital image of a scene as captured by the first image capturing unit 120a and a second digital image of a scene as captured by the second image capturing unit 120b being larger than a threshold value, due to that the first image capturing unit 120a and the second image capturing unit 120b are physically separated by a non-zero distance. Conversely, the far field is defined as the parallax between the first digital image and the second digital image being smaller than the same threshold value, and thus in the far field it appears as if the first digital image and the second digital image were captured from one and the same physical location and thus appear to be centered with respect to each other.

The image processing device <NUM> further comprises processing modules in terms of an image obtainer <NUM>, a model obtainer <NUM>, and a joint object detector and object mapper <NUM>. The image obtainer <NUM> is coupled to the first image capturing unit 120a and the second image capturing unit 120b as well as to the joint object detector and object mapper <NUM>. The model obtainer <NUM> is coupled to the second image capturing unit 120b as well as to the joint object detector and object mapper <NUM>.

<FIG> is a flowchart illustrating embodiments of methods for joint visual object detection and object mapping to a 3D model. The methods are performed by the image processing device <NUM>. The methods are advantageously provided as computer programs <NUM>.

S106: The image processing device <NUM> obtains a first sequence of digital images of a scene <NUM>, <NUM> as captured by a first image capturing unit 120a, and obtains a second sequence of digital images of the scene <NUM>, <NUM> as captured by a second image capturing unit 120b. In <FIG> the sequences of digital images might be obtained by the image obtainer <NUM>.

The second sequence of digital images is time-wise synchronized with the first sequence of digital images by being captured time-wise in parallel with the first sequence of digital images. The first image capturing unit 120a and the second image capturing unit 120b have a known spatial relation <NUM> and the spatial relation between the first sequence of digital images and the second sequence of digital images is defined by this known spatial relation <NUM>. By means of this known spatial relation <NUM> it thereby appears as if the first sequence of digital images and the second sequence of digital images were captured from one and the same physical location and are centered with respect to each other. As noted above, the first image capturing unit 120a has a narrower field of view 130a than the field of view 130b of the second image capturing unit 120b.

S108: The image processing device <NUM> performs joint visual object detection and object mapping to the 3D model.

Performing joint visual object detection and object mapping to the 3D model comprises extracting, S108a, a set of objects 410a:410d, <NUM> from at least the first sequence of digital images by performing visual object detection on at least the first sequence of digital images. In <FIG> joint visual object detection and object mapping might be performed by the joint object detector and object mapper <NUM>.

Performing joint visual object detection and object mapping to the 3D model further comprises mapping, S108b, the extracted set of objects 410a:410d, <NUM> (see, <FIG> and <FIG> as referred to below) to the 3D model in accordance with the second sequence of digital images and the known spatial relation <NUM>. The scene <NUM>, <NUM> is thereby registered to the 3D model.

Visual object detection is thus run at least on frames (defining a sequence of digital images) as captured by the image capturing unit 120a with narrow field of view, whereas in parallel, object mapping is run on frames captured by the image capturing unit 120b with wide field of view to register the current view of the scene to the 3D model. Since the relations between the first image capturing unit 120a and the second image capturing unit 120b, and thus between the respective sequences of images captured by these image capturing units, is known, any object extracted from the first sequences of digital images can be directly mapped onto the 3D model.

Embodiments relating to further details of joint visual object detection and object mapping to a 3D model as performed by the image processing device <NUM> will now be disclosed.

In some aspects, the first image capturing unit 120a and the second image capturing unit 120b capture digital images using one and the same frame rate. In some embodiments, the first sequence of digital images and the second sequence of digital images are thus captured at the same frame rate. If this is not the case, resampling together with interpolation and/or extrapolation might have to be performed on at last one of the sequences of digital images.

There could be different ways for the image processing device <NUM> to perform the mapping from objects 410a:410d, <NUM> to the 3D model. In some embodiments, how to map the extracted set of objects 410a:410d, <NUM> to the 3D model is determined from the known spatial relation <NUM>. In some embodiments, mapping the extracted set of objects 410a:410d, <NUM> to the 3D model (as in S108b) involves the image processing device <NUM> to project each of the extracted objects 410a:410d, <NUM> to the scene <NUM>, <NUM> according to the known spatial relation <NUM>.

There might be different ways for the image processing device <NUM> to obtain the 3D model to which the detected objects are mapped.

In some aspects, an existing 3D model, hereinafter denoted an initial 3D model, is uploaded to the image processing device <NUM> or otherwise made obtainable by the image processing device <NUM>. In other aspects, the 3D model is constructed on the fly. In particular, according to some embodiments, the image processing device <NUM> is configured to perform (optional) steps S102 and S104:.

The 3D model is then obtained from the initial 3D model by the image processing device <NUM> registering the scene <NUM>, <NUM> in the initial 3D model. The scene <NUM>, <NUM> is registered in the initial 3D model by the initial sequence of digital images being matched to the initial 3D model.

Visual object detection is run on frames capture by both image capturing units 120a, 120b. Again, since the spatial relation between the two views is known, all detected objects in both the first sequence of digital images and the second sequence of digital images are projected on the wide view scene and to be registered onto the 3D model. That is, the visual object detection is performed on both the first sequence of digital images and the second sequence of digital images. The set of objects 410a:410d, <NUM> might then comprise objects 410a:410d, <NUM> extracted from both the first sequence of digital images and the second sequence of digital images. The approach is computationally demanding but allows object at different resolutions to be captured.

In some aspects, the ratio of the number of detected objects in different views is used to provide a guidance to the user of the image processing device <NUM> about the optimal recording position of the sequences of images. According to the invention, the image processing device <NUM> is configured to perform step S110:
S110: The image processing device <NUM> provides, as a result of how many objects 410a:410d, <NUM> are detected either in the first sequence of digital images or in the second sequence of digital images, an indication to move the image capturing units closer towards, or farther from, the scene <NUM>, <NUM>.

If no objects are detected in the first sequence of digital images this is an indication that the image capturing units 120a, 120b are too far away from the scene <NUM>, <NUM>. Hence, in some examples, the indication is to move the image capturing units 120a, 120b closer towards the scene <NUM>, <NUM> when no objects 410a:410d, <NUM> are detected in the first sequence of digital images. Further, the indication might be to move the image capturing units 120a, 120b yet closer towards the scene <NUM>, <NUM> until the number of objects 410a:410d, <NUM> detected in the first sequence of digital images decreases.

If no objects are detected in the second sequence of digital images this is an indication that the image capturing units 120a, 120b are too close to the scene <NUM>, <NUM>. Hence, in some examples, the indication is to move the image capturing units 120a, 120b farther from the scene <NUM>, <NUM> when no objects 410a:410d, <NUM> are detected in the second sequence of digital images. Further, the indication might be to move the image capturing units 120a, 120b yet farther from the scene <NUM>, <NUM> until the number of objects 410a:410d, <NUM> detected in the second sequence of digital images decreases.

Thus, assuming that the number of objects detected by the first image capturing unit 120a is x1 and that the number of objects detected by the second image capturing unit 120a is x2. Then the following conditions can be used to guide the user of the image processing device <NUM> towards the optimal recording position of the sequences of images. There are two extreme points. One of the extreme points is given by x1=<NUM>. The other of the extreme points is given by x2=<NUM>.

If x1=<NUM> this implies that the image capturing units 120a, 120b are too far away from the scene <NUM>, <NUM>. The user is thus, by means of the indication provided in S110, guided to move the image processing device <NUM> closer towards the scene <NUM>, <NUM>.

If x2=<NUM> this implies that the image capturing units 120a, 120b are too close to the scene <NUM>, <NUM>. The user is thus, by means of the indication provided in S110, guided to move the image processing device <NUM> farther away from the scene <NUM>, <NUM>.

Further, by recording whether the number of detected objects increase or decrease from after the image processing device <NUM> has been moved, the image processing device <NUM> is enabled to guide the user of the image processing device <NUM> towards the optimal recording position of the sequences of digital images where as many objects as possible are detected in the scene <NUM>, <NUM>. By selectively moving the image capturing units closer towards, or farther from, the scene <NUM>, <NUM>, an optimum might thus be reached where as many objects 410a:410d, <NUM> in the scene as possible can be detected.

Reference is now made to the flowchart of <FIG> illustrating a method for joint visual object detection and object mapping to a 3D model as performed by the image processing device <NUM> based on at least some of the above disclosed embodiments.

S201: An initial 3D model is uploaded to the image processing device <NUM> or otherwise made obtainable by the image processing device <NUM>.

S202: An initial sequence of digital images of the scene <NUM>, <NUM> is obtained from at least the second image capturing unit 120b.

S203: The current orientation and position of the image capturing units 120a, 120b, and thus of the image processing device <NUM> with respect to the 3D model is determined by the initial sequence of digital images being matched to the initial 3D model.

S204: A first sequence of digital images of the scene <NUM>, <NUM> as captured by a first image capturing unit 120a and a second sequence of digital images of the scene <NUM>, <NUM> as captured by a second image capturing unit 120b are obtained.

The image processing device <NUM> performs joint visual object detection and object mapping to the 3D model according to S205 and S206.

S205: A set of objects 410a:410d, <NUM> is extracted from at least the first sequences of images by visual object detection being performed on at least the first sequence of digital images.

S206: The extracted set of objects 410a:410d, <NUM> are mapped to the 3D model in accordance with the second sequence of digital images and the known spatial relation <NUM>. The scene <NUM>, <NUM> is thereby registered to the 3D model.

Reference is now made to <FIG> schematically illustrates the image processing device <NUM> and a scene <NUM>, where the scene comprises objects 410a, 410b, 410c, 41od. In this respect, <FIG> shows the image processing device <NUM> of <FIG> but illustrates how the first field of view 130a and the second field of view 130b appear in the far field. The first sequence of digital images and the second sequence of digital images as captured by the first image capturing unit 120a and the second image capturing unit 120b (not shown) are thus centered at the same point. According the illustrative example of <FIG>, objects 410b and 410c are fully located within the first field of view 130a and these objects are thus fully captured by the first image capturing unit 120a and are thus assumed to be detected upon visual object detection being performed based on a digital image having been captured by the first image capturing unit 120a for this first field of view 130a. Object 410a is only partially located within the first field of view 130a and might thus not be detected upon visual object detection being performed based on a digital image having been captured by the first image capturing unit 120a for this first field of view 130a. Object 410d is fully located outside the first field of view 130a and is thus not detected upon visual object detection being performed based on a digital image having been captured by the first image capturing unit 120a for this first field of view 130a. Objects 410a and 410d might be detected either upon visual object detection being performed based on a digital image having been captured by the second image capturing unit 120b for this second field of view 130b or by movement of the image processing device <NUM> such that the first field of view 130a is rotated such that objects 410a and 410d are, one at the time, fully located within the first field of view 130a.

Reference is now made to <FIG> schematically illustrates content of a digital image 500a of scene <NUM> as captured by the first image capturing unit 120a. <FIG> schematically illustrates content of a digital image 500b of the same scene <NUM> as in <FIG> but captured by the second image capturing unit 120b. The digital image 500a shown in <FIG> is used for visual object detection in a scene whereas the digital image 500b shown in <FIG> is used for object mapping to a 3D model of the scene. In <FIG> an object <NUM> in the form of a laptop computer has been detected, as indicated by its surrounding bounding box <NUM>. Since the two digital images 500a, 500b are centered with respect to each other, the object <NUM> as detected in the digital image 500a captured by the first image capturing unit 120a can be directly mapped to the digital image 500b captured by the second image capturing unit 120b.

<FIG> schematically illustrates, in terms of a number of functional units, the components of an image processing device <NUM> according to an embodiment. Processing circuitry <NUM> is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product <NUM> (as in <FIG>), e.g. in the form of a storage medium <NUM>. The processing circuitry <NUM> may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).

Particularly, the processing circuitry <NUM> is configured to cause the image processing device <NUM> to perform a set of operations, or steps, as disclosed above. For example, the storage medium <NUM> may store the set of operations, and the processing circuitry <NUM> may be configured to retrieve the set of operations from the storage medium <NUM> to cause the image processing device <NUM> to perform the set of operations.

Thus the processing circuitry <NUM> is thereby arranged to execute methods as herein disclosed. The image processing device <NUM> may further comprise a communications interface <NUM>. As such the communications interface <NUM> may comprise one or more transmitters and receivers, comprising analogue and digital components. The processing circuitry <NUM> controls the general operation of the image processing device <NUM> e.g. by sending data and control signals to the communications interface <NUM> and the storage medium <NUM>, by receiving data and reports from the communications interface <NUM>, and by retrieving data and instructions from the storage medium <NUM>. Other components, as well as the related functionality, of the image processing device <NUM> are omitted in order not to obscure the concepts presented herein.

<FIG> schematically illustrates, in terms of a number of functional modules, the components of an image processing device <NUM> according to an embodiment. The image processing device <NUM> of <FIG> comprises a number of functional modules; an obtain module 710c configured to perform step S106, an object detection and object mapping module 710d configured to perform steps S108, S108a, S108b. The image processing device <NUM> of <FIG> may further comprise a number of optional functional modules, such as any of an obtain module 710a configured to perform step S102, an obtain module 710b configured to perform step S710b, and a provide module 710e configured to perform step S110. In general terms, each functional module 710a-710e may in one embodiment be implemented only in hardware and in another embodiment with the help of software, i.e., the latter embodiment having computer program instructions stored on the storage medium <NUM> which when run on the processing circuitry makes the image processing device <NUM> perform the corresponding steps mentioned above in conjunction with <FIG>. It should also be mentioned that even though the modules correspond to parts of a computer program, they do not need to be separate modules therein, but the way in which they are implemented in software is dependent on the programming language used. Preferably, one or more or all functional modules 710a-710e may be implemented by the processing circuitry <NUM>, possibly in cooperation with the communications interface <NUM> and/or the storage medium <NUM>. The processing circuitry <NUM> may thus be configured to from the storage medium <NUM> fetch instructions as provided by a functional module 710a-710e and to execute these instructions, thereby performing any steps as disclosed herein.

The image processing device <NUM> might be a graphics processing unit (GPU), a visual processing unit (VPU) or a tensor processing unit (TPU), and/or be provided in any of: a video card, a mother board, an embedded system, a mobile phone, a vehicle, a personal computer, a workstation, or a game console.

A first portion of the instructions performed by the image processing device <NUM> may be executed in a first device, and a second portion of the of the instructions performed by the image processing device <NUM> may be executed in a second device; the herein disclosed embodiments are not limited to any particular number of devices on which the instructions performed by the image processing device <NUM> may be executed. Hence, the methods according to the herein disclosed embodiments are suitable to be performed by an image processing device <NUM> residing in a cloud computational environment. Therefore, although a single processing circuitry <NUM> is illustrated in <FIG> the processing circuitry <NUM> may be distributed among a plurality of devices, or nodes. The same applies to the functional modules 710a-710e of <FIG> and the computer program <NUM> of <FIG>.

Claim 1:
A method for joint visual object detection and object mapping to a 3D model, the method being performed by an image processing device (<NUM>), the method comprising:
obtaining (S106) a first sequence of digital images of a scene (<NUM>, <NUM>) as captured by a first image capturing unit (120a), and obtaining a second sequence of digital images of the scene (<NUM>, <NUM>) as captured by a second image capturing unit (120b),
wherein the second sequence of digital images is time-wise synchronized with the first sequence of digital images by being captured time-wise in parallel with the first sequence of digital images, wherein the first image capturing unit (120a) has a narrower field of view (130a) than the field of view (130b) of the second image capturing unit (120b), and wherein the first image capturing unit (120a) and the second image capturing unit (120b) have a known spatial relation (<NUM>); and
performing (S108) joint visual object detection and object mapping to the 3D model by:
extracting (S108a) a set of objects (410a:410d, <NUM>) from both the first sequences of digital images and the second sequence of digital images by performing visual object detection on both the first sequence of digital images and the second sequence of digital images;
mapping (S108b) the extracted set of objects (410a:410d, <NUM>) to the 3D model in accordance with the second sequence of digital images and the known spatial relation (<NUM>), and thereby registering the scene (<NUM>, <NUM>) to the 3D model; and
the method being characterised in:
providing (S110), as a result of how many objects (410a:410d, <NUM>) are detected either in the first sequence of digital images or in the second sequence of digital images, an indication to move the image capturing units closer towards, or farther from, the scene (<NUM>, <NUM>).