Patent ID: 12223667

DETAILED DESCRIPTION

The inventive concept will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the inventive concept are shown. This inventive concept may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept to those skilled in the art. Like numbers refer to like elements throughout the description. Any step or feature illustrated by dashed lines should be regarded as optional.

The embodiments disclosed herein relate to mechanisms for joint visual object detection and object mapping to a 3D model. In order to obtain such mechanisms, there is provided an image processing device100, a method performed by the image processing device100, a computer program product comprising code, for example in the form of a computer program, that when run on an image processing device100, causes the image processing device100to perform the method.

As noted above, there is a need for improved joint visual object detection and object mapping.

In more detail, in joint visual object detection and object mapping applications, where objects are automatically detected and registered onto 3D model, might provide some benefits. However, existing mechanism for such joint detection and mapping cannot perform optimally when operating on one and the same sequence of digital images due to the contradicting requirements described above. The scene that is to be subjected to the joint visual object detection and object mapping has to be captured with a narrow field of view (and hence a camera lens with comparatively long focal length should be used) for successful object detection and with a wide field of view (and hence a camera lens with comparatively short focal length should be used) for successful image registration.

According to a non-limiting illustrative example of a technician inspecting an installation, this would require the technician to first get close to the objects of interest and then step away in an attempt to properly register detected object. Such approach is not only time consuming, but is also inaccurate, as for example multiple instances of the same object class could be detected in a close view, but fail to register to the model, due to loss of the surrounding context.

At least some of the herein disclosed embodiments are therefore based on the image processing device having direct access to simultaneously captured data from two image capturing units, for example as provided in a dual camera. Having two image capturing units with different fields of view solve the contradicting requirements described above. Visual data in terms of sequences of images, as simultaneously captured from two image capturing units with different fields of view might thus be used for joint visual object detection and object mapping. Access to simultaneously captured high level of details and wider background allows for improved detector performance without loss in registration of the detected objects.

FIG.1is a schematic diagram illustrating an image processing device100according to an embodiment. The image processing device100comprises a first image capturing unit120a. The first image capturing unit120ahas a first field of view130a. The image processing device100further comprises a second image capturing unit120b. The second image capturing unit120bhas a second field of view130b. The first image capturing unit120aand the second image capturing unit120bmight be part of a dual camera110. The image capturing units120a,120bare configured to capture respective sequences of digital images.

It is assumed that the first image capturing unit120ahas a narrower field of view130athan the field of view130bof the second image capturing unit120b. Hence, the first image capturing unit120ais equipped with a camera lens with longer focal length than the focal length of the camera lens that the second image capturing unit120bis equipped with. The first image capturing unit120aand the second image capturing unit120bhave a known spatial relation170. In some aspects, the known spatial relation170causes a first sequence of digital images as captured by the first image capturing unit120aand a second sequence of digital images as captured by the second image capturing unit120bto, for scenes captured in the far field, be centered at the same point. In this respect,FIG.1illustrates only the first field of view130aand the second field of view130bas appearing in the near field. Here, the near field is defined as the parallax between a first digital image of a scene as captured by the first image capturing unit120aand a second digital image of a scene as captured by the second image capturing unit120bbeing larger than a threshold value, due to that the first image capturing unit120aand the second image capturing unit120bare physically separated by a non-zero distance. Conversely, the far field is defined as the parallax between the first digital image and the second digital image being smaller than the same threshold value, and thus in the far field it appears as if the first digital image and the second digital image were captured from one and the same physical location and thus appear to be centered with respect to each other.

The image processing device100further comprises processing modules in terms of an image obtainer140, a model obtainer150, and a joint object detector and object mapper160. The image obtainer140is coupled to the first image capturing unit120aand the second image capturing unit120bas well as to the joint object detector and object mapper160. The model obtainer150is coupled to the second image capturing unit120bas well as to the joint object detector and object mapper160.

FIG.2is a flowchart illustrating embodiments of methods for joint visual object detection and object mapping to a 3D model. The methods are performed by the image processing device100. The methods are advantageously provided as computer programs820.

S106: The image processing device100obtains a first sequence of digital images of a scene400,510as captured by a first image capturing unit120a, and obtains a second sequence of digital images of the scene400,510as captured by a second image capturing unit120b. InFIG.1the sequences of digital images might be obtained by the image obtainer140.

The second sequence of digital images is time-wise synchronized with the first sequence of digital images by being captured time-wise in parallel with the first sequence of digital images. The first image capturing unit120aand the second image capturing unit120bhave a known spatial relation170and the spatial relation between the first sequence of digital images and the second sequence of digital images is defined by this known spatial relation170. By means of this known spatial relation170it thereby appears as if the first sequence of digital images and the second sequence of digital images were captured from one and the same physical location and are centered with respect to each other. As noted above, the first image capturing unit120ahas a narrower field of view130athan the field of view130bof the second image capturing unit120b.

S108: The image processing device100performs joint visual object detection and object mapping to the 3D model.

Performing joint visual object detection and object mapping to the 3D model comprises extracting, S108a, a set of objects410a:410d,520from at least the first sequence of digital images by performing visual object detection on at least the first sequence of digital images. InFIG.1joint visual object detection and object mapping might be performed by the joint object detector and object mapper160.

Performing joint visual object detection and object mapping to the 3D model further comprises mapping, S108b, the extracted set of objects410a:410d,520(see,FIGS.4and5as referred to below) to the 3D model in accordance with the second sequence of digital images and the known spatial relation170. The scene400,510is thereby registered to the 3D model.

Visual object detection is thus run at least on frames (defining a sequence of digital images) as captured by the image capturing unit120awith narrow field of view, whereas in parallel, object mapping is run on frames captured by the image capturing unit120bwith wide field of view to register the current view of the scene to the 3D model. Since the relations between the first image capturing unit120aand the second image capturing unit120b, and thus between the respective sequences of images captured by these image capturing units, is known, any object extracted from the first sequences of digital images can be directly mapped onto the 3D model.

Embodiments relating to further details of joint visual object detection and object mapping to a 3D model as performed by the image processing device100will now be disclosed.

In some aspects, the first image capturing unit120aand the second image capturing unit120bcapture digital images using one and the same frame rate. In some embodiments, the first sequence of digital images and the second sequence of digital images are thus captured at the same frame rate. If this is not the case, resampling together with interpolation and/or extrapolation might have to be performed on at last one of the sequences of digital images.

There could be different ways for the image processing device100to perform the mapping from objects410a:410d,520to the 3D model. In some embodiments, how to map the extracted set of objects410a:410d,520to the 3D model is determined from the known spatial relation170. In some embodiments, mapping the extracted set of objects410a:410d,520to the 3D model (as in S108b) involves the image processing device100to project each of the extracted objects410a:410d,520to the scene400,510according to the known spatial relation170.

There might be different ways for the image processing device100to obtain the 3D model to which the detected objects are mapped.

In some aspects, an existing 3D model, hereinafter denoted an initial 3D model, is uploaded to the image processing device100or otherwise made obtainable by the image processing device100. In other aspects, the 3D model is constructed on the fly.

In particular, according to some embodiments, the image processing device100is configured to perform (optional) steps S102and S104:

S102: The image processing device100obtains an initial 3D model of the scene400,510. InFIG.1the initial 3D model of the scene400,510might be obtained by the model obtainer150.

S104: The image processing device100obtains an initial sequence of digital images from at least the second image capturing unit120b.

The 3D model is then obtained from the initial 3D model by the image processing device100registering the scene400,510in the initial 3D model. The scene400,510is registered in the initial 3D model by the initial sequence of digital images being matched to the initial 3D model.

In some aspects, visual object detection is run on frames capture by both image capturing units120a,120b. Again, since the spatial relation between the two views is known, all detected objects in both the first sequence of digital images and the second sequence of digital images are projected on the wide view scene and to be registered onto the 3D model. That is, according to an embodiment, the visual object detection is performed on both the first sequence of digital images and the second sequence of digital images. The set of objects410a:410d,520might then comprise objects410a:410d,520extracted from both the first sequence of digital images and the second sequence of digital images. The approach is computationally demanding but allows object at different resolutions to be captured.

In some aspects, the ratio of the number of detected objects in different views is used to provide a guidance to the user of the image processing device100about the optimal recording position of the sequences of images. In particular, according to some embodiments, the image processing device100is configured to perform (optional) step S110:

S110: The image processing device100provides, as a result of how many objects410a:410d,520are detected either in the first sequence of digital images or in the second sequence of digital images, an indication to move the image capturing units closer towards, or farther from, the scene400,510.

If no objects are detected in the first sequence of digital images this is an indication that the image capturing units120a,120bare too far away from the scene400,510. Hence, in some examples, the indication is to move the image capturing units120a,120bcloser towards the scene400,510when no objects410a:410d,520are detected in the first sequence of digital images. Further, the indication might be to move the image capturing units120a,120byet closer towards the scene400,510until the number of objects410a:410d,520detected in the first sequence of digital images decreases.

If no objects are detected in the second sequence of digital images this is an indication that the image capturing units120a,120bare too close to the scene400,510. Hence, in some examples, the indication is to move the image capturing units120a,120bfarther from the scene400,510when no objects410a:410d,520are detected in the second sequence of digital images. Further, the indication might be to move the image capturing units120a,120byet farther from the scene400,510until the number of objects410a:410d,520detected in the second sequence of digital images decreases.

Thus, assuming that the number of objects detected by the first image capturing unit120ais x1and that the number of objects detected by the second image capturing unit120ais x2. Then the following conditions can be used to guide the user of the image processing device100towards the optimal recording position of the sequences of images. There are two extreme points. One of the extreme points is given by x1=0. The other of the extreme points is given by x2=0.

If x1=0 this implies that the image capturing units120a,120bare too far away from the scene400,510. The user is thus, by means of the indication provided in S110, guided to move the image processing device100closer towards the scene400,510.

If x2=0 this implies that the image capturing units120a,120bare too close to the scene400,510. The user is thus, by means of the indication provided in S110, guided to move the image processing device100farther away from the scene400,510.

Further, by recording whether the number of detected objects increase or decrease from after the image processing device100has been moved, the image processing device100is enabled to guide the user of the image processing device100towards the optimal recording position of the sequences of digital images where as many objects as possible are detected in the scene400,510. By selectively moving the image capturing units closer towards, or farther from, the scene400,510, an optimum might thus be reached where as many objects410a:410d,520in the scene as possible can be detected.

Reference is now made to the flowchart ofFIG.3illustrating a method for joint visual object detection and object mapping to a 3D model as performed by the image processing device100based on at least some of the above disclosed embodiments.

S201: An initial 3D model is uploaded to the image processing device100or otherwise made obtainable by the image processing device100.

S202: An initial sequence of digital images of the scene400,510is obtained from at least the second image capturing unit120b.

S203: The current orientation and position of the image capturing units120a,120b, and thus of the image processing device100with respect to the 3D model is determined by the initial sequence of digital images being matched to the initial 3D model.

S204: A first sequence of digital images of the scene400,510as captured by a first image capturing unit120aand a second sequence of digital images of the scene400,510as captured by a second image capturing unit120bare obtained.

The image processing device100performs joint visual object detection and object mapping to the 3D model according to S205and S206.

S205: A set of objects410a:410d,520is extracted from at least the first sequences of images by visual object detection being performed on at least the first sequence of digital images.

S206: The extracted set of objects410a:410d,520are mapped to the 3D model in accordance with the second sequence of digital images and the known spatial relation170. The scene400,510is thereby registered to the 3D model.

Reference is now made toFIG.4.FIG.4schematically illustrates the image processing device100and a scene400, where the scene comprises objects410a,410b,410c,410d. In this respect,FIG.4shows the image processing device100ofFIG.1but illustrates how the first field of view130aand the second field of view130bappear in the far field. The first sequence of digital images and the second sequence of digital images as captured by the first image capturing unit120aand the second image capturing unit120b(not shown) are thus centered at the same point. According the illustrative example ofFIG.4, objects410band410care fully located within the first field of view130aand these objects are thus fully captured by the first image capturing unit120aand are thus assumed to be detected upon visual object detection being performed based on a digital image having been captured by the first image capturing unit120afor this first field of view130a. Object410ais only partially located within the first field of view130aand might thus not be detected upon visual object detection being performed based on a digital image having been captured by the first image capturing unit120afor this first field of view130a. Object410dis fully located outside the first field of view130aand is thus not detected upon visual object detection being performed based on a digital image having been captured by the first image capturing unit120afor this first field of view130a. Objects410aand410dmight be detected either upon visual object detection being performed based on a digital image having been captured by the second image capturing unit120bfor this second field of view130bor by movement of the image processing device100such that the first field of view130ais rotated such that objects410aand410dare, one at the time, fully located within the first field of view130a.

Reference is now made toFIG.5.FIG.5(a)schematically illustrates content of a digital image500aof scene510as captured by the first image capturing unit120a.FIG.5(b)schematically illustrates content of a digital image500bof the same scene510as inFIG.5(a)but captured by the second image capturing unit120b. The digital image500ashown inFIG.5(a)is used for visual object detection in a scene whereas the digital image500bshown inFIG.5(b)is used for object mapping to a 3D model of the scene. InFIG.5(a)an object520in the form of a laptop computer has been detected, as indicated by its surrounding bounding box530. Since the two digital images500a,500bare centered with respect to each other, the object520as detected in the digital image500acaptured by the first image capturing unit120acan be directly mapped to the digital image500bcaptured by the second image capturing unit120b.

FIG.6schematically illustrates, in terms of a number of functional units, the components of an image processing device100according to an embodiment. Processing circuitry610is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product810(as inFIG.8), e.g. in the form of a storage medium630. The processing circuitry610may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).

Particularly, the processing circuitry610is configured to cause the image processing device100to perform a set of operations, or steps, as disclosed above. For example, the storage medium630may store the set of operations, and the processing circuitry610may be configured to retrieve the set of operations from the storage medium630to cause the image processing device100to perform the set of operations. The set of operations may be provided as a set of executable instructions.

Thus the processing circuitry610is thereby arranged to execute methods as herein disclosed. The storage medium630may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory. The image processing device100may further comprise a communications interface620. As such the communications interface620may comprise one or more transmitters and receivers, comprising analogue and digital components. The processing circuitry610controls the general operation of the image processing device100e.g. by sending data and control signals to the communications interface620and the storage medium630, by receiving data and reports from the communications interface620, and by retrieving data and instructions from the storage medium630. Other components, as well as the related functionality, of the image processing device100are omitted in order not to obscure the concepts presented herein.

FIG.7schematically illustrates, in terms of a number of functional modules, the components of an image processing device100according to an embodiment. The image processing device100ofFIG.7comprises a number of functional modules; an obtain module710cconfigured to perform step S106, an object detection and object mapping module710dconfigured to perform steps S108, S108a, S108b. The image processing device100ofFIG.7may further comprise a number of optional functional modules, such as any of an obtain module710aconfigured to perform step S102, an obtain module710bconfigured to perform step S710b, and a provide module710econfigured to perform step S110. In general terms, each functional module710a-710emay in one embodiment be implemented only in hardware and in another embodiment with the help of software, i.e., the latter embodiment having computer program instructions stored on the storage medium630which when run on the processing circuitry makes the image processing device100perform the corresponding steps mentioned above in conjunction withFIG.7. It should also be mentioned that even though the modules correspond to parts of a computer program, they do not need to be separate modules therein, but the way in which they are implemented in software is dependent on the programming language used. Preferably, one or more or all functional modules710a-710emay be implemented by the processing circuitry610, possibly in cooperation with the communications interface620and/or the storage medium630. The processing circuitry610may thus be configured to from the storage medium630fetch instructions as provided by a functional module710a-710eand to execute these instructions, thereby performing any steps as disclosed herein.

The image processing device100might be a graphics processing unit (GPU), a visual processing unit (VPU) or a tensor processing unit (TPU), and/or be provided in any of a video card, a mother board, an embedded system, a mobile phone, a vehicle, a personal computer, a workstation, or a game console.

A first portion of the instructions performed by the image processing device100may be executed in a first device, and a second portion of the of the instructions performed by the image processing device100may be executed in a second device; the herein disclosed embodiments are not limited to any particular number of devices on which the instructions performed by the image processing device100may be executed. Hence, the methods according to the herein disclosed embodiments are suitable to be performed by an image processing device100residing in a cloud computational environment. Therefore, although a single processing circuitry610is illustrated inFIG.6the processing circuitry610may be distributed among a plurality of devices, or nodes. The same applies to the functional modules710a-710eofFIG.7and the computer program820ofFIG.8.

FIG.8shows one example of a computer program product810comprising computer readable storage medium830. On this computer readable storage medium830, a computer program820can be stored, which computer program820can cause the processing circuitry610and thereto operatively coupled entities and devices, such as the communications interface620and the storage medium630, to execute methods according to embodiments described herein. The computer program820and/or computer program product810may thus provide means for performing any steps as herein disclosed.

In the example ofFIG.8, the computer program product810is illustrated as an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-Ray disc. The computer program product810could also be embodied as a memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM) and more particularly as a non-volatile storage medium of a device in an external memory such as a USB (Universal Serial Bus) memory or a Flash memory, such as a compact Flash memory. Thus, while the computer program820is here schematically shown as a track on the depicted optical disk, the computer program820can be stored in any way which is suitable for the computer program product810.

The inventive concept has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the inventive concept, as defined by the appended patent claims.