Patent Description:
Image registration might generally involve transforming different sets of data into one coordinate system. The sets of data might comprise multiple images as captured from different sensors, at different times, depths, or viewpoints. Image registration might be used in application such as computer vision, medical imaging, military automatic target recognition, compiling and analyzing images and data from satellites, super resolution applications, creating a panoramic view of a scene, creating a three-dimensional (3D) model of the scene, etc..

Some aspects of image registration consider the problem of identifying overlapping regions and the geometric transform between pairs of images.

One example of a scheme used for image registration is based on matching key-points across the images. Sets of key-points and their descriptors are extracted from each image. The descriptors can used to establish correspondence between of key-points across pairs of images, and in turn be used to establish the geometric transform.

It might be challenging to create a consistent mapping between the key-points across pairs of images due to there being a large number of key-points and due to there being many key-points with similar descriptors. One solution is to perform a brute force search and reject a large number of incorrect mappings by imposing some geometrical transform and using, e.g., a random sample consensus (RANSAC) algorithm.

Hence, there is still a need for improved mechanisms for image registration, and especially in terms of efficient matching between key-points across pairs of images. <CIT> discloses a method for performing panoramic stitching. The method involves matching objects corresponding to a same object identifier in pairs of adjacent images, and determining a relationship between the images and the objects in the images.

An object of embodiments herein is to provide mechanisms for efficient image registration not suffering from the above noted issues, or at least where the above noted issues have been reduced or mitigated.

According to a first aspect there is presented a computer-implemented method for image registration as defined in claim <NUM>.

According to a second aspect there is presented an image registration entity for image registration as defined in claim <NUM>.

According to a third aspect there is presented a computer program for image registration as defined in claim <NUM>.

According to a fourth aspect there is presented a computer readable storage medium as defined in claim <NUM>.

Advantageously, these aspects provide efficient image registration.

Advantageously, these aspects do not suffer from the above noted issues.

Advantageously, use of the information about matched objects in the images is used to constrain the matching between key-points, which leads to improved accuracy and complexity reduction in the image registration process.

Advantageously, the proposed method and image registration require comparatively low complexity for implementation and execution.

The scope of the invention and its embodiments is defined by the appended claims.

The embodiments disclosed herein relate to mechanisms for image registration. In order to obtain such mechanisms there is provided an image registration entity, a method performed by the image registration entity and a computer program, that when run on an image registration entity, causes the image registration entity to perform the method.

As noted above there is still a need for improved mechanisms for image registration, and especially in terms of efficient matching between key-points across pairs of images.

In this respect, many traditional mechanisms for image registration would perform the matching of key-points and the localization of objects in parallel to understand the environment of the scene since these two tasks typically are regarded as addressing different problems and therefore performed separately.

At least some of the herein disclosed embodiments are based on using the knowledge from having performed object detection when matching key-points between different images to enable more efficient image registration. As will be further disclosed below, this might be realized by splitting the total set of key-points of each image into subsets based on their correspondence to the image areas under detected objects.

<FIG> is a schematic diagram illustrating an image registration entity <NUM> where embodiments presented herein can be applied. Without loss of generality, the image registration entity <NUM> illustrates how a current image Ic is processed. It is assumed that a previous image Ip has been processed in the same manner. The image registration entity <NUM> comprises a key-point extractor <NUM> configured to extract key-points Kc from the current image Ic. In some examples the key-points are represented by spatial coordinates X and the corresponding descriptor D, i.e., K = [X, D]. The descriptor of a given key-point could describe statistics, in terms of gradients, texture, etc., in a surrounding of the given key-point. The image registration entity <NUM> comprises an object detector <NUM> configured to detect objects Oc from the current image Ic. In some examples the objects are represented by a location B (for example given in terms of bounding box coordinates) and an object class C, i.e., O = [B, C]. The image registration entity <NUM> comprises a key-point matcher <NUM> configured to match the key-points Kc from the current image Ic to key-points Kp from the previous image Ip using the objects Oc detected the current image Ic and objects Op detected the previous image Ip. The image registration entity <NUM> comprises an image representation constructor <NUM> configured to, from the matching key-points {Kp, Kc} as found by the key-point matcher <NUM>, the current image Ip, the previous image Ip, the objects Oc from the current image Ic, and the objects Op from the previous image Ip, construct an image representation {Ip, Ic} of the current image Ip and the previous image Ip.

<FIG> in more detail illustrates the key-point matcher <NUM>. The key-point matcher <NUM> comprises a key-point mapper <NUM> configured to map key-points to objects (i.e., key-points Kc are mapped to objects Oc in image Ic). The key-point matcher <NUM> comprises a mapped based key-point matcher <NUM> configured to match key-points Kc to key-points Kp based on which key-points are mapped to which objects, for both images Ic and Ip.

<FIG> is a flowchart illustrating embodiments of methods for image registration. The methods are performed by the image registration entity <NUM>. The methods are advantageously provided as computer programs <NUM>.

The image registration is performed for a first image and a second image of the same scene. The first image comprises a first set of objects and the second image comprises a second set of objects. Since both images are of the same scene it is assumed that at least some of the object are part of both images and thus that there is a matching between some of the first set of objects and some of the second set of objects.

S102: The image registration entity <NUM> obtains a matching between a first set of objects 320P, 330P, 340P, 350P in a first image 310P of a scene and a second set of objects 320C, 330C, 340C, 350C in a second image 310C of the scene. The first set of objects 320P, 330P, 340P, 350P and the second set of objects 320C, 330C, 340C, 350C are extracted using visual object detection in the first image 310P and in the second image 310C, respectively.

Each of the first image and the second image comprises respective sets of key-points.

S104: The image registration entity <NUM> obtains a first set of key-points as extracted from the first image 310P and a second set of key-points as extracted from the second image 310C.

The key-points of the first image are to be matched to the key-points of the second image. The matching is based on using information of the match between the sets of objects in the different images, as obtained in S102.

S106: The image registration entity <NUM> performs image registration. Image registration is performed by the image registration entity <NUM> matching the first set of key-points to the second set of key-points. Those of the first set of key-points that are mapped to objects in the first set of objects 320P, 330P, 340P, 350P and that have matching objects in the second set of objects 320C, 330C, 340C, 350C are restricted to only be matched to those of the second set of key-points that are mapped to any of the matching objects in the second set of objects 320C, 330C, 340C, 350C.

Thus, the first set of key-points and the second set of key-points could each be regarded as being divided in subsets, where each subset of key-points corresponds to an individual object in one of the images. The matching in S106 then is performed individually for each subset in the first image, where the key-points in a subset corresponding to a given object in the first image are matched to the key-points in the subset corresponding to the object in the second image being mapped to the given object in the first image.

Intermediate reference is here made to <FIG> schematically illustrates a first image 310P having a first set of objects 320P, 330P, 340P, 350P. <FIG> further schematically illustrates a second image 310C having a second set of objects 320C, 330C, 340C, 350C. <FIG> further schematically illustrates key-points (one of which is identified at reference numeral 360P) in the first image 310P and key-points (one of which is identified at reference numeral 360C) in the second image, and matching (in terms of directed arrows, one of which is identified at reference numeral <NUM>) between the key-points in the first image 310P and the key-points in the second image 310P. It is for illustrative examples assumed that key-point 360P is mapped to object 320P, that key-point 360C is mapped to object 320C, that object 320P is matched to object 320C, and thus that key-point 360P is matched to key-point 360C, as indicated by arrow <NUM>.

Information of the matching between the key-points is then used when an image representation of the scene is constructed.

S108: The image registration entity <NUM> applies the matching between key-points resulting from the image registration when constructing an image representation of the scene.

Embodiments relating to further details of image registration as performed by the image registration entity <NUM> will now be disclosed.

There could be different ways for the first set of objects to be matched to the second set of objects.

In some aspects, the matching is based on object class identifiers, or identities, denoted object class IDs. That is, each object might be associated with an object class ID. Then, in some embodiments, the first set of objects 320P, 330P, 340P, 350P are matched to the second set of objects 320C, 330C, 340C, 350C by object class ID matching.

Further, the matching between the first set of key-points and the second set of key-points might only be made between key-points mapped to objects of same object class ID.

In some aspects, the matching is based on location of the objects in the first image and in the second image, respectively. Therefore, each object in the first set of objects 320P, 330P, 340P, 350P might be associated with a respective location in the first image 310P, and each object in the second set of objects 320C, 330C, 340C, 350C might be associated with a respective location in the second image 310C. Then, in some embodiments, the first set of objects 320P, 330P, 340P, 350P are matched to the second set of objects 320C, 330C, 340C, 350C by location matching.

In general terms, the location matching between an object in the first image and an object in the second image relates to the distance-wise difference between the location of the object in the first image and the location of the object in the second image; the lower the difference the higher the location matching is. There could be different ways to perform the location matching. In some embodiments, the location matching is based on Jaccard index values computed between the objects in the first set of objects 320P, 330P, 340P, 350P and the objects in the second set of objects 320C, 330C, 340C, 350C. The higher the Jaccard index value is between an object in the first image and an object in the second image, the higher the chance is that these two objects are mapped to each other.

In some aspects, the matching is based on descriptors of the key-points. That is, each key-point might have a descriptor, where each descriptor has a value. Then, in some embodiments, the matching between the first set of key-points and the second set of key-points only is made between key-points having as similar descriptor values as possible.

One purpose of the key-point descriptors is to characterize the image area in the vicinity of the key-points. The descriptor D of a key-point is typically provided as a scalar or a finite vector, which summarizes the properties of that key-point. There are different types of available descriptors. As an example, the speeded up robust features (SURF) descriptor is a <NUM>-dimensional vector of floating-point values based on the sum of the Haar wavelet response around the particular key-point. As an example, the scale-invariant feature transform (SIFT) descriptor is a <NUM>-dimensional vector calculated by aggregation histograms of gradient magnitude around the key-point. The histogram of oriented gradients (HOG) descriptor describes local appearance around a key-point by means of concatenated histograms of gradient directions. The Binary Robust Independent Elementary Features (BRIEF) descriptor is a binary version of the SIFT descriptor in the form of a <NUM>-bit number. Beyond capturing the intensity distribution of the pixels within the neighborhood, the descriptors could be based on color gradients, dominant color in the area, etc. As an example, closeness of key-point descriptors could be measured by means of Euclidean vector distance.

There might be different ways to handle cases where key-points are not matched to any object in the first image but to an object in the second image, or matched to an object in the first image but not to any object in the second image, or neither matched to any object in the first image nor to any object in the second image.

In some aspects, the key-points that do not belong to any object in the first image are mapped to key-points belonging to the background (i.e., not to any object) in the second image. In particular, according to the invention, any of those of the first set of key-points that are not mapped to any object in the first set of objects 320P, 330P, 340P, 350P are restricted to only be matched to those of the second set of key-points that are not mapped to any of the matching objects in the second set of objects 320C, 330C, 340C, 350C.

In some aspects, the key-points that belong to objects in the first image not having any matching objects in the second image are mapped to key-points belonging to the background (i.e., not to any object) in the second image. In particular, some embodiments, any of those of the first set of key-points that are mapped to objects in the first set of objects 320P, 330P, 340P, 350P that do not have any matching objects in the second set of objects 320C, 330C, 340C, 350C are restricted to only be matched to those of the second set of key-points that are not mapped to any of the matching objects in the second set of objects 320C, 330C, 340C, 350C.

If some objects are not detected in a portion of an image, or if the portion of the image does not comprise any object, key-points in that portion of the image will be considered as belonging to the background. This is because the matching is based on correspondence between objects in the first image and in the second image. If an object abruptly disappears from the scene, or it is not detected in one of the images, key-points are considered as belonging to the background. Intermediate reference is here made to <FIG> schematically illustrates the same first image 310P and the same second image 310C as in <FIG> but with the difference that object 320C is no longer found in image 310C. Object 320P therefore does not have any matching object in image 310C, and key-point 360P of object 320P in image 310P is therefore matched to the background in image 310C.

There are different ways to map key-points to objects. In some aspects, this mapping is based on the use of bounding boxes. For example, each object might be associated with a bounding box. Then, in some embodiments, each of the key-points is mapped to its object by being located within the bounding box of its object.

There could be different ways to handle situations where there is a match between one object in the first image and two objects in the second image. When an object in the first set of objects 320P, 330P, 340P, 350P is matched to two or more objects in the second set of objects 320C, 330C, 340C, 350C, each of the key-points mapped to that object in the first set of objects 320P, 330P, 340P, 350P is matched to a key-point mapped to either of these two or more objects in the second set of objects 320C, 330C, 340C, 350C. Additional conditions, such as descriptors, could then be applied to determine the matching.

There are different applications where the image representation of the scene could be used. In some examples, the image representation of the scene is a panoramic view comprising, or represented by, the first image 310P and the second image 310C. In some examples, the image representation of the scene is a 3D model comprising, or represented by, the first image 310P and the second image 310C. In further examples, image representation of the scene could be used in medical imaging (where multiple images of human organs or tumours have to be stitched, or in image super resolution applications, where a higher resolution two-dimensional (2D) scene from a set of low resolution 2D images is constructed.

In view of at least some of the above disclosed embodiments, a method for image registration as performed by the image registration entity <NUM> comprises the following.

Objects are matched, as in S102, between the images IP and IC for example using the class ID (type of the object), i.e., match CC and CP. If there are multiple objects of the same class, the Jaccard index based on BC and BP might be used to determine the matching.

The key-points and the corresponding descriptors, KC = [XC, DC], for image IC are extracted and key-points and the corresponding descriptors, KP = [XP, DP], for image IP are retrieved, as in S104. The location and type (class) of the objects OC = [BC, CC] for image IC might be extracted.

Key-points are, as in S106, matched between objects in image IP and in image IC. Descriptor based key-point association is performed only for the subset of key-points laying in a pair of objects having been matched. Key-points that do not belong to any object might be matched to corresponding background points. The thus matched pairs of key-points might be tested against a geometric transform.

It should be noted that, for the sake of notation but without loss of generality, the notation previous image IP and current image IC has been used, thus indicating that these images have been captured at different points in time, such as in a sequence of images. However, the herein disclosed embodiments are also applicable to images produced at the same time instant, but from different sources, e.g., from two different cameras. Further, the herein disclosed embodiments are also applicable beyond usage of single camera; stereo cameras, or additional sensors, could be also used, which includes depth cameras, thermal cameras, or equipment for active scanning, such as laser or lidar.

<FIG> schematically illustrates, in terms of a number of functional units, the components of an image registration entity <NUM> according to an embodiment. Processing circuitry <NUM> is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product <NUM> (as in <FIG>), e.g. in the form of a storage medium <NUM>. The processing circuitry <NUM> may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).

Particularly, the processing circuitry <NUM> is configured to cause the image registration entity <NUM> to perform a set of operations, or steps, as disclosed above. For example, the storage medium <NUM> may store the set of operations, and the processing circuitry <NUM> may be configured to retrieve the set of operations from the storage medium <NUM> to cause the image registration entity <NUM> to perform the set of operations.

Thus the processing circuitry <NUM> is thereby arranged to execute methods as herein disclosed. The image registration entity <NUM> may further comprise a communications interface <NUM> at least configured for communications with other entities, nodes, functions, and devices. As such the communications interface <NUM> may comprise one or more transmitters and receivers, comprising analogue and digital components. The processing circuitry <NUM> controls the general operation of the image registration entity <NUM> e.g. by sending data and control signals to the communications interface <NUM> and the storage medium <NUM>, by receiving data and reports from the communications interface <NUM>, and by retrieving data and instructions from the storage medium <NUM>. Other components, as well as the related functionality, of the image registration entity <NUM> are omitted in order not to obscure the concepts presented herein.

<FIG> schematically illustrates, in terms of a number of functional modules, the components of an image registration entity <NUM> according to an embodiment. The image registration entity <NUM> of <FIG> comprises a number of functional modules; an obtain module 210a configured to perform step S102, an obtain module 210b configured to perform step S104, an image registration module 210c configured to perform step S106, and an apply module 210d configured to perform step S108. The image registration entity <NUM> of <FIG> may further comprise a number of optional functional modules, as represented by functional module 210e. In general terms, each functional module 210a-210e may in one embodiment be implemented only in hardware and in another embodiment with the help of software, i.e., the latter embodiment having computer program instructions stored on the storage medium <NUM> which when run on the processing circuitry makes the image registration entity <NUM> perform the corresponding steps mentioned above in conjunction with <FIG>. It should also be mentioned that even though the modules correspond to parts of a computer program, they do not need to be separate modules therein, but the way in which they are implemented in software is dependent on the programming language used. Preferably, one or more or all functional modules 210a-210e may be implemented by the processing circuitry <NUM>, possibly in cooperation with the communications interface <NUM> and/or the storage medium <NUM>. The processing circuitry <NUM> may thus be configured to from the storage medium <NUM> fetch instructions as provided by a functional module 210a-210e and to execute these instructions, thereby performing any steps as disclosed herein.

The image registration entity <NUM> might be integrated with, be part of, or collocated with, an image processing device, such as a graphics processing unit (GPU), a visual processing unit (VPU) or a tensor processing unit (TPU), and/or be provided in any of: a video card, a mother board, an embedded system, a mobile phone, a vehicle, a personal computer, a workstation, or a game console.

A first portion of the instructions performed by the image registration entity <NUM> may be executed in a first device, and a second portion of the of the instructions performed by the image registration entity <NUM> may be executed in a second device; the herein disclosed embodiments are not limited to any particular number of devices on which the instructions performed by the image registration entity <NUM> may be executed. Hence, the methods according to the herein disclosed embodiments are suitable to be performed by an image registration entity <NUM> residing in a cloud computational environment. Therefore, although a single processing circuitry <NUM> is illustrated in <FIG> the processing circuitry <NUM> may be distributed among a plurality of devices, or nodes. The same applies to the functional modules 210a-210e of <FIG> and the computer program <NUM> of <FIG>.

Claim 1:
A computer-implemented method for image registration, the method being performed by an image registration entity (<NUM>), the method comprising:
obtaining (S102) a matching between a first set of objects (320P, 330P, 340P, 350P) in a first image (310P) of a scene and a second set of objects (320C, 330C, 340C, 350C) in a second image (310C) of the scene, the first set of objects (320P, 330P, 340P, 350P) and the second set of objects (320C, 330C, 340C, 350C) being extracted using visual object detection in the first image (310P) and in the second image (310C), respectively;
obtaining (S104) a first set of key-points as extracted from the first image (310P) and a second set of key-points as extracted from the second image (310C);
performing (S106) image registration by matching the first set of key-points to the second set of key-points, wherein those of the first set of key-points that are mapped to objects in the first set of objects (320P, 330P, 340P, 350P) and that have matching objects in the second set of objects (320C, 330C, 340C, 350C) are restricted to only be matched to those of the second set of key-points that are mapped to any of the matching objects in the second set of objects (320C, 330C, 340C, 350C), and any of those of the first set of key-points that are not mapped to any object in the first set of objects (320P, 330P, 340P, 350P) are restricted to only be matched to those of the second set of key-points that are not mapped to any of the matching objects in the second set of objects (320C, 330C, 340C, 350C); and
applying (S108) the matching between key-points resulting from the image registration when constructing an image representation of the scene.