Patent Description:
Panoramic images are widely used for a virtual navigation of indoor environments, such as facilities, factories, shopping malls and the like, and outdoor environments, such as Google Street View. In industrial settings, these images are often produced as a result of a laser scan.

The real scene may include a variety of real objects which are related for example to a facility or a panoramic image of a street or a place or the like. Examples of real objects include, but are not limited to, equipment pieces, tools, containers, material pieces, finished or semi-finished products, houses, light posts, trees and other objects and landmarks present in the real scene. During the creation of these images, it is necessary to align their direction of view. This is for example done to allow navigation along a path by means of transitions between panoramic images (transition from one image to the other or respective next one). This alignment is currently done by recording the camera's direction at the moment of the image capture.

Unfortunately, this direction information is not always available. It might not be available at all when the images are captured by a device that does not provide that information. Another scenario is that after a series of images were taken and aligned and sometime after this capture some of the images are recaptured and updated, the original parameters might be unknown. This problem is currently solved by recapturing the entire series of images instead of updating some of them and by aligning the images manually. Further, direction information could be lost during the transfer of the captured images to a host due to unexpected communication network failure or the like.

The following documents describes image registration procedures of the prior art. Publication "<NPL>et al. presents a framework for registering panoramic images; the framework includes a procedure which leverages cadastral 3D information, typically available in urban scenarios. Patent document <CIT> discloses a method for determining the location and orientation of panoramic images. Publication "<NPL>et al. presents a work on image registration where registration is realized through the use of region descriptors. Publication "<NPL>et al. describes an image registration method that combines log-polar transform and SIFT to recover similarity transformations (rotation/ scale/ translation).

It is therefore the objective of the present invention to provide a method and a data processing system for aligning a first panoramic image and a second panoramic image in a navigation procedure that provide a simple way to avoid manual intervention or cumbersome capturing in order to generate a panoramic stream of images during the virtual navigation in indoor and/or outdoor environments.

Various disclosed embodiments of the present invention include methods and corresponding systems and computer-readable mediums. A method for aligning a first panoramic image and a second panoramic image in a navigation procedure is defined in appended claim <NUM>.

In another example, a data processing system is provided. The data processing system contains a processor and an accessible memory, the data processing system is particularly configured to align a first panoramic image and a second panoramic image in a navigation procedure. The data processing system is defined in appended claim <NUM>.

The foregoing has outlined rather broadly the features and technical advantages of the present disclosure so that those skilled in the art may better understand the detailed description that follows. Additional features and advantages of the disclosure will be described hereinafter that form the subject of the claims. Those skilled in the art will appreciate that they may readily use the conception and the specific embodiment disclosed as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure.

Before undertaking the detailed description below, it may be advantageous to set forth definitions of certain words or phrases used throughout this patent document: the terms "include" and "comprise," as well as derivatives thereof, mean inclusion without limitation; the term "or" is inclusive, meaning and/or; the phrases "associated with" and "associated therewith," as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, or the like; and the term "controller" means any device, system or part thereof that controls at least one operation, whether such a device is implemented in hardware, firmware, software or some combination of at least two of the same. Definitions for certain words and phrases are provided throughout this patent document, and those of ordinary skill in the art will understand that such definitions apply in many, if not most, instances to prior as well as future uses of such defined words and phrases. While some terms may include a wide variety of embodiments, the appended claims may expressly limit these terms to specific embodiments.

When reading the claim language, the following definitions apply. When the claim language recites A and/or B it means A alone, B alone or A and B. When the claim language recites at least one of A and B it means A alone, B alone or A and B. When the claim language recites at least one of A or B it means A alone, B alone or A and B.

Other features which are considered as characteristic for the invention are set forth in the appended claims.

The construction and method of operation of the invention, however, together with additional objects and advantages thereof will be best understood from the following description of specific embodiments when read in connection with the accompanying drawings.

Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably arranged device. The numerous innovative teachings of the present application will be described with reference to exemplary non-limiting embodiments.

Referring now to the figures of the drawings in detail and first, particularly to <FIG> thereof, there is shown a block diagram of a data processing system <NUM> in which an embodiment can be implemented, for example as a PDM system particularly configured by software or otherwise to perform the processes as described herein, and in particular as each one of a plurality of interconnected and communicating systems as described herein. The data processing system <NUM> illustrated can include a processor <NUM> connected to a level two cache/bridge <NUM>, which is connected in turn to a local system bus <NUM>. The local system bus <NUM> may be, for example, a peripheral component interconnect (PCI) architecture bus. Also connected to the local system bus <NUM> in the illustrated example are a main memory <NUM> and a graphics adapter <NUM>. The graphics adapter <NUM> may be connected to a display <NUM>.

Other peripherals, such as local area network (LAN) / Wide Area Network / Wireless (e.g. WiFi) adapter <NUM>, may also be connected to the local system bus <NUM>. An expansion bus interface <NUM> connects the local system bus <NUM> to an input/output (I/O) bus <NUM>. The I/O bus <NUM> is connected to a keyboard/mouse adapter <NUM>, a disk controller <NUM>, and an I/O adapter <NUM>. The disk controller <NUM> can be connected to a storage <NUM>, which can be any suitable machine usable or machine readable storage medium, including but not limited to nonvolatile, hard-coded type mediums such as read only memories (ROMs) or erasable, electrically programmable read only memories (EEPROMs), magnetic tape storage, and user-recordable type mediums such as floppy disks, hard disk drives and compact disk read only memories (CD-ROMs) or digital versatile disks (DVDs), and other known optical, electrical, or magnetic storage devices.

Also connected to the I/O bus <NUM> in the example shown is an audio adapter <NUM>, to which non-illustrated speakers may be connected for playing sounds. The keyboard/mouse adapter <NUM> provides a connection for a non-illustrated pointing device, such as a mouse, trackball, trackpointer, touchscreen, etc..

Those of ordinary skill in the art will appreciate that the hardware illustrated in <FIG> may vary for particular implementations. For example, other peripheral devices, such as an optical disk drive and the like, also may be used in addition or in place of the hardware illustrated. The illustrated example is provided for the purpose of explanation only and is not meant to imply architectural limitations with respect to the present disclosure.

A data processing system in accordance with an embodiment of the present disclosure can include an operating system employing a graphical user interface. The operating system permits multiple display windows to be presented in the graphical user interface simultaneously, with each display window providing an interface to a different application or to a different instance of the same application. A cursor in the graphical user interface may be manipulated by a user through the pointing device. The position of the cursor may be changed and/or an event, such as clicking a mouse button, generated to actuate a desired response.

One of various commercial operating systems, such as a version of Microsoft Windows™, a product of Microsoft Corporation located in Redmond, Washington may be employed if suitably modified. The operating system is modified or created in accordance with the present disclosure as described.

The LAN/ WAN/Wireless adapter <NUM> can be connected to a network <NUM> (not a part of data processing system <NUM>), which can be any public or private data processing system network or combination of networks, as known to those of skill in the art, including the Internet. The data processing system <NUM> can communicate over network <NUM> with server system <NUM>, which is also not part of data processing system <NUM>, but can be implemented, for example, as a separate data processing system <NUM>.

<FIG> illustrates a schematic view of a illustrates a schematic view of a series of two panoramic images i1 , i2 that require an alignment of the horizontal direction for their application in a navigation procedure, such as the panoramic view on a street or the like (i.e. Google Street View). During the capturing of the two images i1, i2, the two images i1, i2 have been taken at a different camera orientation, for example due to the trajectory of a camera car bearing the mounted camera on its roof and driving its way down through a specific road/street. When these images i1, i2 shall form part of a series of images that allows a user to navigate along the street, it is required that each image following the previous one is direction-wise aligned with the previous image. Thus, a consistent view on that street while navigating through this street is generated.

The present method for aligning a first panoramic image and a second panoramic image in a navigation procedure therefore exactly satisfies the demand on providing a simple method for the alignment of two images that have been taken at different physical positions without recording the exact camera direction for each image. After capturing the two images i1 and i2, a computer vision algorithm is used for extracting features that are present in the first panoramic images i1 and in the second panoramic image i2. The extracted features are then used to prepare a first feature vector v1 for the first panoramic image i1 and a second feature vector v2 for the second panoramic image i2. Typically, this image matching technique can be applied according to known image matching algorithms, such as scale invariant feature transform (SIFT), speed up robust feature (SURF), robust independent elementary features (BRIEF), oriented FAST, KAZE and rotated BRIEF (ORB) (also see <NPL>).

Then, the first feature vector v1 and the second feature vector v2 are compared in order to identify matches in both the first and the second feature vector v1, v2 that are above a determined threshold in terms of the resolution and/or the intensity of the extracted features. Preferably, the software algorithms are enabled to prioritize those extracted features where the angle of view is expected not to vary significantly from one image to the next image. Typically, these features can relate to those which are not in the lateral proximity of the camera but rather centrally located at the farer end the scene. For this specific image alignment, features f1 and f2 are considered in the following steps.

Once these matches are now identified for each of the features f1, f2 matching in both the first and the second feature vectors v1, v2 for each of the two images i1, i2, the perspective angle αm,n between a reference point R of the respective image and the center of each features f1, f2 according to the identified matches is determined. The index m is the number of the feature and n is the number of the image. For this specific example, the first feature f1 in the first image i1 is located at an angle α<NUM> and the second feature f2 at an angle α<NUM> respectively. Correspondingly, the first feature f1 in the second image i2 is located at an angle α<NUM> and the second feature f2 at an angle α<NUM> respectively.

These angles now allow too calculate the difference d1 and d2 for each of the two feature f1, f2 in the two images i1 and i2. For feature f1, the difference d1 equals to (α<NUM> - α<NUM>), and for feature f2, the difference d2 is (α<NUM> - α<NUM>). These values for the lateral difference in the rotation of the two images relatively to each other can be used to yield a skimmed mean dL for the lateral rotation of image i1 relative image i2. In a panoramic film this skimmed mean dL can be applied to correct the position of image i2 relative to image i1 which leads to the result that the observer of the two images i1 and i2 in a video stream will have the sensation that both images i1 and i2 having been captured at the same orientation of the camera. It is needless to say that this process can be now applied to the couple of the second and a third image of the intended video stream and so on, in order provide a video stream of the passage through a specific street, such as this is intended by Google Street View for example.

This step can now be also repeated for the vertical rotation of the two images i1 and i2 as schematically shown in <FIG>. The vertical rotation difference d3 for the feature f1 equals to (β<NUM> - β<NUM>); correspondingly the vertical rotation difference d4 for the feature f2 equals to (β<NUM> - β<NUM>). The skimmed mean dv for the vertical rotation equals analogously to (d3 + d4)/<NUM>.

<FIG> illustrates a flowchart <NUM> of a method for aligning a first panoramic image and a second panoramic image in a navigation procedure, the method containing the now described following steps.

At step <NUM> a computer vision algorithm is used for extracting features that are present in the first panoramic images and in the second panoramic image. Further, these extracted features are used to prepare a first feature vector for the first panoramic image and a second feature vector for the second panoramic image.

At step <NUM> the first and the second feature vector are compared to each other and matches are identified for those features that are present in both the first and the second feature vector and that are above a determined threshold, i.e. for the precision and/or the intensity.

At step <NUM>, for the identified matches in each of the first and the second feature vectors for each of the two images the perspective angle between a reference point of the respective image and the features according to the identified matches is determined.

At step <NUM> the difference between the determined perspective angles for at least part of the identified matches in the first and the second image is determined yielding to the result that the difference in the rotation of first image relative to the second image can be determined and this rotation difference can be applied in navigation application in order to align the first and the second image using this rotation difference.

Example: We'll start with the two panoramic images according to <FIG> that show a lab scenery taken at two different positions and two different orientations. After running KAZE algorithm on both images, a total of <NUM> features are identified in a first image and <NUM> in a second. A total of <NUM> features have a good match above a threshold between the two images. The KAZE algorithm is described in more detail in:.

The resulting features and matches can be visualized as shown in <FIG> by the horizontal lines and dots in the two images presented side-by-side.

From the resulting array of <NUM> matching features, for each match, an angle of the feature from the center of the image is calculated for both images. For each match the difference of the respective angles in both images is calculated and added into an array holding the values for difference angles. A mean value of the array of the values for the difference angles is calculated and yield a lateral correction in the orientation from the first image relative to the other image. In order to discard outliers - features that were matched by mistake and resulting in an angle more than <NUM>% larger or smaller than the mean value, a trimmed mean is calculated here. Thus, the two images can be presented in a navigation application as if they had been initially take with the same lateral orientation thereby allowing to generate a real eye stream of image as if a person would walk into the direction of the view without moving its head laterally.

One or more of the processor <NUM>, the memory <NUM>, and the alignment program running on the processor <NUM> receive the inputs via one or more of the local system bus <NUM>, the adapter <NUM>, the network <NUM>, the server <NUM>, the interface <NUM>, the I/O bus <NUM>, the disk controller <NUM>, the storage <NUM>, and so on. Receiving, as used herein, can include retrieving from the storage <NUM>, receiving from another device or process, receiving via an interaction with a user, or otherwise.

Of course, those of skill in the art will recognize that, unless specifically indicated or required by the sequence of operations, certain steps in the processes described above may be omitted, performed concurrently or sequentially, or performed in a different order.

Those skilled in the art will recognize that, for simplicity and clarity, the full structure and operation of all data processing systems suitable for use with the present disclosure is not being illustrated or described herein. Instead, only so much of a data processing system as is unique to the present disclosure or necessary for an understanding of the present disclosure is illustrated and described. The remainder of the construction and operation of data processing system <NUM> may conform to any of the various current implementations and practices known in the art.

It is important to note that while the disclosure includes a description in the context of a fully functional system, those skilled in the art will appreciate that at least portions of the mechanism of the present disclosure are capable of being distributed in the form of instructions contained within a machine-usable, computer-usable, or computer-readable medium in any of a variety of forms, and that the present disclosure applies equally regardless of the particular type of instruction or signal bearing medium or storage medium utilized to actually carry out the distribution. Examples of machine usable/readable or computer usable/readable mediums include: nonvolatile, hard-coded type mediums such as read only memories (ROMs) or erasable, electrically programmable read only memories (EEPROMs), and user-recordable type mediums such as floppy disks, hard disk drives and compact disk read only memories (CD-ROMs) or digital versatile disks (DVDs).

Claim 1:
A method for aligning a first panoramic image and a second panoramic image in a navigation procedure, the method comprising the following steps:
a) using a computer vision algorithm for extracting features that are present in the first panoramic images and in the second panoramic image and using the extracted features to prepare a first feature vector for the first panoramic image and a second feature vector for the second panoramic image;
b) comparing the first and the second feature vector and identifying matches in both the first and the second feature vector that are above a determined threshold;
c) determining for the identified matches in each of the first and the second feature vectors for each of the two images the perspective angle between a reference point of the respective image, a vertical or a horizontal line passing through the reference point and the features according to the identified matches wherein the reference point is chosen to be the center of the image; and
d) determining the difference between the determined perspective angles for at least part of the identified matches in the first and the second image.