Image processing method and associated apparatus

An image processing method includes: receiving a plurality of images, the images being captured under different view points; and performing image alignment for the plurality of images by warping the plurality of images, where the plurality of images are warped according to a set of parameters, and the set of parameters are obtained by finding a solution constrained to predetermined ranges of physical camera parameters. In particular, the step of performing the image alignment further includes: automatically performing the image alignment to reproduce a three-dimensional (3D) visual effect, where the plurality of images is captured by utilizing a camera module, and the camera module is not calibrated with regard to the view points. For example, the 3D visual effect can be a multi-angle view (MAV) visual effect. In another example, the 3D visual effect can be a 3D panorama visual effect. An associated apparatus is also provided.

BACKGROUND

The present invention relates to three-dimensional (3D) visual effect reproduction, and more particularly, to an image processing method, and to an associated apparatus.

According to the related art, 3D visual effect reproduction typically requires preparation of source images and complicated calculations. During a preparation stage, no matter whether the resolution of the source images is high or low, some problems may occur. For example, the source images should be captured from a plurality of pre-calibrated cameras, where the pre-calibrated cameras should have been calibrated with respect to predetermined view points or predetermined lines of views, which causes difficulty of the preparation of the source images. In another example, in order to perform the complicated calculations efficiently, it is required to prepare a high end computer having high calculation power, where the high end computer would never be replaced by a conventional multifunctional mobile phone since it seems unlikely that the conventional multifunctional mobile phone can work well under the heavy calculation load of the complicated calculations. That is, the conventional multifunctional mobile phone can never be a total solution to 3D production/reproduction. In conclusion, the related art does not serve the end user well. Thus, a novel method is required for performing image processing regarding 3D visual effect reproduction in a smart and robust manner, in order to implement the preparation of the source images mentioned above and associated calculations within a portable electronic device such as a multifunctional mobile phone.

SUMMARY

It is therefore an objective of the claimed invention to provide an image processing method, and to provide an associated apparatus, in order to solve the above-mentioned problems.

It is another objective of the claimed invention to provide an image processing method, and to provide an associated apparatus, in order to implement the preparation of the source images mentioned above and associated calculations within a portable electronic device such as a multifunctional mobile phone.

It is another objective of the claimed invention to provide an image processing method, and to provide an associated apparatus, in order to carry out a total solution to three-dimensional (3D) production/reproduction by utilizing a portable electronic device (e.g. a mobile phone, a laptop computer, or a tablet).

An exemplary embodiment of an image processing method comprises: receiving image data of a plurality of images, the images being captured under different view points (or along different lines of views); and performing image alignment for the plurality of images by warping the plurality of images according to the image data, wherein the plurality of images are warped according to a set of parameters, and the set of parameters are obtained by finding a solution constrained to predetermined ranges of physical camera parameters.

An exemplary embodiment of an apparatus for performing image processing is provided, where the apparatus comprises at least one portion of an electronic device. The apparatus comprises: a storage and a processing circuit. The storage is arranged to temporarily store information. In addition, the processing circuit is arranged to control operations of the electronic device, to receive image data of a plurality of images, the images being captured under different viewpoints (or along different lines of views), to temporarily store the image data into the storage, and to perform image alignment for the plurality of images by warping the plurality of images according to the image data, wherein the plurality of images are warped according to a set of parameters, and the set of parameters are obtained by finding a solution constrained to predetermined ranges of physical camera parameters.

DETAILED DESCRIPTION

Please refer toFIG. 1, which illustrates a diagram of an apparatus100for performing image processing according to a first embodiment of the present invention. According to different embodiments, such as the first embodiment and some variations thereof, the apparatus100may comprise at least one portion (e.g. a portion or all) of an electronic device. For example, the apparatus100may comprise a portion of the electronic device mentioned above, and more particularly, can be a control circuit such as an integrated circuit (IC) within the electronic device. In another example, the apparatus100can be the whole of the electronic device mentioned above. In another example, the apparatus100can be an audio/video system comprising the electronic device mentioned above. Examples of the electronic device may include, but not limited to, a mobile phone (e.g. a multifunctional mobile phone), a personal digital assistant (PDA), a portable electronic device such as the so-called tablet (based on a generalized definition), and a personal computer such as a tablet personal computer (which can also be referred to as the tablet, for simplicity), a laptop computer, or desktop computer.

As shown inFIG. 1, the apparatus100comprises a processing circuit110and a storage120. The storage120is arranged to temporarily store information, such as information carried by at least one input signal108that is inputted into the processing circuit110. For example, the storage120can be a memory (e.g. a volatile memory such as a random access memory (RAM), or a non-volatile memory such as a Flash memory), or can be a hard disk drive (HDD). In addition, the processing circuit110is arranged to control operations of the electronic device, to receive image data of a plurality of images, the images being captured under different view points (or along different lines of views), to temporarily store the image data into the storage120, and to perform image alignment for the plurality of images by warping the plurality of images according to the image data, where the plurality of images are warped according to a set of parameters, and the set of parameters are obtained by finding a solution constrained to predetermined ranges of physical camera parameters. For example, the images are captured, and more particularly, are arbitrarily captured under different view points by utilizing a camera module of the electronic device mentioned above, where the aforementioned image data can be received through the input signal108that is input into the processing circuit110. This is for illustrative purposes only, and is not meant to be a limitation of the present invention. According to some variations of this embodiment, the images are captured, and more particularly, are arbitrarily captured under different view points by utilizing an external device such as a hand-held camera.

Please note that it is unnecessary for the camera module mentioned above to be calibrated. More particularly, the camera module of this embodiment is not calibrated with regard to the view points (or the lines of views) mentioned above. For example, in a situation where the electronic device is light enough for a user to hold it easily, the user may hold the electronic device to arbitrarily capture the images of some objects under these different view points. Then, the processing circuit110automatically performs the image alignment to reproduce a three-dimensional (3D) visual effect, and more particularly, generates 3D images to reproduce the 3D visual effect, where the 3D images may comprise emulated images that are not generated by utilizing any camera such as the camera module mentioned above. The processing circuit110may output information of the 3D images through at least one output signal128that carries the information of the 3D images. In practice, a screen of the electronic device can be utilized for displaying animation based upon the 3D images to reproduce the 3D visual effect. This is for illustrative purposes only, and is not meant to be a limitation of the present invention. According to some variations of this embodiment, the screen may provide the user with stereoscopic views based upon the 3D images to reproduce the 3D visual effect. No matter whether the screen is designed to provide stereoscopic views or not, examples of the 3D visual effect may comprise (but not limited to) a multi-angle view (MAV) visual effect and a 3D panorama visual effect. According to some variations of this embodiment, the apparatus100can output the information of the 3D images, in order to reproduce the 3D visual effect by utilizing an external display device.

FIG. 2illustrates the apparatus100shown inFIG. 1according to an embodiment of the present invention, where the apparatus100of this embodiment is a mobile phone, and therefore, is labeled “Mobile phone” inFIG. 2. A camera module130(labeled “Camera” inFIG. 2, for brevity) is taken as an example of the camera module mentioned in the first embodiment, and is installed within the apparatus100mentioned above (i.e. the mobile phone in this embodiment), which means the apparatus100comprises the camera module130. According to this embodiment, the camera module130is positioned around an upper side of the apparatus100. This is for illustrative purposes only, and is not meant to be a limitation of the present invention. According to some variations of this embodiment, the camera module130can be positioned around another side of the apparatus100. In addition, a touch screen150(labeled “Screen” inFIG. 2, for brevity) is taken as an example of the screen mentioned in the first embodiment, and is installed within the apparatus100mentioned above, which means the apparatus100comprises the touch screen150. As shown inFIG. 2, the camera module130can be utilized for capturing the plurality of images mentioned above. For example, by analyzing the image data of the images, the processing circuit110can perform feature extraction and feature matching to determine/find out the aforementioned solution constrained to the predetermined ranges of physical camera parameters, such as some predetermined ranges of physical parameters of the camera module130(e.g. directions/angles of the lines of views of the camera module130). As a result, the processing circuit110can generate the 3D images bounded to the aforementioned solution, in order to reproduce the 3D visual effect without introducing visible artifacts.

FIG. 3illustrates the apparatus100shown inFIG. 1according to another embodiment of the present invention, where the apparatus100of this embodiment is a personal computer such as a laptop computer, and therefore, is labeled “Laptop computer” inFIG. 3. The camera module130(labeled “Camera” inFIG. 3, for brevity) is taken as an example of the camera module mentioned in the first embodiment, and is installed within the apparatus100mentioned above (i.e. laptop computer in this embodiment), which means the apparatus100comprises the camera module130. According to this embodiment, the camera module130is positioned around an upper side of the apparatus100. This is for illustrative purposes only, and is not meant to be a limitation of the present invention. According to some variations of this embodiment, the camera module130can be positioned around another side of the apparatus100. In addition, a screen50(e.g. a liquid crystal display (LCD) panel) is taken as an example of the screen mentioned in the first embodiment, and is installed within the apparatus100mentioned above, which means the apparatus100comprises the screen50. Similar descriptions are not repeated in detail for this embodiment.

FIG. 4illustrates a flowchart of an image processing method200according to an embodiment of the present invention. The image processing method200shown inFIG. 4can be applied to the apparatus100shown inFIG. 1, and more particualrly, the apparatus100of any of the embodiments respectively shown inFIG. 3andFIG. 4. The image processing method200is described as follows.

In Step210, the processing circuit110receives image data of a plurality of images, the images being captured under different view points (e.g., the plurality of images disclosed in the first embodiment). In this embodiment, the aforementioned image data can be received through the input signal108that is input into the processing circuit110. For example, the images are captured under these different view points (or along different lines of views), and more particularly, are arbitrarily captured by utilizing a camera module such as the camera module130disclosed above. Please note that it is unnecessary for the camera module mentioned above to be calibrated. More particularly, the camera module of this embodiment is not calibrated with regard to the view points.

In Step220, the processing circuit110performs image alignment for the plurality of images by warping the plurality of images according to the image data, where the plurality of images are warped according to a set of parameters, and the set of parameters are obtained by finding a solution constrained to predetermined ranges of physical camera parameters. More particularly, the image alignment may include vertical alignment and horizontal alignment, where the horizontal alignment is typically performed after the vertical alignment is performed. For example, the horizontal alignment can be performed under disparity analysis, where the disparity analysis is utilized for analyzing warped images of the vertical alignment. This is for illustrative purposes only, and is not meant to be a limitation of the present invention. According to some variations of this embodiment, the preparation/beginning of the horizontal alignment can be performed after the preparation/beginning of the vertical alignment is performed, and the horizontal alignment and the vertical alignment can be completed at the same time when some warping operations are completed.

According to this embodiment, in order to achieve better performance during the operations disclosed above, the processing circuit110is preferably arranged to determine the aforementioned predetermined ranges of physical camera parameters in advance by performing operations of sub-steps (1), (2), (3), (4), and (5) as follows:(1) the processing circuit110controls the camera module130to capture a base image, such as one of the plurality of images mentioned in Step210;(2) the processing circuit110controls the camera module130to capture multiple reference images, such as others within the plurality of images mentioned in Step210;(3) the processing circuit110records one set of physical camera parameters corresponding to each reference image, where the aforementioned one set of physical camera parameters can be some location/coordinate-related physical parameters of the camera module130disclosed above (e.g. directions/angles of the lines of views of the camera module130), and may comprise some physical parameters that are not location/coordinate-related (e.g. focal lengths and some other lens parameters of the camera module130);(4) the processing circuit110records warps the base image to match each reference image according to the recorded set of physical camera parameters, and therefore, generates a series of warped base images corresponding to each reference image; and(5) the processing circuit110determines the aforementioned predetermined ranges of physical camera parameters by finding whether difference (s) between warped base images and the reference images is distinguishable under human vision, where the criterion (or criteria) for determining whether the difference(s) is distinguishable under human vision or not can be predefined based upon some predefined rules.

Thus, the processing circuit110eventually determines the aforementioned predetermined ranges of physical camera parameters, in order to achieve better performance during the operations disclosed inFIG. 4. As a result, for an arbitrary set of physical camera parameters that respectively fall within the aforementioned predetermined ranges of physical camera parameters (e.g. a set of physical camera parameters, each of which falls within the corresponding predetermined ranges of the aforementioned predetermined ranges), no difference between any warped base image corresponding to this set of physical camera parameters and the reference images is distinguishable under human vision.

Please not that, by performing the operations of sub-steps (1), (2), (3), (4), and (5) disclosed above, the solution constrained to the aforementioned predetermined ranges of physical camera parameters (i.e. the solution mentioned in the descriptions for Step220) can be found, where the solution allows the base image to be arbitrarily warped while the associated physical camera parameters of this arbitrarily warping operation keep falling within the aforementioned predetermined ranges of physical camera parameters. As the aforementioned predetermined ranges of physical camera parameters is preferably determined by finding whether any difference between warped base images and the reference images is distinguishable under human vision, the solution guarantees that this arbitrarily warping operation will not cause any artifact that is distinguishable under human vision. Therefore, no artifact will be found.

In this embodiment, the sub-steps (1), (2), (3), (4), and (5) are taken as examples of the operations of determining the aforementioned predetermined ranges of physical camera parameters. This is for illustrative purposes only, and is not meant to be a limitation of the present invention. According to some variations of this embodiment, it is unnecessary to perform all of the sub-steps (1), (2), (3), (4), and (5). According to some variations of this embodiment, other sub-step(s) may be included.

FIG. 5illustrates an input image500and some transformation images512,514,522, and532generated during a learning/training procedure involved with the image processing method200shown inFIG. 4according to an embodiment of the present invention, where the learning/training procedure is utilized for determining a predefined solution space (e.g. a pre-trained solution space).

As shown inFIG. 5, the processing circuit110performs similarity transformation on the input image500by performing a plurality of warping operations to generate the transformation images512,514,522, and532, in order to find out the solution mentioned in the descriptions for Step220. Please note that some of these warping operations performed during the learning/training procedure may cause visible artifacts, which are allowed during the learning/training procedure. In this embodiment, the criterion (or criteria) for determining whether the difference mentioned in the sub-step (5) of the embodiment shown inFIG. 4is distinguishable under human vision or not is predefined based upon some predefined rules, and therefore, the solution mentioned in the descriptions for Step220can be referred to as the predefined solution space. For example, the processing circuit110provides the user with an interface, allowing the user to determine whether a transformation image under consideration (e.g. one of the transformation images512,514,522, and532) has any artifact that is distinguishable under human vision. When the user determines that the transformation image under consideration does not have any artifact, the processing circuit110expands the predefined solution space (e.g., the predefined solution space is expanded to include the ranges of physical camera parameters corresponding to the transformation image under consideration); otherwise, the processing circuit110shrinks the predefined solution space (e.g., the predefined solution space is shrunk to exclude the ranges of physical camera parameters corresponding to the transformation image under consideration). Please note that the input image500can be the base image mentioned in the sub-step (1) of the embodiment shown inFIG. 4. This is for illustrative purposes only, and is not meant to be a limitation of the present invention. According to some variations of this embodiment, the input image500can be one of the reference images mentioned in the sub-step (2) of the embodiment shown inFIG. 4.

As a result of performing the similarity transformation, the processing circuit110eventually determines the aforementioned predetermined ranges of physical camera parameters, where any warped image that is bounded within the predefined solution space will not have any artifact that is distinguishable under human vision. For example, in a situation where the transformation image522is a warped image that is bounded within the predefined solution space, the similarity transformation corresponding to the transformation image522can be considered to be visually insensible 3D similarity transformation with regard to physical camera parameters, where “visually insensible” typically represents “deformation of warped image is hard to be distinguished by human vision”. Similar descriptions are not repeated in detail for this embodiment.

In the following embodiments such as those shown inFIGS. 6-10, with the aid of the learning/training results regarding the aforementioned solution such as the predefined solution space (e.g. the pre-trained solution space), the processing circuit110is capable of performing visually insensible image warping. As a result, the image alignment mentioned in the descriptions for Step220(e.g. the vertical alignment and the horizontal alignment) and the associated image warping (if any, for reproducing the 3D visual effect mentioned above) will not cause any artifact that is distinguishable under human vision.

FIG. 6illustrates some images612,614, and616obtained from multi-view vertical alignment involved with the image processing method200shown inFIG. 4according to an embodiment of the present invention. In this embodiment, the multi-view vertical alignment disclosed inFIG. 6is taken as an example of the vertical alignment mentioned in the descriptions for Step220.

As shown inFIG. 6, some video objects respectively shown in the images612,614, and616are the same object in the real world. For example, each of the images612,614, and616comprises a partial image of the same person and further comprises a partial image of the same logo (which is illustrated with a warped shape of “LOGO”). The processing circuit110performs feature extraction on each of the images612,614, and616and performs feature matching for the images612,614, and616to find out some common feature points in each of the images612,614, and616, such as the feature points illustrated with small circles on the three dashed lines crossing the images612,614, and616withinFIG. 6. For example, in each of the images612,614, and616, one of the common feature points can be located at the upper right corner of the logo, another of the common feature points can be located at the lower left corner of the logo, and another of the common feature points can be located at a junction of something worn by the person.

During the multi-view vertical alignment, the processing circuit110aligns the images612,614, and616by performing rotating and/or shifting operations of their original images, which are multi-view images respectively corresponding to three view points (or three lines of views) and are a portion of the plurality of images mentioned in Step210in this embodiment. As a result of performing the multi-view vertical alignment, the common feature points in the images612,614, and616are aligned to the same vertical locations (or the same horizontal lines such as the three dashed lines shown inFIG. 6), respectively, where the dashed lines crossing the three images612,614, and616withinFIG. 6indicates the alignment results of the multi-view vertical alignment. Thus, the processing circuit110performs optimization over geometry constraint to solve the optimal camera parameters within a predefined solution space such as that mentioned above, and more particularly, a predefined visually insensible solution space. Similar descriptions are not repeated in detail for this embodiment.

FIG. 7illustrates one of the images612,614, and616obtained from the multi-view vertical alignment, such as the image612, and an associated image622obtained from horizontal alignment involved with the image processing method200shown inFIG. 4according to an embodiment of the present invention. In addition,FIG. 8illustrates another of the images612,614, and616obtained from the multi-view vertical alignment, such as the image614, and an associated image624obtained from the horizontal alignment according to this embodiment. Additionally,FIG. 9illustrates another of the images612,614, and616obtained from the multi-view vertical alignment, such as the image616, and an associated image626obtained from the horizontal alignment according to this embodiment.

The horizontal alignment disclosed inFIG. 7,FIG. 8, andFIG. 9is taken as an example of the horizontal alignment mentioned in the descriptions for Step220. In practice, the processing circuit110can perform disparity histogram analysis for two-dimensional (2D) translation of warped images. For example, the processing circuit110calculates the number of pixels with regard to displacement (more particularly, horizontal displacement) for each of the images612,614, and616, in order to generate a disparity histogram for each of the images612,614, and616, such as that shown inFIG. 10. As shown inFIG. 10, the horizontal axis represents the displacement (more particularly, the horizontal displacement) of at least one pixel (e.g., a single pixel or a group of pixels) within the image under consideration in comparison with a certain image, and the vertical axis represents the number of pixels, where the image under consideration can be any of the images612,614, and616, and the aforementioned certain image can be the base image or a specific image selected from the images612,614, and616. By performing the disparity histogram analysis, the processing circuit110can determine whether to or how to crop the image under consideration, in order to perform the horizontal alignment. For example, the processing circuit110performs the horizontal alignment on the image612by cropping a portion of the image612to obtain the image622. In another example, the processing circuit110performs the horizontal alignment on the image614by cropping a portion of the image614to obtain the image624. In another example, the processing circuit110performs the horizontal alignment on the image616by cropping a portion of the image616to obtain the image626. Similar descriptions are not repeated in detail for this embodiment.

According to an embodiment of the present invention, such as a combination of the embodiments respectively shown inFIGS. 5-10, the processing circuit110performs the learning/training procedure, the multi-view vertical alignment, and the horizontal alignment as disclosed above, and further performs sequence reproduction, for reproducing the 3D visual effect mentioned above. For example, the processing circuit110performs the sequence reproduction by generating a series of warped images {613-1,613-2, . . . } that vary from the image612to the image614and by generating a series of warped images {615-1,615-2, . . . } that vary from the image614to the image616to output an image sequence {612, {613-1,613-2, . . . },614, {615-1,615-2, . . . },616}, in order to display animation based upon the images of the image sequence {612, {613-1,613-2, . . . },614, {615-1,615-2, . . . },616}, for reproducing the 3D visual effect. Similar descriptions are not repeated in detail for this embodiment.

FIG. 11illustrates the apparatus100shown inFIG. 1according to an embodiment of the present invention, where the processing circuit110thereof comprises some processing modules involved with the image processing method200shown inFIG. 4, and can selectively operate with aid of motion information generated by some motion sensors130when needed. Examples of the processing modules mentioned above may comprise a feature extraction module1102(labeled “Feature extraction”), a feature matching module1104(labeled “Feature matching”), a trained camera parameter prior module1112(labeled “Trained camera parameter prior”), a multi-view vertical alignment module1114(labeled “Multi-view vertical alignment”), a horizontal alignment module1116(labeled “Horizontal alignment”), a various 3D visual effect user interface (UI) module1118(labeled “UI for various 3D visual effect”), and an image warping module1122(labeled “Image warping”).

Based upon an input sequence that is input into the processing circuit110(more particularly, the image data of the input sequence, such as the image data of the plurality of images mentioned in Step210), the feature extraction module1102and the feature matching module1104are arranged to perform the feature extraction and the feature matching disclosed above, respectively, while the trained camera parameter prior module1112is arranged to store results of the learning/training results regarding the aforementioned solution such as the predefined solution space, and more particularly, some trained camera parameters that are obtained during the learning/training procedure. In addition, the multi-view vertical alignment module1114and the horizontal alignment module1116are arranged to perform at least one portion (e.g. a portion or all) of the multi-view vertical alignment disclosed above and at least one portion (e.g. a portion or all) of the horizontal alignment disclosed above, respectively, where the image warping module1122is arranged to perform image warping (more particularly, the aforementioned visually insensible image warping) when needed. Additionally, the various 3D visual effect UI module1118is arranged to reproduce the 3D visual effect mentioned above, and more particularly, to perform various kinds of 3D visual effects when needed.

According to this embodiment, the motion sensors130can be optional since the processing circuit110can operate properly and correctly without the aid of the aforementioned motion information generated by the motion sensors130, and therefore, the information paths from the motion sensors130to the processing circuit110are illustrated with dashed lines to indicate the fact that the motion sensors130can be optional. However, in a situation where the apparatus100is equipped with the motion sensors130, the calculation load of the processing circuit110can be decreased since the aforementioned motion information may be helpful. Similar descriptions are not repeated in detail for this embodiment.

FIG. 12illustrates two images IMG1 and IMG2 under processing of global/local coordinate transformation involved with the image processing method200shown inFIG. 4according to an embodiment of the present invention, andFIG. 13illustrates the global/local coordinate transformation performed on the two images IMG1 and IMG2 shown inFIG. 12.

As shown inFIG. 12, the processing circuit110can perform image processing on the images IMG1 and IMG2 by performing rotating, shifting, cropping, and/or warping operations on the images IMG1 and IMG2, respectively, where the warped rectangles respectively illustrated in the images IMG1 and IMG2 shown inFIG. 12(i.e. those depicted with non-dashed lines) may represent the processed results of the images IMG1 and IMG2, respectively. As shown inFIG. 13, the processing circuit110may determine a clipping region for each of the processed results of the images IMG1 and IMG2 by virtually “overlapping” the images IMG1 and IMG2 and the processed results thereof on a set of global coordinates, which can be regarded as a common set of global coordinates for processing the images IMG1 and IMG2. For example, on the set of global coordinates, the start point for the image IMG1 can be located at the origin, and the start point for the image IMG2 can be located at a specific point on the horizontal axis, where the clipping region can be a maximum rectangular region available for both of the processed results of the images IMG1 and IMG2.

In practice, the start point for the image IMG1 and the start point for the image IMG2 can be determined based upon the proposed distance from the eyes of a viewer (e.g. the user) to the point of focus, such as the proposed distance (more particularly, the proposed horizontal distance) between the viewer and the convergence point where the sightlines of the respective eyes of the viewer converge. For example, in a situation where the 3D visual effect is supposed to be 3D display (or stereoscopic display), the processing circuit110can align the images IMG1 and IMG2 to background motion(s), where the embodiment shown inFIG. 14is typical of this situation. As a result, the start point for the image IMG1 and the start point for the image IMG2 may be close to each other. In another example, in a situation where the 3D visual effect is supposed to be the MAV visual effect mentioned above, the processing circuit110can align the images IMG1 and IMG2 to foreground motion(s), where the embodiment shown inFIG. 15is typical of this situation. As a result, the start point for the image IMG1 and the start point for the image IMG2 may be far from each other, in comparison with the embodiment shown inFIG. 14.

FIG. 16illustrates a portrait mode of a plurality of display modes involved with the image processing method200shown inFIG. 4according to an embodiment of the present invention, andFIG. 17illustrates a panorama mode of the plurality of display modes according to this embodiment, where the processing circuit110is capable of switching between different display modes within the plurality of display modes, and more particularly, is capable of switching between the portrait mode and the panorama mode.

For example, referring toFIG. 16, some warped images bounded to the aforementioned predefined solution space (e.g. the pre-trained solution space) are aligned to foreground, such as the location where an actor/actress is supposed to be in front of the viewer. When it is detected that switching to the panorama mode is required (e.g. the viewer such as the user triggers the switching operation, or a predetermined timer triggers the switching operation), the processing circuit110rearrange these warped images in a reversed order and utilizes the rearranged warped images as the warped images for the panorama mode, where the leftmost warped image shown inFIG. 16(i.e. the warped image IMG11 thereof) is arranged to be the rightmost warped image shown inFIG. 17(i.e. the warped image IMG11 thereof), and the rightmost warped image shown inFIG. 16(i.e. the warped image IMG17 thereof) is arranged to be the leftmost warped image shown inFIG. 17(i.e. the warped image IMG17 thereof). This is for illustrative purposes only, and is not meant to be a limitation of the present invention. According to a variation of this embodiment, in a situation where the processing circuit110prepares the warped images for the panorama mode first, when it is detected that switching to the portrait mode is required (e.g. the viewer such as the user triggers the switching operation, or a predetermined timer triggers the switching operation), the processing circuit110rearrange the warped images for the panorama mode in a reversed order and utilizes the rearranged warped images as the warped images for the portrait mode. Similar descriptions are not repeated in detail for this variation.

Based upon the embodiments/variations disclosed above, the image processing method200comprises performing automatic multiple image alignment in terms of the aforementioned predefined solution space (e.g. the pre-trained solution space) for reproducing the 3D visual effect from an image sequence, in which each image can be captured with an uncalibrated camera module (e.g. the camera module130) or an uncalibrated hand-held camera. More particularly, the image processing method200comprises learning a model consisted of physical camera parameters, which can be utilized for performing visually insensible image warping. Examples of the parameters under consideration may comprise the extrinsic parameters for rotational variations, the intrinsic parameters for camera calibration matrix and lens distortion. The image processing method200further comprises, from the corresponding feature points of the input sequence, performing the multi-view vertical alignment constrained by the learned camera parameters. The alignment process turns out a constrained optimization problem for image warping that is visually insensible to human vision. In addition, the image processing method200further comprises performing the horizontal alignment through the disparity analysis from the vertically-aligned matching points. Additionally, the image processing method200further comprises utilizing the UI such as a graphical UI (GUI) to reproduce the 3D visual effect by using the extracted alignment information and the warped image sequence.

It is an advantage of the present invention that the present invention method and apparatus can generate warped images bounded to the aforementioned solution such as the predefined solution space (e.g. the pre-trained solution space) to make the associated learned geometric distortion be insensible by human vision, so that there is no artifact in respective reproduced image. In addition, the optimization over the solution space according to the learned geometry constraint can always generate rational results. Regarding the implementation of the present invention method and apparatus, the working flow of the associated calculations can be highly paralleled, and the associated computational complexity is low and the required memory resource is economy. In contrast to the related art, the present invention method and apparatus are robust to the image noises and outliers. Additionally, the present invention method and apparatus preserve the relative disparity/depth information in the warped image sequence, which is very important to image-based 3D applications.