Image correction system, image correction method, and computer program product

This invention is concerning an image correction system that includes an image-capturing unit configured to acquire a captured image by capturing a subject; a tilt recognition unit configured to recognize tilt of an image-capturing direction of the image-capturing unit relative to an image-capturing direction in which the image-capturing unit being directly opposite to the subject captures the subject; a first correction unit configured to generate a corrected image by correcting distortion in the captured image due to the tilt; and a second correction unit configured to acquire a corrected captured image by correcting non-linear distortion in the corrected image.

TECHNICAL FIELD

The present invention relates to an image correction system, an image correction method, and a computer program product.

BACKGROUND ART

It is known that there are two types of distortion, linear distortion (trapezoidal distortion) and non-linear distortion, that can occur in a projected image projected by a projection apparatus such as a projector. Linear distortion occurs, for example, when the projection apparatus deviates from a position directly opposite to a projection surface. In other words, linear distortion occurs when a projection apparatus designed to be disposed in a position perpendicular to a projection surface is not directly opposite to the projection surface.

Non-linear distortion occurs, for example, when a projection surface such as a hanging projector screen is uneven. In general, an uneven projection surface such as a hanging projector screen lacks linearity, thus non-linear distortion occurs.

A projection system (projector and camera system) that includes a projection apparatus and an image-capturing apparatus is a well-known technology to correct distortion in a projected image. For example, this technology corrects distortion in a projected image projected by the projection apparatus such as a projector on the basis of a captured image captured by a portable image-capturing apparatus (such as a digital camera, a web camera, and a camera built in a mobile phone or a smart phone, and hereinafter referred to as an “external camera”).

Patent document 1 is known as a technical document that discloses a technique to correct distortion in a projected image. Patent document 1 discloses an invention to correct distortion in a projected image by using a captured image captured by a mobile phone with a built-in camera.

However, linear distortion occurs in a corrected projected image in some cases when a captured image acquired by an image-capturing apparatus capturing a projected image is used, for example, in order to correct distortion in the projected image due to unevenness of a projection surface.

CITATION LIST

Patent Literature

DISCLOSURE OF INVENTION

Problem to be Solved by the Invention

The present invention has been made in view of the disadvantage described above, and it is an object of the present invention to provide an image correction system, an image correction method and a computer program product with which, if a captured image acquired by an image-capturing apparatus capturing a subject is distorted, distortion of the subject contained in the captured image can be eliminated by utilizing the captured image.

Means for Solving Problem

To achieve the object, an image correction system according to the present invention includes: an image-capturing unit configured to acquire a captured image by capturing a subject; a tilt recognition unit configured to recognize tilt of an image-capturing direction of the image-capturing unit relative to an image-capturing direction in which the image-capturing unit being directly opposite to the subject captures the subject; a first correction unit configured to generate a corrected image by correcting distortion in the captured image due to the tilt; and a second correction unit configured to acquire a corrected captured image by correcting non-linear distortion in the corrected image.

An image correction method according to the present invention comprising: acquiring, by an image-capturing unit, a captured image by capturing a subject; recognizing, by a tilt recognition unit, tilt of an image-capturing direction of the image-capturing unit relative to an image-capturing direction in which the image-capturing unit being directly opposite to the subject captures the subject; generating, by a first correction unit, a corrected image by correcting distortion in the captured image due to the tilt; and acquiring, by a second correction unit, a corrected captured image by correcting non-linear distortion in the corrected image.

A computer program product according to the present invention comprising a non-transitory computer-usable medium having a computer program that causes a computer to function as: an image-capturing unit configured to acquire a captured image by capturing a subject; a tilt recognition unit configured to recognize tilt of an image-capturing direction of the image-capturing unit relative to an image-capturing direction in which the image-capturing unit being directly opposite to the subject captures the subject; a first correction unit configured to generate a corrected image by correcting distortion in the captured image due to the tilt; and a second correction unit configured to acquire a corrected captured image by correcting non-linear distortion in the corrected image.

Effect of the Invention

According to the present invention, if a captured image acquired by an image-capturing apparatus capturing a subject is distorted, the distortion of the subject contained in the captured image can be eliminated by utilizing the captured image.

BEST MODES FOR CARRYING OUT THE INVENTION

Embodiments will now be described in detail below of an image correction system, an image correction method and a computer program product with reference to the accompanying drawings.

First Embodiment

FIG. 1is a block diagram illustrating an example of a configuration of functional blocks in an image correction system1according to a first embodiment. The image correction system1of the first embodiment includes an image-capturing apparatus2and a projection apparatus3. The image-capturing apparatus2includes an image-capturing unit21and a tilt recognition unit22. The projection apparatus3includes a controller31, a projection unit32, a first correction unit33, and a second correction unit34.

The image-capturing unit21captures a projected image projected by the projection unit32. The image-capturing unit21transmits the captured image to the first correction unit33.

The tilt recognition unit22recognizes tilt of an image-capturing direction of the image-capturing unit21relative to an image-capturing direction in which the image-capturing unit21being directly opposite to a projection surface captures the projection surface. The tilt recognition unit22recognizes the tilt with an accelerometer when the image-capturing unit21captures a projected image. The accelerometer, for example, provides such information as the image-capturing unit21is tilted by 10 degrees relative to the ground. The tilt recognition unit22transmits information indicating the tilt (hereinafter referred to as “tilt information”) to the first correction unit33.

The accelerometer will be described. When the accelerometer is a three-axis accelerometer, trapezoidal distortion (linear distortion) in the longitudinal direction in a captured image can be corrected. When the accelerometer is a four-axis to six-axis accelerometer, trapezoidal distortion (linear distortion) in the lateral direction can also be corrected, and correction of gyro-like spin is also possible.

In general, a projection surface such as a screen on which the projection unit32projects an image is perpendicular to the ground. This can lead to an assumption that, when an external camera (image-capturing apparatus2) is directly opposite to the projection surface, the external camera (image-capturing apparatus2) is perpendicular to the ground. Therefore, a corrected image acquired by correcting tilt of a captured image by using tilt information recognized at the tilt recognition unit22is identical to a captured image (a captured image captured from the front) that is acquired when the external camera being directly opposite to the projection surface captures the projection surface such as a screen.

The image-capturing apparatus2may transmit a captured image and tilt information to the projection apparatus3in either of a wireless or wired manner.

The controller31inputs, into the projection unit32, an input image to be projected by the projection unit32as a projected image. The controller31also inputs the input image into the second correction unit34in order to correct non-linear distortion in the input image. The controller31receives, from the second correction unit34, a corrected input image that is generated by correcting the non-linear distortion in the input image. The controller31then inputs the corrected input image into the projection unit32.

The projection unit32projects the input image or the corrected input image as a projected image on a projection surface. The projection surface is, for example, a screen, a wall or a white board. The input image and the corrected input image will be described later in detail.

The first correction unit33generates a corrected image from a captured image on the basis of tilt information. The first correction unit33transmits the corrected image to the second correction unit34. The first correction unit33will be described later in detail.

The second correction unit34corrects non-linear distortion in the input image on the basis of the corrected image, and generates a corrected input image. The second correction unit34will be described later in detail.

FIG. 2is a diagram illustrating an example of an embodiment of the image correction system1according to the first embodiment. The image correction system1inFIG. 2is implemented by an external camera (image-capturing apparatus2) and a short-throw projector (projection apparatus3). The short-throw projector (projection apparatus3) is projecting a pattern image for correcting distortion in a projected image on a screen (projection surface). Intersections of the lines contained in the pattern image inFIG. 2are used by the second correction unit34, as corresponding points for associating a corrected image generated by the first correction unit33with an input image.

FIG. 3is a diagram illustrating an example of a pattern image projected by the projection unit32of the image correction system1according to the first embodiment.FIG. 3is an example of a pattern image with a grid pattern.

In the example ofFIG. 2, the short-throw projector (projection apparatus3) is projecting the pattern image. In the example ofFIG. 2, non-linear distortion occurs in the projected pattern image because of the distortion of the screen (projection surface). The second correction unit34corrects the non-linear distortion in the projected image by using the intersections of the lines contained in the grid pattern image as the corresponding points between a corrected image and an input image.

Described are an input image input by the controller31into the projection unit32, and distortion in a projected image.FIG. 4is a diagram illustrating an example of non-linear distortion in a projected image projected by the projection unit32of the image correction system1according to the first embodiment. The example ofFIG. 4illustrates a case in which a square is projected on the projection surface. In the example ofFIG. 4, non-linear distortion occurs in the square of the projected image because of the distortion of the projection surface.

When the image-capturing unit21acquires a captured image to detect this non-linear distortion, there is a case in which linear distortion occurs in the captured image because the image-capturing unit21deviates from a position directly opposite to the projection surface.

FIG. 5is a diagram illustrating an example of a case in which linear distortion and non-linear distortion are contained in a captured image captured by the image-capturing unit21of the image correction system1according to the first embodiment. The example of the captured image inFIG. 5is acquired, for example, when the external camera (image-capturing apparatus2) rotates horizontally to the ground in the right direction about the y-axis of the coordinate axes of the external camera (image-capturing apparatus2) inFIG. 2, and deviates from a position directly opposite to the screen (projection surface).

The first correction unit33corrects a captured image containing linear distortion and non-linear distortion as illustrated inFIG. 5by eliminating the linear distortion due to deviation of the image-capturing unit21from a position directly opposite to the projection surface, and generates a corrected image from the captured image.

FIG. 6is a diagram illustrating an example of a projected image after correction projected by the projection unit32of the image correction system1according to the first embodiment. The second correction unit34generates a corrected input image on the basis of the input image and the corrected image generated by the first correction unit33from the captured image. When the projection unit32projects the corrected input image, a projected image after correction is displayed on the projection surface. The projected image after correction is identical to the input image.

Next, described in detail is the first correction unit33in the image correction system1according to the first embodiment.FIG. 7is a diagram illustrating an example of functional blocks in the first correction unit33of the image correction system1of the first embodiment. The first correction unit33includes an intrinsic parameter matrix calculation unit331, a rotation matrix calculation unit332, a three-dimensional coordinate calculation unit333, a projection transformation matrix calculation unit334, and a corrected image generation unit335.

Formula 1 is an intrinsic parameter matrix A. Described are components of the intrinsic parameter matrix A of Formula 1, where f denotes the focal length of the image-capturing unit21, cx denotes x-coordinate of the principal point (cx, cy) of the image-capturing unit21, and cy denotes y-coordinate of the principal point (cx, cy) of the image-capturing unit21.

The intrinsic parameter matrix calculation unit331needs to acquire the focal length f and the principal point (cx, cy) of the image-capturing unit21in order to calculate the intrinsic parameter matrix A.

The intrinsic parameter matrix calculation unit331may acquire the focal length f and the principal point (cx, cy) on the basis of exchangeable image file format (Exif) information that is tag information of a captured image. The intrinsic parameter matrix calculation unit331may acquire the focal length f and the principal point (cx, cy) by camera calibration of the image-capturing unit21.

The intrinsic parameter matrix A is used when the three-dimensional coordinate calculation unit333calculates, from two-dimensional coordinates of the captured image, three-dimensional coordinates in the real space corresponding to the two-dimensional coordinates.

The rotation matrix calculation unit332acquires tilt information from the tilt recognition unit22. The tilt information includes information indicating a tilt θ of an image-capturing direction of the image-capturing unit21relative to an image-capturing direction in which the image-capturing unit21being directly opposite to the projection surface captures the projection surface. The rotation matrix calculation unit332calculates a rotation matrix R representing coordinate transformation that rotates coordinates by just the tilt θ.

The rotation matrix R will now be described.FIG. 8is a diagram illustrating an example of the tilt θ, relative to the projection surface, of the image-capturing unit21of the image correction system1according to the first embodiment. The example ofFIG. 8illustrates a case in which an image-capturing direction (z-axis direction) of the image-capturing apparatus2is rotated by just the tilt θ around the x-axis so that the image-capturing unit21is directly opposite to the projection surface. InFIG. 8, x, y1, and z1denote the coordinate axes before rotation by just the tilt θ, and x, y2, and z2denote the coordinate axes after rotation by just the tilt θ.

Coordinates (x, y2, z2) can be calculated from the following formula by using the rotation matrix R that represents rotation by just the tilt θ.

The rotation matrix calculation unit332calculates a matrix that satisfies Formula 2 as the rotation matrix R.

The three-dimensional coordinate calculation unit333calculates three-dimensional coordinates corresponding to coordinates of a captured image by perspective projection transformation in which the coordinates of the captured image and the intrinsic parameter matrix A of the image-capturing unit21are used. Perspective projection transformation will now be described. Capturing a projected image by the image-capturing unit21corresponds to an operation of transforming the three-dimensional coordinates in the real space into the two-dimensional coordinates in the captured image. This transformation is called perspective projection transformation. Perspective projection transformation can be represented by a perspective projection transformation matrix P of the image-capturing unit21. The perspective projection transformation matrix P is expressed by the following formula.

The matrix A in Formula 3 is the above-mentioned intrinsic parameter matrix of the image-capturing unit21. A matrix [Rt] is an extrinsic parameter matrix. The extrinsic parameter matrix represents rotation and translation of the image-capturing unit21.

When the image-capturing unit21is tilted by just the tilt θ as illustrated inFIG. 8and captures a projected image, a perspective projection transformation matrix P1is expressed by the following formula.

Formula 4 defines t=0 because the image-capturing unit21itself is the origin, and because the image-capturing unit21rotates about the origin.

The relation between the three-dimensional coordinates in the real space and the two-dimensional coordinates in the captured image can be represented as follows.

Formula 5 represents equality except for scale. That is, Formula 5 represents equality in a homogeneous coordinate system.

When the image-capturing unit21being directly opposite to the projection surface captures the projection surface, a perspective projection transformation matrix P2is expressed by the following formula.

The relation between the three-dimensional coordinates in the real space and the two-dimensional coordinates in the captured image can be represented as follows.

In general, Formula 7 is equivalent to Formula 8 below. In Formula 7, R is defined as the identity matrix, and t is defined as 0. Thus, Formula 8 below defines R=E (identity matrix) and t=0.

The three-dimensional coordinate calculation unit333calculates three-dimensional coordinates in the real space corresponding to the coordinates of the captured image by using an inverse transformation matrix P1−1that is the inverse of the perspective projection transformation matrix P1. The three-dimensional coordinate calculation unit333does not need to calculate the corresponding three-dimensional coordinates in the real space with respect to all the points in the two-dimensional coordinates of the captured image. In other words, the three-dimensional coordinate calculation unit333may calculate corresponding points in the three-dimensional coordinates corresponding to the points in the captured image as many a number as needed (four) to calculate a projection transformation matrix H to be described later.

The projection transformation matrix calculation unit334calculates the projection transformation matrix H that transforms the coordinates of the captured image into coordinates of the corrected image. The projection transformation matrix H is used by the corrected image generation unit335to calculate the coordinates of the corrected image corresponding to the coordinates of the captured image.

FIG. 9is a diagram for explaining an example of a calculation method of the projection transformation matrix H calculated by the projection transformation matrix calculation unit334of the image correction system1according to the first embodiment. The projection transformation matrix H is expressed by the following formula.

Formula 9 and Formula 10 represent equality except for scale. That is, Formula 9 and Formula 10 represent equality in a homogeneous coordinate system.

Calculating coefficients of the projection transformation matrix H results in calculating Formula 11 below.

Formula 11 includes eight unknown coefficients h1to h8. Formula 11 is obtained for each pair of coordinates (x1, y1) of the captured image and coordinates (x2, y2) of the corrected image corresponding to the coordinates (x1, y1) of the captured image. In order to determine the eight unknown coefficients h1to h8, information on at least four points of the corresponding points in the corrected image corresponding to the points in the captured image. The coordinates (x2, y2) of the corrected image corresponding to the coordinates (x1, y1) of the captured image can be calculated by applying the perspective projection transformation matrix P2to the three-dimensional coordinates in the real space corresponding to the above-described coordinates of the captured image.

The projection transformation matrix calculation unit334may calculate the projection transformation matrix H by the least-square method that uses a plurality of pieces of information on the corresponding points in the corrected image corresponding to the points in the captured image. Thereby, if the above-described corresponding points acquired by the perspective projection transformation matrix calculation unit334contain some errors with respect to correspondence, a projection transformation matrix H can be calculated that contains minimum errors. The least-square method is not the only method for reducing the effect of errors with respect to correspondence, and other methods can be used for this purpose.

The corrected image generation unit335generates a corrected image by applying the projection transformation matrix H to all the pixel coordinates in the captured image.

FIG. 10is a flowchart for explaining an example of a generation method of the corrected image by the image correction system1according to the first embodiment.

The first correction unit33receives a captured image from the image-capturing unit21. The first correction unit33also receives tilt information from the tilt recognition unit22(Step S1). The intrinsic parameter matrix calculation unit331calculates an intrinsic parameter matrix A of the image-capturing unit21on the basis of Exif information that is tag information of the captured image (Step S2). The rotation matrix calculation unit332calculates a rotation matrix R on the basis of the tilt information (Step S3).

The three-dimensional coordinate calculation unit333calculates a perspective projection transformation matrix P1from the intrinsic parameter matrix A of the image-capturing unit21and the rotation matrix R. The three-dimensional coordinate calculation unit333calculates three-dimensional coordinates corresponding to coordinates of the captured image by applying an inverse transformation matrix P1−1that is the inverse of the perspective projection transformation matrix P1to the coordinates of the captured image (Step S4). The projection transformation matrix calculation unit334calculates coordinates of a corrected image corresponding to the three-dimensional coordinates by applying a perspective projection transformation matrix P2to the three-dimensional coordinates (Step S5).

At Step S4and Step S5, the first correction unit33calculates corresponding points of the coordinates in the corrected image corresponding to points of the coordinates in the captured image. The projection transformation matrix calculation unit334calculates a projection transformation matrix H by the least-square method, in which the corresponding points of the coordinates in the corrected image corresponding to the points of the coordinates in the captured image is used (Step S6). The corrected image generation unit335generates the corrected image by applying the projection transformation matrix H to all the pixel coordinates in the captured image (Step S7).

Next, described is the second correction unit34in the image correction system1according to the first embodiment.FIG. 11is a diagram illustrating an example of functional blocks in the second correction unit34of the image correction system1of the first embodiment. The second correction unit34includes a correction region determination unit341, a corresponding point detection unit342, a projection transformation matrix calculation unit343, and a corrected input image generation unit344.

The correction region determination unit341determines a correction region in the corrected image.FIG. 12is a diagram for explaining an example of a correction method used in the second correction unit34of the image correction system1according to the first embodiment. A region41inFIG. 12is an example of the correction region determined by the correction region determination unit341. The correction region determination unit341may determine the region41by any method. For example, the correction region determination unit341determines, as the correction region, a rectangle having the equal aspect ratio to that of the projection region of the projected image.

Pixel values of an input image are associated with pixel values of the correction region. In general, a correction region is smaller than an input image in size. Thus, the input image needs to be reduced so that the input image is fully displayed on the correction region.

Back toFIG. 11, the corresponding point detection unit342detects corresponding points in the pattern image on a corrected input image corresponding to points in a pattern image on the corrected image. For example, in the example ofFIG. 12, the corresponding point detection unit342associates a point43in the corrected image with a point53in the corrected input image. The corresponding point detection unit342associates a point44in the corrected image with a point54in the corrected input image. The corresponding point detection unit342associates a point45in the corrected image with a point55in the corrected input image. The corresponding point detection unit342associates a point46in the corrected image with a point56in the corrected input image. Thereby, a unit region42in the corrected image is associated with a unit region52in the corrected input image.

The projection transformation matrix calculation unit343determines a projection transformation matrix Hn′ for each unit region n. For example, the projection transformation matrix calculation unit343determines the projection transformation matrix Hn′ on the basis of Formula 11 obtained by four points detected by the corresponding point detection unit342.

The corrected input image generation unit344determines pixel values of coordinates obtained by applying the projection transformation matrix Hn′ to coordinates in the correction region on the basis of pixel values of the coordinates in the correction region to which the projection transformation matrix Hn′ is applied. The pixel values are not based on pixel values of the correction region in the corrected image but on pixel values of the input image fitted to the region41.

For example, the corrected input image generation unit344determines a pixel value57of the corrected input image on the basis of a pixel value47of the input image in a position corresponding to a position in the correction region associated by the projection transformation matrix H1′. The corrected input image generation unit344determines a pixel value of a certain point in the corrected input image by the projection transformation matrix Hn′. Thereby, the corrected input image generation unit344generates a corrected input image51.

FIG. 13is a diagram illustrating an example of relations among projection transformation matrices that correspond to projection, image-capturing, and correction by the image correction system1according to the first embodiment. Projecting an input image by the projection unit32corresponds to the transformation represented by an inverse matrix (Hn′)−2that is the inverse of the projection transformation matrix Hn′. Capturing a projected image corresponds to the transformation represented by the inverse matrix H−1that is the inverse of the projection transformation matrix H calculated by the first correction unit33. Therefore, the projected image and the corrected image are identical. That is, a matrix E is the identity matrix.

The second correction unit34transforms coordinates of the correction region in the corrected image into coordinates of the corrected input image by the projection transformation matrix Hn′. The second correction unit34determines pixel values of the coordinates in the corrected input image to be the same pixel values as those in the input image that has been fitted to the correction region. Thus, an image in the correction region and a projected image after correction are identical. That is, a matrix Enis the identity matrix.

FIG. 14is a flowchart for explaining an example of a correction method used in the second correction unit34of the image correction system1according to the first embodiment.

The corresponding point detection unit342calculates corresponding points between a corrected image and a corrected input image by using grid points in the grid pattern image (Step S11). This associates unit regions in the corrected image with unit regions in the corrected input image. The correction region determination unit341determines a correction region to be corrected by the second correction unit34(Step S12). The correction region determination unit341associates pixel values of the input image with pixel values of the correction region (Step S13). The projection transformation matrix calculation unit343calculates projection transformation matrices for respective unit regions in the correction region (Step S14). The corrected input image generation unit344transforms certain coordinates in the correction region in the corrected image into coordinates of the corrected input image by the projection transformation matrix Hn′. The corrected input image generation unit344generates the corrected input image by determining pixel values of the coordinates in the corrected input image to be the same pixel values as those in the input image that has been fitted to the correction region (Step S15).

FIG. 15is a diagram for explaining an operation of the image correction system1according to the first embodiment.

The controller31transmits to the projection unit32a signal indicating the start of calibration of a projected image (Step S21). The projection unit32projects a pattern image as the projected image on a projection surface (Step S22). The image-capturing unit21captures the projected image and acquires a captured image (Step S23). The tilt recognition unit22recognizes a tilt θ of the image-capturing unit21at the time of capturing at Step S23(Step S24). The first correction unit33generates a corrected image by correcting the captured image on the basis of an intrinsic parameter matrix of the image-capturing unit21and the tilt θ recognized at the time of capturing (Step S25).

The second correction unit34calculates, as distortion correction parameters, a correction region in the corrected image, and a projection transformation matrix calculated for each unit region and used for transforming a corrected image into a corrected input image (Step S26). The following describes S26in detail. First, the correction region in the corrected image is described. The correction region in the corrected image will be a projection region in a corrected projected image. Thus, the second correction unit34determines an aspect ratio of the correction region to be equal to that of an input image. In general, a corrected projected image (corrected input image) projected on a projection surface corresponding to a correction region in a corrected image is a reduced input image.

Next, described is the projection transformation matrix calculated for each unit region and used for transforming a corrected image into a corrected input image. The second correction unit34detects corresponding points between a pattern image on the corrected image and that on the input image. The second correction unit34calculates a projection transformation matrix for each unit region defined by line segments joining the corresponding points.

The second correction unit34stores the distortion correction parameter into a memory of the image correction system1(Step S27). The second correction unit34receives an input image from the controller31(Step S28). The second correction unit34reads out the distortion correction parameter that has been stored at Step S27from the memory (Step S29). The second correction unit34corrects the input image on the basis of the distortion correction parameter (Step S30).

The controller31determines whether or not it has received a command that instructs termination of calibration of the projected image (Step S31). When the controller31receives a command that instructs termination of calibration (Yes at Step S31), the controller31terminates the process. When the controller31does not receive a command that instructs termination of calibration (No at Step S31), the process returns to Step S28.

The pattern image used in the image correction system1according to the first embodiment may be a pattern image other than the grid pattern image.FIG. 16is a diagram illustrating a first modification of the pattern image projected, by the projection unit32of the image correction system1of the first embodiment. The example ofFIG. 16illustrates a pattern image of a chessboard pattern.FIG. 17is a diagram illustrating a second modification of the pattern image projected by the projection unit32of the image correction system1of the first embodiment. The example ofFIG. 17illustrates a pattern image with circles.

The corresponding point detection unit342of the second correction unit34may detect corresponding points without using a pattern image. For example, the corresponding point detection unit342may apply, to a content image of a user, a corresponding-point extraction method in which scale invariant feature transform (SIFT) is used. When the controller31determines that the content image of a user has few image features, the controller31may determine not to apply the corresponding-point extraction method by SIFT to the content image. Specifically, the controller31may calculate frequencies of content images, and determine not to use a content image with few high-frequency components.

The correction processing by the first correction unit33and the second correction unit34may be performed in an external computer connected through a network instead of in the projection apparatus3. The image correction system1may be connected with the external computer in either of a wired or wireless manner.

Using an accelerometer is not the only method for the tilt recognition unit22to recognize tilt of the image-capturing apparatus2. For example, the tilt recognition unit22may recognize tilt of the image-capturing apparatus2on the basis of image processing in which a captured image captured by the image-capturing apparatus2is used. In the image processing, for example, the tilt recognition unit22may recognize the tilt of the image-capturing apparatus2by detecting an upper and a lower black edges of a projection surface (such as a screen) contained in a captured image, and using the ratio of the length of the upper black edge to that of the lower black edge.

Second Embodiment

Next, described is an image correction system1according to a second embodiment.FIG. 18is a block diagram illustrating an example of a configuration of functional blocks in the image correction system1of the second embodiment. The image correction system1of the second embodiment differs from the image correction system1of the first embodiment in that the image-capturing apparatus2includes the first correction unit33and the second correction unit34. This allows the projection apparatus3to have a simple configuration that includes only basic functions for projection. The image correcting system1of the second embodiment also differs in that it includes a storage unit23that stores a corrected captured image acquired by correcting non-linear distortion in a corrected image by the second correction unit34. Detailed description of the operation of the image correction system1according to the second embodiment is omitted because the description thereof is the same as that of the operation of the image correction system1according to the first embodiment.

In the description of the image correction system1of the first embodiment, the projection apparatus3projects an image on a screen as a projection surface, and the image-capturing apparatus2captures the projection surface, for example. However, the projection surface for the projection apparatus3and the subject for the image-capturing apparatus2are not limited to a screen. In the description of the image correction system1of the second embodiment, described is a case in which the subject of the image-capturing apparatus2is not the screen.

FIG. 19is a diagram illustrating an example of a subject of the image-capturing apparatus2of the image correction system1according to the second embodiment. The example ofFIG. 19illustrates a case in which the projection apparatus3projects the pattern image with circles illustrated inFIG. 17on a book, and the image-capturing apparatus2acquires a captured image by capturing the book. Following description is a case of correcting a captured image acquired by capturing a book. First, the image-capturing unit21acquires a captured image by capturing a book on which a pattern image is projected. Then, the first correction unit33acquires a corrected image by correcting trapezoidal distortion (linear distortion) in the captured image because of the image-capturing apparatus2not being directly opposite to the book as the subject when capturing it.

Next, the second correction unit34acquires a corrected captured image by correcting non-linear distortion caused by, for example, a bend in the book. Specifically, the second correction unit34receives the pattern image illustrated inFIG. 17as an input image, and calculates the above-described projection transformation matrix Hn′ from the input image and the corrected image. In other words, the second correction unit34calculates the above-described projection transformation matrix Hn′ by establishing correspondences between the circles in the pattern image illustrated inFIG. 17and the circles in the pattern image contained in the corrected image. The pattern image projected by the projection unit32is not limited to the pattern image illustrated inFIG. 17, but any pattern image can be used in order for the second correction unit34to calculate the projection transformation matrix Hn′. The second correction unit34stores projection transformation matrices Hn′ for respective regions n (regions including respective circles in the pattern image) in the storage unit23. Thereby, the second correction unit34can correct non-linear distortion identical to the non-linear distortion caused by, for example, a bend in the book illustrated inFIG. 19. For example, the projection transformation matrices Hn′ stored in the storage unit23can be used in a case in which the subject is a book that has the same shape as that of the book inFIG. 19and on which the same non-linear distortion occurs as that on the book inFIG. 19.

The second correction unit34acquires a corrected captured image by transforming the respective regions n in the corrected image by the projection transformation matrices Hn′, and stores the corrected captured image in the storage unit23. The image-capturing apparatus2may transmits the corrected captured image to the projection apparatus3, and then the projection apparatus3may project the corrected captured image. Thereby, when a book is captured as a subject to acquire a captured image, and then the first correction unit33and the second correction unit34correct the captured image to acquire a corrected captured image, the corrected captured image can be displayed as a projected image on a projection surface such as a screen.

Third Embodiment

Next, described is an image correction system1according to a third embodiment.FIG. 20is a block diagram illustrating an example of a configuration of functional blocks in the image correction system1of the third embodiment. The image correction system1of the third embodiment includes the image-capturing apparatus2and a display apparatus4. The image correction system1of the third embodiment differs from the image correction systems1of the first and the second embodiments in that it includes the display apparatus4instead of the projection apparatus3. The display apparatus4includes a display unit5that displays a corrected captured image.

The configuration of the image-capturing apparatus2of the image correction system1of the third embodiment is the same as that of the second embodiment, and thus the description thereof is omitted. The detailed description of the operation of the image correction system1of the third embodiment is also omitted because the description thereof is the same as that of the image correction systems1of the first and the second embodiments.

FIGS. 21 and 22are diagrams each illustrating an example of a subject of the image-capturing apparatus2of the image correction system1according to the third embodiment.FIG. 21is an example of a book containing pages on which a pattern image is printed.FIG. 22is an example of a sheet of paper on which a pattern image is printed. Following description is an example of a case in which the subject of the image-capturing apparatus2is the book illustrated inFIG. 21. It should be noted that the following description is also applicable to a case in which the subject of the image-capturing apparatus2is the sheet of paper illustrated inFIG. 22.

First, the image-capturing unit21acquires a captured image by capturing a book on which a pattern image is printed. Then, the first correction unit33acquires a corrected image by correcting trapezoidal distortion (linear distortion) in the captured image because of the image-capturing apparatus2not being directly opposite to the book as the subject when capturing it. The second correction unit34reads out the same pattern image as the pattern image printed on the book from the storage unit23, and then calculates the above-described projection transformation matrix Hn′ from the pattern image and the corrected image.

The second correction unit34stores, in the storage unit23, projection transformation matrices Hn′ for respective regions n (regions including respective circles in the pattern image) in the corrected image. Thereby, the second correction unit34can correct non-linear distortion that is identical to the non-linear distortion caused by, for example, a bend in the book illustrated inFIG. 21. The second correction unit34acquires a corrected captured image by transforming the respective regions n in the corrected image by the projection transformation matrices, Hn′, and stores the corrected captured image in the storage unit23. The image-capturing apparatus2transmits the corrected captured image to the display apparatus4to display the corrected captured image on the display unit5of the display apparatus4. Thereby, when a book is captured as a subject to acquire a captured image, and the first correction unit33and the second correction unit34corrects the captured image to acquire a corrected captured image, the corrected captured image can be displayed on the display unit5of the display apparatus4.

The tilt recognition unit22, the controller31, the first correction unit33, and the second correction unit34of the image correction systems1according to the first to the third embodiments may be implemented either by hardware such as an integrated circuit (IC), or by software (computer program). The tilt recognition unit22, the controller31, the first correction unit33, and the second correction unit34of the image correction systems1of the first to the third embodiments may be implemented by combining hardware and software (computer program).

FIG. 23is a diagram for explaining an example of a hardware configuration to execute software (computer program) in the image correction systems1according to the first to the third embodiments. The image-capturing apparatus2of the first to the third embodiments, and the projection apparatus3of the first and the second embodiments include an auxiliary memory61such as a memory card, a main memory62such as a random access memory (RAM), a processor63such as a central processing unit (CPU), and a communication interface (I/F)64for communicating with other apparatuses. The auxiliary memory61, the main memory62, the processor63, and the communication I/F64are connected one another through a bus65.

The auxiliary memory61stores software (computer program). However, the software (computer program) may be provided as a computer program product recorded in a file in an installable or executable format in a recording medium, such as a CD-ROM, a flexible disk (FD), a CD-R, and a digital versatile disk (DVD), that is readable by a computer.

The software (computer program) may be stored on a computer connected to a network such as the Internet, and may be provided by downloading through the network. The software (computer program) may also be provided or distributed through a network such as the Internet.

The processor63reads out the software (computer program) stored in the auxiliary memory61such as a memory card, a storage medium such as a CD-ROM, or a computer on a network to execute it on the main memory62. Thereby, when the tilt recognition unit22, the controller31, the first correction unit33, and the second correction unit34are implemented by software (computer program), these functional blocks are implemented on the main memory62.

INDUSTRIAL APPLICABILITY

The image correction systems1according to the first to the third embodiments can provide an image correction system, an image correction method and a computer program that can eliminate distortion in a subject contained in a captured image by using the captured image even if the captured image acquired by the image-capturing apparatus2capturing the subject is distorted.

EXPLANATION OF LETTERS OR NUMERALS