Projection system, image processing device and projection method

A projection system includes a visible light projector, a camera and an image processor. The visible light projector projects an image with visible light onto the object. The camera captures an image of the object and has an optical axis not coinciding with an optical axis of the visible light projector. The image processor generates an image to be projected with visible light onto the object based on the image captured by the camera. The image processor includes a corrector and an image generator. The corrector corrects a deviation between a projection image and the object. The corrector is configured to calculate a projection region on the object onto which an image is projected by the visible light projector. The image generator is configured to generate image data to project the image to be projected onto the object onto the projection region.

BACKGROUND

1. Technical Field

The present disclosure relates to a projection system, an image processor and a projection method for projecting onto an object an image corresponding to a position of the projection target object.

2. Related Art

JP 2013-192189 A discloses a controller, a projection system, a program and an image processing method for mapping objects such as patterns and colors onto the surface of the object by applying the irradiation light onto the object. The controller of JP 2013-192189 A includes imaging means, an image acquisition unit for acquiring an image including an object captured by the imaging unit, region extraction means for extracting a projection region of the object from the image acquired from the image acquisition means, and mapping means for mapping an object corresponding to the projection region onto the projection region. Thus, even if the object undergoes shape change, movement, and the like, the image of the object is mapped onto the object.

SUMMARY

The present disclosure provides a projection system, an image processor and a projection method for correcting a deviation between a projection image and the object generated when the object moves, and for preventing the deviation between the projection image and the object from occurring even if the object moves.

One aspect of the present disclosure provides a projection system including:a visible light projector for projecting an image with visible light onto the object;a camera for capturing an image of the object, the camera having an optical axis not coinciding with an optical axis of the visible light projector;an image processor for generating an image to be projected with visible light onto the object based on the image captured by the camera.

The image processor includes:a corrector for correcting a deviation between a projection image and the object caused by a difference between the optical axis of the visible light projector and the optical axis of the camera according to a position of the object, the corrector configured to calculate a projection region on the object onto which an image is projected by the visible light projector, andan image generator configured to generate image data to project the image to be projected onto the object onto the projection region.

Another aspect of the present disclosure provides an image processor including: an image output configured to output image data indicating an image with visible light projected onto an object by a visible light projector;an image input configured to input image data of the object captured by a camera having an optical axis not coinciding with an optical axis of the visible light projector; anda controller configured to generate the image data based on an image indicated by the image data.

The controller includes:a corrector for correcting a deviation between a projection image and the object caused by a difference between the optical axis of the visible light projector and the optical axis of the camera according to a position of the object, the corrector configured to calculate a projection region on the object onto which an image is projected by the visible light projector, andan image generator configured to generate image data to project the image to be projected onto the object onto the projection region.

Still another aspect of the present disclosure provides a projection method including:preparing a visible light projector at a position where an image with visible light is configured to be projected onto an object;preparing a camera configured to capture an image of the object to have an optical axis not coinciding with an optical axis of the visible light projector;correcting a deviation between a projection image and the object caused by a difference between the optical axis of the visible light projector and the optical axis of the camera according to a position of the object,determining a projection region on the object onto which an image is projected by the visible light projector;generating image data to project an image to be projected onto the object onto the projection region; andprojecting an image indicated by the generated image data by the visible light projector.

According to the present disclosure, a projection system for correcting a deviation between a projection image and the object generated when the object moves, and for preventing a deviation between the projection image and the object from occurring even if the object moves is obtained.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENT

Embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. However, description in more detail than is necessary can be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially identical constituent elements are omitted so as to avoid unnecessarily redundant description and enable those of skill in the art to readily understand the embodiments herein.

It should be noted that the inventors provide the accompanying drawings and description below to allow those of skill in the art to satisfactory understand the present disclosure. Accordingly, the drawings and the description are not intended to limit the subject matter defined in the claims.

First Embodiment

FIG. 1shows a configuration of a projection system100according to the first embodiment. The projection system100is a system that performs projection mapping for projecting, onto a screen2and an object3such as a person or thing that may move between the projection system100and the screen2, images corresponding to their positions and shapes.

A texture image including a pattern, a color, a mark, a symbol, and the like is projected onto the object3.

As shown inFIG. 1, for convenience of illustration, the vertical direction downward is defined as the y-axis, the horizontal direction perpendicular to the y-axis is defined as the x-axis, and the direction perpendicular to the x-axis and the y-axis is defined as the z-axis.

In the example inFIG. 1, the screen2has a rectangular planar shape on the x-y plane. However, the shape of the screen2is not limited thereto, and may be a sphere, a curved surface, or the like, or may have irregularities. In addition, the screen2is obtained by, for example, processing paper or cloth into a rectangle for projection, but is not limited thereto, and may be a wall surface of a building or a naturally formed rock wall.

The projection system100includes a screen2and a camera4for capturing the object3. The camera4is a camera which has an image sensor such as a CCD or CMOS image sensor and is sensitive to visible light and infrared rays. The projection system100further includes an image processor5. Based on the image captured by the camera4, the image processor5generates images to be projected onto the screen2and the object3, and controls each part of the projection system100.

The projection system100further includes a visible light projector6for projecting an image generated by the image processor5onto the screen2and the object3with visible light, and an infrared projector7for projecting a pattern image with infrared rays for measuring their shapes and positions onto the object3.

The visible light projector6is a projector for projecting an image with visible light using a technology such as DLP, 3LCD, LCOS, or DMD technology, or the like. The visible light projector6projects an image including various image contents based on the image data input from, for example, the image processor5.

The infrared projector7is a projector for projecting an image with infrared rays using a technology such as DLP, 3LCD, LCOS, or DMD technology, or the like. The infrared projector7projects a pattern image for measuring the shape and position of the object3by using the space encoding method based on the image data input from, for example, the image processor5.

The visible light projector6and the infrared projector7are arranged so as to be capable of projecting an image with light onto an identical region, for example, the screen2and the object3. The optical axis of the visible light projector6and the optical axis of the infrared projector7do not coincide with each other inFIG. 1, but they may be optically coupled so as to coincide with each other.

The camera4is arranged at a position capable of capturing an image of a region (for example, the screen2and the object3) onto which an image is projected by the visible light projector6and the infrared projector7. In addition, the camera4is arranged so that the optical axis of the camera4does not coincide with any of the optical axis of the visible light projector6and the optical axis of the infrared projector7.

The projection system100projects a content image with visible light from the visible light projector6onto the screen2and the object3. The position and the shape of the object3are measured by using the infrared projector7and the camera4so that an image (texture image) to be projected onto the object3of the content images is projected onto the object3even if the object3moves.

FIG. 2shows a specific configuration of the image processor5. The image processor5includes a controller10, a storage20, an image input11for receiving an image captured by the camera4, and an image output18for outputting image data generated in the controller10.

The controller10is a device that controls the overall operation of the image processor5. The controller10includes a general-purpose processor, such as a CPU or an MPU, which achieves a predetermined function by executing a program. The controller10achieves various kinds of control in the image processor5by calling and executing the control program stored in the storage20. The controller10is not limited to those that achieve a predetermined function in cooperation between hardware and software, and may be a hardware circuit designed exclusively for achieving a predetermined function. That is, the controller10can be achieved by various processors such as a CPU, an MPU, a GPU, an FPGA, a DSP, and an ASIC.

The storage20is a medium for storing various pieces of information. Specifically, the storage20is achieved only with a storage device, such as a semiconductor memory device such as a flash memory or an SSD or a disk device such as a hard disk, or achieved with appropriate combination of these. In the storage20, a control program executed by the controller10, image data, and the like are stored. In addition, the storage20may act as a workspace for the controller10.

The image input11is an interface circuit (module) for connecting the image processor5and a peripheral device (for example, the camera4). In addition, the image output18is an interface circuit (module) for connecting the image processor5and peripheral devices (for example, the visible light projector6and the infrared projector7). Various interfaces such as Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI) (registered trademark), IEEE 1394, Bluetooth (registered trademark), and the like are used as the image input11and the image output18.

The controller10includes a detector12for detecting the position of the object3based on the image input into the image input11.

The controller10further includes a calibrator13for calculating a coordinate transformation matrix that associates each pixel of the image with the visible light projected by the visible light projector6with each pixel of the camera4based on the image captured by the camera4. The coordinate transformation matrix transforms the coordinate system viewed from the camera4(hereinafter referred to as “camera coordinate system”) into the coordinate system for image projection by the visible light projector6(hereinafter referred to as “visible light projective coordinate system”). The coordinate transformation matrix is stored in the storage20.

Furthermore, the calibrator13can also associate each pixel of the image with infrared rays projected by the infrared projector7with each pixel of the camera4. Thus, the pixels of the visible light image, the infrared image, and the camera are associated with each other.

The controller10further includes a coordinate transformer14for calculating the position of the object3in the visible light projective coordinate system in the position of the object3in the camera coordinate system detected by the detector12by applying the coordinate transformation matrix.

As will be described below, even if an image is projected so as to match the position of the object3in the visible light projective coordinate system calculated by the coordinate transformer14, when the object3moves in the z direction from the position where the calibration is performed, the position of the texture image is deviated from the position of the object3. Thus, the controller10further includes a corrector15for correcting such deviation and making the position of the texture image coincide with the position of the object3. Details of the correction method will be described below.

The controller10further includes an image generator16for generating image data to be projected onto the screen2and the object3from the content image data given in advance according to the corrected position of the object3obtained by the corrector15. The image data generated by the image generator16is transmitted to the visible light projector6via the image output18and is projected onto the screen2and the object3by the visible light projector6.

The controller10further includes a pattern image generator17that generates pattern image data for measuring the shape and position of the object3by using the space encoding method. The pattern image data generated by the pattern image generator17is transmitted to the infrared projector7via the image output18and is projected onto the object3by the infrared projector7.

In the following, the calibration operation performed by the projection system100according to the present embodiment will be described.

FIG. 3is a flowchart showing the flow of the calibration processing by the projection system100. With reference toFIGS. 1 to 3, the calibration processing will be described. First, the visible light projector6projects a feature point onto the screen2(S101). Next, the feature point is captured with the camera4(S102).

InFIG. 4, four feature points in the visible light projective coordinate system are shown. The feature point to be projected by the visible light projector6is, for example, two line segments which intersect as shown inFIG. 4.

When the feature point projected by the visible light projector6is viewed in the camera coordinate system, in the case where the calibration processing is not performed, the feature point is seen as inFIG. 5. Thus, unless the calibration processing is performed, the image viewed from the camera4appears distorted. That is, when projection mapping is performed without performing the calibration processing, for a spectator in a position different from the visible light projector6, the projection image appears distorted and the stage effect of projection mapping is impaired.

Thus, the calibrator13of the image processor5calculates a coordinate transformation matrix that associates each pixel of the image with the visible light projected by the visible light projector6with each pixel of the camera4(S103). The coordinate transformation is, for example, a projective transformation, an affine transformation, or a perspective projection transformation. The calculated coordinate transformation matrix is stored in the storage20and is used when projection mapping is performed. Applying this coordinate transformation matrix to the data to be projected when performing the projection mapping solves the problem that the image seen from the camera4appears distorted.

Before or after the above calibration processing, calibration processing for associating each pixel of the image with infrared rays projected by the infrared projector7with each pixel of the camera4may be performed. Thus, the pixels of the visible light image, the infrared image, and the camera are associated with each other.

[1-3. Measurement Principle of Position and Shape]

In the present embodiment, an active stereo measurement method and a space encoding method are adopted as means for measuring the position and shape (depth) of the object3.

FIG. 6illustrates an active stereo measurement method. InFIG. 6, the camera4and the infrared projector7are arranged in the same z position at an interval C in the x direction. The infrared rays emitted from the infrared projector7is incident on and reflected by the surface of the object3, and the reflected light is incident on the camera4. As shown inFIG. 6, the angle formed by the incident optical path and the reflection optical path (hereinafter referred to as “parallax”) is denoted as θ. Thus, when the image projected by the infrared projector7is captured from the camera4, the image is deviated by the parallax θ. Then, as shown inFIG. 6, as the distance in the z direction between the camera4and the object3changes, the x coordinate of the position of the infrared reflection point in the captured image of the camera4changes.

Based on such change in coordinates, the controller10of the image processor5performs calculation based on the triangulation with the interval C between the camera4and the infrared projector7as the base line length and calculates the distance in the z direction between the camera4and the object3. Changes in coordinates are measured by using a measurement pattern based on the space encoding method.

The correcting operation to the deviation of the texture image after the calibration processing, which is performed by the corrector15shown inFIG. 2, is performed in the projection mapping operation to be described below. The correcting operation is one of the characteristics of the present embodiment, so that an description thereof is particularly provided here.

FIG. 7illustrates that the position of the texture image and the position of the object3are deviated when the object3moves from the position where the calibration is performed.FIG. 7shows the screen2, the object3, the camera4, and the visible light projector6as seen from above. InFIG. 7, the object3is not on the screen2but at the position of a point P on a plane, parallel to the screen2, apart from the screen2in the z direction by a distance d (hereinafter referred to as “mapping plane”).

The above-described calibration operation is performed by projecting feature points onto the screen2by the visible light projector6and capturing the feature points with the camera4. That is, the calibration operation is performed with the screen2as a reference. Therefore, when recognizing the position of the object3based on the image captured by the camera4, the controller10recognizes that the object3is at the position of the point Q where the straight line passing through the camera4and the point P intersects the screen2.

Therefore, the image processor5controls the visible light projector6to project the texture image onto the position of the point Q. However, since the mapping surface of the object3is in front of the screen2, the texture image is projected onto the position of the point R where the straight line passing through the visible light projector6and the point Q intersects the mapping surface. For this reason, the texture image essentially supposed to be projected onto the position P on the mapping surface is projected onto the position R, causing a deviation in the projection position.

In this way, when the object3is positioned in front of the plane (calibration plane, screen2in the illustrated example) on which the calibration is performed (that is, between the calibration plane and the camera4), the position of the texture image and the position of the object3deviate from each other.

For example, when the object3is in the calibration plane, the image is properly projected onto the object3as shown inFIG. 8A. On the other hand, when the object3is positioned in front of the calibration plane, as shown inFIG. 8B, the position of the texture image and the position of the object3deviate from each other by Mm1.

The deviation amount Mm1of the projection image, that is, the distance between the point P and the point R inFIG. 7is expressed by the following mathematical expression.

where B is the distance in the x direction between the camera4and the visible light projector6, D is the distance in the z direction between the camera4and the screen2, and d is the distance in the z direction between the screen2and the mapping surface or object3.

The direction of deviation of the projection image (the position of the point R with respect to the point P) depends on the positional relationship between the camera4and the visible light projector6. As shown inFIG. 7, when the visible light projector6is in the +x direction (right direction on the page) of the camera4, the direction of deviation of the projection image is also in the +x direction.

In addition, the width in the x direction of the image that can be projected by the visible light projector6on the mapping surface (hereinafter referred to as “projection width”) hp and the correction amount Mx1obtained by converting the deviation amount Mm1of the projection image into the number of pixels are expressed by the following mathematical expressions.

where Hp is the projection width on the screen2, and Ph is the number of horizontal pixels (the number of pixels in the x direction) of the visible light projector6.

In order to eliminate the deviation of the projection position as described above, the corrector15can cause the visible light projector6to project the image properly onto the object3by correcting the position of the image to deviate by the correction amount Mx1.

The distance D and the distance d can be measured by the space encoding method by using the pattern image generated by the pattern image generator17. Therefore, for example, storing the projection width Hp on the screen2, the distance B in the x direction between the camera4and the visible light projector6, and the horizontal pixel number Ph of the visible light projector6in the storage20before executing projection mapping allows the projection system100to correct the correction amount Mx1at any time during projection mapping.

In the above description, the correction of deviation in the horizontal direction (x direction) is described, but the deviation in the vertical direction (y direction) can also be similarly corrected.

FIG. 9is a flowchart showing the projection mapping operation by the projection system100. With reference toFIGS. 1, 2 and 9, the projection mapping operation will be described. First, the infrared projector7projects a pattern image for measurement with infrared rays onto the object3(S201). The pattern image for measurement is generated by the pattern image generator17based on the data stored in the storage20.

The camera4captures the object3and captures the pattern image projected by the infrared projector7(S202).

The controller10receives the image data via the image input11. Based on the received pattern image data, the detector12of the controller10detects the position of the object3in the camera coordinate system, in particular, the distance (depth) in the z direction between the camera4and the object3(S203).

The coordinate transformer14applies the coordinate transformation matrix obtained in the calibration operation shown inFIG. 3to the position of the object3in the camera coordinate system detected in step S203to calculate the position of the object3in the visible light projective coordinate system (S204).

However, as described above, the position of the object3calculated in step S204is deviated from the actual position of the object3when the object3is not within the calibration plane. Thus, the corrector15calculates a correction coefficient (S205) in order to correct such deviation and match the position of the texture image with the actual position of the object3. The correction coefficient is, for example, Mx1given by (Expression 3) based on the distance d in the z direction between the screen2and the object3.

The image generator16corrects the position of the object3calculated in step S204based on the correction coefficient obtained in step S205(S206).

Video data to be projected onto the screen2and the object3is generated from the content image data given in advance (S207) so that the texture image is projected onto the corrected position. That is, the image generator16generates image data for projection based on the position of the object3after correction.

The image generator16transmits the generated image data to the visible light projector6via the image output18, and projects the image onto the screen2and the object3(S208).

The projection system100repeatedly performs the above processing at a predetermined frame rate (S209). Thus, the image content projected from the visible light projector6can be made to accurately follow the movement of the object3.

[1-6. Effects and the Like]

As described above, in the present embodiment, the projection system100includes: a visible light projector6for projecting an image with visible light onto the object3; a camera4for capturing an image of the object3, the camera4having an optical axis not coinciding with an optical axis of the visible light projector6; and an image processor5for generating an image to be projected by visible light onto the object3based on the image captured by the camera4.

The image processor5includes a corrector15for correcting a deviation between the projection image and the object3caused by a difference between the optical axis of the visible light projector6and the optical axis of the camera4according to a position of the object3, the corrector15configured to calculate a projection region on the object3onto which an image is projected by the visible light projector6. The image processor5further includes an image generator16for generating image data so as to project the image to be projected onto the object3onto the projection region.

The projection system100can correct the deviation between the projection image and the object3generated when the object3moves during the projection mapping to prevent the deviation between the projection image and the object3from occurring even when the object3moves.

Second Embodiment

The second embodiment discloses a projection system200in which after the execution of the calibration processing and before the demonstration of the projection mapping, a person measures (actually measures) the amount of deviation between the position of the object3and the projection position of the texture image, and which corrects the deviation during demonstration of the projection mapping by using the amount of deviation actually measured.

FIG. 10shows a configuration of the projection system200according to the second embodiment. In the projection system200of the present embodiment, in addition to the configuration of the image processor5of the first embodiment, the image processor205further includes an operation unit21for accepting input instructions. The input instructions include a deviation amount Mm2of the projection image in the actual measurement position described below. Hereinafter, the deviation amount Mm2is referred to as “actual measurement amount”. In addition, the operation of the controller210of the projection system200, particularly the operation of the corrector215is different from that of the first embodiment.

The operation unit21is an input interface device for accepting input instructions from the user. The operation unit21converts the input instructions received from the user and the contents of the operation into an electric signal and transmits the electric signal to the controller10. The operation unit21includes a mouse, a keyboard, a touch panel, buttons, and the like.

In the present embodiment, after the execution of calibration processing, an actual measurement operation for measuring the actual measurement amount Mm2is performed. The actual measurement operation by the projection system200will be described with reference toFIGS. 10 to 12.FIG. 11illustrates an actual measurement operation by the projection system200.FIG. 12is a flowchart showing a flow of an actual measurement operation by the projection system200. The actual measurement operation is performed before the demonstration of projection mapping as preparation.

After the execution of calibration processing, the object3is arranged in a position (hereinafter referred to as “actual measurement position”) away by A in the z direction from the calibration plane (screen plane2in the example shown inFIG. 11) (S300).

Next, the infrared projector7projects the pattern image for measurement onto the object3(S301), and the camera4captures the pattern image (S302).

The controller10receives the image data via the image input11, and the detector12detects the position of the object3in the camera coordinate system based on the received pattern image data (S303).

The coordinate transformer14applies the coordinate transformation matrix obtained in the calibration operation to the position of the object3in the camera coordinate system detected in step S303to calculate the position of the object3in the visible light projective coordinate system (S304).

Based on the position of the object3calculated in step S304, the image generator16generates image data for actual measurement to be projected onto the screen2end the object3from the image data for actual measurement given in advance (S305).

The image data generated by the image generator16is transmitted to the visible light projector6via the image output18and is projected onto the screen2and the object3by the visible light projector6(S306).

When the object3is within the calibration plane, the image for actual measurement is properly projected onto the object3. However, when the object3is positioned in the actual measurement position in front of the calibration plane, as shown inFIG. 11, the position of the texture image and the position of the object3deviate by Mm2.

Then, the user specifies how many pixels worth of movement of the projection position of the image by the visible light projector6will eliminate the deviation between the position of the texture image and the position of the object3. That is, the user specifies an converted actual measurement amount Mx2obtained by converting the actual measurement amount Mm2into the number of pixels, and inputs the converted actual measurement amount Mx2by using the operation unit21. The controller10acquires the converted actual measurement amount Mx2input by the user to store it in the storage20(S307).

As shown inFIG. 11, during the projection mapping operation, when the object3is in a position (mapping plane) at a distance d from the screen2(calibration plane), the position of the texture image and the position of the object3deviate by Mm3without correction. The correction amount Mx3obtained by converting the deviation amount Mm3of the projection image on the mapping plane into the number of pixels of the visible light projector6is expressed by the following mathematical expression.

In the projection system200according to the present embodiment, correcting the position of the image to deviate by the correction amount Mx3with the corrector215allows the visible light projector6to properly project the image onto the object3. Furthermore, in the present embodiment, actually measuring the amount of deviation between the position of the object3and the projection position of the texture image on the actual measurement plane before the demonstration of the projection mapping allows correction to be performed with higher accuracy as compared with the calculation of the correction amount based on the calculation using the parameters (the distance B between the camera4and the visible light projector6shown inFIG. 2, the distance D in the z direction between the camera4and the screen2, the distance d in the z direction between the screen2and the object3, the projection width Hp, and the like).

In the above description, the correction of deviation in the horizontal direction (x direction) is described, but the deviation in the vertical direction (y direction) can also be similarly corrected.

[2-4. Effects and the Like]

As described above, in the present embodiment, the projection system200further includes an operation unit21. The corrector15corrects the position of the object3based on the measurement value, input by the operation unit21, of the deviation amount between the position of the image projected by the visible light projector6and the position of the object3.

Thus, measuring the deviation amount between the position of the texture image and the position of the object3at at least one actual measurement position allows the deviation between the projection image and the object3to be prevented from occurring even if the object moves during the projection mapping.

Third Embodiment

FIG. 13shows a configuration of the projection system300according to the third embodiment. The projection system300includes an image processor305including a storage20, an operation unit21, and a controller310. The controller310includes a corrector315.

FIG. 14illustrates a correcting operation by the projection system300. In the first and second embodiments, it is described that the position of the texture image and the position of the object3deviate when the object3moves in the z direction from the screen2(calibration plane) during the projection napping operation. However, this positional deviation can also be caused by the movement of the object3in the x direction or the y direction in the calibration plane. This is due to, for example, the low accuracy of the calibration processing, and the optical axis of the visible light projector6and the optical axis of the camera4being not parallel to each other.

For example, after the calibration processing, as shown inFIG. 14, when the object3is positioned on the left side of the center of the screen2as seen from the visible light projector6toward the calibration plane, the image may be projected onto the left side of the proper position, and when the object3is positioned on the right side from the center of the screen2, the image may be projected onto the right side of the proper position.

Thus, in the present embodiment, the actual measurement operation and the correcting operation similar to those in the second embodiment are also performed on the deviation of the projection image caused by the movement of the object3in the calibration plane.

With reference toFIG. 15, an actual measurement operation by the projection system300will be described. First, the object3is arranged at a predetermined reference point C (0, 0) in the calibration plane spreading in the x-y directions. Next, an image for actual measurement is projected onto the object3by the same means as in steps S301to S307inFIG. 12in the second embodiment. The user specifies the converted actual measurement amount Mx (0) obtained by converting the deviation amount between the position of the texture image and the position of the object3into the number of pixels, and inputs the converted actual measurement amount Mx (0) into the controller310by using the operation unit21. The converted actual measurement amount Mx (0) is stored in the storage20.

Next, the object3is arranged at a point D (L, 0) away by L in the x direction from the reference point C (0, 0), and an image for actual measurement is projected onto the object3. The user specifies the converted actual measurement amount Mx (L) obtained by converting the deviation amount into the number of pixels and inputs the converted actual measurement amount Mx (L) into the controller310by using the operation unit21. The converted actual measurement amount Mx (L) is stored in the storage20.

From the result obtained by the above actual measurement operation, when the object3is at the point X (x, 0), the correction amount Mx (x) for correcting the deviation is expressed by the following mathematical expression.

In the projection system300, correcting the position of the image to deviate by the correction amount Mx (x) with the corrector315allows the visible light projector6to properly project the image onto the object3.

In the above description, the correction of deviation in the horizontal direction (x direction) is described, but the deviation in the vertical direction (y direction) can also be similarly corrected.

Other Embodiments

As described above, the first to third embodiments are described as examples of the technique disclosed in the present application. However, the technique in the present disclosure is not limited to this, and can also be applied to embodiments in which changes, substitutions, additions, omissions, and the like are made as appropriate. In addition, it is also possible to combine each component described in the first to third embodiments to form a new embodiment. Thus, in the following, other embodiments will be exemplified.

In the first to third embodiments, the camera4including an image sensor such as a CCD or a CMOS image sensor is described as an example of imaging means (camera). The imaging means has only to capture the object image to generate image data. Therefore, the imaging means is not limited to a camera including an image sensor such as a CCD or a CMOS image sensor.

In addition, as an example of the camera4, a camera having sensitivity to visible light and infrared rays is described. However, the projection system100,200, or300may include, for example, a first camera having sensitivity to visible light and a second camera having sensitivity to infrared rays.

In the first to third embodiments, the infrared projector7is described as an example of the invisible light projector. The invisible light projector has only to project an invisible light image which cannot be seen by a person. Therefore, the invisible light projector is not limited to the infrared projector7for projecting an image with infrared rays. For example, the invisible light projector may be an ultraviolet projector for projecting an image with ultraviolet rays.

In the first to third embodiments, as an example of the image output18, the image output18for outputting both the image data indicating an image projected with infrared rays and the image data indicating an image projected with visible light is described. However, the image processor5,205, or305may include, for example, a first image output for outputting image data indicating an image projected with visible light and a second image output for outputting image data indicating an image projected with infrared rays.

In the first to third embodiments, it is described that the projection system100,200, or300projects a pattern image from the infrared projector7, detects the position and shape of the object3by using the image obtained by capturing the pattern image with the camera4to detect a projection region, and projects a texture image onto the projection region. However, the projection region onto which the texture image is projected has only to conform to the position of the object3. For example, a retroreflective material functioning as a marker is attached to the object3, and the detector12detects the position of the marker by using the image obtained by capturing the retroreflective material with the camera4. If the positional relationship between the marker position and the texture image is determined in advance, it is possible to project the texture image so as to comfort to the position of the object3according to the detected marker position.

In the first to third embodiments, the calibrator13for calculating a coordinate transformation matrix that associates each pixel of the image with the visible light projected by the visible light projector6with each pixel of the camera4is described. However, the calibration processing by the calibrator13has only to associate each pixel of the image with visible light projected by the visible light projector6with each pixel of the camera4. Therefore, the calibrator13is not limited to calculating the coordinate transformation matrix, and has only to determine the coordinate transformation information that associates each pixel of the image with visible light projected by the visible light projector6with each pixel of the camera4.

In the first to third embodiments, as an example of the detection means, the detector12for measuring the shapes and positions (what is called, depths) of the screen2and the object3by using the space encoding method is described. The detection means has only to be capable of measuring the shapes and positions of the screen2and the object3. Therefore, the detection means is not limited to means using the space encoding method. For example, the detection means may use a stereo method in which the same feature point is captured by two cameras and feature point matching is performed. In addition, the detection means may use the Time of Flight (TOF) method.

In the first embodiment, an example in which a plane on which calibration is performed (calibration plane) coincides with the screen plane2is described. However, the calibration plane may be a plane away from the screen plane2by a predetermined distance S and parallel to the screen plane2. For example, the calibration plane is set to a position (for example, S=1 m) where the object3such as a person moves during demonstration of the projection mapping. In this case, the deviation amount Mm1and the projection width hp of the projection image are respectively expressed by the following mathematical expression 6 and mathematical expression 7 instead of mathematical expression 1 and mathematical expression 2.

where D is the distance in the z direction between the camera4and the screen2and d is the distance in the z direction between the calibration plane and the mapping plane or object3.

In the second embodiment, means for measuring the actual measurement amount Mm2of the deviation only at one actual measurement position is described as an example of the actual measurement operation. However, the actual measurement amount of the deviation at a plurality of positions with different positions in the z direction may be measured, and the correction amount for correcting the deviation may be calculated by linear interpolation based on a plurality of measurement amounts.

As described above, the embodiments are described as the exemplification of the technique in the present disclosure. For that, the accompanying drawings and the detailed description are provided.

Therefore, among the components described in the accompanying drawings and the detailed description, not only the components essential for solving the problem, but also the component not essential for solving the problem may be included in order to exemplify the above technique. Therefore, it should not be recognized that these non-essential components are essential immediately because these non-essential components are described in the accompanying drawings and the detailed description.

In addition, since the above embodiments are for illustrating the technique in the present disclosure, various changes, substitutions, additions, omissions, and the like can be made within the scope of the claims or the equivalent thereof.

The present disclosure is applicable to various uses for projecting an image onto an object.