Abstract:
A projection apparatus includes a matrix expression generation unit which generates viewpoint correction matrices used for a conversion calculation on each of the projection surfaces for an image to be projected, a mask generation unit which generates correction mask data indicating regions corresponding to the viewpoint correction matrices generated by the matrix expression generation unit for the image, and a projection control unit which executes the conversion calculation on the image using each viewpoint correction matrix generated for each corresponding region in the correction mask data generated by the mask generation unit, and thereafter controls the projection unit to project the image.

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
       [0001]    This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2016-123603, filed Jun. 22, 2016, the entire contents of which are incorporated herein by reference. 
       BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
       [0002]    The present invention relates to a projection apparatus, a projection system, a projection method, and a computer readable storage medium. 
       2. Description of the Related Art 
       [0003]    Jpn. Appln. KOKAI Publication No. 2013-120347 proposes an image display technique of enabling a viewer who is at a position off of the front of a display screen of an image display apparatus to see an undistorted image. 
         [0004]    The technique described in the above patent document is a technique of correcting distortions in an image displayed on a planar surface seen from an viewer. 
         [0005]    With a diversity of video viewing methods, commercialization of games using VR (virtual reality) technology, and the popularization of home planetariums, so-called full-spherical-type projection apparatuses for continuously projecting an image not only on a single surface, but on multiple surfaces (for example, a ceiling, four walls, and a floor of a room) have been proposed. 
         [0006]    In a full spherical-type projection apparatus, with the exception of a surface in the direction of the proximal end of the mounting for the apparatus, it is possible to continuously project an image on almost all the surfaces by performing projection by, for example, fixing the main body of the apparatus on the ceiling with mounting brackets. 
         [0007]    However, if an image is projected with such a projection apparatus, a viewer may see a distorted image, such as a curved line that was originally a straight line in the original image, especially when the image is projected on a corner where walls intersect, because the location of the projection lens and the position of the viewer&#39;s viewpoint are different. 
         [0008]    The present invention has been achieved in view of such circumstances, and it is an object of the present invention to provide a projection apparatus, a projection system, a projection method, and a computer readable storage medium, which are capable of projecting an image that is easy for a viewer to see when the image is projected on a full-spherical surface. 
       SUMMARY OF THE INVENTION 
       [0009]    According to one aspect of the present invention, there is provided a projection apparatus comprising: a projection unit which projects images; a surface information obtaining unit which obtains information of a plurality of projection surfaces on which the projection unit performs projection; a viewpoint information obtaining unit which obtains position information of a viewer&#39;s viewpoint with respect to the projection unit; and a processor configured to: generate based on the information of the plurality of projection surfaces obtained by the surface information obtaining unit and the position information of the viewer&#39;s viewpoint obtained by the viewpoint information obtaining unit, viewpoint correction matrices used for a conversion calculation on each of the projection surfaces for an image to be projected by the projection unit; generate correction mask data indicating regions corresponding to the viewpoint correction matrices for the image; and execute the conversion calculation on the image using each viewpoint correction matrix for each corresponding region in the correction mask data, and thereafter control the projection unit to project the image. 
         [0010]    Additional objects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objects and advantages of the invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out hereinafter. 
     
    
     
       BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING 
         [0011]    The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention, and together with the general description given above and the detailed description of the embodiments given below, serve to explain the principles of the invention. 
           [0012]      FIG. 1  is a block diagram showing the function and configuration mainly of electronic circuitry of the projector according to the first embodiment; 
           [0013]      FIG. 2  is a drawing showing an example of an environment of a viewer and a room where a projector according to the first embodiment is assumed to be used; 
           [0014]      FIG. 3  is a drawing showing projection surfaces that the projector should recognize in the projection environment shown in  FIG. 2  according to the first embodiment; 
           [0015]      FIG. 4  is a flowchart showing a basic process of an initial setting operation which is performed the first time the projector according to the first embodiment is set; 
           [0016]      FIG. 5  is a drawing showing correction mask data according to the first embodiment; 
           [0017]      FIG. 6  is a flowchart showing a basic process at the time of a projection operation according to the first embodiment; and 
           [0018]      FIG. 7  is a flowchart showing a process at the time of performing an initial setting operation according to the second embodiment of the present invention. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0019]    Hereinafter, embodiments of the present invention will be described. 
       First Embodiment 
       [0020]    Hereinafter, the first embodiment, wherein the present invention is applied to a projector  10  that is fixed with mounting brackets at approximately the center of the ceiling of a room, will now be described with reference to the accompanying drawings. 
         [0021]      FIG. 1  explains the function and configuration of mainly the electronic circuitry of the projector  10 . In  FIG. 1 , image data to be projected is read from a data storage unit  11  and sent to a projection image driving unit  12  via a bus B. 
         [0022]    The data storage unit  11  also stores correction mask data and viewpoint correction matrices data, etc., which will be described later. 
         [0023]    The projection image driving unit  12  drives micromirror elements  13 , which are display elements, to display the sent image data by a more rapid time-divisional drive, which is a result of multiplying a frame rate in accordance with a predetermined format (e.g., a multiplication of 120 frames per second) by the number of color components and the number of grayscale levels to display, in accordance with the sent image data. 
         [0024]    The micromirror element  13  generates an optical image with reflected light produced by quickly changing each of the tilt angles of multiple micromirrors arranged in an array, for example, micromirrors of WXGA (1280 pixels×768 pixels) between an on or off state. 
         [0025]    On the other hand, light in primary colors (R, G, B) are circularly emitted from the light source unit  14  in a time-divisional manner. The light source unit  14  has a semiconductor LED, and repeatedly emits light in primary colors (R, G, B) in a time-divisional manner. The LED of the light source unit  14  may be an LD (semiconductor laser) and an organic EL element as an LED in a broad sense. 
         [0026]    Light in a primary color having a wavelength different from original light, which is excited when light emitted from the LED is reflected on a phosphor, may be used. This light in a primary color from the light source unit  14  is completely reflected on a mirror  15  and irradiated onto the micromirror elements  13 . 
         [0027]    A light figure is generated with light reflected from the micromirror elements  13 , and the generated light figure is projected onto the outside of the projector through the projection lens unit  16 . 
         [0028]    The projection lens unit  16  projects a full spherical image in all directions, using a super-wide-angle lens, a so-called fisheye lens, except for the proximal direction. For example, the projection angle of view 270 degrees on the surface along the projection light axis, and the projection angle of view 360 degrees on the surface orthogonal to the projection light axis can be realized. 
         [0029]    The projection image driving unit  12 , the micromirror elements  13 , the light source unit  14 , the mirror  15 , and the projection lens unit  16  constitute a projection system PI. 
         [0030]    All the operations on each of the foregoing circuits are controlled by the CPU  17 . The CPU  17  is directly connected to a main memory  18  and a program memory  19 . The main memory  18  is configured with SRAM, etc., and functions as a work memory of the CPU  17 . The program memory  18  is configured with an electrically-rewritable non-volatile memory, such as a flash ROM, and stores operation programs executed by the CPU  17 , and a variety of typical data, etc. 
         [0031]    The CPU  17  reads the operation programs and typical data, etc. stored in the program memory  19 , and engages and stores the read program or data on the main memory  19  to execute the program, so as to integrally control the projector  10 . 
         [0032]    The CPU  17  performs various types of projection operations in accordance with operation signals from an operation unit  20 . The operation unit  20  includes some operation keys of the main body of the projector  10  and a receiving unit which performs key operation signals from a remote controller (not shown in the drawing) dedicated for the projector  10 , and sends signals in accordance with the received key operation signals to the CPU  17 . 
         [0033]    The CPU  17  is further connected to a sound processing unit  21  and a communication unit  22  via the bus B. 
         [0034]    The sound processing unit  21  comprises a sound source circuit, such as a PCM sound source, converts sound signals given at the time of a projection operation, and drives the speaker unit  23  to generate sound, or a beep sound, etc. as needed. 
         [0035]    The communication unit  22  carries out transmission and reception of data with external devices of the projector through the antenna  24  by, for example, a wireless LAN communication in compliance with IEEE 802.11a/11b/11g/11n/11ac and a short-distance wireless communication in compliance with IEEE 802.15. 
         [0036]    Next, the operation of the above embodiment will be described. 
         [0037]      FIG. 2  is a drawing showing an example of an environment of a room and a viewer EJ where the projector  10  is assumed to be used. In the same drawing, the projector  10  is set at approximately the center of the ceiling of the room RM by dedicated ceiling-hung mounting brackets (not shown in the drawing), and the projection optical axis of the projection lens unit  16  is directed directly underneath the ceiling. 
         [0038]    Herein, the viewer EJ sitting on a sofa SF which is arranged approximately underneath the projector  10  is facing the wall shown in the left of the drawing. A shelf SL is arranged in the corner at the right back of the room viewed from the viewer EJ. 
         [0039]    As the initial setting operation at the time of setting the projector  10  for the first time, a viewpoint correction process is carried out to set a viewpoint position of the viewer EJ. Furthermore, it is necessary to carry out the viewpoint correction process to change a projection surface when the room RM is rearranged including the shelf SL. 
         [0040]      FIG. 3  shows a projection surface that the projector  10  should recognize in the projection environment shown in  FIG. 2 . In other words, the projection surface herein consists of a ceiling surface TP, wall surfaces WL 1 , WL 2 , WL 3 , and WL 4 , a floor surface FL, a shelf top surface ST, and shelf side surfaces SS 1  and SS 2 . Although the wall surface WL 4  is not shown in  FIG. 3  which is a developed view of the room RM, the wall surface WL 4  is a wall facing the wall WL 2  and adjacent to the wall WL 1  and the wall WL 3 . 
         [0041]    Next, with reference to  FIG. 4 , a flow of the basic process of the above-described initial setting operation, which is operated when the projector  10  is arranged for the first time, is explained. 
         [0042]    The operation which will be described below is executed after the CPU  17  of the projector  10  retrieves an operation program, etc. from the program memory  19 , and activates it on the main memory, as described above. The operation programs, etc. stored in the program memory  19  include not only programs stored in the program memory  19  at the time of shipping the projector  10  from the factory, but also programs that can be installed by a user after the purchase of the projector  10  for upgrading. 
         [0043]    First, the CPU  17  obtains shape information of the room RM from an external device, for example, a personal computer onto which an application program dedicated for the projector  10  is installed, via the antenna  24  and the communication unit  22  (step S 101 ). 
         [0044]    In other words, in the application program, for example, the shape of the room, the length and width of the floor, the height of the room, the location and direction in which the projector is set, and locations and dimensions of major objects are input in accordance with a procedure, and necessary information is generated and sent to the projector  10 . The CPU  17  of the projector  10  which obtains the information stores the obtained information in the data storage unit  11 . 
         [0045]    Next, the CPU  17  obtains information of a viewpoint position of the viewer EJ from an external device via the antenna  24  and the communication unit  22  (step S 102 ). 
         [0046]    In other words, in the application program, the location where the viewer EJ sits in the room RM, the height of the viewpoint, and directions of major lines of sight are input in accordance with a procedure, and the necessary information is generated and sent to the projector  10 . Thus, the position information of the viewer&#39;s viewpoint with respect to the projector  10  can be obtained. Similarly to the manner described above, the CPU  17  of the projector  10  which obtains the information stores the obtained information in the data storage unit  11 . 
         [0047]    Next, the CPU  17  which has obtained the information of the room shape and the position information of the viewer&#39;s viewpoint generates, for each projection surface, viewpoint correction matrices for converting a projection location of the image so that the image looks continuous from the viewer&#39;s viewpoint, and the CPU  17  further stores the generated viewpoint correction matrices in the data storage unit  11  (step S 103 ). Surface information of a projection surface may be obtained not only for a planar surface, but also a curved surface. 
         [0048]    Furthermore, the CPU  17  generates correction mask data to allocate, from the generated viewpoint correction matrices, a viewpoint correction matrix to a projection surface in accordance with a pixel location in the image to be projected, and the CPU  17  further stores the generated correction mask data in the data storage unit  11  (step S 104 ), and the process shown in  FIG. 4  is finished. 
         [0049]      FIG. 5  is a drawing illustrating correction mask data generated by the CPU  17  through the process shown in  FIG. 4 . As described above, the projection lens unit  16  is a fisheye lens, and it has the projection angle of view on a surface along the projection light axis of the order of 270 degrees, and the projection angle of view on a surface orthogonal to the projection light axis of the order of 360 degrees. 
         [0050]    The original image used for projection has a circular shape, and the image is divided into regions in accordance with the projection surfaces as shown in  FIG. 5 . The viewpoint correction matrices are stored in the data storage unit  11  for respective regions corresponding to the projection surfaces, and the image viewed from the viewpoint of the viewer EJ will look natural on each projection surface by performing coordinate conversion using the viewpoint correction matrices. 
         [0051]      FIG. 6  is a drawing showing a flow of the basic process that is carried out, after the above-described initial setting of the projector  10  is finished, for one individual frame of image data, when video data actually stored in the data storage unit  11 , or external video data anytime sent through the antenna  24  and the communication unit  22  are reproduced. 
         [0052]    In other words, after reading individual image data constituting the video data from the data storage unit  11  prior to projection, or after obtaining the image data through the antenna  24  and the communication unit  22  (step S 201 ), the CPU  17  performs viewpoint correction calculation using the correction mask data and the viewpoint correction matrices (step S 202 ). 
         [0053]    In this viewpoint correction calculation, for the original image data as described in the foregoing, calculation to convert a coordinate position of projection using a corresponding viewpoint correction matrix is performed on each pixel information for each of the projection surfaces that are divided by the correction mask data. The conversion calculation makes an entire image viewed from a viewpoint of the viewer EJ look continuous and normal. 
         [0054]    When synchronously performing the display operation in the micromirror elements  13  by the projection image driving unit of the projection system PI and the light emitting operation by the light source unit  14 , using the projection image on which the conversion has been performed, the projection image is projected on almost all surfaces of the room RM by the projection lens unit  16  by reflective light of the micromirror elements  13  (step S 203 ). 
         [0055]    According to the present embodiment as described above, it is possible to project a natural image and that is easy for a viewer to see when a full-spherical-type image is projected. 
         [0056]    In the foregoing embodiment, the shape information of the room RM where the projector  10  is set and the viewpoint position information of the viewer EJ are input from external devices; however, the embodiment is not limited thereto. In order to obtain shape information of the room RM and viewpoint position information of the viewer, the projector  10  itself may comprise an imaging unit including a lens optical system having optical properties equivalent to those of the foregoing projection lens unit  16  to capture an image of the room RM. 
         [0057]    In this case, a color opening filter consisting of color filters of two different colors may be arranged at the lens opening of the imaging unit, for example, to detect a distance to an object for each pixel by performing image analysis on blurriness and color registration errors which occur in an image captured by the lens depending on a distance to an object. 
         [0058]    Thus, while location information of each divided projection surface is extracted in accordance with continuity of changes in a distance between neighboring pixels based on an obtained distance image, it is possible to obtain position information of a viewer&#39;s viewpoint from the distance image and a location of the viewer in the captured image on which an object extracting process and a facial recognition process are performed. 
         [0059]    Thus, the projector  10  comprising an imaging unit enables itself to perform the initial setting operation explained with reference to  FIG. 2  to  FIG. 6 , thereby facilitating the handling of the projector  10 . 
       Second Embodiment 
       [0060]    In the following, it is assumed that the present invention is a portable projector, and the second embodiment in which the present invention is applied to a projection system that performs initial setting using a dedicated application program installed onto a smartphone, for example, will be explained. 
         [0061]    The functional circuit configuration of the projector is basically the same as that shown in  FIG. 1 , and the same reference symbols are used for the same elements, and repetitive illustration and descriptions will be omitted. 
         [0062]    Similarly, assuming that the configuration of the hardware circuit of the smartphone is similar to that of a well-known technique, illustrations and descriptions thereof will be omitted. 
         [0063]      FIG. 7  is a flowchart showing a process for the initial setting of the projector  10  by an application program installed on the smartphone. According to  FIG. 7 , after setting the portable projector  10  at a location where image projection is performed as desired and turning on the power, the user as a viewer who is at a viewing location starts the application program on the smartphone to perform the initial setting. 
         [0064]    At the beginning of the process, a command instructing projection of a test pattern image is sent from the smartphone to the projector  10  (step S 301 ). 
         [0065]    On the projector  10  side where the command is received via the antenna  24  and the communication unit  22 , the CPU  17  reads image data of test patterns which are stored in the data storage unit  11  in advance, and projects a test pattern image using the projection system PI. 
         [0066]    On the smartphone, among camera modes, a panorama capture mode in which a capture range is moved while capturing an image is automatically set after sending the command so as to accept a holding operation of the shutter button, and capturing an image of the projected test pattern image is performed (step S 302 ). 
         [0067]    When capturing an image in the panorama mode, the user of the smartphone continues capturing an image with the smartphone, rotating their body in a horizontal direction from the viewing position, and turns the shutter operation off at the time of finishing capturing an image in 360 degrees. For example, capturing the image continues from when the finger presses the shutter button until the finger is removed. 
         [0068]    If the image capture range of the vertical direction (from the floor to the ceiling) is narrower than the range of the projected test pattern image while the capture angle of view of the lens optical system of the smartphone is set to be the widest at the time of capturing an image in the panorama mode, the user may continue capturing an image for two rotations to capture an image of 720 degrees, i.e., 360 degrees for the upper part and 360 degrees for the lower part of the test pattern image. 
         [0069]    In the smartphone, whether or not the image capture is finished is determined based on whether or not the off operation is performed for the shutter button (step S 303 ). Herein, if it is determined that the off operation for the shutter button has not been performed and the image capture has not been finished (“No” in step S 303 ), the image capture in the panorama mode is continued to await the off operation for the shutter button. 
         [0070]    In step S 303 , when the off operation is performed for the shutter button and it is determined that the image capture is finished (“Yes” in step S 303 ), a synthesis process for the test pattern image captured by the smartphone is performed in the smartphone, so that the boundary portions are put together as a continuous image in a spherical shape in the range of 360 degrees (step S 304 ). 
         [0071]    Subsequently, a command instructing to stop the projection of the test pattern image is sent to the projector  10  from the smartphone (step S 305 ). 
         [0072]    On the projector  10  side where the command is received via the antenna  24  and the communication unit  22 , the CPU  17  stops the projection of the test pattern image by the projection system PI upon the reception of the command. 
         [0073]    Furthermore, while an image obtained by the synthesis process is displayed on a display of the smartphone, the smartphone accepts on a touchscreen integrally provided on the display, tapping to input a point which constitutes a corner of a projection surface, scrolling to input a location of a side of the projection surface, and flicking to move the displayed image on the display, thereby accepting the inputs to define the projection surfaces in the captured image as inputs of coordinate positions corresponding to the test pattern image (step S 306 ). 
         [0074]    In this case, the image of the test pattern image displayed on the display may be enlarged or reduced by the spread operation and the pinch operation using two fingers on the touchscreen. 
         [0075]    While acceptance of those inputs is being carried out, it is determined whether or not an operation indicating that the inputs to define all the projection surfaces are finished is input (step S 307 ). 
         [0076]    If it is determined that the inputs to define all the projection surfaces have not been yet finished and the operation indicating the finish of the input has not been yet input (“No” in step S 307 ), the process returns to step S 306 , and the step of accepting the inputs to define the projection surfaces will be successively performed. 
         [0077]    While the process at step S 306  and step S 307  is repeated in such a manner to accept the inputs to define the projection surfaces, the process awaits an operation indicating that the input is finished. 
         [0078]    Furthermore, in step S 307 , when it is determined that the inputs to define all the projection surfaces are finished and an operation indicating the finish is input (“Yes” in step S 307 ), the smartphone calculates and obtains from the touch inputs made on the test pattern image captured by the smartphone, coordinate information of position for each of the plurality of projection surfaces constituting the environment where the projector  10  is set (step S 308 ). 
         [0079]    Next, with the smartphone, coordinate information of the viewpoint position of the user (viewer) surrounded by the projection surfaces is calculated and obtained (step S 309 ). 
         [0080]    In the smartphone which obtains the coordinate information of position for the projection surfaces of the room and the coordinate information of the viewer&#39;s viewpoint position, for each projection surface, viewpoint correction matrices for converting a projection location of the image so that the image looks continuous from the viewer&#39;s viewpoint, is generated by calculation (step S 310 ). 
         [0081]    Furthermore, in the smartphone, correction mask data is generated from the generated viewpoint correction matrices by calculation in order to allocate a viewpoint correction matrix to a projection surface in accordance with a pixel location in the image to be projected (step S 311 ). 
         [0082]    Finally, the smartphone sends the generated viewpoint correction matrices data and the correction mask data to the projector  10  (step S 312 ), and finishes the application program for the initial setting of the projector shown in  FIG. 7 . 
         [0083]    Hereinafter, the CPU  17  of the projector  10 , which has received the viewpoint correction matrices data and the correction mask data from the smartphone via the antenna  24  and the communication unit  22 , transforms the image based on the information, thereby enabling the projector to project the image in accordance with the shape of the projection environment and the viewer&#39;s viewpoint position. 
         [0084]    In the foregoing embodiment, the process of generating the viewpoint correction matrices data and the correction mask data by calculation has been explained; however, the calculation process may be executed on the projector  10  side. 
         [0085]    According to the present embodiment as described above, since it is possible to carry out the initial setting for a portable projector  10  using a mobile information device having an imaging unit and a communication unit such as a smartphone, an image that is natural and easy for a viewer to see can be projected for a full spherical-type image projection, while simplifying the handling. 
         [0086]    In addition, in the present embodiment, inputs to define the projection surfaces can be accepted through an operation of a touchscreen which is integrally provided with a display of a mobile information device, thereby projecting an image easy to view and simplifying the handling. 
         [0087]    A mobile information device which carries out the above-described initial setting of the projector  10  is not limited to a smartphone; it may be a device having an imaging unit and a communication unit, such as a tablet personal computer, or a general feature phone. 
         [0088]    Other than the above, the present invention is not limited to the above-described embodiments, and can be modified in various manners in practice when implementing the invention without departing from the gist of the invention. The functions carried out by the foregoing embodiments may be realized in a combination to the greatest extent possible. The above-described embodiments include various stages, and a variety of inventions can be derived by properly combining structural elements disclosed in connection with the embodiments. For example, if the advantages of the invention are attained even after some of the structural elements disclosed in connection with the embodiments are deleted, the structure made up of the resultant structural elements can be extracted as an invention.