Abstract:
Devices that provide stereoscopic vision by placing a lens array or a pinhole array on a display have a problem. The resolution of an object in the background present at a great distance is degraded. With respect to a three-dimensional object of interest, an intermediate stereoscopic image. This intermediate image is synthesized with a two-dimensionally projected background image separately prepared to obtain another image, and this image is displayed on a stereoscopic display.

Description:
BACKGROUND  
       [0001]     The present invention relates to a device that provides stereoscopic vision and in particular to an autostereoscopic vision device wherein stereoscopic vision can be implemented with the naked eye.  
         [0002]     Recently, the resolutions of displays have been enhanced, and fabrication techniques for microlenses have been improved. In this context, there is a market trend toward displays with autostereoscopic vision utilizing integral photography (hereafter, referred to as “IP”) (M. G. Lippmann, “Epreuves reversibles donnant la sensation du relief,” J. de Phys., vol. 7, 4th, series, pp. 821-825, November 1908). The IP is a system based on the reproduction of three-dimensional light sources.  
         [0003]     Also, there are multi-view stereoscopic display devices that create a stereoscopic effect only in the lateral direction using lenticular lenses or parallax barriers. As illustrated in  FIG. 14 , the method on which these devices are based is basically intended to present an image that gives a binocular parallax to the left and right eyes  20  and  21 . It can be thought to be a special form of the IP.  
         [0004]     In these stereoscopic display devices, the pixels of a display have a finite quantity and a finite size. Therefore, these display devices have a problem. When a far-off object in the background is depicted, the resolution is degraded.  
         [0005]     To cope with this, a technique to dispose a variable-focal length lens in front of a microlens to widen the range in which depth is represented has been developed, as disclosed in Japanese Unexamined Patent Publication No. 2001-238229.  
         [0006]     To provide a three-dimensional sense and further improve the amount of data and rendering speed with a conventional two-dimensional display unit, there is known the following method: a two-dimensionally projected image is used for the background, and a character of interest is realistically rendered using a three-dimensional model; and this character is synthesized into the two-dimensional background.  
       SUMMARY  
       [0007]     Description will be given to the IP with reference to  FIG. 11  to  FIG. 13 . A lens array  2  in which convex lenses are arranged in an array is installed on the front face of a display device  1 .  FIG. 11  illustrates the stereoscopic positional relation, and  FIGS. 12 and 13  show a section taken therefrom. In a case where the pixels of the display are very small relative to the lenses, the following takes place when only the pixel in the position indicated by an open circle in  FIG. 11  is displayed in some color with some brightness on the display  1 : light is focused on the position indicated by the open circle, marked with numeral  50 , by the effect of the lens array  2 , and it becomes rays of light that are widened therefrom.  
         [0008]     When an observer  20  views it within the range of view angle, indicated by numeral  51 , it is possible to make the observer perceive as if a point light source, that is, an object were present in the position marked with numeral  50 . Pinholes may be used in place of these lenses.  
         [0009]      FIG. 12  shows an ideal state in which the display device  1  comprises a large amount of very small pixels. In reality, the pixels of the display device  1  have a finite size and a finite quantity, as illustrated in  FIG. 13 . The following is apparent from this with respect to the three-dimensional position of a light source that can be represented as a region obtained by connecting the center of a lens and both the ends of a pixel of the display device  1 : the representable resolution is poorer in region  36  farther from the lens array  2  than region  35 .  
         [0010]     That is, there is the following problem: the farther an object to be depicted is positioned from lenses, the more the resolution is degraded. It can be conversely thought that the resolution is highest on the plane of the lens array. As an example, it will be assumed that the image  54  of a wall as the projected background is present in the position of the plane of the lens array  2 , as illustrated in  FIG. 15 . When this image is displayed as a stereoscopic image, the stereoscopic display can be accomplished with the highest resolution. When the display is made as mentioned above, however, a problem arises. Part of a three-dimensional object  8  present behind the background image, such as region  55  in  FIG. 15 , cannot be displayed.  
         [0011]     With the technique disclosed in the above-mentioned patent document 1, the range of representation of depth can be widened to some extent by utilizing a lens aside from the lens array  2 . However, the technique cannot solve the problem that, when infinity is represented, a blur occurs. When actually implementing the device is considered, another problem arises. The number of components is increased, and this makes the device expensive.  
         [0012]     According to the present invention, stereoscopic vision can be provided with the enhanced apparent resolution of the background by the taking the following procedure: humans&#39; illusion and the like are utilized, and an stereoscopic image is created by ray tracking or with multiple camera parameters and a two-dimensionally projected background image, separately created, are synthesized together. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0013]      FIG. 1  is a block diagram illustrating an example of an stereoscopic image generation and output device and an autostereoscopic display.  
         [0014]      FIG. 2  is a drawing illustrating an example of the flow of processing carried out when stereoscopic image is displayed using the device illustrated in  FIG. 1 .  
         [0015]      FIG. 3  is a sectional view explaining a technique to generate a stereoscopic image.  
         [0016]      FIG. 4  is a conceptual rendering illustrating the way a background image is acquired.  
         [0017]      FIG. 5  is a drawing illustrating an example of the flow of processing of Step S 5  in  FIG. 2  in detail.  
         [0018]      FIG. 6  is an explanatory drawing illustrating an example in which the image of a three-dimensional object is picked up from multiple viewpoints by live action shot.  
         [0019]      FIG. 7  is a block diagram illustrating an example in which a stereoscopic image generation device and a stereoscopic image output device exist separately from each other.  
         [0020]      FIG. 8  is a sectional view illustrating a method of generating a stereoscopic-image by parallel projecting 3D data onto multiple planes of projection.  
         [0021]      FIG. 9  is a sectional view illustrating a method of generating 3D data from an image perspectively projected from multiple viewpoints.  
         [0022]      FIG. 10  is an explanatory drawing illustrating an example of a case where a 360° background image is generated.  
         [0023]      FIG. 11  is a three-dimensional explanatory drawing illustrating the principle of the IP based autostereoscopic display.  
         [0024]      FIG. 12  is a two-dimensional explanatory drawing illustrating a section taken from  FIG. 11 .  
         [0025]      FIG. 13  is an explanatory drawing illustrating the resolution of a reproduced light source according to the distance from a display in the IP.  
         [0026]      FIG. 14  is a conceptual rendering of the multi-view stereoscopic vision.  
         [0027]      FIG. 15  is an explanatory drawing illustrating a problem that arises when the resolution is enhanced by displaying a background image plane of the lens array on the plane of the lens array.  
     
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS  
       [0028]     Description will be given to stereoscopic display obtained by displaying an image generated by a program that generates an intermediate image for stereoscopically displaying a three-dimensional object, a program that generates a background image, and a program that synthesizes an intermediate image and a background image.  
         [0029]     The present invention produces the effect of producing display in which the resolution of a three-dimensional image looks enhanced. This will be described with reference to, for example,  FIG. 14 . When multi-view stereoscopic vision is provided, image information having a parallax is displayed with respect to a pixel viewed from viewpoints  20  and  21 , and the stereoscopic effect is thereby produced. Meanwhile, it will be assumed that a two-dimensionally projected background image is displayed on a display  1 . In this case, the image that can be viewed at viewpoints  20  and  21  is not information having a parallax but information on the background in different positions. For this reason, it is expected that correspondence cannot be established between the left and right eyes, and the image is one difficult to recognize. In reality, however, the following takes place possibly because the pieces of the information are those on adjacent pixels and they have similar pixel values: the background image is perceived as if it were displayed on the display surface, and its resolution is sensed to be high.  
         [0030]     When a wall formed by projecting a background image is positioned on the plane of the lens array as mentioned above, a three-dimensional object behind the wall cannot be displayed. According to the present invention, a three-dimensional object positioned behind can also be displayed. In this case, a problem arises under ordinary circumstances. The anteroposterior relation between a three-dimensional object positioned behind and a background image is inverted. However, this is negligible. Humans perceive an image looking like a background as a background, and thus the stereoscopic effect without problems can be produced as a whole.  
       First Embodiment  
       [0031]      FIG. 1  is a block diagram illustrating a first embodiment, and arrows of dotted line in the figure show the conceptual flow of data. Description will be given to individual components and the relation between the components with reference to this figure.  
         [0032]     A stereoscopic display  3  is a combination of a display  1  that displays an ordinary two-dimensional image and a convex lens array  2 . An observer  20  observes the stereoscopic display  3  from the convex lens array  2  side.  
         [0033]     For an image generation and output device  4  for stereoscopic vision, for example, a commonly used computer is used. A storage device  6  stores data and programs, which are loaded to a main memory  18  through OS (Operating System) as required. Computation is carried out at CPU  17 .  
         [0034]     The CPU  17  is a computing unit and may be constructed of multiple processors. Or, it may be DSP (Digital Signal Processor) or GPU (Graphics Processor Unit). In a case where the CPU  17  is GPU, the main memory  18  maybe a memory on a graphics board.  
         [0035]     First, according to the specifications of the stereoscopic display  3 , an stereoscopic image  10  is generated from 3D data defined at the storage device  6  by a stereoscopic image generation program  9 . The generation method will be described later. The stereoscopic image  10  may be generated from live action shot images, picked up by cameras from multiple view points, by the stereoscopic image generation program  9 .  
         [0036]     Next, part of a live action shot image  11  is defined as a background image  12 . The stereoscopic image  10  and the background image  12  are synthesized together to generate a synthesized image  15  by a synthesized image generation program  14 . The synthesis method will be described later. The background image  12  may be generated by a background image generation program  13  utilizing the 3D data  8 .  
         [0037]     The synthesized image  15  is loaded to a frame memory  16  by a synthesized image display program  19  through the OS, and is outputted to the stereoscopic display  3  via an input/output IF  5 .  
         [0038]     Description will be given to an embodiment that can be implemented with the construction illustrated in  FIG. 1  with reference to the flowchart in  FIG. 2  from the aspect of program.  
         [0039]     There are three different methods for generating a stereoscopic image  10  in the case illustrated in  FIG. 2 . This embodiment uses the following method: utilizing the 3D data  8 , rendering is carried out on rays of light that connect pixels and lens centers by the stereoscopic image generation program  9  in  FIG. 1 , and a stereoscopic image is thereby generated (Step S 1 ). This method will be described with reference to  FIG. 3 .  
         [0040]     Like  FIG. 12 ,  FIG. 3  is a sectional view of the stereoscopic display  3 . It will be assumed that at this time, there is the 3D data  8  represented by a circle and a triangle as illustrated in  FIG. 3 . A ray of light is drawn from the center of each pixel of the display  1  so that the ray of light passes through the center of the corresponding lens. The rays of light intersecting the 3D data  8  at this time are indicated by broken lines of the points of intersection of the rays of light and the surface of the three-dimensional object, points closest to the observer are indicated by filled circles  38 . That is, in the generation method for stereoscopic image at Step S 1  in  FIG. 2 , a stereoscopic image can be generated by determining the color and brightness of each of the filled circles  38  in  FIG. 3 .  
         [0041]     Description will be given with reference to the flowchart in  FIG. 2  again.  
         [0042]     In this embodiment, a background live action shot image  26  is used as the background image. This is obtained by defining the live action shot image  11  in  FIG. 1  as the background image  12 . As illustrated in  FIG. 4 , for example, an image  44  obtained by shooting a landscape embracing a mountain  40 , a tree  41 , and a house  42  with a camera  43  is taken as the background live action shot image.  
         [0043]     The stereoscopic image  10  and the background live action shot image  26  are synthesized together by the synthesized image generation program  14  in  FIG. 1  (Step S 5 ). Description will be given to this method for synthesis with reference to  FIG. 3  and the flowchart in  FIG. 5 .  FIG. 5  illustrates the details of Step S 5  in  FIG. 2 .  
         [0044]     When the stereoscopic image is generated through the processing of Step S 1 , of the pixels on the display  1 , the pixels  37  indicated by hatching in  FIG. 3  have no relation to the processing to represent the 3D data. Therefore, when the stereoscopic image  10  is generated at Step S 1 , the pixel values of the pixels  37  irrelevant to the representation of the 3D data are set to, for example, −1.  
         [0045]     It is examined whether all the processing has been completed or not with respect to the stereoscopic image  10  (Step S 10 ). If completed, the synthesizing process is ended (S 17 ). If not, the pixel values of the pixels are examined one by one, and it is determined whether each pixel is irrelevant to the representation of the three-dimensional object (Step S 11 ). (In this embodiment, an irrelevant pixel has a pixel value of −1.) When a pixel is irrelevant, the pixel value in the same pixel position in the background live action shot image  26  is written as a pixel value of the synthesized image  12  (Step S 14 ). When a pixel is judged not to be irrelevant at Step S 11 , its pixel value in the stereoscopic image  10  can be written as a pixel value in the synthesized image  12  (Step S 13 ).  
         [0046]     The synthesized image  15  generated by the above-mentioned technique is displayed on the stereoscopic display by the synthesized image display program  19  in  FIG. 1  (Step S 6 ).  
         [0047]     In the above-mentioned embodiment, stereoscopic 3D data can be displayed as stereoscopic vision over a live action shot background of high resolution. Thus, the apparent resolution can be enhanced to improve the quality of stereoscopic display.  
       Second Embodiment  
       [0048]     Description will be given to another embodiment in which an stereoscopic image  10  is generated, with reference to  FIG. 2 .  
         [0049]     First, an intermediate image is generated from 3D data  8  with virtual camera parameters (position, number of pixels, angle of view, aspect ratio, etc.) at multiple viewpoints (Step S 2 ). As illustrated in  FIG. 8 , for example, planes  61  to  63  of projection are prepared, and a multiview intermediate image  24  as a projection of the 3D data  8  is generated by parallel projection. The number of pixels on the display  1  assigned to one lens is basically taken as the number of planes of projection. However, it may be required to increase the number of planes of projection depending on the disposition of lenses.  
         [0050]     As another method for generating a multiview intermediate image, the following procedure may be taken on the assumption of the principle of multi-view stereoscopic display: a multiview intermediate image  24  is generated as such an image as is obtained by observing the 3D data  8  from the positions of view points  65  to  67  by perspective projection, as illustrated in  FIG. 9 . The number of pixels on the display  1  assigned to one lens is basically taken as the number of viewpoints. However, it may be required to increase the number of viewpoints depending on the disposition of lenses.  
         [0051]     With respect to each pixel on the multiview image  24  generated as mentioned above, a pixel value is assigned to the corresponding pixel on the display  1 , and a stereoscopic image  10  is thereby generated (Step S 3 ).  
         [0052]     The subsequent steps are the same as in the first embodiment.  
         [0053]     Use of this embodiment obviates necessity for preparing a rendering program dedicated to stereoscopic display. A stereoscopic image  10  can be generated by utilizing commercially available CG rendering software.  
       Third Embodiment  
       [0054]     Description will be given to another embodiment in which an stereoscopic image  10  is generated, with reference to  FIG. 2 .  
         [0055]     As at Step S 2  mentioned above, a multiple viewpoint live action shot image  25  is prepared on the assumption of the principle of multi-view stereoscopic display. (The live action shot image corresponds to part of the live action shot image  11  in  FIG. 1 .) The number of pixels on the display  1  assigned to one lens is basically taken as the number of these viewpoints. Instead, a method in which an image of intermediate viewpoint is estimated from an image of a smaller number of viewpoints may be used.  
         [0056]     At this time, to divide a background and a three-dimensional object of interest from each other, a chroma key extraction technique for movies and television can be utilized. As illustrated in  FIG. 6 , a background  47  in one color, for example, green is placed behind a three-dimensional object  48 , and the object is shot with cameras  44  to  46 .  
         [0057]     As in the second embodiment, pixels that correspond to pixels on the display  1  are picked up from the multiple viewpoint live action shot image  25 , or the multiview image, prepared as mentioned above. Pixel values are assigned to these pixels, and a stereoscopic image  10  is thereby generated (Step S 3 ).  
         [0058]     The subsequent steps are the same as in the first embodiment. However, since there is a slight difference in the processing of Step S 5 , it will be described with reference to  FIG. 5 .  
         [0059]     In the judgment at Step S 11 , whether a pixel is irrelevant to the representation of a three-dimensional object is determined by the color of the shot background. (In the example described in connection with  FIG. 6 , this color is green.)  
         [0060]     In addition, the following procedure is taken in the third embodiment: when a live action shot picture is used, the boundary between a three-dimensional object and a background can be shot in one pixel. As a result, the outline of the three-dimensional object becomes obscure. To cope with this, the following processing can be performed: with respect to a pixel judged to be relevant at Step S 11 , it is further examined whether the three-dimensional object is blended with the background in the pixel (Step S 15 ). With respect to a pixel in which the three-dimensional object is blended with the background, such processing as in conventional chroma key synthesis techniques is performed. That is, the blend ratio is estimated, and a synthesized image  15  is generated using a pixel value obtained by mixture with the pixel value of the corresponding pixel in the background live action shot image  26  (Step S 16 ).  
         [0061]     According to this embodiment, as mentioned above, the following can be implemented by using only a live action shot image: the unnaturalness of a three-dimensional object at the boundary is eliminated, and further stereoscopic vision is displayed with the resolution of an image as the background being high.  
       Fourth Embodiment  
       [0062]     Description will be given to another embodiment relating to background image with reference to  FIG. 2 .  
         [0063]     In the above-mentioned embodiments, a live action shot image is used as the background image. The recent advancement of rendering techniques has made high-resolution and realistic rendering possible. In this embodiment, consequently, a background image is generated from 3D data by the background image generation program  13  in  FIG. 1 . Like the background shot in live action described with reference to  FIG. 4 , for example, the pieces of 3D data for a mountain and a house are disposed in a three-dimensional space in a computer. Then, a background image  12  with high resolution is generated by a rendering technique that obtains high image quality (Step S 4 ).  
         [0064]     The same processing as in the first to third embodiments mentioned above can be performed except that the background image  12  is used as the background image, in place of the background live action shot image  26 . That is, the processing of Step S 15  and Step S 16  maybe performed or may not be performed. For example, in a case where antialiasing has been enabled when an stereoscopic image  10  is generated, a blue wall or the like is defined as 3D data and the stereoscopic image  10  is generated as in the case of live action shot. Thus, the outline of the three-dimensional object is blended with blue in the background. In this case, the processing of Step S 15  and Step S 16  can be performed as in the third embodiment.  
         [0065]     According to this embodiment, a world that is impossible in live action shot can be utilized as the background. In a case where a stereoscopic image is generated from 3D data, contents without the sense of unnaturalness can be created. In a case where a stereoscopic image is generated from a live action shot image, contents that look as if a three-dimensional object shot in live action enters a CG world can be created.  
       Fifth Embodiment  
       [0066]     Further, description will be given to another embodiment relating to background image.  
         [0067]     In a case where a background image  12  is generated from 3D data  8  as in the fourth embodiment, a 360° background image can be generated by placing pieces of 3D data around a virtual camera  43  for background rendering, as illustrated in  FIG. 10 .  
         [0068]     Also, in live action shot, a 360° background live action shot image can be generated by panning the camera 360° to pick up the image or picking up the image with multiple cameras set with one point at the center.  
         [0069]     Such a 360° background image or a background live action shot image is prepared, and at Step S 5 , it is synthesized with an stereoscopic image  10  generated with respect to a three-dimensional object in motion.  
         [0070]     In a case where a three-dimensional object is present in the direction of arrow  70  in  FIG. 10 , for example, the following procedure can be taken: the background image in the direction of arrow  70  is cut out of the 360° background image in accordance with a predetermined angle of view for background image. Then, the three-dimensional object and the background image are synthesized together.  
         [0071]     In this embodiment, the background behind a moving three-dimensional object changes according to the position of the three-dimensional object, and thus stereoscopic vision can be displayed over a wide range.  
       Sixth Embodiment  
       [0072]     Description will be given another embodiment relating to configuration with reference to  FIG. 5 .  
         [0073]     In this embodiment, the stereoscopic image generation and output device  4  is divided into a stereoscopic image output device  21  and a stereoscopic image generation device  22 . In this embodiment, the steps up to the generation of a synthesized image  15  can be carried out by the stereoscopic image generation device  22  similarly with those in the above-mentioned embodiments.  
         [0074]     The generated synthesized image  15  is transmitted through the input/output IF  88  of the stereoscopic image generation device  22  and the input/output IF  84  of the stereoscopic image output device  21 , and is stored in the storage device  80 . This storage device  80  may be ROM in which information can be written only once or a hard disk or the like on which it can be rewritten any number of times.  
         [0075]     In the stereoscopic image output device  21 , the synthesized image  15  stored in the storage device  80  is loaded to the frame memory  81  by the synthesized image display program  19 . It is transmitted through the input/output IF  84 , and is displayed on the stereoscopic display  3 . This display program  19  may change synthesized images  15  with predetermined timing in predetermined order and cause them to be displayed. Or, it may change synthesized images  15  according to interaction with a user, inputted through the input/output IF  84 .  
         [0076]     According to this embodiment, the stereoscopic image output device  21  can be reduced in size, and its application to a game machine or the like is facilitated.  
         [0077]     With a stereoscopic display device according to the present invention, the following advantages are brought: the apparent resolution of a background can be enhanced without adding any hardware, and the effect of displaying a three-dimensional image so that its resolution looks improved is obtained.