System, method and software for converting images captured by a light field camera into three-dimensional images that appear to extend vertically above or in front of a display medium

System, method and software for producing imagery with enhanced 3D effects that is viewed on a display medium. The imagery contains elements that appear, at least in part to extend vertically above, or in front of the surface of the display medium. To create the imagery with enhanced 3D effects, a light field camera images a scene that contains at least one subject. At a given moment in time, the subject is positioned in an initial orientation. Upon imaging, the light field camera produces light field image data that represents the scene. The light field image data is edited to add enhanced 3D effects. Once edited, the enhanced image data is played on a display medium, where the added 3D effects cause the subject to appear, at least in part, to extend vertically above, or in front of, the display medium.

BACKGROUND OF THE INVENTION

1. Field of the Invention

In general, the present invention relates to systems, methods, and software that are used to create three-dimensional images that are viewed on a display medium, such as an electronic display or printed paper. More particularly, the present invention relates to systems, methods and software that create three-dimensional images with enhanced effects that cause the three-dimensional images to appear to extend above, or in front of, the display medium being viewed.

2. Prior Art Description

Many systems exist for creating images that appear three-dimensional when viewed on a two-dimensional electronic display. However, traditional prior art systems create three-dimensional images that appear to exist behind or below the plane of the display medium. It is far more difficult to create a three-dimensional image that will appear to stand above, or in front of, the screen on which it is viewed. To create a three-dimensional image that appears to be above or in front of a display medium, sophisticated adjustments have to be incorporated into the creation of the image. Such adjustments often include using stereoscopic cameras and creating complex adjustments to the parallax of the stereoscopic cameras used in the creation of the image. Prior art systems that modify the parallax of stereoscopic images are exemplified by U.S. Pat. No. 7,589,759 to Freeman, U.S. Patent Application Publication No. 2012/0263372 to Adachi and U.S. Patent Application No. 2011/0063420 to Masuda.

In the prior art, creating a three-dimensional image that appears to extend in front of or above a display medium is primarily accomplished by creatively altering the parallax of the imaging stereoscopic cameras as the object is imaged. Only minor adjustments are made to the virtual object being imaged prior to the imaging.

It has been discovered that three-dimensional images can be created more realistically and with more clarity by using a light field camera and manipulating the camera image data to add enhanced 3D effects. The added 3D effects cause the images to appear to extend vertically above, or in front of, a display medium. The improved technique represents an advancement in the art as described and claimed below.

SUMMARY OF THE INVENTION

The present invention is a system, method and software for producing imagery with enhanced 3D effects that is viewed on a display medium. The imagery with enhanced 3D effects can be a fixed image or a dynamic video. The imagery contains elements that appear, at least in part to extend vertically above, or in front of the display medium that is showing the imagery.

To create the imagery with enhanced 3D effects, a light field camera images a scene that contains at least one subject. At a given moment in time, the subject is positioned in an initial orientation. Upon imaging, the light field camera produces light field image data that digitally represents the scene being imaged. The light field image data may be interpreted dimensionally different ways, including converting to left and right eye stereo images, or converting to left and right eye side-by-side images.

The light field image data corresponding to the subject in the scene is edited to add enhanced 3D effects. Once edited, the light field image data becomes enhanced image data. When the enhanced image data is printed as an image or played on a display, the added 3D effects cause the subject to appear, at least in part, to extend vertically above, or in front of, the display medium depending upon the orientation of the display medium.

DETAILED DESCRIPTION OF THE DRAWINGS

Although the present invention system, method and software can be used to image most any object, the embodiment illustrated shows the present invention being used to image of a toy dinosaur. This embodiment is selected for the purposes of description and explanation only. The toy dinosaur is intended to represent any object that can be imaged and presented through the system. However, the illustrated embodiment is purely exemplary and should not be considered a limitation when interpreting the scope of the appended claims.

Referring toFIG. 1, it will be understood that the present invention is used to produce imagery10with enhanced 3D effects that is viewed on a display medium, such as a printed page or the illustrated example of a display12of an electronic device14. The imagery10can be a stationary image or a video. Regardless, the imagery10appears to have elements that are three-dimensional when viewed on the display12. Furthermore, at least part of some of the elements embody enhanced 3D effects that cause those elements to appear to extend vertically above, or in front of, the surface plane of the display12. If the electronic device14has a traditional LED or LCD display, the imagery10will have to be viewed with 3D glasses in order to observe the three-dimensional effects in the imagery10. If the electronic device14has an autostereoscopic display, then the three-dimensional effects in the imagery10can be observed with the naked eye.

The imagery10that contains enhanced 3D effects starts as a physical scene15that is captured by a light field camera17. The physical scene15captured by the light field camera17typically contains a primary subject20. In the shown example, the primary subject20is a toy dinosaur22. However, it will be understood that any subject or collection of subjects can be imaged. Upon imaging, the light field camera17creates light field image data16. The light field data16is read by a computer19. The computer19runs specialized image editing software18that adds enhanced 3D effects to the light field image data16.

Referring toFIG. 2in conjunction withFIG. 1, it will be understood that the light field image data16is a digital representation of the original physical scene15. The light field image data16can be viewed, wherein the resulting image would appear as it does inFIG. 2. Using the image editing software18, the light field image data16is viewed and a reference plane24is selected for the light field image data16. The reference plane24can be any plane in the light field image data16from which objects are to appear above and/or below. In the shown embodiment, the reference plane24is oriented with the ground upon which the toy dinosaur22stands. The reference plane24of the light field image data16, when displayed on an electronic display12, is going to be oriented along the surface plane of the electronic display12. As such, when the imagery10is viewed, any part of a subject imaged above the reference plane24will project vertically above, or in front of the display12, depending on the orientation of the display12. Conversely, any part of a subject imaged below the reference plane24will appear to be rearwardly projected and will appear below or behind the plane of the display12, when the imagery10is viewed.

If the imagery10is to be printed, then the reference plane24is selected to correspond with the plane of the paper upon which the imagery10is printed.

The light field camera17is positioned at an angle in front of and above the primary subject20in the physical scene15. The angle of elevation A1of the light field camera17is dependent upon the height of the subjects being imaged and the degree of surrounding area desired to be captured by the light field camera17. The field of view imaged by the light field camera17is limited and it is understood that the light field image data16is intended to be shown on an electronic display12. Most electronic displays are rectangular, having a width that is between 50% and 80% of the length. Accordingly, the light field image data16is created within boundaries appropriate in size and scale for a typical electronic display12. The boundaries include a front boundary27, a rear boundary28, and two side boundaries29,30. Any imagery10that is to be displayed on the electronic display12must exist within the boundaries27,28,29,30in order to be seen.

A rear image boundary28is set for the light field image data16. All of the subjects to appear in the final imagery10must be positioned forward of the rear image boundary28. The primary subject20has a height H1. The light field camera17is set to a second height H2. The second height H2is a function of the subject height H1and the rear image boundary28. The second height H2of the light field camera17is high enough so that the top of the primary subject20, as viewed from the light field camera17does not extend above the rear image boundary28. Accordingly, the elevation angle of the light field camera17depends upon the scene boundaries27,28,29,30and height H1of the primary subject20.

Referring toFIG. 3in conjunction withFIG. 2andFIG. 1, it can be explained that the light field image data16from the light field camera17is not used directly. Rather, the light field image data16digitally manipulated using the custom image editing software18being run by the computer19. The custom image editing software18is used to produce various enhance 3D effects. The digital manipulations performed by the custom image editing software18includes, but is not limited to:i. tilt manipulations of the reference plane of the imaged scene;ii. tilt manipulations of the primary and secondary subjects in the imaged scene;iii. bend manipulations;iv. taper manipulations;v. stretch manipulations.
The digital manipulations that are used depend upon the details of the subjects being imaged.

FIG. 3illustrates the possible tilt manipulations that can be used. In one tilt manipulation, the light field image data16that corresponds to the primary subject20and/or other imaged subjects can be digitally tilted toward or away from the position of the light field camera17. Alternatively, the primary subject20itself can be physically tilted prior to imaging. Accordingly, one or more of the various subjects can be titled with an angle of inclination A1. The preferred tilt angle is generally between 1 degree and 20 degrees from the original orientation.

In a second tilt manipulation, the whole of the reference plane24can be tilted toward or away from the light field camera17. This also can be achieved either digitally or physically. The preferred tilt angle A2is generally between 1 degree and 20 degrees from the horizontal, depending upon the final perceived height of the primary subject20.

Using a point P under the primary subject20as a fulcrum point, the reference plane24can be digitally and/or physically manipulated to tilt forward or backward. The tilt angle T2of the reference plane24and the tilt angle T1of the primary subject20are independent of one another. The tilting of the reference plane24changes the position of the rear image boundary28relative to the perceived position of the primary subject20. This enables the height of the primary subject20to be increased proportionately within the confines of the mathematical relationship.

Referring toFIG. 4, in conjunction withFIG. 1andFIG. 1, a preferred bend manipulation is explained. InFIG. 4, the primary subject20B is shown as a rectangle, rather than a dinosaur, for ease of explanation. A bend in the complex shape of a dinosaur would be difficult to perceive. A bend point B1is selected along the height of the primary subject20B. The bend point B1is between ⅓ and ⅔ the overall height of the primary subject20B. The primary subject20B is also divided into three regions31,33,35along its height. In the first region31, the primary image20B is not manipulated. In the second region33, no manipulation occurs until the bend line B1. Any portion of the primary subject20B above the bend line B1and within the second region33is digitally tilted by a first angle AA1. In the third region35, the primary subject20B is tilted at a second angle AA2, which is steeper than the first angle AA1. The first angle AA1and the second angle AA2are measured in relation to an imaginary vertical plane that is parallel to the vertical plane in which the light field camera17is set. The result is that the imagery10can be made larger and taller without extending above the rear image boundary28. When viewed from the perspective of the light field camera17, the primary subject20B appears taller and has a more pronounced forward or vertical projection.

Referring toFIG. 5, in conjunction withFIG. 1andFIG. 2, a preferred taper manipulation is explained. Again, the primary subject20B is shown as a representative rectangle, rather than a dinosaur for ease of explanation. The primary subject20B is divided into two regions37,39along its height. In the first region37, the primary subject20B is not manipulated. In the second region39, the primary subject20B is reduced in size using a taper from front to back of an angle AA3of between 1 degree and 25 degrees. The point where the taper begins is positioned between ⅓ and ⅔ up the height of the primary subject20B. The result is that the imagery10can be made wider without extending beyond the side image boundaries29,30. When viewed, the primary subject20B appears taller and has a more pronounced forward or vertical projection.

Referring toFIG. 6in conjunction withFIG. 1andFIG. 2, a preferred stretch manipulation is explained. Again, the primary subject20B is shown as a representative rectangle, rather than a dinosaur for ease of explanation. There is an imaginary line41that extends from the light field camera17to the rear boundary28of the light field image data16. This imaginary line41represents the upper limit of what can be viewed with enhanced 3D effects. The various subjects, including the primary subject20have heights that may not reach the imaginary line41. If the height of a subject, such as the primary subject20, is below the imaginary line41, then the height of the primary subject20B can be stretched vertically until it approaches the height of the imaginary line41. The result is that all or some of the imaged subjects can be made taller without extending beyond the image boundaries. When viewed, the primary subject20B appears taller and has a more pronounced forward or vertical projection.

Once the light field image data16is digitally adjusted in one or more of the manners described, digitally enhanced image data43is created. The digitally enhanced image data43is used to create two offset images40,42. Since the digitally enhanced image data43is initially obtained from a light field camera17, this data can be skewed to produce two images from two different viewpoints in the same manner as traditional light field camera data.

Referring toFIG. 7andFIG. 8in conjunction withFIG. 1andFIG. 2, it can be seen that the two offset images40,42can be considered as stereoscopic, with one being the left skewed image40(FIG. 7) and one being the right skewed image42(FIG. 8). Each offset image40,42has a fading perspective due to the elevated orientation of the light field camera17. This causes the front image boundary27to appear to be wider than the rear image boundary28in both offset images40,42.

Referring toFIG. 9, a top view of one of the offset images40,42fromFIG. 7orFIG. 8is shown. Although only one of the offset images is shown, it will be understood that the described process is performed on both of the offset images40,42. Thus, the reference numbers40,42of both offset images are used to indicate that the processes affect both.

Temporary reference guides are superimposed upon the stereoscopic images40.42. The reference guides include a set of inner guidelines44and a set of outer guidelines46. The inner guidelines44are parallel lines that extend from the rear image boundary28to the front image boundary27. The inner guidelines44begin at points P2where in stereoscopic images40,42met the rear boundary line28. The outer guidelines46are also parallel lines that extend from the rear image boundary28to the front image boundary27. The position of the outer guidelines46depends upon the dimensions of the electronic display12upon which the imagery10is to be displayed. The width between the outer guidelines46corresponds to the pixel width of the electronic display to be used.

Referring toFIG. 10in conjunction withFIG. 9, it can be seen that the peripheries of the offset images40,42are digitally altered to fit within the parameters of the outer guidelines46. As such, the offset images40,42are widened toward the rear image boundary28and compressed toward the front image boundary27. This creates corrected offset images40A,42A. The inner guidelines44remain on the corrected offset images40A,42A.

Referring toFIG. 11, in conjunction withFIG. 10, the corrected left and right offset images40A,42A are superimposed. The inner guidelines44from both corrected offset images40A,42A are aligned. Once alignment is achieved, the inner guidelines44are removed. This creates the final imagery10. Depending upon how the final imagery48is to be viewed, the corrected offset images40A,42A can be colored in red or cyan. Alternatively, the offset images40A,42A can be can be oppositely polarized or set into a side-by-side format. In this manner, when the final imagery10is viewed using 3D glasses or is viewed on an autostereoscopic display, the final imagery48will appear to be three-dimensional.

Referring toFIG. 12in view of all earlier figures, the software methodology for the overall system can now be summarized. As is indicated in Block50, a content producer uses a light field camera17to image one or more subjects20that are to appear having enhanced 3D effects. See prior description ofFIG. 1andFIG. 2. The content producer also selects a reference plane24for the light field image data produced by the light field camera. See Block52. Using the reference plane16and the selected subjects20, the content producer can determine the boundaries of the image field to be used by the final imagery10. See Block54.

Knowing the boundaries of the image field, the reference plane24, and the position of the light field camera17. The light field image data16collected by the light field camera17can be enhanced using the image editing software18. As is indicated by Blocks58,60,61and62, the light field image data16is digitally altered using tilt manipulations, bend manipulations, taper manipulations and stretch manipulations. See prior description ofFIG. 3,FIG. 4.FIG. 5. andFIG. 6. This produces enhanced image data43. The enhanced image data43is then utilized to create two offset images40,42. See Block64. Also see prior description ofFIG. 7andFIG. 8. The offset images40,42are then corrected to fit the boarder guidelines. See Block66. Also see prior description ofFIG. 9andFIG. 10. Lastly, the corrected offset images are superimposed. See Block68. Also see prior description ofFIG. 11. The result is final imagery10having enhanced 3D effects that appear to extend above, or in front of, the display12when viewed by a user.

It will be understood that the embodiment of the present invention that is illustrated and described is merely exemplary and that a person skilled in the art can make many variations to that embodiment. All such embodiments are intended to be included within the scope of the present invention as defined by the appended claims.