Patent Description:
Head-mounted displays (HMD), also commonly known as near-to-eye displays (NED) or head-worn displays (HWD), have gained significant interest in recent years and stimulated tremendous efforts to push the technology forward for a broad range of consumer applications. For instance, a lightweight optical see-through HMD (OST-HMD), which enables optical superposition of digital information onto a user's direct view of the physical world and maintains see-through vision to the real-world, is one of the key enabling technologies to augmented reality (AR) applications. A wide field-of-view (FOV), immersive HMD, which immerses a user in computer-generated virtual world or a high-resolution video capture of a remote real-world, is a key enabling technology to virtual reality (VR) applications. HMDs find a myriad of applications in gaming, simulation and training, defense, education, and other fields.

Despite the high promises and the tremendous progress made recently toward the development of both VR and AR displays, minimizing visual discomfort involved in wearing HMDs for an extended period remains an unresolved challenge. One of the key contributing factors to visual discomfort is the vergence-accommodation conflicts (VAC) due to the lack of the ability to render correct focus cues, including accommodation cue and retinal image blur effects. The VAC problem in HMDs stems from the fact that the image source is mostly a 2D flat surface located at a fixed distance from the eye. <FIG> shows a schematic layout of a typical monocular HMD, which mainly includes a 2D microdisplay as the image source and an eyepiece that magnifies the image rendered on the microdisplay and forms a virtual image appearing at a fixed distance from the eye. An OST-HMD requires an optical combiner (e.g. beamsplitter) placed in front of the eye to combine the optical paths of the virtual display and real scene. The conventional HMDs, whether monocular or binocular, see-through or immersive, lack the ability to render correct focus cues for the digital information which may appear at other distances than that corresponding to the virtual image plane. As a result, conventional HMDs fail to stimulate natural eye accommodation response and retinal blurry effects. The problem of lacking correct focus cues in HMDs causes several visual cue conflicts. For instance, a conventional stereoscopic HMD stimulates the perception of 3D space and shapes from a pair of two-dimensional (2D) perspective images, one for each eye, with binocular disparities and other pictorial depth cues of a 3D scene seen from two slightly different viewing positions. Therefore, conventional stereoscopic HMDs force an unnatural decoupling of the accommodation and convergence cues. The cue for the accommodation depth is dictated by the depth of the 2D image plane while the convergence depth of the 3D scene is dictated by the binocular disparities rendered by the image pair. The retinal image blurring cues for virtual objects rendered by the display is mismatched from those created by the natural scene. Many studies have provided strong supportive evidence that these conflicting visual cues related to incorrectly rendered focus cues in conventional HMDs may contribute to various visual artifacts and degraded visual performance.

Several approaches proposed previously may overcome the drawbacks of conventional stereoscopic displays, including volumetric displays, super-multi-view auto-stereoscopic displays, Integral-Imaging-based displays, holographic displays, multi-focal-plane displays, and computational multi-layer displays. Due to their enormous hardware complexity, many of these different display methods are not suitable for implementation in HMD systems. On the other hand, the multi-focal-plane display, integral-imaging, and computational multi-layer approaches are commonly referred to be light field displays and are suitable for head-mounted applications. Their use in HMDs is referred to as head-mounted light field displays.

Head-mounted light field displays render a true 3D scene by sampling either the projections of the 3D scene at different depths or the directions of the light rays apparently emitted by the 3D scene and viewed from different eye positions. They are capable of rendering correct or nearly correct focus cues and addressing the vergence-accommodation mismatch problem in conventional VR and AR displays. For instance, an integral imaging (InI) based display reconstructs the light fields of a 3D scene by angularly sampling the directions of the light rays apparently emitted by the 3D scene and viewed from different eye positions. As illustrated in <FIG>, a simple InI-based display typically includes a display panel and a 2D array which can be a microlens array (MLA) or pinhole array. The display renders a set of 2D elemental images, each of which represents a different perspective of a 3D scene. The conical ray bundles emitted by the corresponding pixels in the elemental images intersect and integrally create the perception of a 3D scene that appears to emit light and occupy the 3D space. The InI-based display using 2D arrays allows the reconstruction of a 3D shape with full-parallax information in both horizontal and vertical directions, which is its main difference from the conventional auto-stereoscopic displays with only horizontal parallax using one-dimensional parallax barriers or cylindrical lenticular lenses. Since its publication by Lippmann in <NUM>, the InI-based technique has been widely explored for both capturing the light fields of real scenes and for its use in eyewear-free auto-stereoscopic displays. It has been known for its limitations in low lateral and longitudinal resolutions, narrow depth of field (DOF), and narrow view angle. Compared with all other non-stereoscopic 3D display techniques, the simple optical architecture of an InI technique makes it attractive to integrate with HMD optical system and create a wearable light field display.

However, like other integral-imaging based display and imaging technologies, the current InI-based HMD method suffers from several major limitations: (<NUM>) narrow field of view (<<NUM>° diagonally); (<NUM>) low lateral resolution (about <NUM> arc minutes in the visual space); (<NUM>) low longitudinal resolution (about <NUM> diopters in the visual space); (<NUM>) narrow depth of field (DOF) (about <NUM> diopter for a <NUM>-arc minute resolution criteria); (<NUM>) limited eyebox for crosstalk-free viewing(<<NUM>); and (<NUM>) limited resolution of viewing angle (><NUM> arc minutes per viewing). These limitations not only create significant barriers for adopting the technologies as high-performance solutions, but also potentially undermine the effectiveness of the technology for addressing the accommodation-convergence discrepancy problem.

<NPL> describes a prototype design using tunable lens and aperture array to render 3D scenes over a large depth range while maintaining high image quality and minimizing cross-talk.

<NPL> describes a vivid reconstruction of a 3D scene using computational and optical methods.

<NPL> describes a depth enhancing technique for integral imaging using a varifocal lens array.

<NPL> describes an enhanced integral imaging system with an electrically controllable image plane.

Thus, the present disclosure details methods, design and embodiment of a high-performance head-mounted light field display based on integral imaging that overcomes some aspects of the performance limits of the state of the art summarized above.

There is provided a method for rendering light field images of a 3D scene in a head-mounted display, HMD, using an integral-imaging-based light field display in accordance with claim <NUM>. Other aspects of the invention are set forth in the dependent claims.

The foregoing summary and the following detailed description of exemplary embodiments of the present invention may be further understood when read in conjunction with the appended drawings, in which:.

Referring now to the figures, wherein like elements are numbered alike throughout, as shown in <FIG>, a HMD system <NUM> in accordance with the present invention may include three key subsystems: I) a microscopic InI unit (micro-InI) <NUM>, II) a relay group <NUM> with a vari-focal element (VFE) <NUM> disposed therein for receiving the light fields from the InI unit <NUM>, and III) eyepiece optics <NUM> for receiving the tuned intermediate 3D scene from the relay group <NUM>. As illustrated in <FIG>, the micro-InI unit <NUM> can reproduce the full-parallax light fields of a 3D scene seen from a constrained viewing zone, where the full-parallax light fields offer the change of view perspectives of a 3D scene from both horizontal and vertical viewing directions. The constrained viewing zone optically corresponds to limiting the aperture of the micro-InI unit <NUM>, and the constrained viewing zone is optically conjugate to the exit pupil of the display system <NUM> where a viewer's eye is placed to view the reconstructed 3D scene. The relay group <NUM> creates an intermediate image of the 3D scene reconstructed by the micro-InI unit <NUM> with a tunable position of its central depth plane (CDP). Depending on the magnification power of the eyepiece <NUM>, the position of the CDP may be tunable in the range from about <NUM> to as large as hundreds of millimeters to create the perception of a 3D scene with a large depth range spanning from the optical infinity (<NUM> diopter) to as close as <NUM> (<NUM> diopters). The relay group <NUM> may also facilitate the flip of the concavity of the reconstructed 3D scene AOB. The eyepiece optics <NUM> reimages the tunable 3D light fields into a viewer's eye and enlarges the tunable depth range of the 3D light fields into a large depth volume spacing from meters far to as close as a few centimeters. A see-through unit (not shown), which may be optics with a beamsplitter function, may optically communicate with the eyepiece optics <NUM> to optically enable non-obtrusive view of a real-world scene if a see-through view is desired. The micro-InI unit <NUM> of <FIG>, as further illustrated in <FIG>, may include a high-resolution microdisplay and a micro-lens array (MLA) <NUM>. The focal length of the lenslets <NUM> in the MLA <NUM> is denoted as fMLA and the gap between the microdisplay <NUM> and the MLA <NUM> is noted as g. A set of 2D elemental images, each representing a different perspective of a 3D scene AOB, may be displayed on the high-resolution microdisplay <NUM>. Through the MLA <NUM>, each elemental image works as a spatially-incoherent object and the conical ray bundles emitted by the pixels in the elemental images intersect and integrally create the perception of a 3D scene that appears to emit light and occupy the 3D space. The central depth plane (CDP) of the reconstructed miniature scene, with a depth range of z<NUM>, is located by the distance lcdp measured from the MLA <NUM>. Such an InI system <NUM> allows the reconstruction of a 3D surface shape AOB with parallax information in both horizontal and vertical directions. The light field of the reconstructed 3D scene (i.e., the curve AOB in <FIG>) may be optically coupled into eyepiece optics <NUM> via the relay group <NUM> for viewing by a user. In a resolution priority InI system (fMLA ≠ g), the central depth plane CDP of the reconstructed 3D scene is optically conjugate to the microdisplay <NUM> and its location is given by <MAT> Where MMLA is the magnification of the micro-InI unit <NUM>, which may be expressed by <MAT>.

As shown in <FIG>, <FIG>, optionally, an aperture array <NUM>, including a group of ray-limiting apertures that matches the pitch of the MLA <NUM>, may be inserted between the microdisplay <NUM> and MLA <NUM>. The small aperture corresponding to each microlens <NUM> allows rays within the designed viewing window to propagate through the optics and reach the eyebox while blocking unwanted rays from reaching an adjacent microlens <NUM> or while blocking rays from neighboring elemental images to reach a microlens <NUM>. For instance, the black zone between the aperture A1 and A2 blocks the dashed rays originated from point P1 from reaching the MLA2 adjacent to the lenslet MLA1. These blocked rays are typically the main source of view cross-talk and ghost images observed in an InI display system. The distance from the microdisplay <NUM> to the aperture array <NUM> is denoted as ga and the diameter of aperture opening is denoted as pa, which may be constrained by <MAT> <MAT> Where ga-max and pa-max are the maximum allowable gap and aperture size, respectively, pei is the dimension of the elemental image, and pmla is the pitch of the MLA <NUM>.

One drawback in using an aperture array <NUM> with a fixed aperture size is that it can partially block rays for pixels located near the edge of each elemental images if the size of the elemental image changes. As illustrated in <FIG>, a small part of the rays from point P1 which are supposed to propagate through lenslet MLA1 are blocked by the black zone between aperture A1 and aperture A2, causing vignetting-like effects such that viewer may observe reduction of image brightness for points near the edge of each elemental images. <FIG> shows an alternative configuration to that of <FIG> in which the aperture array <NUM> is replaced by a programmable spatial light modulator (SLM) <NUM> so that the size and shape of each aperture can be dynamically adapted to avoid partially blocking desired rays. <FIG> shows another embodiment of a micro-InI unit in accordance with the present invention in which the microdisplay <NUM> and aperture array <NUM> are replaced by a display source <NUM> with controllable directional emissions, where the light emission direction can be controlled precisely so that the rays from each pixel will only reach their corresponding MLA lenslet <NUM>. <FIG> demonstrates one possible configuration of such display source <NUM> where a spatial light modulator <NUM> is inserted between a backlight source <NUM> with non-direction emission and non-self-emissive microdisplay <NUM>. The spatial light modulator <NUM> may be set to program and control the cone angle of the rays that illuminate the microdisplay <NUM> and reach the MLA <NUM>.

A conventional InI-based display system can typically suffer from a limited depth of field (DOF) due to the rapid degradation of spatial resolution as the depths of 3D reconstruction points shift away from that of the CDP. For instance, the 3D scene volume may need to be limited to less than <NUM> diopters in order to maintain a spatial resolution of <NUM> arc minutes or better in the visual space. In order to render a much larger 3D scene volume while maintaining a high spatial resolution, such as in the exemplary configuration of <FIG>, a relay group <NUM> with an electronically-controlled vari-focal element <NUM> sandwiched inside is inserted between the micro-InI <NUM> and the eyepiece <NUM>. Exemplary VFE's <NUM> include liquid lenses, liquid crystal lenses, deformable mirrors, or any other tunable optical technology, such as electrically tunable optical technology. By dynamically controlling the optical power, φR, of the relay group <NUM> by applying different voltages to the VFE <NUM>, the relay group <NUM> forms an intermediate image A'O'B' of the reconstructed miniature 3D scene created by the micro-InI <NUM>. The central depth position CDP of the relayed intermediate scene is tunable axially (along the optical axis) with respect to the eyepiece <NUM>. As a result, the depth volume of the magnified 3D virtual scene by the eyepiece <NUM> can be shifted axially from very close (e.g. <NUM> diopters) to very far (e.g. <NUM> diopter) while maintaining high lateral and longitudinal resolutions.

<FIG> schematically illustrates an exemplary configuration of the vari-focal relay group <NUM>, such as the relay group <NUM> of <FIG>, including a front lens group "Front Relay" <NUM> adjacent to the micro-InI unit <NUM>, VFE optics <NUM> located in the middle functioning as the system stop, and rear lens group "Rear Relay" <NUM> adjacent to the eyepiece <NUM>. The compound power, φR, of the relay group <NUM> is given by <MAT> Where φ<NUM>, φVFE, and φ<NUM> are the optical power of the front lens group <NUM>, VFE <NUM>, and the rear lens group <NUM>, respectively. t<NUM> and t<NUM> are the spaces between the front lens group <NUM> and VFE <NUM> and between the VFE <NUM> and the rear lens group <NUM>. z<NUM> is the axial distance between the front lens group and the 3D scene reconstructed by the micro-InI unit <NUM>. The axial position of the relayed intermediate scene is given by <MAT>.

The lateral magnification of the vari-focal relay system is given by <MAT>.

Assuming φe is the optical power of the eyepiece <NUM> and ZRCDP is the distance from the relayed CDP to the eyepiece <NUM>, the apparent CDP position of the reconstructed 3D virtual scene through the eyepiece <NUM> is given by <MAT>.

The lateral magnification of the entire system through the eyepiece <NUM> is given by <MAT>.

The field of view (FOV) of the entire system through the eyepiece <NUM> is given by, FOV = <MAT> Where t<NUM> is the spacing between the eyepiece <NUM> and rear relay lens <NUM>; zxp is the spacing between the exit pupil and the eyepiece <NUM>; h<NUM> is the image height of the reconstructed scene, and we further define uvfe = [(<NUM> - zxpφe) - (xxp + (<NUM> - zxpφe)t<NUM>)φ<NUM>] , and hvfe = [(<NUM> - zxpφe) - (zxp + (<NUM> - zxpφe)t<NUM>)φ<NUM>] - [(zxp + (<NUM> - zxpφe)t<NUM>)φ<NUM> + ((<NUM> - zxpφe) - (zxp + (<NUM> - zxpφe)t<NUM>)φ<NUM>)]t<NUM>.

When the VFE <NUM> is set to be an optical conjugate to the exit pupil of the eyepiece <NUM> (i.e. hvfe=<NUM>) where the entrance pupil of the eye is placed to view the display <NUM>, we have hvfe=<NUM> and the FOV is independent of the optical power of the VFE <NUM>. The equation in Eq. (<NUM>) is simplified into: <MAT>.

As illustrated in <FIG>, a preferred embodiment of the vari-focal relay group <NUM> is the placement of the VFE <NUM> at the back focal length of the front relay group <NUM> (i.e. t<NUM>=<NUM>/φ<NUM>) to make the VFE <NUM> an optical conjugate to the exit pupil of the eyepiece <NUM> (i.e. hvfe=<NUM>). With this preferred embodiment, the compound power, φR, of the relay group <NUM> given by Eq. (<NUM>) is simplified into: <MAT>.

The lateral magnification of the vari-focal relay system given by Eq. (<NUM>) is simplified into <MAT>.

And so does the lateral magnification of the entire system given by Eq. (<NUM>).

When t<NUM>=<NUM>/φ<NUM> and hvfe=<NUM>, the FOV of the system is further simplified into <MAT>.

As demonstrated by Eqs. (<NUM>) through (<NUM>), the careful position of the VFE <NUM> in the preferred manner ensures that the compound optical power of the relay group <NUM> is maintained constant, independent of the optical power of the VFE <NUM> due to constant chief ray directions owing to the property of object-space telecentricity. As further demonstrated by Eq. (<NUM>), the subtended field angle of the display through the eyepiece <NUM> is further maintained constant, independent of the optical power of the VFE <NUM>. Maintaining a constant optical power for the relay group <NUM> helps the virtually reconstructed 3D scene achieve constant field of view regardless of the focal depths of the CDP. Therefore a much larger volume of a 3D scene could be visually perceived without seams or artifacts in a gaze-contingent or time-multiplexing mode. It is worth noting that the lateral magnification of the relay group <NUM> given by Eq. (<NUM>) can be further maintained constant if t<NUM>=<NUM>/φ<NUM> is satisfied, which makes the vari-focal relay group <NUM> a double-telecentric system.

The eyepiece <NUM> in <FIG> can take many different forms. For instance, to achieve a compact optical design of an optical see-through HMD, a wedge-shaped freeform prism can be adopted, through which the 3D scene reconstructed by the micro-Inl unit <NUM> and relay group <NUM> is magnified and viewed. To enable see-through capability for AR systems, a freeform corrector lens with one of the surfaces coated with beamsplitter coating can be attached to the freeform prism eyepiece to correct the viewing axis deviation and undesirable aberrations introduced by the freeform prism to the real-world scene.

In another aspect of the present invention, part of the relay group <NUM> may be incorporated into the eyepiece optics <NUM>, such as freeform eyepiece, such that the tunable intermediate 3D scene is formed inside the freeform eyepiece. In such a context, the eyepiece may be a wedge-shaped freeform waveguide prism, for example. <FIG> schematically illustrates the concept of a freeform waveguide-like prism <NUM> formed by multiple freeform optical surfaces. The exit pupil is located where the use's eye is placed to view the magnified 3D scene. In the design, part of a traditional relay group <NUM> following the VFE <NUM> is incorporated into the prism <NUM> and fulfilled by the top portion <NUM> of the freeform waveguide prism <NUM> contained within the box labeled "Relay Group with VFE. " A light ray emitted from a 3D point (e.g. A) is first refracted by a closest optical element <NUM> of the relay group <NUM> and transmitted into the prism <NUM>, followed by a reflection by one or multiple freeform surfaces to create an intermediate image (e.g. A'). The axial position of the intermediate image (e.g. A') is tunable by the VFE <NUM>. Multiple consecutive reflections by the subsequent surfaces and a final refraction through the exit surface 855allow the ray reaching the exit pupil of the system. Multiple bundles of rays from different elemental images may exist, but do so apparently from the same object point, each of which bundles represents a different view of the object, impinging on different locations of the exit pupil. These ray bundles integrally reconstruct a virtual 3D point (e.g. "A") located in front of the eye. Rather than requiring multiple optical elements, the optical path is naturally folded within a multi-surface prism <NUM>, which helps reduce the overall volume and weight of the optics substantially when compared with designs using rotationally symmetric elements. Compared with a design using a traditional wedge-shaped <NUM>-surface prism, the waveguide-like eyepiece design incorporates part of the relay function, enabling a much more compact system than combining a standalone relay group <NUM> with a <NUM>-surface prism. Besides the advantage of compactness, the waveguide-like multi-fold eyepiece design offers a much more favorable form factor, because it enables the ability to fold the remaining relay group and micro-InI unit horizontally to the temple sides. The multiple folding not only yields a much more weight-balanced system, but also enables a substantially larger see-through FOV than using a wedge-shaped prism.

To enable see-through capability for AR systems, the bottom part <NUM> of the rear surface, marked as the eyepiece portion, of the prism <NUM> in <FIG> can be coated as a beamsplitting mirror, and a freeform corrector lens <NUM> including at least two freeform optical surfaces, may be attached to the rear surface of the prism <NUM> to correct the viewing axis deviation and undesirable aberrations introduced by the freeform prism <NUM> to the real-world scene. The see-through schematic layout is shown in <FIG>. The rays from the virtual light field are reflected by the rear surface of the prism <NUM> while the rays from a real-world scene are transmitted through the freeform corrector lens <NUM> and prism <NUM>. The front surface of the freeform corrector lens <NUM> matches the shape of the rear surface of the prism <NUM>. The back surface of the freeform corrector lens <NUM> may be optimized to minimize the shift and distortion introduced to the rays from a real-world scene when the lens is combined with the prism <NUM>. The additional corrector lens "compensator" does not noticeably increase the footprint and weight of the overall system.

In another aspect of the present invention, the bottom part <NUM> of the rear surface, marked as the eyepiece portion, of the prism <NUM> in <FIG> may be divided into two segments, the segment <NUM>-<NUM> and the segment <NUM>-<NUM>. As schematically illustrated in <FIG>, the segment of <NUM>-<NUM> may be a reflective or partial reflective surface which receives the light fields generated by the micro-InI unit. A beamsplitting mirror coating on the segment of <NUM>-<NUM> also allows the transmission of the light rays from a real-world scene. The segment <NUM>-<NUM> is a transmissive or semi-transmissive surface which only receives the light rays from a real-world scene, while it does not receive the light fields generated by the micro-InI unit <NUM>. <FIG> schematically illustrates a front view of the rear surface of the prism <NUM>. The two surface segments, <NUM>-<NUM> and <NUM>-<NUM>, intersect at an upper boundary of the aperture window required to receive the reconstructed 3D light fields by the micro-InI unit <NUM>, and they may be made by two separate freeform surfaces. The division of the bottom part of the rear surface <NUM> into two separate segments <NUM>-<NUM>, <NUM>-<NUM> with different light paths provides the ability to substantially enlarge the FOV of the see-through view beyond the FOV of the display path without being subject to the constraints of the virtual display path. As shown in <FIG>, a freeform corrector lens <NUM> may be attached to the rear surface of the prism <NUM> to correct the viewing axis deviation and undesirable aberrations introduced by the freeform prism <NUM> to the real-world scene. The rays from the virtual light field are reflected by the segment <NUM>-<NUM> of the rear surface of the prism <NUM> while the rays from a real-world scene are transmitted through both the segments <NUM>-<NUM> and <NUM>-<NUM> of the prism <NUM> and the freeform corrector lens <NUM>. The surface segment <NUM>-<NUM> may be optimized to minimize visual artifacts of see-through view when it is combined with the freeform corrector lens <NUM>. The front surface of the freeform corrector lens <NUM> matches the shape of the surface segments <NUM>-<NUM> and <NUM>-<NUM> of the prism <NUM>. The back surface of the freeform corrector lens <NUM> may be optimized to minimize the shift and distortion introduced to the rays from a real-world scene when the freeform corrector lens <NUM> is combined with the prism <NUM>.

In accordance with yet another aspect of the present invention, <FIG> schematically illustrates an optical design of a physical system <NUM> that embodies the conceptual system of <FIG>. <FIG> illustrates the 2D optical layout of the light field display path, and <FIG> shows the optical layout of the see-through path. The optical system <NUM> of the light field display includes a micro-InI unit, a relay group with VFE, and a freeform waveguide. A part of the relay group may be incorporated into the waveguide. The Micro-InI unit may include a microdisplay S0, a pinhole array S1, and a microlens array S2. The relay group may include four lenses, a commercially available VFE (Electrical Lens EL <NUM>-<NUM> by Optotune Inc. ), and two freeform surfaces (Surface S19 and S20). The freeform waveguide prism <NUM> may be formed by multiple freeform optical surfaces which are labeled as S19, S20, S21, and S22, respectively. In the design, part of a traditional relay group following the VFE may be incorporated into the prism <NUM> and fulfilled by the Surface S19 and S20. A light ray emitted from a 3D point (e.g. A) is first refracted by the surface S19 of the prism <NUM>, followed by a reflection by the surface S20 to create an intermediate image (e.g. A'). The axial position of the intermediate image (e.g. A') is tunable by the VFE. Two more consecutive reflections by the surfaces S21' and S22-<NUM> and a final refraction through the surface S21 allow the ray to reach the exit pupil of the system <NUM>. There exist multiple bundles of rays from different elemental images but apparently from the same object point, each of which represents a different view of the object, impinging on different locations of the exit pupil. These ray bundles integrally reconstruct a virtual 3D point located in front of the eye. The rays reflected by the Surface S21' of the waveguide are required to satisfy the condition of total internal reflection. The rear surfaces S22-<NUM>, S22-<NUM> of the prism <NUM> may be coated with a mirror coating for building an immersive HMD system which blocks the view of the real-world scene. Alternatively, the surface S22-<NUM> may be coated with a beamsplitting coating if optical see-through capability is desired using the auxiliary lens, as shown in <FIG>.

It should be noted that in the design disclosed hereby the Z-axis is along the viewing direction, the Y-axis is parallel to the horizontal direction aligning with interpupilary direction, and the X-axis is in the vertical direction aligning with the head orientation. As a result, the overall waveguide system is symmetric about the horizontal (YOZ) plane, and the optical surfaces (S19, S20, S21, and S22) are decentered along the horizontal Y-axis and rotated about the vertical X-axis. The optical path is folded in the horizontal YOZ plane. This arrangement allows the micro-InI unit and the vari-focal relay group to be mounted on the temple side of the user's head, resulting in a balanced and ergonomic system packaging.

Table <NUM> highlights some of the key performance specifications for the system <NUM> of <FIG>. The system <NUM> offers the ability to render the true 3D light field of a 3D scene which subtends a diagonal FOV of <NUM>° and achieves an optical resolution as high as <NUM> arc minutes per pixel in the visual space. Furthermore, the system <NUM> offers a large depth range, tunable from <NUM> to <NUM> diopters, with a high longitudinal resolution of about <NUM> diopters for a monocular display. Moreover, the system <NUM> achieves a high view density of about <NUM>/mm<NUM>, where the view density, σ, is defined as the number of unique views per unit area on the exit pupil, given by: <MAT> where N is the total number of views and AXP is the area of the exit pupil of the display system. A view density of <NUM>/mm<NUM> is equivalent to a viewing angle resolution of approximately <NUM> arc minute for objects at distance of <NUM> diopters. The exit pupil diameter for crosstalk-free viewing, also known as the eyebox of the display, is about <NUM>. In this embodiment, the exit pupil diameter is limited by the aperture size of the commercial VFE and it can be increased if another larger-aperture VFE is adopted. Finally, the system offers a large see-through FOV, greater than <NUM>° horizontally and <NUM>° vertically. The microdisplay utilized in our prototype is a <NUM>" organic light emitting display (OLED) with an <NUM> color pixel and pixel resolution of 1920x1080 (ECX335A by Sony). The optics design itself, however, is able to support OLED panels of different dimensions or other type of microdisplays such as liquid crystal displays that have a color pixel size greater than <NUM>.

An exemplary implementation of the system <NUM> of <FIG> is provided, Tables <NUM> through <NUM>, in form of the optical surface data. Table <NUM> summarizes the basic parameters of the display path (units: mm). Tables <NUM> through <NUM> provide the optimized coefficients defining the non-spherical optical surfaces.

A high resolution microdisplay with pixels as small as <NUM> is adopted to achieve a high resolution virtual reconstructed 3D image. To achieve such high-resolution imaging for the micro-InI unit, a microlens array (MLA) formed by aspherical surfaces may specifically be designed. Each of the aspherical surfaces of the MLA may be described as, <MAT> where z is the sag of the surface measured along the z-axis of a local x, y, z coordinate system, c is the vertex curvature, r is the radial distance, k is the conic constant, A through E are the 4th, 6th, 8th, 10th and 12th order deformation coefficients, respectively. The material of the MLA is PMMA. Table <NUM> provides the coefficients for the surfaces S1 and S2.

To enable enlarged see-through FOV, the freeform waveguide prism <NUM> may be formed by five freeform surfaces, labeled as surface S19, S20, S21/S21', S22-<NUM>, and S22-<NUM>, respectively. The freeform corrector lens may be formed by two freeform surfaces, in which the front surface shares the same surface specifications as the surfaces S22-<NUM> and S22-<NUM> of the waveguide prism <NUM> and the rear surface is denoted as surface S23. The surface segment of S22-<NUM> is a reflective or partial reflective surface which receives the light fields generated by the micro-InI unit. A beamsplitting mirror coating on the segment of S22-<NUM> also allows the transmission of the light rays from a real-world scene for see-through capability. The surface segment S22-<NUM> is a transmissive or semi-transmissive surface which only receives the light rays from a real-world scene, while it does not receive the light fields generated by the micro-InI unit.

The freeform surfaces, including S19, S20, S21/S21', S22-<NUM>, and S23 may be described mathematically as <MAT> where z is the sag of the free-form surface measured along the z-axis of a local x, y, z coordinate system, c is the vertex curvature (CUY), r is the radial distance, k is the conic constant, and Cj is the coefficient for xmyn. The material for both the waveguide prism and compensation lens is PMMA. Tables <NUM> through <NUM> provide the coefficients for the surfaces S19 through S21, S22-<NUM>, and S23, respectively, and Table <NUM> provides the surface references of each optical surface.

During the design process, the specifications for the Surface segment S22-<NUM> were obtained after the optimization of the light field display path through the prism <NUM> composed of the micro-InI unit, the relay lens group, and the surfaces S19. S20, S21/<NUM>', and S22-<NUM>. The required aperture dimensions of Surfaces S20 and S22-<NUM> were determined first for the light field display path. Then Surfaces S20, S21 and S22-<NUM> were imported into 3D modeling software such as Solidworks® from which the Surface S22-<NUM> was created. The shape of the Surface S22-<NUM> was created in the modeling software by satisfying the following requirements: (<NUM>) it intersects with Surface S22-<NUM> along or above the upper boundary line of the required aperture for surface S22-<NUM> defined by the display path; (<NUM>) along the intersection line between the surface S22-<NUM> and S22-<NUM>, the surface slopes at the intersection points on the surface S22-<NUM> approximately match, if not equal, with those corresponding points on the surface S22-<NUM> to ensure the two surfaces to appear to be nearly continuous, which minimizes visual artifacts to the see-through view when it is combined with a matching freeform corrector lens; (<NUM>) the Surface S22-<NUM> intersects with the surface S20 along or below the lower boundary line of the required aperture for surface S20, defined by the display path; and (<NUM>) the overall thickness between the surface S21 and S22-<NUM> is minimized. Finally, a freeform shape of the Surface S22-<NUM> is obtained in the 3D modeling software which is combined with the surfaces S19, S20, S21/<NUM>', and S22-<NUM> to create an enclosed freeform waveguide prism. <FIG> demonstrated a substantially enlarged see-through FOV through the method described above.

During the design process, three representative wavelengths, <NUM>, <NUM>, and <NUM> were selected which correspond to the peak emission spectra of the blue, green and red emitters within the selected OLED microdisplay. A total of <NUM> lenslets in the MLA were sampled with each representing <NUM> element image points, which added up a total of <NUM> field samples. To evaluate the image quality, an ideal lens with the same power as the eyepiece is placed at the exit pupil of the system (viewing window), which resulted in a cut-off frequency of <NUM> lp/mm for the final image, limited by the pixel size of the microdisplay. The optical performance of the designed system was assessed at representative field angles for the three design wavelengths. By changing the power of the tunable lens VFE, the central depth plane could be shifted axially in a large range, for example, from <NUM> to <NUM> diopters, without noticeable degeneration of optical performance. <FIG> plot the polychromatic modulation transfer function (MTF) for points reconstructed on the CDP set at the depth of <NUM>, <NUM>, and <NUM> diopters, respectively. For each CDP position, two sets of MTFs were plotted, one for fields corresponding to the on-axis MLA and one for fields correspond to the furthest MLA near the edge.

On the other hand, it is equally important to assess how the image quality of a 3D reconstruction point degrades when the reconstructed image is shifted away from the central depth plane for a specific tunable state. This can be evaluated by shifting the central depth plane a small amount of distance without changing the power of the tunable lens. <FIG> plot the polychromatic MTF for reconstructed points shifted away from the CDP by <NUM>, <NUM>, <NUM>, and <NUM> diopters, respectively. For each depth, two sets of MTFs were plotted, one for fields corresponding to the on-axis MLA and one for fields corresponding to the furthest MLA near the edge.

<FIG> plots the polychromatic MTF for the <NUM>°x40° FOV. Across the entire the FOV, the see-through path achieved an average MTF value of over <NUM>% at <NUM> cycles/degree frequency, corresponding to <NUM>/<NUM> normal vision, and nearly <NUM>% at <NUM> cycles/degree frequency, corresponding to <NUM>/<NUM> vision or <NUM> arc minute of visual acuity.

A prototype system ("InI-HMD prototype") was constructed of the InI-HMD <NUM> of <FIG> and Tables <NUM>-<NUM> and associated text.

In a further of its aspects, the present invention may provide methods for rendering light field images for an integral-imaging-based light field display. As one exemplary method, the flowchart of <FIG> illustrates rendering of a light field of a 3D virtual scene <NUM>, where the InI-HMD optics <NUM> creates a virtual central depth plane (CDP) <NUM> at a fixed depth (ZCDP) from the VIEWER measured in diopters, referred to as a fixed-depth mode light field display. The virtual CDP <NUM> is the optical conjugate plane of the microdisplay <NUM> in the visual space. Usually the highest contrast and resolution of the 3D light field could be reconstructed for 3D objects located at the depth of the CDP <NUM>.

To render the light field of a 3D target scene <NUM>, the exemplary fixed-depth mode method of the present invention may start with determining the depth of the virtual CDP <NUM> of the InI-HMD optics <NUM> with respect to the eye position of the VIEWER. A virtual camera array <NUM> composed of I by J pinhole cameras may then be simulated. Each of the virtual cameras in the array <NUM> may be positioned in the simulation in such a way that each location corresponds to the intersection of the chief ray direction of a corresponding lenslet of the microlens array (MLA) <NUM> with the exit pupil of the InI-HMD optics <NUM>, and each virtual camera's viewing axis matches the chief ray direction of the corresponding lenslet seen through the InI-HMD optics <NUM>. Corresponding to the simulated virtual camera array <NUM> is a simulated virtual camera sensor array <NUM> composed of I by J virtual sensors. Each of the virtual sensors may have a pixel resolution of K by L. The projection plane <NUM> of the virtual cameras is set to coincide with the depth of the virtual CDP <NUM> of the InI-HMD optics <NUM>, and the separation between the simulated virtual camera array <NUM> and the sensor array <NUM>, known as the camera equivalent focal length (EFL), f, is set such that the field of view (FOV) of each camera-sensor pair matches the FOV of each lenslet of the MLA <NUM>. A virtual 3D scene <NUM> may be computed using the simulated virtual camera array <NUM> as its reference. For the convenience of reference, hereafter the depths, Z, of 3D scene objects measured in diopters are referenced with respect to the VIEWER or equivalently to the simulated virtual camera array <NUM>. Each pair of the virtual cameras <NUM> and sensors <NUM> may correspond to a computed (rendered) 2D elemental image (EI) of the 3D light field of the 3D scene, representing a slightly different perspective of the 3D scene seen by the simulated virtual cameras <NUM>. These EIs may then be mosaicked to create a full-resolution light field image mosaic <NUM> of I*K by J*L pixels for the microdisplay <NUM>. (It should be noted that element <NUM>, <NUM>, <NUM>, <NUM> are non-physical elements that are computationally simulated to provide data to be delivered to the physical display <NUM>. ) The full-resolution image <NUM> may be displayed via the microdisplay <NUM> of the InI-HMD optics <NUM>. Through the InI-HMD optics <NUM>, a reconstructed virtual 3D scene <NUM> may be reconstructed for a VIEWER to view at the depth Z. For instance, in the present exemplary implementation, following the conventional rendering pipeline of 3D computer graphics (such as, <NPL>), an array of 15x9 elemental images of a 3D target scene <NUM> are simulated, each of which consists of 125x125 color pixels. These EIs may be mosaicked to create the full-resolution image of 1920x1080 pixels for the microdisplay <NUM>.

Using the InI-HMD prototype, a demonstration was performed by fixing the optical power of the tunable lens <NUM>, S10-S16 so that the CDP <NUM> of the display system <NUM>, <NUM> was set at a fixed distance of <NUM> diopter from the VIEWER, which simulates the display properties of a conventional InI-based HMD. (For purposes of the instant fixed-depth mode method a tunable lens is not required, and so its optical power was fixed. ) To demonstrate the optical performance of the light field optics <NUM> in a fixed-depth CDP mode, the virtual 3D target scene <NUM> having three depth planes located at <NUM>, <NUM> and <NUM> diopters away from the viewer or the exit pupil of the InI-HMD optics was created, <FIG>. On each depth plane three groups of Snellen letter E's with different spatial resolutions (<NUM>, <NUM>, and <NUM> arcmins for the individual strokes or gaps of the letters) and orientations (horizontal and vertical) as well as the depth indicators ('3D', '1D' and '<NUM>. 5D') were rendered. The images were rendered using the method described above in connection with <FIG>. <FIG> shows the exemplary mosaic <NUM> of 11x <NUM> EIs of the virtual 3D scene <NUM> generated for the microdisplay <NUM>, where the virtual CDP <NUM> was set at <NUM> diopter. For qualitative assessment of focus cues, three spoke resolution targets were physically placed at the corresponding depths of three depth planes of the virtual 3D scene <NUM>. A camera (not shown) with a <NUM>/<NUM>" color sensor of <NUM> by <NUM> pixels and a <NUM> lens was used in the place of the VIEWER. The camera system overall yielded a spatial resolution of <NUM> arcmin per pixel, which was substantially better than that of the display optics <NUM>. The entrance pupil diameter of camera lens was set to about <NUM> such that it is similar to that of the human eye. <FIG> shows the captured images of the reconstructed virtual 3D scene overlaying with the real-world targets where the camera was focusing on <NUM> diopter. It can be observed that only the targets, both the real (indicated by the arrow) and virtual (indicated by the box) ones, located at the same depth of the focus plane of the camera are correctly and clearly resolved, which suggests the ability of the InI-based HMD <NUM>, <NUM> to render correct focus cues to the VIEWER. The ability to resolve the smallest Snellen letters on the top row of the <NUM> diopter targets further suggests the spatial resolution of the prototype matches with the designed nominal resolution of <NUM> arcmins. In this configuration of fixed lens focus, it can be further observed that the EIs of the virtual targets at the depths (e.g. 3D and <NUM>. 5D) different from the focus plane of the camera do not converge properly, causing multiple copies of the letters being captured in <FIG>. These targets can properly converge when the camera focus is adjusted to focus on their corresponding depths, as demonstrated in <FIG>, which show the captured images of the same virtual and real-world scene with camera being focused at <NUM> and <NUM> diopters, respectively. The targets corresponding to the camera focus depth were marked by a box, respectively. However, alike a traditional InI-based HMD, the image contrast and resolution of the targets reconstructed at the depth plane other than the CDP can only maintain in a relatively short, limited DOF and degrade severely beyond that, even though the EIs of these targets converge correctly and located at the same depth as the focus plane of the camera. For instance, the captured images in <FIG> can still resolve the letters corresponding up to <NUM> arcmins while that in <FIG> can only resolve the letters corresponding to <NUM> arcmins and the EIs start to converge improperly.

With the assistance of tunable lens <NUM>, <NUM> (<FIG>, <FIG>) in accordance with the present invention, the depth of the CDP <NUM> can be dynamically adjusted. This capability allows the system <NUM> of the present invention to operate in two different modes: vari-depth mode (<FIG>, <FIG>) and time-multiplexed multi-depth mode (<FIG>, <FIG>). In the vari-depth mode, the depth of the CDP <NUM> may be adaptively varied according to the average depth of the displayed contents or the depth of interest. In multi-depth mode, the power of the tunable lens <NUM>, <NUM> may be rapidly switched among several states corresponding to several discrete CDP depths, while in synchronization the light field rendering is updated at the same speed such that the contents of different depths are time-multiplexed and viewed as an extended volume if the switching occurs at flickering-free rate.

The method for rendering the light field of 3D virtual scene in a vari-depth mode is illustrated in the flowchart of <FIG>. The vari-depth mode starts with determining the depth of interest, ZDOI, of a 3D target scene <NUM> measured in diopters, which can be either determined by the point of the interest of VIEWER or specified by a computer algorithm. The point of interest of the VIEWER can be determined by an eyetracking device if available in the HMD system or other user input devices such as a computer mouse. Alternatively, instead of relying upon an eyetracking device or other input devices, a computer algorithm can specify the depth of interest of the target scene based on the average depth of the virtual 3D scene obtained from a depth map associated therewith or based on feature points of the virtual 3D scene detected by image processing algorithms. Once the depth of interest (DOI) of the scene <NUM> is determined, a controller <NUM>, such as a PC, may apply an electrical control signal, V, to the VFE element <NUM> of the vari-focal relay group <NUM> which adaptively varies the distance, ZRCDP(V), between the relayed intermediate miniature 3D scene <NUM> and the eyepiece <NUM> of the InI-HMD optics <NUM> measured in diopters. Consequently, the depth, ZCDP(V), of the virtual CDP <NUM> of the InI-HMD optics <NUM>, which is measured in diopters, is adaptively set such that it coincides with the depth of interest of the target scene <NUM>. The simulated virtual camera array <NUM> and the virtual camera sensor array <NUM> are configured in a similar fashion to the fixed-depth one shown in <FIG> except that the camera projection plane <NUM> coincides with the depth of interest of the 3D scene <NUM>. The rest of the rendering method remains the same as that discussed in connection with <FIG>.

For the purpose of demonstrating the vari-depth mode, the optical power of the tunable lens <NUM> was varied so that the CDP <NUM> of the display optics <NUM> was set to the depth of <NUM> diopters. The virtual camera and virtual sensor arrays <NUM>, <NUM> were adapted to match the adjusted depth of the virtual CDP <NUM> of the display optics <NUM>. The EIs were then re-rendered for targets at <NUM> and <NUM> diopters with the camera projection plane adjusted to match the depth of <NUM> diopters. <FIG> show the captured images through the HMD with the camera (not shown) located at VIEWER focused at the depth of <NUM> and <NUM> diopters, respectively. By correctly adjusting the optical power of the tunable lens <NUM> as well as regenerating the contents on the microdisplay <NUM>, the system <NUM> was able to maintain the same level of the spatial resolution of <NUM> arcmins and image quality for the targets located at the depth of <NUM> diopters, <FIG>, as well as for the targets located at <NUM> diopter in <FIG>. The vari-depth mode, however, only achieves high-resolution display for targets near the specific depth dictated by the CDP of the display hardware. As shown in <FIG>, the targets at the depth of <NUM> diopters show more severely degraded resolution than in <FIG> due to its increased separation from the given CDP, even when the camera is focused at the depth of these <NUM>-diopter targets.

In still a further of its aspects, a multi-depth mode method in accordance with the present invention for rendering the light field of a 3D virtual scene <NUM> is illustrated in the flowchart of <FIG>. In the multi-depth mode, we started with selecting multiple depths of interest, ZDOI (n) (n=<NUM>. N), of a 3D target scene <NUM> distributed along the visual axis measured in diopters, where ZDOI (<NUM>) may define the closest depth plane <NUM>-<NUM> in diopters to the VIEWER and ZDOI (N) the furthest depth plane <NUM>-N. The placement of the multiple depths of interests may be constrained by multiple factors. The most important factors may be the angular resolution requirements, the depth of field requirements, the threshold tolerance to eye accommodation errors, and the longitudinal resolution requirements. Other factors that may affect the selection of the depths of interests include the depth range affordable by the vari-focal VFE <NUM> and the depth distribution of the 3D scene <NUM>. The total number of depth planes, N, may be constrained by the hardware design. For instance, in a time-multiplexed implementation where the different depths of interests are rendered in a time-sequential fashion, the update frame rates of the VFE <NUM>, the microdisplay <NUM>, and the graphics hardware, may be expressed as <MAT> where fc is the threshold refresh rate required for flickering-free view, fVFE is the maximum response speed of the VFE <NUM> to an electrical signal for optical power change, fdisplay is the maximum refresh rate of the microdisplay <NUM>, and fc is the maximum frame rate of the graphics rendering hardware. The number of depth planes can be increased if a spatial-multiplexing method can be implemented where the hardware can afford to render multiple depth planes concurrently. Once the placement and the number of the depths of interests are determined, the rest of the rendering method may be implemented as follows. For each of the selected depths of interests, ZDOI (n) (n=<NUM>. N), a controller <NUM> applies an electrical control signal, V(n), to the VFE element <NUM> of the vari-focal relay group <NUM>, which adaptively varies the distance, ZRIM(Vn), between the relayed intermediate miniature 3D scene <NUM> and the eyepiece <NUM> of the InI-HMD optics <NUM>. Consequently, the depth of the virtual CDP <NUM> of the InI-HMD optics <NUM>, ZCDP(Vn), is adaptively set such that it coincides with the given depths of interest, ZDOI (n) (n=<NUM>. The simulated virtual camera array <NUM> and the virtual camera sensor array <NUM> may be configured in a similar fashion to that described in <FIG> such that the camera projection plane <NUM> coincides with the depth of interest, ZDOI(n) (n=<NUM>. N) <NUM>-<NUM>, <NUM>-N, for example. To render the 2D elemental images of the 3D scene <NUM> for the given depth of interest, a depth map of the 3D virtual scene <NUM> is created to obtain depth information of the scene objects with respect to the VIEWER. Instead of rendering the 2D elemental images of the entire 3D scene <NUM>, we may only render the 2D elemental images located in the depth range defined by <MAT>.

Where ZDOI(n - <NUM>) - ZDOI(in) and ZDOI(n - <NUM>) - ZDOI(ni) define the dioptric spacings between the given depth of interests and its adjacent depth planes. When n=<NUM>, ZDOI(n - <NUM>) defines the nearest depth limit <NUM>-<NUM> to be rendered by the display1602, while when n=N, ZDOI(n + <NUM>) defines the furthest depth limit <NUM>-N to be rendered by the display <NUM>. The rendered 2D elemental images may be mosaiced together in the same way as in the fixed-depth or vari-depth modes to create the nth frame of full-resolution light field image which is then sent to the microdisplay <NUM> for update. The same rendering method may repeat for the next depth of interest until all of the N depth planes are rendered. As stated earlier, all of the N depth planes may be rendered in a time-sequential fashion or in a concurrent manner or a hybrid of the two methods.

To demonstrate the multi-depth mode of <FIG>, we decided to create an implementation of two time-multiplexed depth planes, one placed at <NUM> diopters and the other placed at <NUM> diopters. The optical power of the tunable lens VFE <NUM> was electrically controlled by two different signals V1 and V2 sequentially such that the virtual CDP <NUM> of the display system <NUM> was set to the depths of <NUM> and <NUM> diopters accordingly. At each of the two virtual CDP placements, we re-rendered the EIs for the target scene <NUM> which included two resolution targets placed at <NUM> and <NUM> diopters. For this simple case, the EIs rendered for the <NUM> diopter CDP placement only rendered the target object placed at <NUM> diopters and similarly the EIs rendered for the <NUM> diopter CDP placement only rendered the target object placed at <NUM> diopters. The separately-rendered EIs were displayed in a time-multiplexing fashion at a frame rate of about <NUM> while in synchronization the CDP <NUM> of the display <NUM> was rapidly switched between the depths of <NUM> and <NUM> diopters. The refresh speed of <NUM> was due to the limit of the highest <NUM> refresh rate of the OLED microdisplay <NUM>. <FIG> show the captured images through the HMD with the camera (not shown) placed at the location of the VIEWER and focused at the depths of <NUM> and <NUM> diopters, respectively. Along with the virtual display, two spoke resolution targets were physically placed at the corresponding depths of the letters. As shown in <FIG>, when the camera was focused at the near depth of <NUM> diopters, both of the virtual and real objects at the near depth (the letters and the spoke on the left) appears to be in sharp focus, while the far objects (the letters and the spoke on the right) show noticeable out-of-focus blurring as expected. <FIG> demonstrates the case when the camera focus was switched to the far depth of <NUM> diopters. It can be clearly observed that both of the letters at far and near depths are comparably sharp at the corresponding focus of the camera. By driving the display in a dual-depth mode, the system achieved high-resolution displays of targets with a large depth separation of nearly <NUM> diopters while rendering focus cues comparable to their real counterparts.

The vari-depth and multi-depth modes of the InI-based light field rendering methods of the present invention may share the feature that the depth of the CDP <NUM>, <NUM> is either adaptively varied according to the depth of interest in the vari-depth mode or is rapidly switched among several discrete depths in the multi-depth mode. However, their visual effects and implications on focus cues are noticeably different. For instance, as demonstrated in <FIG>, in the vari-depth mode of an InI-HMD (<FIG>), the contents away from the CDP <NUM> are rendered with correct blurring cues, though in potentially degraded resolution, due to the nature of light field rendering, while in a conventional vari-focal HMD the contents away from its focal plane can be as high resolution as the contents on the focal depth unless artificially blurred but do not show proper focus cues due to its 2D rendering nature. In the multi-depth mode (<FIG>), a significant advantage over the traditional multi-focal plane HMD approach is the requirement of much less number of depth switch to render correct focus cues in the same depth range, while depth blending is necessary in a multi-focal system to render focus cues for contents away from the physical focal planes. In the case of InI-based light field rendering, covering a depth range of <NUM> diopters only requires <NUM> focal depth and the focus cues generated in this case are also more accurate and continuous.

Claim 1:
A method for rendering light field images of a 3D scene in a head-mounted display, HMD, using an integral-imaging-based light field display, comprising:
providing integral imaging, Inl, optics including a microdisplay (<NUM>), the Inl optics having a central depth plane, CDP, (<NUM>) associated therewith, the Inl optics including a microlens array (<NUM>) of lenslets;
providing an eyepiece in optical communication with the InI optics, the eyepiece and the Inl optics together providing InI-HMD optics (<NUM>);
sampling the 3D scene using a simulated virtual array of cameras so that each camera captures a respective portion of the 3D scene to create a plurality of elemental images, the elemental images collectively comprising image data for display on the microdisplay (<NUM>), wherein the step of sampling the 3D scene comprises positioning each virtual camera such that each virtual camera location corresponds to the intersection of the chief ray of a corresponding lenslet of the microlens array (<NUM>) with an exit pupil of the InI-HMD optics (<NUM>) , wherein each simulated virtual camera's optical axis matches the chief ray direction of a corresponding lenslet seen through the InI-HMD optics; and displaying the image data on the microdisplay (<NUM>).