Systems and methods for lightfield reconstruction utilizing contribution regions

A method for rendering a view from a lightfield includes identifying a ray associated with a portion of the view and selecting a set of camera views from a plurality of camera views representing the lightfield based on an intersection point of the ray with a plane. Each camera view has an associated contribution region disposed on the plane. The associated contribution region overlaps contribution regions associated with other camera views of the set of camera views at the intersection point. The method also includes determining a characteristic of the ray based on a contribution factor for each camera view of the set of camera views. The contribution factor is determined based on the relative position of the intersection point within the associated contribution region.

BACKGROUND

Virtual reality (VR) and augmented reality (AR) applications often seek to enable a user to move throughout a scene (virtual or real-world) and enable the user to view the scene from the current pose of the user's head mounted device (HMD) or other VR/AR viewing device. Lightfields have been proposed as a light content format to enable rendering of imagery of a scene from many different views. However, a lightfield suitable for realistic depiction can utilize considerable amounts of processing resources, including a considerable number of processing cycles of a central processing unit (CPU) and a graphics processing unit (GPU). The amount of resources required for use of lightfields for VR and AR applications often is particularly problematic for mobile consumer devices due to their relatively limited storage, transmission, and processing resources.

DETAILED DESCRIPTION

The following description is intended to convey a thorough understanding of the present disclosure by providing a number of specific embodiments and details involving rendering views in a lightfield. It is understood, however, that the present disclosure is not limited to these specific embodiments and details, which are examples only, and the scope of the disclosure is accordingly intended to be limited only by the following claims and equivalents thereof. It is further understood that one possessing ordinary skill in the art, in light of known systems and methods, would appreciate the use of the disclosure for its intended purposes and benefits in any number of alternative embodiments, depending upon specific design and other needs.

A lightfield (also known as a lumigraph or photic field) may be conceptualized as the amount of light flowing in every direction through every point in a defined space or environment, wherein the direction of each ray in the lightfield is specified by a five-dimensional plenoptic function and the magnitude of each ray is specified by a corresponding radiance. A common approach to parameterizing a lightfield for computer-graphics implementations is via a two-plane parameterization, in which a lightfield is represented as a collection of perspective images of an st plane (often referred to as the “focal plane”), with each image representing the perspective of a virtual camera from a corresponding position on a uv plane (often referred to as the “camera plane”) that is parallel to the st plane.

As illustrated inFIG. 1, a view from the perspective of an observer126can be rendered utilizing arrays of images associated with a surface, such as a planar surface, e.g., a focal plane110. The view of the observer126can be constructed from rays (e.g., ray122and ray124) extending from the surface (e.g., focal plane110) to the observer126. While the surfaces (e.g., the focal plane110or the camera plane108described below) are illustrated as planes, the surfaces can have the form of other non-planar manifolds.

In the illustrated embodiment ofFIG. 1, the direction of the rays is expressed in terms of intersection points of the rays on two parallel planes (e.g., a camera plane108and the focal plane110). In an example, the camera plane108having the designated coordinates “u” and “v” and the focal plane110having the designated coordinates “s” and “t” are utilized to specify the direction of a ray extending between the focal plane110and the observer126, through the camera plane108.

The view of an observer126is displayed on a display device120. Portions of a view to be displayed on the display device120are rendered based on characteristics of a ray (e.g., ray122or ray124) extending between the focal plane110and the observer126, through the camera plane108and the modeled display device120. While a single display and observer are illustrated, more than one display device can be used. For example, two display devices, one for a left-eye view and one for a right-eye view, can be used and views or images can be rendered for each eye of the observer. A portion of the view can be a pixel or a set of pixels. In an example, the characteristic includes radiance. In a further example, the characteristic can include a color value or alpha.

The characteristics of the ray can be determined based on corresponding portions of camera views. A selection of a subset of cameras or camera positions can be associated with a ray based on an intersection of the ray with the camera plane108. Camera views associated with the subset of cameras or camera positions are selected based on the intersection of the ray with the focal plane110. In an example, a two-dimensional array of camera positions can be associated with the camera plane108. A two-dimensional array of camera views can be associated with a camera position from the plane108and camera view positions on the focal plane110. For example, a two-dimensional array of camera views in the focal plane110is uniquely associated with each camera position of the array of camera positions associated with the camera plane108.

While the focal plane110and camera plane108are illustrated as being planar, other non-planar manifolds can be used. For example, cameras in the camera plane108can be arranged on a sphere, with the cameras facing outward. A viewer can be anywhere inside of the sphere, and any view can be reconstructed from inside the sphere. In such an example, there are several ways to represent the focal plane onto which to project the lightfield images: the camera view images can be projected onto a sphere larger than the sphere of cameras, the camera view images can be projected onto a single plane that is in front of the viewer, or the camera view images can be projected onto planes in front of each camera or camera position, among others. The methods described herein allow for various non-planar manifolds on which a camera array is distributed or camera view images are projected, offering different user viewing experiences.

FIG. 2illustrates a typical two-plane representation of a lightfield in accordance with some embodiments. An example lightfield100is composed of a two-dimensional (2D) array102of images (e.g., images103,104,105), also referred to herein as camera views, wherein each image of the array102represents the rays arriving at a corresponding point (e.g., point106) on a camera plane108(having dimensions “u” and “v”) from all points on a focal plane110(having dimensions “s” and “t”). In such implementations, the images (camera views) of the array102represent off-axis, or sheared, perspective views of a scene or environment. Thus, the 2D array102may be represented as an array of images, with each image i having a position in the array defined by the coordinate (si, ti). Further, each image is represented by a 2D array of pixels. The array of pixels of an image may be conceived as an array of image tiles, with each image tile representing a corresponding pixel region. To illustrate with reference to image105, each image is composed of, for example, a 4×4 array of tiles (e.g., tiles114,115). While tiles114and115are illustrated as quadrilateral tiles, tiles can have a polygonal shape, for example, triangular. Each tile (e.g., tiles114,115) can be sent separate from the other tiles to central and graphical processing units for rendering.

Thus, as shown by lightfield100ofFIG. 2, a lightfield may be represented by an array of images or camera views of a scene, with each image representing a slightly planar-shifted perspective of the scene relative to the perspectives of the other images of the lightfield. Often, a lightfield contains a relatively large number of such images, and these images may be rendered or captured in relatively high resolution.

When rendering a view from a lightfield, portions of the view are rendered based on the characteristics of rays extending to the observer126from the camera plane108and focal plane110. The characteristics of each ray are determined based on a subset of images or camera views. The camera views are associated with camera positions and are selected based on an intersection of the ray with contribution regions associated with camera positions on the camera plane108and based on an intersection of the ray with the focal plane110.

In an embodiment, a subset of camera positions and associated camera views can be selected based on a position of the intersection of the ray with contribution regions associated with each camera position in the camera plane108. In an alternative embodiment, contribution regions can be associated with each camera view in the focal plane110. The characteristics of the ray (e.g., radiance, color value, or alpha) can be determined based on weighted contributions of camera views associated with each of the subset of camera positions having contribution regions overlapping the intersection of the ray within the camera plane108. For example, the characteristics of the ray can be determined based on a sum of weighted characteristics derived from the camera views associated with the selected camera positions. Weights, also referred to as contribution factors, associated with the camera views can be determined based on the relative position of the ray intersection within the contribution region. Optionally, the weights (contribution factors) associated with the contributions can be normalized, for example, using a sum of contribution factors of the selected camera views.

For example, as illustrated inFIG. 3, camera positions within an array of camera positions associated with the camera plane108can have associated contribution regions302or304. While the contribution regions302or304are illustrated as being circular disks, the contribution regions can have other shapes. For example, the contributions can be circular, elliptical, quadrilateral, or polygonal disks.

As illustrated, a ray intersects the camera plane at a position (e.g., position306or position308). Camera positions having contribution regions overlapping the intersection position are selected to provide a subset of camera positions. The subset of camera positions is used to select camera views that are used to determine characteristics of the ray. For example, a ray intersecting at position306can result in the selection of camera positions C0, C1, C2, and C4based on the contribution regions overlapping with the position306of the intersection of the ray. In another example, camera positions C0, C4, C6, and C7are selected when a ray intersects the camera plane at position308.

Camera views can be selected based on the subset of camera positions and the projection of the ray onto the focal plane. Contributions from the camera views can be weighted based on the relative position of the ray intersection within the contribution region. For example, the weight or contribution factor of the contribution can vary with radial distance from the camera position within the contribution region. In an example, the contribution factor can vary from 1 to 0 in a smooth manner with increasing distance from the camera position. For example, a contribution can decrease linearly with increasing distance from the camera position. In another example, a contribution can decrease in a Gaussian relationship with radial distance from the camera position.

To further illustrate the rendering of a view using selected camera views, a method400illustrated inFIG. 4includes determining a point of view of an observer, as illustrated at block402. Based on the position and orientation of an observer, rays extending into the lightfield are utilized to render a view to be displayed to the observer and can be generated, for example, utilizing the four-dimensional coordinates disposed on two parallel planes, such as a camera plane and a focal plane.

As illustrated at block404, to render a display image or view, a ray can be selected. The ray extends to the observer from the focal plane through the camera plane and through a display plane, representing the display device. The ray is associated with a portion of a view or image to be displayed. The portion of the view can be a pixel or can be a set of pixels. Characteristics of the ray can be utilized to reconstruct a portion of the display, such as specifying characteristics of light to be displayed by the pixel.

As illustrated at block406, a set of camera positions of an array of camera positions can be identified. A select ray intersects with the camera plane. Camera positions having contribution regions overlapping with point of intersection of the ray within the camera plane can be selected. Contribution regions can be used to determine both which camera positions to associate with a ray and to determine a contribution factor to associate with camera views associated with the selected camera positions.

Based on the selected camera positions and the intersection of the ray with the focal plane, a set of camera views can be selected, as illustrated at block408. Portions of the camera views associated with the portions of the view intersecting the ray can be utilized to generate characteristics of the ray, which are used in determining the portion of the view or displayed image.

As illustrated at410, a characteristic of the ray can be determined using the set of camera views. In an example, the characteristic can include radiance, color value, or alpha. In particular, a portion is selected from each camera view that corresponds with the portion of the view or image associated with the ray, and the characteristic of the portion of the camera view is used when determining the characteristic of the ray. For example, a normalized weighted sum of characteristics of the portion of the selected camera views can be utilized to specify the characteristic of the ray.

The weight or contribution factor of the characteristic from each camera view can be determined based on a relative distance of a camera position associated with the camera view to the intersection point of the ray within the contribution region. Alternatively, the contribution factor can be determined based on a relative distance of a ray intersection with the focal plane and a position of the camera view within the focal plane. Each of the contribution factors can be normalized based on a sum of the contribution factors. For example, the contribution of each camera view to the characteristic of the ray is the product of the contribution factor associated with the camera view and a value of the characteristic of the portion of the camera view. The contribution of each selected camera view is added and optionally normalized based on the sum of contribution factors of the selected camera views.

In other words, the characteristic of the ray can be the sum of contributions from each camera view. The contribution from each camera view is determined by multiplying the contribution factor by the characteristic value of the camera view and dividing the product by a sum of each of the contribution factors to normalize the contributions. Alternatively, the products of each contribution factor and camera view value can be summed and the sum divided by the sum of each of the contribution factors for normalization.

As illustrated at block412, the portion of the display image can be rendered based on the characteristic of the ray. For example, the radiance values associated with the characteristic of the ray can be applied to the portion of the display image. In a particular example, a pixel value can be set based on the characteristics of the ray.

The process of selecting a ray can be repeated for each portion of the display image. Rendered portions of the view or image can be displayed on a display device, as illustrated at block414, and can represent a point of view of an observer.

Such an approach provides technical advantages, including utilizing less processor resources and accelerating the rendering of views or images to be displayed to an observer. For example, visibility computation and the render shaders are simpler and generally faster than conventional methods. The reconstruction filter is flexible and the method allows for a trade between edge doubling with depth of field aperture blur by altering the size of the contribution regions. The size and shape of the contribution region can be the size and shape of the bokeh. Accordingly, the contribution region an imitate a camera aperture.

Further, methods using contribution regions provide for improved adaptive lightfield level of detail. For example, the method allows for hierarchical level of detail rendering. For example, camera positions or associated contributions regions can be organized in a hierarchy. Example hierarchies include a kd-tree, octree, or bounding volume hierarchy (BVH), among others. In at least one embodiment, the images of a two-dimensional array of images representing a lightfield are organized into a binary tree structure.

The use of a hierarchy of contribution regions to identify camera views and determine the characteristics of a ray provides further advantages when adapting the level of detail. For example, the level of detail can be progressively enhanced while budget exists within the processing resources to render progressively more detailed views or images for display. In another example, an increasingly detailed view can be calculated as an observer is positioned closer to a scene. In a further example, progressively increasing levels of detail can be determined until a criteria is met.

Conceptually, progressively greater detail can be provided by increasing camera density and the density or proximity of camera views or by utilizing increasingly higher resolution camera views, or both. For example, as illustrated inFIG. 5, camera density can be selectively changed by selecting subsets of camera positions. In an initial level of detail, camera positions502can be selected and camera views from the camera positions502can be utilized in determining the characteristic of the ray or rays. To increase the level of detail, additional cameras or camera positions can be added. For example, an intermediate level of detail can be generated using camera views from both camera positions502and504. Optionally, at the intermediate level of detail, middle resolution images can be used for each of the camera views generated from camera positions502or504. A further level of detail can be improved utilizing a greater density of cameras or camera positions, for example, utilizing the camera positions502,504, and506to select camera views useful in generating the characteristics of the rays. Optionally, when a higher number or density of cameras or camera positions are selected, higher resolution camera views can also be selected.

As described above, increasing camera density can be further augmented using progressively higher resolutions of camera views. For example, when generating characteristic of rays using camera positions502, low resolution camera views can also be used. When utilizing an increased camera density, such as when using camera positions502and504, a midrange resolution can be utilized for the camera views. In a further example, when a high-density configuration utilizing camera positions502,504, and506are used, higher resolution images can be utilized for the camera views. In an example, the sides of each image of higher resolution can be approximately twice the size of the preceding lower resolution image.

Such an approach can advantageously be used to provide improved image quality when permitted by a processor utilization budget or can be used to generate lower resolution images at an apparent distance and higher resolution images when the observer is in apparent close proximity.

The further illustrate the utilization of a camera hierarchy,FIG. 6illustrates a camera array600in which camera positions602are selected based on a selected level within a hierarchy. The camera positions602and camera views associated with camera positions602can be selected based on the intersection of a ray with the contribution region (e.g., contribution region604) and the contribution factors of the associated camera views can be derived from a relative position of the intersection point of the ray with the camera plane and within the contribution region604.

As illustrated, when selecting a subset or lower density arrangement of camera views or cameras602, the contribution region604can be relatively large overlapping other camera positions606within the array that are not part of the selected level of the hierarchy. Optionally, low resolution camera views can be associated with the camera positions602when determining ray characteristics at the low-density level of the hierarchy.

A higher level of detail can be achieved using an increased density of camera positions and optionally higher resolution camera views associated with those camera positions. For example, as illustrated inFIG. 7, a camera array700of camera positions702having a greater camera density than the selected camera positions602of the camera array600can be selected. A size of the contribution region704can be reduced to accommodate the higher density of camera positions selected for the array700. Optionally, higher resolution images can be used as camera views to further enhance the level of detail to be rendered. As illustrated, the contribution region704is smaller than the contribution region604.

For example, a method800illustrated inFIG. 8can include selecting a detail level, as illustrated at block802. Based on the selected detail level, a subarray of camera positions of the camera array can be selected, as illustrated at block804. The subarray can include a lower density set of camera positions or can include less than all of the camera positions.

As illustrated at block806, a size of the contribution regions associated with each camera position on the subarray can be set based on the density of the subarray. For example, the lower-density subarray of camera positions have contribution regions of greater size.

As illustrated at block808, a view or display image can be rendered. For example, the view or display image can be rendered by determining rays associated with portions of the display image, selecting cameras or camera positions based on contribution regions intersecting with the selected rays, and determining characteristics of the rays based on weighted contributions from camera views associated with the selected camera positions. The ray characteristics can be utilized to render portions of the view or display image. Optionally, the view can be displayed on a display device, as illustrated at block812.

Alternatively, an iterative process can be utilized to determine whether a criteria is met or whether a budget, such as a processing budget, has been exhausted, as illustrated block810. For example, if a low-resolution view is rendered, the system can determine whether additional processing budget is available and can utilize additional processing budget to improve the level of detail of a view or image. For example, a more detailed level can be selected, as illustrated at block802, a greater density camera subarray can be selected, as illustrated at block804, smaller contribution regions can be set for the greater density subarray, as illustrated at block806, and a more detailed display image or view can be rendered based on the increased density of the camera subarray and the associated camera views.

Other criteria can, for example, include relative distance of an observer from the camera plane or camera view plane, sharpness of an image, or functions that account for movement of the observer. As images or views having desirable levels of detail are rendered, the views or images can be displayed on the display device, as illustrated at block812.

The above methods further advantageously allow for a more efficient generation of views having a change in focus. In an example, the focus of a view can be altered by changing the relative position of the focal plane110and the camera plane108. For example, as illustrated inFIG. 9andFIG. 10, focal positions of a view can be altered by moving the position of the focal plane110relative to the camera plane108to increase the distance or decrease the distance. For example, if a focal plan110is moved from position A to position B, the intersection of the ray with the camera plane108does not change. However, the position of the ray intersection with the focal plane110changes. For example, moving the focal plane110from position A to position B (FIG. 9) translates the intersection points in the focal plane110from points A to points B (seeFIG. 10), which results in a different selection of camera views that contribute to the characteristics of the ray. When the contribution region resides in the camera plane, adjusting the relative distance of the focal plane maintains the selection of camera positions and contribution factors. The camera views are reselected. Alternatively, when contribution factors are based on the position of an intersection with a contribution region in the focal plane110, the change in relative position of the focal plane110can result in different contribution factors associated with the camera views, while the selected camera positions remain constant.

For example, a method1100illustrated inFIG. 11includes repositioning a focal plane110, as illustrated at block1102. Based on the change in position of the intersection of a ray, a different subset of camera views is selected, as illustrated at block1104. When the contribution regions are associated with the camera plane, selected camera positions and associated contribution factors remain constant, reducing the processing burden for changing a focus.

As illustrated at block1106, ray characteristics can be determined using the revised subset of camera views. In particular, ray characteristics can be determined using a weighted combination of characteristics of portions of the revised subset of camera views. For example, a weighted sum of characteristics of the portions of the camera views can be used to determine the ray characteristics. The contribution factors can be determined based on the position of an intersection of the ray with a contribution region. In an example, the contribution factors are normalized based on a sum of contribution factors. In a particular example, the contribution factors are determined based on the position of the intersection of the ray with a contribution region in the camera plane. Alternatively, the contribution factors are determined based on the position of the intersection of the ray with a contribution region in the focal plane.

The process of selecting subsets of camera views based on the altered position of the ray intersection can be repeated to generate portions of a display image or view, as illustrated at block1108. The display image or view rendered by the process can be displayed on a display device, as illustrated at block1112.

In another example, the field of view and bokeh can be influenced by the size and shape of the contribution region. In particular, the contribution region can imitate camera aperture.

The above methods can be implemented in a system or systems, optionally utilizing a network.FIG. 12illustrates an example display system1200. The display system1200includes a lightfield generation device1202, a lightfield compression component1206, a lightfield decompression/rendering component1208, and a display device1210.

In an example, the lightfield generation device1202generates or obtains a lightfield1212representing a scene1213. To illustrate, the lightfield generation device1202can comprise, for example, a lightfield camera configured to capture the lightfield1212, such as via a multiple camera rig configured to capture multiple images of the scene1213from different perspectives on a camera plane, with the resulting multiple images arranged in an 2D image array format, such as that illustrated by 2D image array102ofFIG. 2. Alternatively, the lightfield generation device1202may comprise a graphics rendering device configured to generate the lightfield1212of a VR or AR implementation of the scene1213by rendering the images of the image array of the lightfield1212using lightfield rendering techniques, such as the contribution region technique described above. Whether generated through image capture or graphics rendering, the lightfield1212is buffered or otherwise stored in the storage device1204(for example, random access memory, a disc drive, etc.) for further processing.

The lightfield compression component1206can operate to compress the lightfield1212to generate a compressed lightfield1214represented by less data than the lightfield1212, and thus better suited for efficient storage, transmission and processing.

In an example, the compressed lightfield1214is provided to the lightfield decompression/rendering component1208for further processing, whereby the compressed lightfield1214is provided by storing the compressed lightfield1214in a storage device1216accessible by both components1206,1208, by transmitting a representation of the compressed lightfield1214from the lightfield compression component1206to the lightfield decompression/rendering component1208via one or more networks1218or other data communication interconnects (e.g., data transmission cables), or the combination thereof.

To illustrate, the components1206,1208may be implemented as components of a larger device, and the storage device1216may comprise system memory or a hard disc of the larger device. As another example, the components1206,1208may be remote to each other, and the compressed lightfield1214is provided from the component1206to the component1208via a server located on the network1218.

In a further example, the lightfield decompression/rendering component208can operate to identify a view of the scene1213to be displayed at the display device1210, and from this identified view can identify which tiles of which images of the compressed lightfield1214are to be used to render imagery representative of the view (that is, which image tiles are “visible” in the identified view). The lightfield decompression/rendering component1208can access the identified image tiles from the compressed lightfield1214and decompresses the tiles to generate decompressed image tiles. From the decompressed image tiles, the lightfield decompression/rendering component1208can render one or more display images (rendered imagery1220), which are provided to the display device1210for display.

To illustrate, in some embodiments the display device1210comprises a head mounted display (HMD) device, and the view of the scene1213to be rendered is based on the current pose of the HMD device relative to the coordinate frame of reference of the scene1213. With this pose identified, the lightfield decompression/rendering component1208identifies the tiles of the compressed lightfield1214that represent imagery visible from the given pose, decompresses the identified tiles, renders a left-eye image and a right-eye image from the decompressed tiles, and provides the left-eye image and right-eye image to the HMD device for concurrent display so as to provide a user of the HMD a 3D view of the scene1213.

In some implementations, the lightfield decompression/rendering component1208is implemented on a mobile device or other device with relatively limited processing, storage, or transmission resources. Accordingly, to facilitate efficient processing of the compressed lightfield1214for use in generating the rendered imagery1220, the lightfield compression component1206utilizes one or more lightfield compression techniques, such as disparity predicted replacement (DPR) compression technique1222, to reduce the amount of data required to represent the compressed lightfield1214as provided to the lightfield decompression/rendering component1208.

The lightfield compression component1206may utilize one or more other compression techniques in addition to the DPR compression technique1222, and the lightfield decompression/rendering component1208may utilize one or more complementary decompression techniques in addition to the DPR compression technique1222. As an example, the lightfield compression component1206can employ a disparity compensated prediction (DCP) compression process1226and the lightfield decompression/rendering component1208can employ a corresponding DCP decompression process1228.

FIG. 13illustrates an example hardware implementation of the lightfield compression component1206in accordance with at least some embodiments. In the depicted example, the lightfield compression component1206includes an application processor1302having an interface coupled to a non-transitory computer-readable storage medium1304, an interface coupled to the storage device1204, and interfaces coupled to one or both of the network1218and the storage device1216. The application processor1302may comprise one or more central processing units (CPUs), graphics processing units (GPUs), or combinations thereof. The computer-readable storage medium1304includes, for example, one or more random access memories (RAMs), one or more read only memories (ROMs), one or more hard disc drives, one or more optical disc drives, one or more Flash drives, and the like.

The computer-readable storage medium1304stores software in the form of executable instructions configured to manipulate the application processor1302to perform one or more of the processes described herein. To illustrate, this software may include, for example, a DCP module1310to perform DCP encoding processes, a tree operations module1312to perform the binary tree generation and traversal processes, a motion search module1314to calculate the DDV for each reference tile on the basis of one or more motion search processes, and a tile replacement module1316to perform the selective tile elimination processes.

FIG. 14illustrates an example hardware implementation of the lightfield decompression/rendering component1208in accordance with at least some embodiments. In the depicted example, the lightfield decompression/rendering component1208includes a CPU1402having an interface coupled to a non-transitory computer-readable storage medium1404and an interface coupled to an inertial measurement unit (IMU)1406, and interfaces coupled to one or both of the network1218and the storage device1216(not shown inFIG. 14). The lightfield decompression/rendering component1208further includes a GPU1408having an interface coupled to the CPU1402, an interface coupled to a non-transitory computer-readable storage medium1410, an interface coupled to a frame buffer1412, and an interface coupled to a display controller1414. The display controller in turn is coupled to one or more display panels, such as a left-eye display panel1416and a right-eye display panel1418of a HMD device.

In at least one embodiment, the workloads of the lightfield decompression process and rendering process are split between the CPU1402and the GPU1408. To illustrate, the computer-readable storage medium1404stores software in the form of executable instructions configured to manipulate the CPU1402identify and DCP-decode (on demand) the lightfield tiles to render imagery representative of a particular view of a scene, while the computer-readable storage medium1410stores software in the form of executable instructions configured to manipulate the GPU1408to reconstruct any eliminated tiles needed to render the imagery, and then render the imagery using the accessed and reconstructed tiles. To illustrate, the CPU-side software can include a view determination module1420to determine the pose of the HMD device1210(FIG. 12) via the IMU1406and from the pose determine the current view into the scene1213, a visibility analysis module1422to identify a set image tiles of the compressed lightfield1214that are “visible” from the current view or otherwise are to render imagery of the scene1213from the current view as described above, and a tile decoder module1424to access those tiles of the set that are present in the compressed lightfield1214, DCP decode them, and provide the present tiles (e.g., tile1426) to the GPU1408.

In an example implementation of the contribution region methods, the lightfield tiles can be sent to the GPU and the disk geometry can be sent as shader variables to be used to determine per-fragment weight or contribution factor. A selection of which lightfield tiles to render can be made by the CPU. More specifically, efficient GPU implementation can be accomplished by passing the tiles quads as the geometry to the GPU. The position and shape of the disk is passed as shader arguments. Then projective texturing is used to texture the tile. An intercept of the ray from the eye to the shade point on the tile with the disk or contribution region is used to determine the weight for that sample.

EXAMPLE

A method of rendering images using contribution regions is compared to a method using a triangular mesh with barycentric interpolation. For purposes of the test, the same lightfield including camera positions and camera views is used when implementing both methods.

Table 1 illustrates the render times per frame for each method on different devices.

TABLE 1Render TimesRender Times (ms)DeviceContribution RegionsMeshHTC1043.21358.1638Nexus 6P108.724157.245Workstation19.0467 (15.2*)16.1767*adjusting the contribution region radius provides better performance will maintaining full coverage of the contribution regions.

As illustrated in TABLE 1, considerable performance gains are found when using the contribution regions method, particularly when implemented on mobile devices.