Finite aperture omni-directional stereo light transport

In various embodiments, a finite aperture omni-directional camera is modeled by aligning a finite aperture lens and focal point with the omni-directional part of the projection. For example, each point on an image plane maps to a direction in camera space. For a spherical projection, the lens can be orientated along this direction and the focal point is picked along this direction at focal distance from the lens. For a cylindrical projection, the lens can be oriented along the projected direction on the two dimensional (2D) xz-plane, as the projection is not omni-directional in the y direction. The focal point is picked along the (unprojected) direction so its projection on the xz-plane is at focal distance from the lens. The final outgoing ray can be constructed by sampling of point on this oriented lens and shooting a ray from there through the focal point.

BACKGROUND

Omni-directional virtual reality (VR) applications such as used by youtube's 360 video format require omni-directional stereo rendering. However, one disadvantage associated with conventional omni-directional stereo rendering is that it lacks depth of field effects where certain things are in focus and certain things are out of focus depending on how far they are from the focal distance. It is pointed out that depth of field effects can be another important depth cue besides stereo vision.

SUMMARY

Various embodiments in accordance with the present disclosure can at least address the disadvantage described above that is associated with conventional omni-directional stereo rendering.

In various embodiments, a finite aperture omni-directional camera is modeled by aligning a finite aperture lens and focal point with the omni-directional part of the projection. For example, each point on an image plane maps to a direction in camera space. For a spherical projection, the lens can be orientated along this direction and the focal point is picked along this direction at focal distance from the lens. For a cylindrical projection, the lens can be oriented along the projected direction on the two dimensional (2D) xz-plane, as the projection is not omni-directional in the y direction. The focal point is picked along the (unprojected) direction so its projection on the xz-plane is at focal distance from the lens. The final outgoing ray can be constructed by sampling of point on this oriented lens and shooting a ray from there through the focal point.

In various embodiments, the above described finite aperture lens model can be combined with the stereo omni-directional rendering process. Therefore, in various embodiments, this Finite Aperture Omni-directional Stereo Camera can be then used in light transport rendering algorithms to render photorealistic omni-directional stereo images. Light transport rendering algorithms such as bidirectional path tracing and photon mapping involve mapping points on an image plane to primary rays in camera space in the inverse of mapping scene points back to image plane coordinates. For a given stereo separation distance, finite aperture lens coordinates and image plane point, the method described above gives the outgoing camera ray. Note that various embodiments can involve an inverse method of mapping one or more camera space scene points back to image coordinates for fixed stereo separation distance and finite aperture lens coordinates.

In various embodiments, the present disclosure can include a method of generating an image. The method can include accessing a first data model of an environment captured by a virtual reality camera system, the data model representing an image backplate. In addition, the method can include accessing a second data model of objects within a virtualized three dimensional (3D) space, wherein the image backplate is within the space. The method can also include simulating a presence of at least one light source, or camera, within the space and casting a ray from the light source, or the camera into the space, to a point, p, in the space, possibly after interacting with objects in the second data model, using ray tracing techniques. Furthermore, the method can include simulating a finite aperture with respect to a camera capturing a two dimensional (2D) projection of a 3D scene by projecting a ray from the point p, through a focal point of the camera, to a lens point l within the finite aperture of the camera. The method can also include, provided the point p lies within the backplate and is not occluded from a camera performing the following: based on the point p, performing inverse mapping to obtain a orientation in camera space corresponding to the point p and the lens point l, wherein the orientation is defined by two values; based on the orientation, performing inverse mapping to obtain a pixel, s, on a capture plane of the camera corresponding to the point p and the lens point l; and using the pixel s, looking up a color value within the first data model corresponding to pixel s, and using this color value to compute a ray tracing value at the point p. Additionally, the method can include, provided the point p lies within an object of the objects and is not occluded from a camera, performing the following: based on the point p, performing inverse mapping to obtain a orientation in camera space corresponding to the point p and the lens point l, wherein the orientation is defined by two values; based on the orientation, performing inverse mapping to obtain a pixel, s, on a capture plane of the camera corresponding to the point p and the lens point l; and at the pixel s of the capture plane, rendering energy contribution from the point p.

In various embodiments, the present disclosure can include a method of generating an image. The method can include a) accessing a first data model of an environment captured by a virtual reality camera system, the data model representing an image backplate. Furthermore, the method can include b) accessing a second data model of objects within a virtualized three dimensional (3D) space, wherein the image backplate is within the space. Moreover, the method can include c) provided a given pixel, s, perform a mapping to a camera orientation, wherein the orientation is defined by two values. Additionally, the method can include d) provided a lens point, l, generating a camera ray through a focal point f, of the oriented camera. In addition, the method can include e) computing incoming energy along the ray using ray tracing techniques. The method can also include f) rendering energy contribution at the pixel s of a capture plane. The method can also include g) repeating the c)-f) for a second lens position l for each pixel s.

In various embodiments, the present disclosure can include a computer system including a processor and a memory coupled to the processor. The memory includes instructions for implementing a method of generating an image. The memory includes accessing a first data model of an environment captured by a virtual reality camera system, the data model representing an image backplate. Moreover, the method can include accessing a second data model of objects within a virtualized three dimensional (3D) space, wherein the image backplate is within the space. Additionally, the method can include simulating a presence of at least one light source, or camera, within the space and casting a ray from the light source, or the camera into the space, to a point, p, in the space, possibly after interacting with objects in the second data model, using ray tracing techniques. The method can also include simulating a finite aperture with respect to a camera capturing a 2D projection of a 3D scene by projecting a ray from the point p, through a focal point of the camera, to a lens point l within the finite aperture of the camera. Furthermore, the method can include, provided the point p lies within the backplate and is not occluded from a camera performing the following: based on the point p, performing inverse mapping to obtain a orientation in camera space corresponding to the point p and the lens point l, wherein the orientation is defined by two values; based on the orientation, performing inverse mapping to obtain a pixel, s, on a capture plane of the camera corresponding to the point p and the lens point l; and using the pixel s, looking up a color value within the first data model corresponding to pixel s, and using this color value to compute a ray tracing value at the point p. In addition, the method can include provided the point p lies within an object of the objects and is not occluded from a camera, performing the following: based on the point p, performing inverse mapping to obtain a orientation in camera space corresponding to the point p and the lens point l, wherein the orientation is defined by two values; based on the orientation, performing inverse mapping to obtain a pixel, s, on a capture plane of the camera corresponding to the point p and the lens point l; and at the pixel s of the capture plane, rendering energy contribution from the point p.

In various embodiments, the present disclosure can include a method of generating an image. The method can include accessing a first data model of an environment captured by a virtual reality camera system, the data model representing an image backplate. In addition, the method can include accessing a second data model of objects within a virtualized three dimensional (3D) space, wherein the image backplate is within the space. The method can also include simulating a presence of at least one light source, or camera, within the space and casting a ray from the light source, or the camera into the space, to a point, p, in the space, possibly after interacting with objects in the second data model, using ray tracing techniques. Furthermore, the method can include simulating a finite aperture with respect to a camera capturing a two dimensional (2D) projection of a 3D scene by projecting a ray from the point p, through a focal point of the camera, to a lens point l within the finite aperture of the camera. The method can also include, provided the point p lies within the backplate and is not occluded from a camera performing the following: based on the point p, performing inverse mapping to obtain a orientation in camera space corresponding to the point p and the lens point l, wherein the orientation is defined by two values; based on the orientation, performing inverse mapping to obtain a pixel, s, on a capture plane of the camera corresponding to the point p and the lens point l; and using the pixel s, looking up a color value within the first data model corresponding to pixel s, and using this color value to compute a ray tracing value at the point p. Additionally, the method can include, provided the point p lies within an object of the objects and is not occluded from a camera, performing the following: based on the point p, performing inverse mapping to obtain a orientation in camera space corresponding to the point p and the lens point l, wherein the orientation is defined by two values; based on the orientation, performing inverse mapping to obtain a pixel, s, on a capture plane of the camera corresponding to the point p and the lens point l; scale the energy contribution from the point p by a factor w; and at the pixel s of the capture plane, rendering the scaled energy contribution. In various embodiments, it is noted that the scaling factor w can be computed using Multiple Importance Sampling (MIS) techniques, but is not limited to such.

In various embodiments, the present disclosure can include a method of generating an image. The method can include a) accessing a first data model of an environment captured by a virtual reality camera system, the data model representing an image backplate. Furthermore, the method can include b) accessing a second data model of objects within a virtualized three dimensional (3D) space, wherein the image backplate is within the space. Moreover, the method can include c) provided a given pixel, s, perform a mapping to a camera orientation, wherein the orientation is defined by two values. Additionally, the method can include d) provided a lens point, l, generating a camera ray through a focal point f, of the oriented camera. In addition, the method can include e) scale the incoming energy contribution from the point p by a factor w. The method can also include f) rendering the scaled energy contribution at the pixel s of a capture plane. The method can also include g) repeating the c)-f) for a second lens position l for each pixel s. In various embodiments, note that the scaling factor w can be computed using Multiple Importance Sampling (MIS) techniques, but is not limited to such.

In various embodiments, it is noted that the methods described in the above two paragraphs can be combined into a method. Each method would render to the same capture plane but scale the energy that is rendered to the capture plane by factors computed using Multiple Importance Sampling. Note that each method of the above two paragraphs would run independently. They just share the same scene data and capture plane. Therefore, the first data model, the second data model, and the capture plane are shared by the two methods of the above two paragraphs.

While particular embodiments in accordance with the present disclosure have been specifically described within this Summary, it is noted that the present disclosure and the claimed subject matter are not limited in any way by these embodiments.

DETAILED DESCRIPTION

Reference will now be made in detail to various embodiments in accordance with the present disclosure, examples of which are illustrated in the accompanying drawings. While the present disclosure will be described in conjunction with various embodiments, it will be understood that these various embodiments are not intended to limit the present disclosure. On the contrary, the present disclosure is intended to cover alternatives, modifications and equivalents, which may be included within the scope of the present disclosure as construed according to the Claims. Furthermore, in the following detailed description of various embodiments in accordance with the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be evident to one of ordinary skill in the art that the present disclosure may be practiced without these specific details or with equivalents thereof. In other instances, well known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects of the present disclosure.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present disclosure, discussions utilizing terms such as “accessing,” “simulating,” “casting,” “projecting,” “performing,” “computing,” “mapping,” “looking up,” “using,” “projecting,” “rendering,” “determining,” “implementing,” “inputting,” “operating,” “analyzing,” “identifying,” “generating,” “extracting,” “receiving,” “processing,” “acquiring,” “producing,” “providing,” “storing,” “altering,” “creating,” “loading” or the like, refer to actions and processes of a computing system or similar electronic computing device or processor. The computing system or similar electronic computing device manipulates and transforms data represented as physical (electronic) quantities within the computing system memories, registers or other such information storage, transmission or display devices.

Portions of the detailed description that follow may be presented and discussed in terms of one or more methods. Although steps and sequencing thereof are disclosed in figures herein describing the operations of one or more methods, such steps and sequencing are exemplary. Any method is well suited to performing various other steps or variations of the steps recited in the flowchart of the figure herein, and in a sequence other than that depicted and described herein.

Various embodiments described herein may be discussed in the general context of computer-executable instructions residing on some form of computer-readable storage medium, such as program modules, executed by one or more computers or other devices. By way of example, and not limitation, computer-readable storage media may comprise non-transitory computer storage media and communication media. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or distributed as desired in various embodiments.

Finite Aperture Omni-Directional Camera

Current approaches to omni-directional rendering use a pinhole camera model. In various embodiments in accordance with the present disclosure, this model is extended to support finite aperture, allowing for depth of field rendering. For example, in various embodiments, a finite aperture omni-directional camera is modeled by aligning the finite aperture lens and focal point with the omni-directional part of the projection. Each point on the image plane maps to a direction in camera space. For a spherical projection, the lens is orientated along this direction and the focal point is picked along this direction at focal distance. For a cylindrical projection, the lens is oriented along the projected direction on the two dimensional (2D) xz-plane, as the projection is not omni-directional in the y direction. The focal point is picked along the (unprojected) direction so its projection on the xz-plane is at focal distance from the lens. The final outgoing ray is constructed by sampling of a point on this oriented lens and shooting a ray from there through the focal point.

Finite Aperture Omni-Directional Stereo Light Transport

In various embodiments, the above described finite aperture lens model is combined with the stereo omni-directional rendering process. In various embodiments, this new Finite Aperture Omni-directional Stereo Camera is then used in light transport rendering algorithms to render photorealistic Omni-directional Stereo images. Light transport rendering algorithms such as bidirectional path tracing and photon mapping involve mapping points on the image plane to primary rays in camera space and the inverse of mapping scene points back to image plane coordinates. For a given stereo separation distance, finite aperture lens coordinates and image plane point, the method described above gives the outgoing camera ray. Note that described herein in accordance with various embodiments is an inverse method of mapping a camera space scene points back to image coordinates for fixed stereo separation distance and finite aperture lens coordinates.

Spherical Camera

In various embodiments, each image coordinate maps to spherical coordinates (theta, phi). For given lens coordinates, each spherical coordinate maps to a single camera ray. All rays for a fixed theta together form a hyperboloid. The theta is searched for which rotates the hyperboloid so it contains a scene point, which corresponds to rotating the scene point in the opposite orientation until it lies on the hyperbola. The intersection is searched for of the hyperboloid with the 2D circle in the xz-plane through the scene point. This circle lies on the sphere around the origin containing the scene point. Intersecting this sphere with the hyperboloid will give a circle in the yz-plane with its center on the x-axis. The radius is found in the x coordinate of this circle by solving the hyperboloid-sphere intersection in the y=0 plane. The intersection between the hyperboloid and the y=0 plane gives a hyperbola. The intersection between the sphere in the y=0 plane gives a circle. Intersecting the circle and hyperbola gives the radius and x coordinate of the hyperboloid-sphere intersection circle. This circle contains the rotated scene point. The scene point is rotated around the y-axis, so intersecting the circle with the xz-plane through the scene point gives the rotated scene point. Sega is the angle between the scene point and rotated scene point within the xz-plane. Phi is the angle between the scene point and the xy-plane through the origin.

Cylindrical Camera

In various embodiments, each image coordinate maps to cylindrical coordinates (theta, height). For given lens coordinates, each cylindrical coordinate maps to a single camera ray. Theta is found by projecting the scene point on the xz-plane. The method of inversion for a pinhole stereoscopic cylindrical camera is first described. In the xz-plane, each theta maps to a single ray, touching the circle around the origin with a radius equal to the stereo separation. The theta is searched for which rotates this ray to pass through the projected scene point, which equals to rotating the projected scene point in the opposite orientation until it lies on the ray. Therefore, the intersection is searched for of the vertical ray with the circle through the projected scene point. The angle between this intersection point and the projected scene point equals theta. In the case of a finite aperture lens, the origin of the ray lies on a finite aperture lens in the ray does not touch the circle with radius equal to the stereo separation. However, the ray does touch another circle around the origin with adjusted radius r:

l=the horizontal lens coordinate in units camera space,

Then
r=(l+d)/sqrt(((l*l)/(f*f))+1)
Theta is found by solving for the pinhole stereoscopic cylindrical camera with adjusted stereo separation r. The found theta is adjusted to obtain the theta for the original finite aperture camera:
theta=(l>0)?theta′+acos(r/(d+l))
otherwise: theta′−acos(r/(d+l))
Photo to Geometry Projection

In various embodiments, the inverse scene to camera projection can also be used to map points on 3D geometry (such as matte objects in a computer generated (CG) scene) to 2D images (such as a backplate photo) captured using a physical Omni-directional Stereo Camera.

In various embodiments in accordance with the present disclosure add depth of field effects to omni-directional cameras. Furthermore, various embodiments extend this to include stereo omni-directional cameras and at least provide an efficient analytical method for mapping scene points back to camera space, desired for the integration of the new camera model with advanced physically based rendering algorithms such as bidirectional path tracing.

FIG. 1is a flow diagram of a method100for rendering an image in accordance with various embodiments of the present disclosure. Although specific operations are disclosed inFIG. 1, such operations are examples. The method100may not include all of the operations illustrated byFIG. 1. Also, method100may include various other operations and/or variations of the operations shown. Likewise, the sequence of the operations of flow diagram100can be modified. It is appreciated that not all of the operations in flow diagram100may be performed. In various embodiments, one or more of the operations of method100can be controlled or managed by software, by firmware, by hardware or by any combination thereof, but is not limited to such. Method100can include processes of embodiments of the present disclosure which can be controlled or managed by a processor(s) and/or electrical components under the control of computer or computing device readable and executable instructions (or code). The computer or computing device readable and executable instructions (or code) may reside, for example, in data storage features such as computer or computing device usable volatile memory, computer or computing device usable non-volatile memory, and/or computer or computing device usable mass data storage. However, the computer or computing device readable and executable instructions (or code) may reside in any type of computer or computing device readable medium or memory.

At operation102, construct scene point p through a ray tracing technique. It is pointed out that operation102can be implemented in a wide variety of ways. For example, in various embodiments, the ray tracing technique at operation102can include, but is not limited to, Monte Carlo ray tracing technique, and the like. Note that operation102can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation104ofFIG. 1, project p back to screen pixel s for 360 VR camera model. It is pointed out that operation104can be implemented in a wide variety of ways. For example, in various embodiments, at operation104a cylindrical camera method300ofFIG. 3can be performed. Furthermore, in various embodiments, at operation104a spherical camera method400ofFIG. 4can be performed. It is noted that operation104can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation106, render energy contribution to the camera framebuffer at location s. Note that operation106can be implemented in a wide variety of ways. For example, operation106can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

FIG. 2is a flow diagram of a method200for rendering an image augmented with a photograph made by an actual physical 360 VR camera in accordance with various embodiments of the present disclosure. Although specific operations are disclosed inFIG. 2, such operations are examples. The method200may not include all of the operations illustrated byFIG. 2. Also, method200may include various other operations and/or variations of the operations shown. Likewise, the sequence of the operations of flow diagram200can be modified. It is appreciated that not all of the operations in flow diagram200may be performed. In various embodiments, one or more of the operations of method200can be controlled or managed by software, by firmware, by hardware or by any combination thereof, but is not limited to such. Method200can include processes of embodiments of the present disclosure which can be controlled or managed by a processor(s) and/or electrical components under the control of computer or computing device readable and executable instructions (or code). The computer or computing device readable and executable instructions (or code) may reside, for example, in data storage features such as computer or computing device usable volatile memory, computer or computing device usable non-volatile memory, and/or computer or computing device usable mass data storage. However, the computer or computing device readable and executable instructions (or code) may reside in any type of computer or computing device readable medium or memory.

At operation202, construct scene point p through a ray tracing technique. It is noted that operation202can be implemented in a wide variety of ways. For example, in various embodiments, the ray tracing technique at operation202can include, but is not limited to, Monte Carlo ray tracing technique, and the like. Note that operation202can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation204ofFIG. 2, project p back to screen pixel s for 360 VR camera model. It is pointed out that operation204can be implemented in a wide variety of ways. For example, in various embodiments, at operation204a cylindrical camera method300ofFIG. 3can be performed. Moreover, in various embodiments, at operation204a spherical camera method400ofFIG. 4can be performed. Note that operation204can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation206, lookup value at pixel s in 360 VR camera framebuffer (e.g., backplate photograph). It is noted that operation206can be implemented in a wide variety of ways. For example, operation206may be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation208ofFIG. 2, use lookup value to compute ray tracing color (e.g., as background). It is pointed out that operation208can be implemented in a wide variety of ways. For example, operation208may be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

FIG. 3is a flow diagram of a method300including a cylindrical 360 VR camera model, a cylindrical finite aperture model, and a cylindrical inversion process in accordance with various embodiments of the present disclosure. Although specific operations are disclosed inFIG. 3, such operations are examples. The method300may not include all of the operations illustrated byFIG. 3. Also, method300may include various other operations and/or variations of the operations shown. Likewise, the sequence of the operations of flow diagram300can be modified. It is appreciated that not all of the operations in flow diagram300may be performed. In various embodiments, one or more of the operations of method300can be controlled or managed by software, by firmware, by hardware or by any combination thereof, but is not limited to such. Method300can include processes of embodiments of the present disclosure which can be controlled or managed by a processor(s) and/or electrical components under the control of computer or computing device readable and executable instructions (or code). The computer or computing device readable and executable instructions (or code) may reside, for example, in data storage features such as computer or computing device usable volatile memory, computer or computing device usable non-volatile memory, and/or computer or computing device usable mass data storage. However, the computer or computing device readable and executable instructions (or code) may reside in any type of computer or computing device readable medium or memory.

FIG. 3is described in combination with a portion ofFIG. 5. It is noted that a portion ofFIG. 5includes at least an overview of a cylindrical 360 VR camera model, a cylindrical finite aperture model, and a cylindrical inversion process in accordance with various embodiments of the present disclosure. For example,FIG. 5includes and indicates thatFIGS. 6, 6a,6b, and8can be associated with a cylindrical 360 VR camera model in various embodiments. In addition,FIG. 5includes and indicates thatFIGS. 6cand9can be associated with a cylindrical finite aperture model in various embodiments. Furthermore,FIG. 5includes and indicates thatFIGS. 6d, 6e, and 6fcan be associated with a cylindrical inversion process in various embodiments. It is noted that at least each ofFIGS. 6e, 7h, 7i, and9includes additional information not shown withinFIG. 5.

At operation302ofFIG. 3, randomly sample point l on camera lens. It is noted that operation302can be implemented in a wide variety of ways. For example, in various embodiments, randomly sample point l on camera lens902at operation302can be implemented in a manner as shown inFIG. 9. Note that withinFIG. 9, f is at focal distance from the camera lens902. Operation302can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation304, construct circle C in xz-plane centered at o with radius r. It is pointed out that operation304can be implemented in a wide variety of ways. For example, in various embodiments, construct circle C in xz-plane centered at o with radius r at operation304can be implemented in a manner as shown inFIG. 6e. Note that an equation for radius r is shown withinFIG. 6ewhere d is equal to the stereo separation, l is equal to the horizontal lens coordinate in units camera space, and f is equal to the focal distance. Operation304can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation306ofFIG. 3, project scene point p to p′ on xz-plane. Note that operation306can be implemented in a wide variety of ways. For example, in various embodiments, project scene point p to p′ on xz-plane at operation306can be implemented in a manner as shown inFIG. 6c. It is pointed out that an equation for h is shown withinFIG. 6c. Operation306can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation308, construct line L parallel with z-axis at distance r from o. It is noted that operation308can be implemented in a wide variety of ways. For example, in various embodiments, construct line L parallel with z-axis at distance r from o at operation308can be implemented in a manner as shown inFIG. 6f. Operation308can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation310ofFIG. 3, rotate p′ around o until it intersects the line parallel L in q. Note that operation310can be implemented in a wide variety of ways. For example, in various embodiments, rotate p′ around o until it intersects the line parallel L in q at operation310can be implemented in a manner as shown inFIG. 6f. Operation310can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation312, compute Θ′ as angle between vectors op′ and oq (e.g., seeFIG. 6f). It is pointed out that operation312can be implemented in a wide variety of ways. For example, in various embodiments, compute Θ′ as angle between vectors op′ and oq at operation312can be implemented in a manner as shown inFIG. 6f. Operation312can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation314ofFIG. 3, compute Θ from Θ′ using Θ=(lx>0)?Θ′+acos(r/(d+lx)): Θ′−acos(r/(d+lx)). It is noted that operation314can be implemented in a wide variety of ways. For example, in various embodiments, compute Θ from Θ′ using Θ=(lx>0)?Θ′+acos(r/(d+lx)): Θ′−acos(r/(d+lx)) at operation314can be implemented in a manner as shown inFIG. 6f. Note that equations for Θ are shown withinFIG. 6f. Operation314can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation316, compute h from p, p′, e, l and f. Note that operation316can be implemented in a wide variety of ways. For example, in various embodiments, compute h from p, p′, e, l and f at operation316can be implemented in a manner as shown inFIG. 6c. It is pointed out that an equation for h is shown withinFIG. 6c. Operation316can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation318ofFIG. 3, compute screen space pixels from Θ and h (e.g., seeFIG. 6a). It is noted that operation318can be implemented in a wide variety of ways. For example, in various embodiments, compute screen space pixel s from Θ and h at operation318can be implemented in a manner as shown inFIG. 6a. Operation318can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

FIG. 4is a flow diagram of a method400including a spherical 360 VR camera model, a spherical finite aperture model, and a spherical inversion process in accordance with various embodiments of the present disclosure. Although specific operations are disclosed inFIG. 4, such operations are examples. The method400may not include all of the operations illustrated byFIG. 4. Also, method400may include various other operations and/or variations of the operations shown. Likewise, the sequence of the operations of flow diagram400can be modified. It is appreciated that not all of the operations in flow diagram400may be performed. In various embodiments, one or more of the operations of method400can be controlled or managed by software, by firmware, by hardware or by any combination thereof, but is not limited to such. Method400can include processes of embodiments of the present disclosure which can be controlled or managed by a processor(s) and/or electrical components under the control of computer or computing device readable and executable instructions (or code). The computer or computing device readable and executable instructions (or code) may reside, for example, in data storage features such as computer or computing device usable volatile memory, computer or computing device usable non-volatile memory, and/or computer or computing device usable mass data storage. However, the computer or computing device readable and executable instructions (or code) may reside in any type of computer or computing device readable medium or memory.

FIG. 4is described in combination with a portion ofFIG. 5. It is point out that a portion ofFIG. 5includes at least an overview of a spherical 360 VR camera model, a spherical finite aperture model, and a spherical inversion process in accordance with various embodiments of the present disclosure. For example,FIG. 5includes and indicates thatFIGS. 6, 7a,7b, and8can be associated with a spherical 360 VR camera model in various embodiments. Moreover,FIG. 5includes and indicates thatFIGS. 7cand9can be associated with a spherical finite aperture model in various embodiments. Additionally,FIG. 5includes and indicates thatFIGS. 7d, 7e, 7f, 7g, 7h, and 7ican be associated with a spherical inversion process in various embodiments. It is noted that at least each ofFIGS. 6e, 7h, 7i, and9includes additional information not shown withinFIG. 5.

At operation402, randomly sample point l on camera lens. It is noted that operation402can be implemented in a wide variety of ways. For example, in various embodiments, randomly sample point l on camera lens902at operation402can be implemented in a manner as shown inFIG. 9. Note that withinFIG. 9, f is at focal distance from the camera lens902. Operation402can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation404ofFIG. 4, construct line L through l and focal point f. It is pointed out that operation404can be implemented in a wide variety of ways. For example, in various embodiments, construct line L through l and focal point f at operation404can be implemented in a manner as shown inFIG. 7d. Operation404can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation406, rotate L around the x axis to obtain hyperboloid. Note that operation406can be implemented in a wide variety of ways. For example, in various embodiments, rotate L around the x axis to obtain hyperboloid at operation406can be implemented in a manner as shown inFIG. 7e. It is noted that the hyperboloid is not explicitly drawn withinFIG. 7e. Instead, the hyperboloid is indicated by the rotation (circle of arrows) of the line L around the x axis. Operation406can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation408ofFIG. 4, rotate scene point p around x-axis until it intersects the xz-plane in q. It is pointed out that operation408can be implemented in a wide variety of ways. For example, in various embodiments, rotate scene point p around x-axis until it intersects the xz-plane in q at operation408can be implemented in a manner as shown inFIG. 7g. Operation408can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation410, intersect hyperboloid with xz-plane to obtain two dimensional (2D) hyperbola H. It is noted that operation410can be implemented in a wide variety of ways. For example, in various embodiments, intersect hyperboloid with xz-plane to obtain 2D hyperbola H at operation410can be implemented in a manner as shown inFIG. 7f. Operation410can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation412ofFIG. 4, rotate q around the y-axis until it intersects the hyperbola H in t. Note that operation412can be implemented in a wide variety of ways. For example, in various embodiments, rotate q around the y-axis until it intersects the hyperbola H in t at operation412can be implemented in a manner as shown inFIG. 7g. Operation412can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation414, compute the angle Θ between vectors oq and ot. It is pointed out that operation414can be implemented in a wide variety of ways. For example, in various embodiments, compute the angle Θ between vectors oq and ot at operation414can be implemented in a manner as shown inFIG. 7g. Operation414can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation416ofFIG. 4, rotate t around x-axis to obtain circle with origin w in yz-plane. Note that operation416can be implemented in a wide variety of ways. For example, in various embodiments, rotate t around x-axis to obtain circle702with origin w in yz-plane at operation416can be implemented in a manner as shown inFIG. 7h. Operation416can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation418, find intersection point u of circle with xz-plane through p. It is pointed out that operation418can be implemented in a wide variety of ways. For example, in various embodiments, find intersection point u of circle702with xz-plane through p at operation418can be implemented in a manner as shown inFIG. 7h. Operation418can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation420ofFIG. 4, construct vector v=f+l. It is noted that operation420can be implemented in a wide variety of ways. For example, in various embodiments, construct vector v=f+l at operation420can be implemented in a manner as shown inFIG. 7i. Operation420can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation422, compute ϕ as angle between vector v and vector wu. Note that operation422can be implemented in a wide variety of ways. For example, in various embodiments, compute ϕ as angle between vector v and vector wu at operation422can be implemented in a manner as shown inFIG. 7i. Operation422can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation424ofFIG. 4, compute screen space pixel s from Θ and ϕ (e.g., seeFIG. 7a). It is pointed out that operation424can be implemented in a wide variety of ways. For example, in various embodiments, compute screen space pixel s from Θ and ϕ at operation424can be implemented in a manner as shown inFIG. 7a. Operation424can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

With reference toFIGS. 6, 6a, and7a, please note that they can include mapping from [theta,h] and [theta,phi] spaces to a pixel at coordinates (i,j) in an n by m pixel image.

FIG. 10is a flow diagram of a method1000for generating an image in accordance with various embodiments of the present disclosure. Although specific operations are disclosed inFIG. 10, such operations are examples. The method1000may not include all of the operations illustrated byFIG. 10. Also, method1000may include various other operations and/or variations of the operations shown. Likewise, the sequence of the operations of flow diagram1000can be modified. It is appreciated that not all of the operations in flow diagram1000may be performed. In various embodiments, one or more of the operations of method1000can be controlled or managed by software, by firmware, by hardware or by any combination thereof, but is not limited to such. Method1000can include processes of embodiments of the present disclosure which can be controlled or managed by a processor(s) and/or electrical components under the control of computer or computing device readable and executable instructions (or code). The computer or computing device readable and executable instructions (or code) may reside, for example, in data storage features such as computer or computing device usable volatile memory, computer or computing device usable non-volatile memory, and/or computer or computing device usable mass data storage. However, the computer or computing device readable and executable instructions (or code) may reside in any type of computer or computing device readable medium or memory.

At operation1002, accessing a first data model of an environment captured by a virtual reality camera system, the data model representing an image backplate. Note that operation1002can be implemented in a wide variety of ways. For example, in various embodiments, at operation1002the virtual reality (VR) camera system is a cylindrical VR 360 camera system. In various embodiments, at operation1002the virtual reality camera system is a spherical VR 360 camera system. In various embodiments, at operation1002the first data model includes pixels that are referenced by the two values. Operation1002can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation1004ofFIG. 10, accessing a second data model of objects within a virtualized three dimensional (3D) space, wherein the image backplate is within the space. Note that operation1004can be implemented in a wide variety of ways. For example, in various embodiments, at operation1004the second data model includes pixels that are referenced by the two values. Operation1004can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation1006, simulating a presence of at least one light source, or camera, within the space and casting a ray from the light source, or the camera into the space, to a point, p, in the space, possibly after interacting with objects in the second data model, using ray tracing techniques. Note that operation1006can be implemented in a wide variety of ways. For example, in various embodiments, at operation1006the ray tracing techniques are Monte Carlo ray tracing techniques, but are not limited to such. Operation1006can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation1008ofFIG. 10, simulating a finite aperture with respect to a camera capturing a two dimensional (2D) projection of a 3D scene by projecting a ray from the point p, through a focal point of the camera, to a lens point l within the finite aperture of the camera. Note that operation1008can be implemented in a wide variety of ways. For example, in various embodiments, at operation1008the simulating the finite aperture with respect to the camera capturing a 2D projection of a 3D scene includes projecting a plurality of rays from the point p, through the focal point of the camera, to a plurality of lens points l across the finite aperture of the camera and further comprising performing a first group of operations1010,1012,1014, and1016(described herein) and/or a second group of operations1018,1020,1022, and1024(described herein) for each of the plurality of lens points lto determine a corresponding plurality of pixels s. In various embodiments, it is noted that the first group of operations1010,1012,1014, and1016, might be projecting to a different camera model. In this case, it projects to a plurality of lens points l across the finite aperture of the camera model corresponding to the physical camera system used to capture the backplate, not the camera used to capture the 2D projection of the virtual scene. Operation1008can be implemented in any manner similar to that described and/or shown herein, but is not limited to such. After completion of operation1008, method1000can proceed to operation1010and/or operation1018.

At operation1010, provided the point p lies within the backplate and is not occluded from a camera, performing operations1012,1014, and1016. Note that operation1010can be implemented in a wide variety of ways. For example, operation1010can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation1012ofFIG. 10, based on the point p, performing inverse mapping to obtain an orientation in camera space corresponding to the point p and the lens point l, wherein the orientation is defined by two values. Note that operation1012can be implemented in a wide variety of ways. For example, in various embodiments, at operation1012in a cylindrical VR 360 camera system the two values include theta and h and wherein theta is a camera angle within an xz-plane of the space associated with point p and wherein h is a height in a y coordinate in the camera space associated with the point p. In various embodiments, at operation1012in a spherical VR 360 camera system the two values include theta and phi and wherein theta is a camera angle within an xz-plane of the space associated with point p and wherein phi is tilt angle of the camera system with respect to y-axis of the space and associated with the point p. Operation1012can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation1014, based on the orientation, performing inverse mapping to obtain a pixel, s, on a capture plane of the camera corresponding to the point p and the lens point l. Note that operation1014can be implemented in a wide variety of ways. For example, operation1014can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation1016ofFIG. 10, using the pixel s, looking up a color value within the first data model corresponding to pixel s, and using this color value to compute a ray tracing value at the point p. Note that operation1016can be implemented in a wide variety of ways. For example, operation1016can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation1018, provided the point p lies within an object of the objects and is not occluded from a camera, performing operations1020,1022, and1024. Note that operation1018can be implemented in a wide variety of ways. For example, operation1018can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation1020ofFIG. 10, based on the point p, performing inverse mapping to obtain an orientation in camera space corresponding to the point p and the lens point l, wherein the orientation is defined by two values. Note that operation1020can be implemented in a wide variety of ways. For example, in various embodiments, at operation1020in a cylindrical VR 360 camera system the two values include theta and h and wherein theta is a camera angle within an xz-plane of the space associated with point p and wherein h is a height in a y coordinate in the camera space associated with the point p. In various embodiments, at operation1020in a spherical VR 360 camera system the two values include theta and phi and wherein theta is a camera angle within an xz-plane of the space associated with point p and wherein phi is tilt angle of the camera system with respect to y-axis of the space and associated with the point p. Operation1020can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation1022, based on the orientation, performing inverse mapping to obtain a pixel, s, on a capture plane of the camera corresponding to the point p and the lens point l. Note that operation1022can be implemented in a wide variety of ways. For example, operation1022can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation1024ofFIG. 10, at the pixel s of the capture plane, rendering energy contribution from the point p. Note that operation1024can be implemented in a wide variety of ways. For example, operation1024can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

In various embodiments, it is noted that method1000can be performed by at least a processor and/or by at least a graphics processing unit, but is not limited to such.

FIG. 11is a flow diagram of a method1100for generating an image in accordance with various embodiments of the present disclosure. Although specific operations are disclosed inFIG. 11, such operations are examples. The method1100may not include all of the operations illustrated byFIG. 11. Also, method1100may include various other operations and/or variations of the operations shown. Likewise, the sequence of the operations of flow diagram1100can be modified. It is appreciated that not all of the operations in flow diagram1100may be performed. In various embodiments, one or more of the operations of method1100can be controlled or managed by software, by firmware, by hardware or by any combination thereof, but is not limited to such. Method1100can include processes of embodiments of the present disclosure which can be controlled or managed by a processor(s) and/or electrical components under the control of computer or computing device readable and executable instructions (or code). The computer or computing device readable and executable instructions (or code) may reside, for example, in data storage features such as computer or computing device usable volatile memory, computer or computing device usable non-volatile memory, and/or computer or computing device usable mass data storage. However, the computer or computing device readable and executable instructions (or code) may reside in any type of computer or computing device readable medium or memory.

At operation1102, accessing a first data model of an environment captured by a virtual reality camera system, the data model representing an image backplate. Note that operation1102can be implemented in a wide variety of ways. For example, in various embodiments, at operation1102the virtual reality (VR) camera system is a cylindrical VR 360 camera system. In various embodiments, at operation1102the virtual reality camera system is a spherical VR 360 camera system. In various embodiments, at operation1102the first data model includes pixels that are referenced by the two values. Operation1102can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation1104ofFIG. 11, accessing a second data model of objects within a virtualized three dimensional (3D) space, wherein the image backplate is within the space. It is noted that operation1104can be implemented in a wide variety of ways. For example, operation1104can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation1106, provided a given pixel, s, perform a mapping to a camera orientation, wherein the orientation is defined by two values. Note that operation1106can be implemented in a wide variety of ways. For example, in various embodiments, at operation1106, in a cylindrical VR 360 camera system the two values include theta and h and wherein theta is a camera angle within an xz-plane of the space associated with pixel s and wherein h is a height in a y coordinate in the camera space associated with the pixel s. In various embodiments, at operation1106in a spherical VR 360 camera system the two values include theta and phi and wherein theta is a camera angle within an xz-plane of the space associated with pixel s and wherein phi is tilt angle of the camera system with respect to y-axis of the space and associated with the pixel s. Operation1106can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation1108ofFIG. 11, provided a lens point, l, generating a camera ray through a focal point f, of the oriented camera. It is noted that operation1108can be implemented in a wide variety of ways. For example, operation1108can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation1110, Computing incoming energy along the ray using ray tracing techniques. Note that operation1110can be implemented in a wide variety of ways. For example, in various embodiments, at operation1110the ray tracing techniques are Monte Carlo ray tracing techniques, but are not limited to such. Operation1110can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation1112ofFIG. 11, rendering energy contribution at the pixel s of a capture plane. It is noted that operation1112can be implemented in a wide variety of ways. For example, operation1112can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

At operation1114, repeating operations1106,1108,1110and1112for a second lens position l (or a plurality of lens positions l) for each pixel s. Note that operation1114can be implemented in a wide variety of ways. For example, operation1114can be implemented in any manner similar to that described and/or shown herein, but is not limited to such.

In various embodiments, it is noted that method1100can be performed by at least a processor and/or by at least a graphics processing unit, but is not limited to such.

Example Computing System

FIG. 12is a block diagram of an example of a computing system1200upon which one or more various embodiments described herein may be implemented in accordance with various embodiments of the present disclosure. In a basic configuration, the system1200can include at least one processing unit1202(e.g., graphics processing unit (GPU), central processing unit (CPU), processor, and the like) coupled to memory1204. This basic configuration is illustrated inFIG. 12by dashed line1206. In various embodiments, the at least one processing unit1202can be coupled to the memory1204via an address/data bus1203(or other interface), but is not limited to such. The system1200may also have additional features and/or functionality. For example, the system1200may also include additional storage (e.g., removable and/or non-removable) including, but not limited to, magnetic or optical disks or tape. Such additional storage is illustrated inFIG. 12by removable storage1208and non-removable storage1220. The system1200may also contain communications connection(s)1222that allow the device to communicate with other devices, e.g., in a networked environment1228using logical connections to one or more remote computers. In various embodiments, the removable storage1208, non-removable storage1220, and communications connection(s)1222can be coupled to the address/data bus1203(or other interface).

The system1200may also includes input device(s)1224such as a keyboard, mouse, pen, voice input device, touch input device, etc. Output device(s)1226such as a display device, speakers, printer, etc., may also be included. In various embodiments, the input device(s)1224and the output device(s)1226can be coupled to the address/data bus1203(or other interface). It is noted that the address/data bus1203enables communication between those devices which are coupled to it.

In the example ofFIG. 12, the memory1204can include computer-readable instructions, data structures, program modules, and the like associated with one or more various embodiments1250in accordance with the present disclosure. However, the embodiment(s)1250may instead reside in any one of the computer storage media used by the system1200, or may be distributed over some combination of the computer storage media, or may be distributed over some combination of networked computers, but are not limited to such.

It is noted that the computing system1200may not include all of the elements illustrated byFIG. 12. In addition, the computing system1200can be implemented to include one or more elements not illustrated byFIG. 12. It is pointed out that the computing system1200can be utilized or implemented in any manner similar to that described and/or shown by the present disclosure, but is not limited to such.

The foregoing descriptions of various specific embodiments in accordance with the present disclosure have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed, and many modifications and variations are possible in light of the above teaching. The present disclosure is to be construed according to the Claims and their equivalents.