Patent ID: 12260493

In accordance with common practice the various features illustrated in the drawings may not be drawn to scale. Accordingly, the dimensions of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method, or device. Finally, like reference numerals may be used to denote like features throughout the specification and figures.

SUMMARY

Various implementations disclosed herein include devices, systems, and methods for multi-camera hole filling, which results in an image that is a near representation of the user's field-of-view with a reduced number of holes (e.g., occlusions and disocclusions). According to some implementations, a method for multi-camera hole filling may obtain an occlusion mask based on a warped image of an environment captured by a respective image sensor, wherein the warped image accounts for the POV difference between a user's eye and the respective image sensor as mentioned above. In some implementations, the method for multi-camera hole filling may fill holes of the occlusion mask based on images from other image sensors different from the respective image sensor, wherein the aforementioned images are normalized to account for different intrinsic camera characteristics between the respective image sensor and the other image sensors. In some implementations, the method for multi-camera hole filling may also fill holes of the occlusion mask based on a diffusion and/or feathering process, wherein the diffusion and/or feathering process may be associated with a diffusion kernel that accounts for depth and/or displacement/distance.

According to some implementations, the method is performed at a computing system including non-transitory memory and one or more processors, wherein the computing system is communicatively coupled to a first image sensor and a second image sensor. The method includes: obtaining a first image of an environment from a first image sensor, wherein the first image sensor is associated with first intrinsic parameters; performing a warping operation on the first image according to perspective offset values to generate a warped first image in order to account for perspective differences between the first image sensor and a user of the electronic device; determining an occlusion mask based on the warped first image that includes a plurality of holes; obtaining a second image of the environment from a second image sensor, wherein the second image sensor is associated with second intrinsic parameters; normalizing the second image based on a difference between the first and second intrinsic parameters to produce a normalized second image; and filling a first set of one or more holes of the occlusion mask based on the normalized second image to produce a modified first image.

In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors and the one or more programs include instructions for performing or causing performance of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions, which, when executed by one or more processors of a device, cause the device to perform or cause performance of any of the methods described herein. In accordance with some implementations, a device includes: one or more processors, a non-transitory memory, and means for performing or causing performance of any of the methods described herein.

In accordance with some implementations, a computing system includes one or more processors, non-transitory memory, an interface for communicating with a display device and one or more input devices (e.g., the first and second images sensors), and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors and the one or more programs include instructions for performing or causing performance of the operations of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions which when executed by one or more processors of a computing system with an interface for communicating with a display device and one or more input devices (e.g., the first and second images sensors), cause the computing system to perform or cause performance of the operations of any of the methods described herein. In accordance with some implementations, a computing system includes one or more processors, non-transitory memory, an interface for communicating with a display device and one or more input devices (e.g., the first and second images sensors), and means for performing or causing performance of the operations of any of the methods described herein.

DESCRIPTION

Numerous details are described in order to provide a thorough understanding of the example implementations shown in the drawings. However, the drawings merely show some example aspects of the present disclosure and are therefore not to be considered limiting. Those of ordinary skill in the art will appreciate that other effective aspects and/or variants do not include all of the specific details described herein. Moreover, well-known systems, methods, components, devices, and circuits have not been described in exhaustive detail so as not to obscure more pertinent aspects of the example implementations described herein.

A physical environment refers to a physical world that someone may interact with and/or sense without the use of electronic devices. The physical environment may include physical features such as a physical object or physical surface. For example, a physical environment may include a physical city that includes physical buildings, physical streets, physical trees, and physical people. People may directly interact with and/or sense the physical environment through, for example, touch, sight, taste, hearing, and smell. An extended reality (XR) environment, on the other hand, refers to a wholly or partially simulated environment that someone may interact with and/or sense using an electronic device. For example, an XR environment may include virtual reality (VR) content, augmented reality (AR) content, mixed reality (MR) content, or the like. Using an XR system, a portion of a person's physical motions, or representations thereof, may be tracked. In response, one or more characteristics of a virtual object simulated in the XR environment may be adjusted such that it adheres to one or more laws of physics. For example, the XR system may detect a user's movement and, in response, adjust graphical and auditory content presented to the user in a way similar to how views and sounds would change in a physical environment. In another example, the XR system may detect movement of an electronic device presenting an XR environment (e.g., a laptop, a mobile phone, a tablet, or the like) and, in response, adjust graphical and auditory content presented to the user in a way similar to how views and sounds would change in a physical environment. In some situations, the XR system may adjust one or more characteristics of graphical content in the XR environment responsive to a representation of a physical motion (e.g., a vocal command).

Various electronic systems enable one to interact with and/or sense XR environments. For example, projection-based systems, head-mountable systems, heads-up displays (HUDs), windows having integrated displays, vehicle windshields having integrated displays, displays designed to be placed on a user's eyes (e.g., similar to contact lenses), speaker arrays, headphones/earphones, input systems (e.g., wearable or handheld controllers with or without haptic feedback), tablets, smartphones, and desktop/laptop computers may be used. A head-mountable system may include an integrated opaque display and one or more speakers. In other examples, a head-mountable system may accept an external device having an opaque display (e.g., a smartphone). The head-mountable system may include one or more image sensors and/or one or more microphones to capture images or video and/or audio of the physical environment. In other examples, a head-mountable system may include a transparent or translucent display. A medium through which light representative of images is directed may be included within the transparent or translucent display. The display may utilize OLEDs, LEDs, μLEDs, digital light projection, laser scanning light source, liquid crystal on silicon, or any combination of these technologies. The medium may be a hologram medium, an optical combiner, an optical waveguide, an optical reflector, or a combination thereof. In some examples, the transparent or translucent display may be configured to selectively become opaque. Projection-based systems may use retinal projection technology to project graphical images onto a user's retina. Projection systems may also be configured to project virtual objects into the physical environment, for example, on a physical surface or as a hologram.

FIG.1is a block diagram of an example operating architecture100in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein. To that end, as a non-limiting example, the operating architecture100includes an optional controller110and an electronic device120(e.g., a tablet, mobile phone, laptop, near-eye system, wearable computing device, or the like).

In some implementations, the controller110is configured to manage and coordinate an XR experience (sometimes also referred to herein as a “XR environment” or a “virtual environment” or a “graphical environment”) for a user150and zero or more other users. In some implementations, the controller110includes a suitable combination of software, firmware, and/or hardware. The controller110is described in greater detail below with respect toFIG.2. In some implementations, the controller110is a computing device that is local or remote relative to the physical environment105. For example, the controller110is a local server located within the physical environment105. In another example, the controller110is a remote server located outside of the physical environment105(e.g., a cloud server, central server, etc.). In some implementations, the controller110is communicatively coupled with the electronic device120via one or more wired or wireless communication channels144(e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In some implementations, the functions of the controller110are provided by the electronic device120. As such, in some implementations, the components of the controller110are integrated into the electronic device120.

In some implementations, the electronic device120is configured to present audio and/or video (A/V) content to the user150. In some implementations, the electronic device120is configured to present a user interface (UI) and/or an XR environment128to the user150. In some implementations, the electronic device120includes a suitable combination of software, firmware, and/or hardware. The electronic device120is described in greater detail below with respect toFIG.3.

According to some implementations, the electronic device120presents an XR experience to the user150while the user150is physically present within a physical environment105that includes a table107within the field-of-view (FOV)111of the electronic device120. As such, in some implementations, the user150holds the electronic device120in his/her hand(s). In some implementations, while presenting the XR experience, the electronic device120is configured to present XR content (sometimes also referred to herein as “graphical content” or “virtual content”), including an XR cylinder109, and to enable video pass-through of the physical environment105(e.g., including the table107) on a display122. For example, the XR environment128, including the XR cylinder109, is volumetric or three-dimensional (3D).

In one example, the XR cylinder109corresponds to display-locked content such that the XR cylinder109remains displayed at the same location on the display122as the FOV111changes due to translational and/or rotational movement of the electronic device120. As another example, the XR cylinder109corresponds to world-locked content such that the XR cylinder109remains displayed at its origin location as the FOV111changes due to translational and/or rotational movement of the electronic device120. As such, in this example, if the FOV111does not include the origin location, the XR environment128will not include the XR cylinder109. For example, the electronic device120corresponds to a near-eye system, mobile phone, tablet, laptop, wearable computing device, or the like.

In some implementations, the display122corresponds to an additive display that enables optical see-through of the physical environment105including the table107. For example, the display122correspond to a transparent lens, and the electronic device120corresponds to a pair of glasses worn by the user150. As such, in some implementations, the electronic device120presents a user interface by projecting the XR content (e.g., the XR cylinder109) onto the additive display, which is, in turn, overlaid on the physical environment105from the perspective of the user150. In some implementations, the electronic device120presents the user interface by displaying the XR content (e.g., the XR cylinder109) on the additive display, which is, in turn, overlaid on the physical environment105from the perspective of the user150.

In some implementations, the user150wears the electronic device120such as a near-eye system. As such, the electronic device120includes one or more displays provided to display the XR content (e.g., a single display or one for each eye). For example, the electronic device120encloses the FOV of the user150. In such implementations, the electronic device120presents the XR environment128by displaying data corresponding to the XR environment128on the one or more displays or by projecting data corresponding to the XR environment128onto the retinas of the user150.

In some implementations, the electronic device120includes an integrated display (e.g., a built-in display) that displays the XR environment128. In some implementations, the electronic device120includes a head-mountable enclosure. In various implementations, the head-mountable enclosure includes an attachment region to which another device with a display can be attached. For example, in some implementations, the electronic device120can be attached to the head-mountable enclosure. In various implementations, the head-mountable enclosure is shaped to form a receptacle for receiving another device that includes a display (e.g., the electronic device120). For example, in some implementations, the electronic device120slides/snaps into or otherwise attaches to the head-mountable enclosure. In some implementations, the display of the device attached to the head-mountable enclosure presents (e.g., displays) the XR environment128. In some implementations, the electronic device120is replaced with an XR chamber, enclosure, or room configured to present XR content in which the user150does not wear the electronic device120.

In some implementations, the controller110and/or the electronic device120cause an XR representation of the user150to move within the XR environment128based on movement information (e.g., body pose data, eye tracking data, hand/limb/finger/extremity tracking data, etc.) from the electronic device120and/or optional remote input devices within the physical environment105. In some implementations, the optional remote input devices correspond to fixed or movable sensory equipment within the physical environment105(e.g., image sensors, depth sensors, infrared (IR) sensors, event cameras, microphones, etc.). In some implementations, each of the remote input devices is configured to collect/capture input data and provide the input data to the controller110and/or the electronic device120while the user150is physically within the physical environment105. In some implementations, the remote input devices include microphones, and the input data includes audio data associated with the user150(e.g., speech samples). In some implementations, the remote input devices include image sensors (e.g., cameras), and the input data includes images of the user150. In some implementations, the input data characterizes body poses of the user150at different times. In some implementations, the input data characterizes head poses of the user150at different times. In some implementations, the input data characterizes hand tracking information associated with the hands of the user150at different times. In some implementations, the input data characterizes the velocity and/or acceleration of body parts of the user150such as his/her hands. In some implementations, the input data indicates joint positions and/or joint orientations of the user150. In some implementations, the remote input devices include feedback devices such as speakers, lights, or the like.

FIG.2is an illustrative diagram of an image capture architecture200in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein. According to some implementations, the image capture architecture200includes a camera220that captures an image (e.g., relative to an image plane230) that includes at least a 3D object210within an associated 3D world scene. For example, the camera220may be an image sensor implemented in the electronic device120with the field-of-view111of the physical environment105, as discussed herein with respect toFIG.1.

As shown inFIG.2, the image capture architecture200includes the 3D object210in the 3D world scene (e.g., the physical environment105inFIG.1). The 3D object is associated with 3D world coordinates212: [xwywzw]. The image capture architecture200also includes a camera220(e.g., a pinhole camera or the like) with a focal point224. The camera220is associated with 3D camera coordinates222: [xcyczc]. The image capture architecture200further includes an image plane230separated from the focal point224of the camera220by a focal length214. The image plane230(or features thereon) is associated with 2D pixel coordinates232: [u v].

According to some implementations, the camera220is a simple camera without a lens and with a single small aperture (e.g., the focal point224). Light rays pass through the aperture and project an inverted image onto the image plane230on the opposite side of the camera220. According to some implementations, a virtual image plane240is illustrated for ease of reference as being in front of the camera220with an upright image of the 3D world scene.

The camera parameters are represented by a camera matrix, which is shown below as equation (1). The camera matrix maps the 3D world scene into the image plane230. The camera matrix includes both extrinsic and intrinsic parameters. The extrinsic parameters represent the location of the camera220in the 3D scene (e.g., the 3D camera coordinates222). The intrinsic parameters represent the focal point224(e.g., the optical center or aperture) and the focal length214of the camera220. In other words, the camera matrix is used to denote a projective mapping from the 3D world coordinates212to the 2D pixel coordinates232.

zc[uv1]=K[RT][xwywzw1](1)

[u v 1]trepresents a 2D point in the 2D pixel coordinates232, and [xwywzw1]trepresents a 3D point position in the 3D world coordinates212, where the exponent t represents the transposition operator. Both are expressed in the augmented notation of homogeneous coordinates, which is the most common notation in robotics and rigid body transforms.

The intrinsic parameters are represented by the intrinsic matrix K, which is shown below as equation (2). The parameters αx=f·mxand αy=f·myrepresent focal length in terms of pixels, where mxand myare scale factors relating pixels to distance and f is the focal length214in terms of distance. γ represents a skew coefficient between the x- and y-axis and is often 0. u0and v0represent the principal point.

K=[αxγu000αyv0000100001](2)

The extrinsic parameters are represented by the extrinsic matrix [R T], which is shown below as equation (3). R3×3is sometimes referred to as the rotation matrix, and Taxi is sometimes referred to as the translation vector. [R T] encompasses the extrinsic parameters, which denote coordinate system transformations from the 3D world coordinates212to the 3D camera coordinates222. Equivalently, the extrinsic parameters define the position of the camera center and the camera's heading in the 3D world coordinates212. T corresponds to the position of the origin of the world coordinate system expressed in coordinates of the camera-centered coordinate system.

[RT]4⁢x⁢4=[R3⁢x⁢3T3⁢x⁢101⁢x⁢31](3)

As such, according to some implementations, a rigid 3D-to-3D transformation252from the 3D world coordinates212to the 3D camera coordinates222(or vice versa) exists based on extrinsic parameters associated with three rotational degrees of freedom (DOFs) and three translational DOFs (e.g., the extrinsic matrix [R T]). According to some implementations, a projective 3D-to-2D transformation254from the set of camera coordinates222to the 2D pixel coordinates232(or vice versa) exists based on the intrinsic parameters associated with the camera220(e.g., the intrinsic matrix K). One of ordinary skill in the art will appreciate how the image capture architecture200inFIG.2and the explanation thereof may be adapted to the multi-camera architecture300described below with reference toFIG.3A.

FIG.3Ais an illustrative diagram of a multi-camera architecture300in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations, the multi-camera architecture300corresponds to a near-eye system305with a plurality of image sensors including a left main (forward-facing) camera310(sometimes also referred to as “MCAML”), a right (forward-facing) main camera320(sometimes also referred to as “MCAMR”), a left side-facing (peripheral) camera330(sometimes also referred to as “SCAML”), a right side-facing (peripheral) camera340(sometimes also referred to as “SCAMR”), a left downward-facing camera350(sometimes also referred to as “DCAML”), and a right downward-facing camera360(sometimes also referred to as “DCAMR”). As one example, the electronic device120inFIGS.1and10may correspond to the near-eye system305.

As shown inFIG.3A, each of the plurality of image sensors of the near-eye system305captures an image of an environment (e.g., a physical environment, a partially XR environment, a fully XR environment, and/or the like). In this example, the left main (forward-facing) camera310captures an image312of the environment according to its intrinsic parameters (e.g., 2616×2136 resolution), and the right main (forward-facing) camera320captures an image322of the environment according to its intrinsic parameters (e.g., 872×712 resolution).

Continuing with this example, the left side-facing camera330captures an image332of the environment according to its intrinsic parameters (e.g., 1280×1280 resolution), and the right side-facing camera340captures an image342of the environment according to its intrinsic parameters (e.g., 1280×1280 resolution). Furthermore, continuing with this example, the left downward-facing camera350captures an image352of the environment according to its intrinsic parameters (e.g., 1280×1280 resolution), and the right downward-facing camera360captures an image362of the environment according to its intrinsic parameters (e.g., 1280×1280 resolution). The intrinsic parameters for the plurality of image sensors of the near-eye system305are described below in more detail with reference toFIG.3C. One of ordinary skill in the art will appreciate that the number and location of the image sensors of the near-eye system305inFIG.3Ais merely an example and may be different in other implementations.

FIG.3Bis an illustrative diagram of post-warp occlusion in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example,FIG.3Bshows an image390of the environment captured by the MCAML310that includes an obstacle391(e.g., a table-top or other planar surface) and a first object393located above the obstacle391.FIG.3Balso shows an representative image392of the environment from the perspective of the left eye of the user150that includes the obstacle391, the first object393located above the obstacle391, and a second object395located below the obstacle391. As such, for example, the image390does not include the second object395located below the obstacle391due to an offset/difference between the POVs of the MCAML310and the left eye of the user150.

As such, images from a respective MCAM (e.g., the MCAML310associated with a left eye) are warped to account for the aforementioned POV differences in order to provide a more comfortable experience for the user150. However, this warping operation may introduce holes in the warped images from the respective MCAM. As one example,FIG.3Balso shows a warped image394that corresponds to a warped version of the image390. As shown inFIG.3B, the warped image394includes an occluded area397. In some implementations, hole filling and/or diffusion processes are performed on the warped image394to fill-out or complete the occluded area397, which is described in more detail below with reference toFIGS.4A and4B.

FIG.3Cis a block diagram of an example data structure for an intrinsic parameters library431for the multi-camera architecture300inFIG.3Ain accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations, the intrinsic parameters library431includes values for the following parameters for the each of the main MCAML310, the auxiliary MCAMR320, the SCAMs330and340, and the DCAMs350and360: translational coordinates372, rotational coordinates374, resolution376, channel(s)378, field-of-view (FOV)380, frame rate382, and frame delay384.

As shown inFIG.3C, the main MCAML310is associated with values373A,375A,377A (e.g., 2616×2136),379A (e.g., RGB),381A (e.g., 109×89),383A (e.g., 90 fps), and385A (e.g., ˜0 ms) for the translational coordinates372, the rotational coordinates374, the resolution376, the channel(s)378, the FOV380, the frame rate382, and the frame delay384intrinsic parameters, respectively.

As shown inFIG.3C, the auxiliary MCAMR320is associated with values373B,375B,377B (e.g., 872×712),379B (e.g., YCC),381B (e.g., 109×89),383B (e.g., 30-45 fps), and385B (e.g., ˜1-2 ms) for the translational coordinates372, the rotational coordinates374, the resolution376, the channel(s)378, the FOV380, the frame rate382, and the frame delay384intrinsic parameters, respectively.

As shown inFIG.3C, the SCAMs330and340are associated with values373C,375C,377C (e.g., 1280×1280),379C (e.g., grayscale),381C (e.g., 160×160),383C (e.g., 30-45 fps), and385C (e.g., ˜1-2 ms) for the translational coordinates372, the rotational coordinates374, the resolution376, the channel(s)378, the FOV380, the frame rate382, and the frame delay384intrinsic parameters, respectively.

As shown inFIG.3C, the DCAMs350and360are associated with values373D,375D,377D (e.g., 1280×1280),379D (e.g., grayscale),381D (e.g., 160×160),383D (e.g., 30-45 fps), and385D (e.g., ˜1-2 ms) for the translational coordinates372, the rotational coordinates374, the resolution376, the channel(s)378, the FOV380, the frame rate382, and the frame delay384intrinsic parameters, respectively.

One of ordinary skill in art will appreciate that the values for the aforementioned values and parameters are merely examples and may be different in other implementations. One of ordinary skill in art will appreciate thatFIG.3Cis described from the perspective of the first image processing pipeline architecture400inFIG.4Afor a first eye (e.g., left eye) of the user150. To that end, one of ordinary skill in art will appreciate that the MCAMs are switched for the second image processing pipeline architecture475inFIG.4Bfor a second eye (e.g., right eye) of the user150with the MCAMR320as the main MCAM and the MCAML310as the auxiliary MCAM.

FIG.4Aillustrates a block diagram of a first example image processing pipeline architecture400in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, the first image processing pipeline architecture400corresponds to a first displayL482for a first eye (e.g., left eye) of the user150.

As shown inFIG.4A, the image312from the MCAML310is fed to the warp engine410. In some implementations, the warp engine410performs a warping operation/technique (e.g., dense depth reprojection or the like) on the image312from the MCAML310to generate a first warped image412. For example, the first warped image412accounts for the offset/difference between the translational/rotational coordinates of the user's eye (e.g., left eye) and the translational/rotational coordinates of the MCAML310.

As shown inFIG.4A, the occlusion mask generator420determines/generates an occlusion mask422based on the first warped image412and depth information404relative to at least one of the MCAML310and the first eye (e.g., left eye) of the user150. According to some implementations, the occlusion mask422indicates holes in the first warped image412. In some situations, the warping process generates occlusions and disocclusions with respect to physical objects in the physical environment. In particular, disocclusions are problematic as disocclusions are regions in the warped image that were previously not visible from the original POV but are now “visible” because the POV change also changed the position with respect to an occluding physical object or the like. This effectively creates “holes” to be filled. For example, an occlusion mask generation process is described below in more detail with reference toFIG.5.

As shown inFIG.4A, the image322from the MCAMR320, the image332from the SCAML330, the image342from the SCAMR340, the image352from the DCAML350, and the image362from the DCAMR360are fed to the normalization engine430. In some implementations, the normalization engine430normalizes each of the aforementioned images322,332,342,352, and362based on a difference between the intrinsic parameters for the MCAML310and the respective image sensor in order to produce a set of normalized images432. In some implementations, the normalization engine430obtains (e.g., receives or retrieves) the intrinsic parameters for the various image sensors from the intrinsic parameters library431.

As shown inFIG.4A, the hole filling engine440fills holes of the occlusion mask422based on at least one of the set of normalized images432to produce a modified first image442. As shown inFIG.4A, the diffusion engine450performs a pixelwise diffusion process on the modified first image442using a diffusion kernel to fill additional holes of the occlusion mask422to generate a diffused first image456. In some implementations, the pixelwise kernel determiner452determines/generates the diffusion kernel based on the depth of a subject pixel according to the depth information404. In some implementations, the pixelwise kernel determiner452determines/generates the diffusion kernel based on a determination as to whether a subject pixel is within a focus region, wherein the focus region is determined based at least in part on a gaze direction402. In some implementations, the featherer454performs a feathering operation on pixels associated with the pixelwise diffusion process in order to smooth discontinuities therein.

As shown inFIG.4A, the renderer462renders XR content465from the virtual content library463relative to a current camera pose from the camera pose determiner466. In some implementations, the renderer462provides an indication461of the location of the rendered XR content465to the hole filling engine440. In some implementations, the hole filling engine440identifies a plurality of pixels in the warped first image412that will be covered by the XR content based on the indication461and foregoes filling holes associated with the identified plurality of pixels.

As shown inFIG.4A, the compositor464composites the rendered XR content465with the diffused first image456based at least in part on the depth information404(e.g., to maintain z-order) to generate a rendered frame467of the XR environment. In turn, the first displayL482for the first eye (e.g., the left eye) of the user150displays the rendered frame467of the XR environment. In some implementations, the compositor464obtains (e.g., receives, retrieves, determines/generates, or otherwise accesses) the depth information404(e.g., a point cloud, depth mesh, or the like) associated with the scene (e.g., the physical environment105inFIG.1) to maintain z-order between the rendered XR content and physical objects in the physical environment.

FIG.4Billustrates a block diagram of a second example image processing pipeline architecture475in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, the second image processing pipeline architecture475corresponds to a second displayR484for a second eye (e.g., right eye) of the user150. The second image processing pipeline architecture475inFIG.4Bis similar to and adapted from the first image processing pipeline architecture400inFIG.4A. As such, similar reference numbers are used herein and only the differences therebetween will be described for the sake of brevity.

As shown inFIG.4B, the image322from the MCAMR320is fed to the warp engine410. In some implementations, the warp engine410performs a warping operation/technique (e.g., dense depth reprojection or the like) on the image322from the MCAMR320to generate a first warped image412. For example, the first warped image accounts for the offset/difference between the translational/rotational coordinates of the user's eye (e.g., right eye) and the translational/rotational coordinates of the MCAMR320.

As shown inFIG.4B, the image312from the MCAML310, the image332from the SCAML330, the image342from the SCAMR, the image352from the DCAML350, and the image362from the DCAMRare fed to the normalization engine430. In some implementations, the normalization engine430normalizes each of the aforementioned images312,332,342,352, and362based on a difference between the intrinsic parameters for the MCAMR320and the respective image sensor in order to produce a set of normalized images432. In some implementations, the normalization engine430obtains (e.g., receives or retrieves) the intrinsic parameters for the various image sensors from the intrinsic parameters library431.

As shown inFIG.4B, the compositor464composites the rendered XR content465with the diffused first image456based at least in part on the depth information404(e.g., in order to maintain z-order) to generate a rendered frame467of the XR environment. In turn, the second displayR484for the second eye (e.g., the right eye) of the user150displays the rendered frame467of the XR environment.

FIG.5is an illustrative diagram of an occlusion mask generation process in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein.

As shown inFIG.5, a point (xe, ye) in the 2D eye plane520may be transformed to a point (xc, yc) in the 2D camera plane510based on equation (4) below.

[xcyc1]←Pc⁢Pe-1⁢⌊xeye11depth⁢(xe,ye)⌋(4)

In equation (4),

Pc=Kc[RcTc01]⁢and⁢Pe=Ke[ReTe01]
are 4×4 projection matrices mapping 3D scene points in projective space to the camera and the eye pixels, respectively. One of ordinary skill in the art will appreciate how equations (4)-(6) correlate with the transformations252and254described above with respect toFIG.2.

As such, in this example, the occlusion mask is generated based on known depth values relative to both the camera plane510and the eye plane520. In greater detail, according to some implementations, an eye pixel (xe, ye) may be flagged as an occlusion if the difference between its depth and the visible depth at the corresponding camera pixel (xc, yc) is too large. One way to verify this is to perform a roundtrip check outlined by equations (5) and (6) below.

(xe,ye)-(x^e,y^e)>τ⁢with(5)[x^ey^e1]←Pe⁢Pc-1⁢⌊xcyc11depth⁢(xc,yc)⌋(6)
where τ corresponds to a pixel distance threshold.

One of ordinary skill in the art will appreciate that the occlusion mask may also be generated based on a depth value relative to one of the camera plane510and the eye plane520in some implementations. In other words, in some situations, one of depth(xe, ye) and depth(xe, ye) is known.

FIGS.6A and6Billustrate example diffusion techniques in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein.

According to some implementations, as shown inFIG.6A, a diffusion kernel602(e.g., the size and weights thereof) is based at least in part on the depth of pixels in the diffusion kernel as well as the distance of subject pixels to the reference pixel. For example, in this way, pixels within the foreground are associated with a smaller diffusion kernel for greater resolution. As shown inFIG.6A, the diffusion kernel602is a 3×3 matrix. In this example, x0is the reference pixel in the center of the diffusion kernel602, which is surrounded by other pixels xi,jthat fill-out the 3×3 matrix associated with the diffusion kernel602(sometimes also referred to as Ω3×3).

A weight w(x) is assigned to each pixel in the diffusion kernel602using a neighborhood weight function defined as equation (7) below.
w(x,x0)=ws(x,x0)·wd(depth(x),depth(x0))  (7)

The neighborhood weight function in equation (7) may be separated into a depth weighting function that corresponds to equation (8) and a spatial weighting function that corresponds to equation (9).

wd(depth⁢(x),depth⁢(x0))=depth⁢(x)P(8)ws(x,x0)=e-12⁢(x-x0σs)2(9)

In equation (8), P is an exponent that puts more weight on the background where depth values are larger (e.g., when P>>1). In equation (9), σsrepresents a standard deviation with respect to the center of the diffusion kernel602. As such, pixels further away from the reference pixel x0are given lower weights according to the inverse distance function604illustrated inFIG.6A. Furthermore, the resultant value Ii+1(x0) for the reference pixel x0may be defined by equation (10) below.

Ii+1(x0)=∑x∈Ωiw⁡(x,x0)·Ii(x))∑x∈Ωiw⁡(x,x0)(10)
In equation (10), Ωirepresents the current iteration neighborhood, whereas Ω0includes non-occluded pixels but is gradually filled in by the diffusion kernel602at each iteration. As such, in some implementations, the diffusion process is iterative in nature and gradually fills occluded areas with varying neighborhoods of pixels at each iteration.

According to some implementations, as shown inFIG.6B, a diffusion kernel (e.g., the size and weights thereof) is foveated based at least in part on a focal point/region determination. To this end, the system identifies a focal region652of an image650based on gaze direction and weights pixels within the focal region652higher than pixels in the balance654of the image650. As such, diffusion may be performed at a higher resolution on pixels within the focal region652.

For example, the reference pixel x0within the focal region652may be associated with a diffusion kernel Ω+, and pixels outside of the focal region652may be associated with a diffusion kernel Ω−. In some implementations, the diffusion kernel Ω+may be performed in color with a higher resolution than the diffusion kernel Ω−, which may be performed at a lower resolution with luma values.

As such, the following set of equations (11) addresses the overall treatment of the image650during the foveated diffusion process.

f⁡(x,xgaze)=e-12⁢(x-xgazeσe)2(11)I⁡(x)=(f⁡(x)·Ω+)+((1-f⁡(x))·Ω-)
In equation (11), σerepresents a standard deviation with respective to the gaze direction defines as xgaze.

FIG.7is a flowchart representation of a method700of multi-camera hole filling in accordance with some implementations. In various implementations, the method700is performed by at a computing system including non-transitory memory and one or more processors, wherein the computing system is communicatively coupled to a first image sensor and a second image sensor (e.g., the controller110inFIGS.1and9; the electronic device120inFIGS.1and10; or a suitable combination thereof), or a component thereof. For example, the electronic device corresponds to the near-eye system305inFIG.3Awith one or more forward-facing MCAMs, one or more side-facing SCAMs, and/or one or more downward-facing DCAMs. In some implementations, the method700is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method700is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). In various implementations, some operations in method700are, optionally, combined and/or the order of some operations is, optionally, changed.

As noted above, in some instances, a near-eye system (e.g., with video-pass through of a physical environment) may include a plurality of exterior-facing image sensors (i.e., cameras) such as one or more forward-facing cameras (MCAMs), one or more side-facing cameras (SCAMs), and/or one or more downward-facing cameras (DCAMs) that may be associated with different intrinsic camera characteristics (e.g., resolution, frame rate, field-of-view (FOV), frame delay, color space, and/or the like). As a first problem, the points of view (POVs) of the forward-facing image sensor and a user of the near-eye system are different, for example, the forward-facing image sensor is closer to the physical environment than the user's POV and may be offset from the position of the user's eyes. As such, images from a respective MCAM (e.g., associated with a left eye) are warped to account for the aforementioned POV differences in order to provide a more comfortable experience for the user. However, this warping process may introduce holes (e.g., including occlusions and disocclusions) in the warped images from the respective MCAM. The warping process generates occlusions and disocclusions with respect to physical objects in the physical environment. In particular, disocclusions are problematic as disocclusions are regions in the warped image that were previously not visible from the original POV but are now “visible” because the POV change also changed the position with respect to an occluding physical object or the like. This effectively creates “holes” to be filled.

Thus, as described herein, in some implementations, images from the other MCAM (e.g., associated with a right eye), the SCAM(s), and the DCAM(s) may be used to fill the holes but are normalized to account for the different intrinsic camera characteristics. Finally, a diffusion and feathering process may be performed on the images from the other MCAM to fill any remaining holes, wherein the diffusion and feathering process is based on a diffusion kernel that accounts for depth and/or displacement/distance.

As represented by block7-1, the method700includes obtaining a first image of an environment from a first image sensor associated with first intrinsic camera parameters. For example, with reference toFIGS.3A-3C, the left main (forward-facing) camera310captures an image312of the environment according to its intrinsic parameters (e.g., 2616×2136 resolution). In some implementations, the first intrinsic camera parameters include a first resolution, a first FOV, a first frame rate, a first frame delay, a first color space, and/or the like. In some implementations, the environment corresponds to a physical environment, a partially XR environment, a fully XR environment, and/or the like.

As represented by block7-2, the method700includes performing a warping operation on the first image to generate a warped first image. For example, with reference toFIG.4A, the warp engine410performs a warping operation/technique (e.g., dense depth reprojection or the like) on the image312from the MCAML310to generate a first warped image412. For example, the first warped image accounts for the offset/difference between the translational/rotational coordinates of the user's eye (e.g., left eye) and the translational/rotational coordinates of the MCAML310. In some implementations, the warping operation corresponds to dense depth reprojection. According to some implementations, the warping operation accounts for rotational and/or translational offsets between the first image sensor's POV and the POV of an associated eye of the user.

As represented by block7-3, the method700includes determining an occlusion mask based on the warped first image. In some implementations, the occlusion mask includes a plurality of holes. For example, with reference toFIG.4A, the occlusion mask generator420determines/generates an occlusion mask422based on the first warped image412and depth information404relative to at least one of the MCAML310and the first eye (e.g., left eye) of the user150. According to some implementations, the occlusion mask422indicates holes in the first warped image412. For example, an occlusion mask generation process is described above in more detail with reference toFIG.5. In some implementations, the occlusion mask is determined based at least in part on a first set of depth values relative to a first perspective associated with the first image sensor and a second set of depth values relative to a second perspective associated with the second image sensor different from the first perspective. For example, the second perspective corresponds to an eye of a user of the device.

As represented by block7-4, the method700includes obtaining a second image of the environment from a second image sensor associated with second intrinsic camera parameters. For example, with reference toFIGS.3A-3C, the right main (forward-facing) camera320captures an image322of the environment according to its intrinsic parameters (e.g., 872×712 resolution). In some implementations, the second intrinsic camera parameters include a second resolution, a second FOV, a second frame rate, a second frame delay, a second color space, and/or the like. As one example, the first and second intrinsic parameters include at least one similar resolution, FOV, frame rate, frame delay, or color space. As another example, the first and second intrinsic parameters include mutually exclusive resolutions, FOVs, frame rates, frame delays, and color spaces.

As represented by block7-5, the method700includes normalizing the second image based on a difference between the first and second intrinsic camera parameters to generate a normalized second image. For example, with reference toFIG.4A, the normalization engine430normalizes the image322from the right main (forward-facing) camera320based on a difference between the intrinsic parameters for the MCAML310and the intrinsic parameters for the MCAMR320in order to produce a second normalized image. In some implementations, the normalization engine430obtains (e.g., receives or retrieves) the intrinsic parameters for the various image sensors from the intrinsic parameters library431. In some implementations, normalizing the second image corresponds to linear up-sampling using a guided or joint bilateral filter. In some implementations, normalizing the second image corresponds to linear down-sampling using a guided or joint bilateral filter.

As represented by block7-6, the method700includes filling holes in the occlusion mask based on the normalized second image. For example, with reference toFIG.4A, the hole filling engine440fills a first set of one or more holes of the occlusion mask422based on the normalized second image to produce a modified first image442.

In some implementations, the method700includes skipping hole filling for pixels in the occlusion mask that are slated to be covered by virtual content. For example, with reference toFIG.4A, the hole filling engine440identifies a plurality of pixels in the warped first image412that will be covered by the XR content based on the indication461and foregoes filling holes associated with the plurality of pixels.

In some implementations, as represented by block7-7, the method700includes performing a diffusion process on a pixelwise basis to fill holes using a diffusion kernel. For example, with reference toFIG.4A, the diffusion engine450performs a pixelwise diffusion process on the modified first image442using a diffusion kernel to fill a second set of one or more holes of the occlusion mask422to generate a diffused first image456. According to some implementations, the first and second sets of one or more holes may be associate with mutually exclusive holes. According to some implementations, the first and second sets of one or more holes may be associated with at least some similar/overlapping holes.

In some implementations, as represented by block7-7a, the method700includes modifying the diffusion kernel (e.g., the size and weights thereof) based on pixelwise depth. For example, with reference toFIG.4A, the pixelwise kernel determiner452determines/generates the diffusion kernel based on the depth of a subject pixel according to the depth information404. As such, pixels within the foreground are associated with a smaller diffusion kernel for greater resolution. Dynamic generation of the diffusion kernel based on depth and/or displacement/distance is described in more detail above with reference toFIG.6A.

In some implementations, as represented by block7-7b, the method700includes modifying the diffusion kernel (e.g., the size and weights thereof) based on a pixelwise focus region determination associated with gaze direction. As such, pixels within the focus region are associated with a smaller diffusion kernel for greater resolution. For example, with reference toFIG.4A, the pixelwise kernel determiner452determines/generates the diffusion kernel based on a determination as to whether a subject pixel is within a focus region, wherein the focus region is determined based at least in part on a gaze direction402. Foveated diffusion is described in more detail above with reference toFIG.6B.

In some implementations, as represented by block7-7c, the method700includes performing a feathering operation. example, with reference toFIG.4A, the featherer454performs a feathering operation on pixels associated with the pixelwise diffusion process in order to smooth discontinuities therein.

FIGS.8A and8Bshow a flowchart representation of a method800of multi-camera hole filling in accordance with some implementations. In various implementations, the method800is performed by at a computing system including non-transitory memory and one or more processors, wherein the computing system is communicatively coupled to a first image sensor and a second image sensor (e.g., the controller110inFIGS.1and9; the electronic device120inFIGS.1and10; or a suitable combination thereof), or a component thereof. For example, the electronic device corresponds to the near-eye system305inFIG.3Awith one or more forward-facing MCAMs, one or more side-facing SCAMs, and/or one or more downward-facing DCAMs. In some implementations, the method800is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method800is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). In various implementations, some operations in method800are, optionally, combined and/or the order of some operations is, optionally, changed.

As described above, in some instances, a near-eye system (e.g., with video-pass through of a physical environment) may include a plurality of exterior-facing image sensors (i.e., cameras) such as one or more MCAMs, one or more SCAMs, and/or one or more DCAMs that may be associated with different intrinsic camera characteristics (e.g., resolution, frame rate, FOV, frame delay, color space, and/or the like). As a first problem, the POVs of the forward-facing image sensor and a user of the near-eye system are different, for example, the forward-facing image sensor is closer to the physical environment than the user's POV and may be offset from the position of the user's eyes. As such, images from a respective MCAM (e.g., associated with a left eye) are warped to account for the aforementioned POV differences in order to provide a more comfortable experience for the user. However, this warping process may introduce holes in the warped images from the respective MCAM. Thus, as described herein, in some implementations, images from the other MCAM (e.g., associated with a right eye), the SCAM(s), and the DCAM(s) may be used to fill the holes but are normalized to account for the different intrinsic camera characteristics.

As represented by block8-1, inFIG.8A, the method800includes obtaining a first image of an environment from a first MCAM. For example, with reference toFIG.4A, the MCAML310captures the image312of the environment.

As represented by block8-2, the method800includes performing a warping operation on the first image from the first MCAM to generate a warped first image. For example, with reference toFIG.4A, the warp engine410performs a warping operation/technique (e.g., dense depth reprojection or the like) on the image312from the MCAML310to generate a first warped image412.

As represented by block8-3, the method800includes determining an occlusion mask based on the warped first image. For example, with reference toFIG.4A, the occlusion mask generator420determines/generates an occlusion mask422based on the first warped image412and depth information404relative to at least one of the MCAML310and the first eye (e.g., left eye) of the user150. According to some implementations, the occlusion mask422indicates holes in the first warped image412. For example, an occlusion mask generation process is described above in more detail with reference toFIG.5.

As represented by block8-4, the method800includes obtaining a second image of the environment from a second MCAM. For example, with reference toFIG.4A, the MCAMR320captures the image322of the environment.

As represented by block8-5, the method800includes normalizing the second image from the second MCAM based on a difference of intrinsic camera parameters between first and second MCAMs to generate a normalized second image. For example, with reference toFIG.4A, the normalization engine430normalizes the image322based on a difference between the intrinsic parameters for the MCAML310and the MCAMR320to produce a normalized second image.

As represented by block8-6, the method800includes filling holes in the occlusion mask based on the normalized second image. For example, with reference toFIG.4A, the hole filling engine440fills holes of the occlusion mask422based on the normalized second image.

As represented by block8-7, the method800includes determining whether hole filling criteria are satisfied. In some implementations, the hole filling criteria are satisfied when at least a threshold percentage of holes in the occlusion mask have been filled (e.g., 75%, 90%, 99.99%, etc.). If the hole filling criteria are satisfied, the method800continues to block8-13. However, if the hole filling criteria are not satisfied, the method800continues to block8-8.

As represented by block8-8, inFIG.8B, the method800includes obtaining images of the environment from one or more SCAMs and/or one or more DCAMs For example, with reference toFIG.4A, the SCAML330captures the image332of the environment, the SCAMR340captures the image342of the environment, the DCAML350captures the image352of the environment, and/or from the DCAMR360captures the image362of the environment.

In some implementations, images from the from one or more SCAMs and/or one or more DCAMs are prioritized (or weighted) based on the image processing pipeline. For example, if the image processing pipeline corresponds to a right eye of the user150(e.g., as shown inFIG.4B), the MCAMR320corresponds to the main MCAM and the MCAML310corresponds to the auxiliary MCAM. Moreover, in this example, images from the SCAMR340and the DCAMR360may be prioritized (or weighted more heavily) than the images from the SCAML330and the DCAML350.

As represented by block8-9, the method800includes normalizing the images from the one or more SCAMs and/or the one or more DCAMs based on a difference of intrinsic camera parameters between the first MCAM and the one or more SCAMs and/or the one or more DCAMs to generate one or more normalized images. For example, with reference toFIG.4A, the normalization engine430normalizes at least some of the images332,342,352, and/or362based on a difference between the intrinsic parameters for the MCAML310and the SCAML330, the SCAMR340, the DCAML350, and/or the DCAMR360to produce one or more normalized images.

As represented by block8-10, the method800includes filling holes in the occlusion mask based on at least some of the one or more normalized images from block8-9. For example, with reference toFIG.4A, the hole filling engine440fills holes of the occlusion mask422based on the one or more normalized images.

As represented by block8-11, the method800includes determining whether hole filling criteria are satisfied. In some implementations, the hole filling criteria are satisfied when at least a threshold percentage of holes in the occlusion mask have been filled (e.g., 75%, 90%, 99.99%, etc.). If the hole filling criteria are satisfied, the method800continues to block8-13. However, if the hole filling criteria are not satisfied, the method800continues to block8-12.

As represented by block8-12, the method800includes performing a diffusion and feathering process on a pixelwise basis to additional fill holes. For example, with reference toFIG.4A, the diffusion engine450performs a pixelwise diffusion process on the modified first image442using a diffusion kernel to fill additional holes of the occlusion mask422to generate a diffused first image456.

As represented by block8-13, the method800includes rendering virtual content based on a current camera pose. For example, with reference toFIG.4A, the renderer462renders XR content from the virtual content library463according to a current camera pose from the camera pose determiner466relative thereto.

As represented by block8-14, the method800includes compositing the hole filled image of the environment with the rendered virtual content. For example, with reference toFIG.4A, the compositor464composites the rendered XR content465with the diffused first image456based at least in part on the depth information404(e.g., to maintain z-order) to generate a rendered frame467of the XR environment.

In some implementations, as represented by block8-15, the method800optional includes presenting or causing presentation of the composited content from block8-14. For example, with reference toFIG.4A, the first displayL482for the first eye (e.g., the left eye) of the user150displays the rendered frame467of the XR environment. One of ordinary skill in the art will appreciate that the first image processing pipeline architecture400inFIG.4Acorresponds to a first displayL482for a first eye (e.g., left eye) of the user150and that an image processing pipeline architecture for a second eye (e.g., right eye) of the user may be performed in parallel (e.g., as shown inFIG.4B).

FIG.9is a block diagram of an example of the controller110in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations, the controller110includes one or more processing units902(e.g., microprocessors, application-specific integrated-circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), central processing units (CPUs), processing cores, and/or the like), one or more input/output (I/O) devices906, one or more communication interfaces908(e.g., universal serial bus (USB), IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, global system for mobile communications (GSM), code division multiple access (CDMA), time division multiple access (TDMA), global positioning system (GPS), infrared (IR), BLUETOOTH, ZIGBEE, and/or the like type interface), one or more programming (e.g., I/O) interfaces910, a memory920, and one or more communication buses904for interconnecting these and various other components.

In some implementations, the one or more communication buses904include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices906include at least one of a keyboard, a mouse, a touchpad, a touch-screen, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and/or the like.

The memory920includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDR RAM), or other random-access solid-state memory devices. In some implementations, the memory920includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory920optionally includes one or more storage devices remotely located from the one or more processing units902. The memory920comprises a non-transitory computer readable storage medium. In some implementations, the memory920or the non-transitory computer readable storage medium of the memory920stores the following programs, modules and data structures, or a subset thereof including an optional operating system930and a presentation architecture940.

The operating system930includes procedures for handling various basic system services and for performing hardware dependent tasks.

In some implementations, the presentation architecture940is configured to render, present, and modify an XR environment. To that end, in some implementations, the presentation architecture940includes a data obtainer942, a mapper and locator engine944, the warp engine410, the occlusion mask generator420, the normalization engine430, the hole filling engine440, the diffusion engine450, a rendering engine460, a data transmitter982.

In some implementations, the data obtainer942is configured to obtain data (e.g., captured image frames of the physical environment105, presentation data, input data, user interaction data, camera pose tracking information, eye tracking information, head/body pose tracking information, hand/limb tracking information, sensor data, location data, etc.) from at least one of the I/O devices906of the controller110, the electronic device120, and the optional remote input devices. To that end, in various implementations, the data obtainer942includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the warp engine410is configured to perform a warping operation/technique (e.g., dense depth reprojection or the like) on a first image of an environment from a first image sensor (e.g., the main MCAMLinFIGS.3A-3C) associated with first intrinsic parameters to generate a first warped image. For example, the first warped image accounts for the offset/difference between the translational/rotational coordinates of the user's eye (e.g., left eye) and the translational/rotational coordinates of the first image sensor. To that end, in various implementations, the warp engine410includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the occlusion mask generator420is configured to obtain (e.g., receive, retrieve, or determine/generate) an occlusion mask based on the first warped image that indicates holes in the first warped image. For example, an occlusion mask generation process is described above in more detail with reference toFIG.5. To that end, in various implementations, the occlusion mask generator420includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the normalization engine430is configured to normalize a second image of the environment from a second image sensor associated with second intrinsic parameters based on a difference between the first and second intrinsic parameters to produce a normalized second image. In some implementations, the normalization engine430is also configured to normalize images of the environment from other image sensors (e.g., the SCAMs330and340, and the DCAMs350and360inFIGS.3A-3C). To that end, in various implementations, the normalization engine430includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the intrinsic parameters library431includes the intrinsic parameters for the various image sensors. In some implementations, the intrinsic parameters library431is stored locally and/or remotely. In some implementations, the intrinsic parameters library431is pre-populated or populated on-the-fly by polling the various image sensors. For example, the intrinsic parameters library431is described above in more detail with reference toFIG.3C.

In some implementations, the hole filling engine440is configured to fill a first set of one or more holes of the occlusion mask based on the normalized second image to produce a modified first image. In some implementations, the hole filling engine440is also configured to fill additional holes of the occlusion mask based on other normalized images associated with other image sensors (e.g., the SCAMs330and340, and the DCAMs350and360inFIGS.3A-3C). To that end, in various implementations, the hole filling engine440includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the diffusion engine450is configured to perform a pixelwise diffusion process on the modified first image using a diffusion kernel to fill a second set of one or more holes of the occlusion mask. To that end, in various implementations, the diffusion engine450includes instructions and/or logic therefor, and heuristics and metadata therefor. In some implementations, the diffusion engine450includes the pixelwise kernel determiner452and the featherer454.

In some implementations, the pixelwise kernel determiner452is configured to obtain (e.g., receive, retrieve, or determine/generate) the diffusion kernel based on the depth of a subject pixel. In some implementations, the pixelwise kernel determiner452is configured to obtain (e.g., receive, retrieve, or determine/generate) the diffusion kernel based on a determination as to whether a subject pixel is within a focus region, wherein the focus region is determined based at least in part on a gaze direction. To that end, in various implementations, the pixelwise kernel determiner452includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the featherer454is configured to perform a feathering operation on pixels associated with the pixelwise diffusion process in order to smooth discontinuities therein. To that end, in various implementations, the featherer454includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the rendering engine460is configured to render the XR environment. To that end, in various implementations, the rendering engine460includes instructions and/or logic therefor, and heuristics and metadata therefor. In some implementations, the rendering engine460includes the renderer462, the compositor464, and the camera pose determiner466.

In some implementations, the renderer462is configured to render XR content from the virtual content library463according to a current camera pose relative thereto. To that end, in various implementations, the renderer462includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the virtual content library463includes a plurality of XR object, items, scenery, and/or the like. In some implementations, the virtual content library463is stored locally and/or remotely. In some implementations, the virtual content library463is pre-populated.

In some implementations, the compositor464is configured to composite the rendered XR content with the modified first image. In some implementations, the compositor464obtains (e.g., receives, retrieves, determines/generates, or otherwise accesses) depth information (e.g., a point cloud, depth mesh, or the like) associated with the scene (e.g., the physical environment105inFIG.1) to maintain z-order between the rendered XR content and physical objects in the physical environment. To that end, in various implementations, the compositor464includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the camera pose determiner466is configured to determine a current camera pose of the electronic device120and/or the user150relative to the XR content. To that end, in various implementations, the camera pose determiner466includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the data transmitter982is configured to transmit data (e.g., presentation data such as rendered image frames associated with the XR environment, location data, etc.) to at least the electronic device120. To that end, in various implementations, the data transmitter982includes instructions and/or logic therefor, and heuristics and metadata therefor.

Although the data obtainer942, the mapper and locator engine944, the warp engine410, the occlusion mask generator420, the normalization engine430, the hole filling engine440, the diffusion engine450, the rendering engine460, the data transmitter982are shown as residing on a single device (e.g., the controller110), it should be understood that in other implementations, any combination of the data obtainer942, the mapper and locator engine944, the warp engine410, the occlusion mask generator420, the normalization engine430, the hole filling engine440, the diffusion engine450, the rendering engine460, the data transmitter982may be located in separate computing devices.

In some implementations, the functions and/or components of the controller110are combined with or provided by the electronic device120shown below inFIG.10. Moreover,FIG.9is intended more as a functional description of the various features which be present in a particular implementation as opposed to a structural schematic of the implementations described herein. As recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. For example, some functional modules shown separately inFIG.9could be implemented in a single module and the various functions of single functional blocks could be implemented by one or more functional blocks in various implementations. The actual number of modules and the division of particular functions and how features are allocated among them will vary from one implementation to another and, in some implementations, depends in part on the particular combination of hardware, software, and/or firmware chosen for a particular implementation.

FIG.10is a block diagram of an example of the electronic device120(e.g., a mobile phone, tablet, laptop, near-eye system, wearable computing device, or the like) in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations, the electronic device120includes one or more processing units1002(e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, and/or the like), one or more input/output (I/O) devices and sensors1006, one or more communication interfaces1008(e.g., USB, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, and/or the like type interface), one or more programming (e.g., I/O) interfaces1010, one or more displays1012, an image capture device1070(e.g., one or more optional interior- and/or exterior-facing image sensors), a memory1020, and one or more communication buses1004for interconnecting these and various other components.

In some implementations, the one or more communication buses1004include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices and sensors1006include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a magnetometer, a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptics engine, a heating and/or cooling unit, a skin shear engine, one or more depth sensors (e.g., structured light, time-of-flight, or the like), a localization and mapping engine, an eye tracking engine, a body/head pose tracking engine, a hand/limb tracking engine, a camera pose tracking engine, and/or the like.

In some implementations, the one or more displays1012are configured to present the XR environment to the user. In some implementations, the one or more displays1012are also configured to present flat video content to the user (e.g., a 2-dimensional or “flat” AVI, FLV, WMV, MOV, MP4, or the like file associated with a TV episode or a movie, or live video pass-through of the physical environment105). In some implementations, the one or more displays1012correspond to touchscreen displays. In some implementations, the one or more displays1012correspond to holographic, digital light processing (DLP), liquid-crystal display (LCD), liquid-crystal on silicon (LCoS), organic light-emitting field-effect transitory (OLET), organic light-emitting diode (OLED), surface-conduction electron-emitter display (SED), field-emission display (FED), quantum-dot light-emitting diode (QD-LED), micro-electro-mechanical system (MEMS), and/or the like display types. In some implementations, the one or more displays1012correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays. For example, the electronic device120includes a single display. In another example, the electronic device120includes a display for each eye of the user. In some implementations, the one or more displays1012are capable of presenting AR and VR content. In some implementations, the one or more displays1012are capable of presenting AR or VR content.

In some implementations, the image capture device1070correspond to one or more RGB cameras (e.g., with a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), IR image sensors, event-based cameras, and/or the like. In some implementations, the image capture device1070includes a lens assembly, a photodiode, and a front-end architecture. In some implementations, the electronic device120corresponds to the near-eye system305inFIG.3Adescribed in more detail above. In turn, the image capture device1070includes a plurality of image sensors such as the left main (forward-facing) camera310(e.g., MCAML), the right (forward-facing) main camera320(e.g., MCAMR), the left side-facing (peripheral) camera330(e.g., SCAML), the right side-facing (peripheral) camera340(e.g., SCAMR), the left downward-facing camera350(e.g., DCAML), and the right downward-facing camera360(e.g., DCAMR).

The memory1020includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, the memory1020includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory1020optionally includes one or more storage devices remotely located from the one or more processing units1002. The memory1020comprises a non-transitory computer readable storage medium. In some implementations, the memory1020or the non-transitory computer readable storage medium of the memory1020stores the following programs, modules and data structures, or a subset thereof including an optional operating system1030and a presentation engine1040.

The operating system1030includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the presentation engine1040is configured to present XR content (and/or other content) to the user via the one or more displays1012. To that end, in various implementations, the presentation engine1040includes a data obtainer1042, a presenter1044, an interaction handler1046, and a data transmitter1050.

In some implementations, the data obtainer1042is configured to obtain data (e.g., presentation data such as rendered image frames associated with the XR environment, input data, user interaction data, head tracking information, camera pose tracking information, eye tracking information, sensor data, location data, etc.) from at least one of the I/O devices and sensors1006of the electronic device120, the controller110, and the remote input devices. To that end, in various implementations, the data obtainer1042includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the presenter1044is configured to present and update XR content (e.g., the rendered image frames associated with the XR environment) via the one or more displays1012. To that end, in various implementations, the presenter1044includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the interaction handler1046is configured to detect user interactions with the presented XR content. To that end, in various implementations, the interaction handler1046includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the data transmitter1050is configured to transmit data (e.g., presentation data, location data, user interaction data, head tracking information, camera pose tracking information, eye tracking information, etc.) to at least the controller110. To that end, in various implementations, the data transmitter1050includes instructions and/or logic therefor, and heuristics and metadata therefor.

Although the data obtainer1042, the presenter1044, the interaction handler1046, and the data transmitter1050are shown as residing on a single device (e.g., the electronic device120), it should be understood that in other implementations, any combination of the data obtainer1042, the presenter1044, the interaction handler1046, and the data transmitter1050may be located in separate computing devices.

Moreover,FIG.10is intended more as a functional description of the various features which be present in a particular implementation as opposed to a structural schematic of the implementations described herein. As recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. For example, some functional modules shown separately inFIG.10could be implemented in a single module and the various functions of single functional blocks could be implemented by one or more functional blocks in various implementations. The actual number of modules and the division of particular functions and how features are allocated among them will vary from one implementation to another and, in some implementations, depends in part on the particular combination of hardware, software, and/or firmware chosen for a particular implementation.

While various aspects of implementations within the scope of the appended claims are described above, it should be apparent that the various features of implementations described above may be embodied in a wide variety of forms and that any specific structure and/or function described above is merely illustrative. Based on the present disclosure one skilled in the art should appreciate that an aspect described herein may be implemented independently of any other aspects and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and/or a method may be practiced using any number of the aspects set forth herein. In addition, such an apparatus may be implemented and/or such a method may be practiced using other structure and/or functionality in addition to or other than one or more of the aspects set forth herein.

It will also be understood that, although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first node could be termed a second node, and, similarly, a second node could be termed a first node, which changing the meaning of the description, so long as all occurrences of the “first node” are renamed consistently and all occurrences of the “second node” are renamed consistently. The first node and the second node are both nodes, but they are not the same node.

The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of the claims. As used in the description of the implementations and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

As used herein, the term “if” may be construed to mean “when” or “upon” or “in response to determining” or “in accordance with a determination” or “in response to detecting,” that a stated condition precedent is true, depending on the context. Similarly, the phrase “if it is determined [that a stated condition precedent is true]” or “if [a stated condition precedent is true]” or “when [a stated condition precedent is true]” may be construed to mean “upon determining” or “in response to determining” or “in accordance with a determination” or “upon detecting” or “in response to detecting” that the stated condition precedent is true, depending on the context.